Mirrors
/
cpython
mirror of https://github.com/python/cpython
3
0
Fork 0
Code Releases Activity
Browse Source

#4153: update the Unicode howto.

pull/2332/head
Ezio Melotti 13 years ago
parent
663a9e2f25
commit
410eee566f
1 changed files with 74 additions and 66 deletions
Whitespace
Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL
Split View
Diff Options
Show Stats Download Patch File Download Diff File
  1. 140
      Doc/howto/unicode.rst

140
Doc/howto/unicode.rst
View File

@ -44,7 +44,7 @@ machines assigned values between 128 and 255 to accented characters. Different
machines had different codes, however, which led to problems exchanging files.
Eventually various commonly used sets of values for the 128--255 range emerged.
Some were true standards, defined by the International Standards Organization,
and some were **de facto** conventions that were invented by one company or
and some were *de facto* conventions that were invented by one company or
another and managed to catch on.
255 characters aren't very many. For example, you can't fit both the accented
@ -62,8 +62,8 @@ bits means you have 2^16 = 65,536 distinct values available, making it possible
to represent many different characters from many different alphabets; an initial
goal was to have Unicode contain the alphabets for every single human language.
It turns out that even 16 bits isn't enough to meet that goal, and the modern
Unicode specification uses a wider range of codes, 0 through 1,114,111 (0x10ffff
in base 16).
Unicode specification uses a wider range of codes, 0 through 1,114,111 (
``0x10FFFF`` in base 16).
There's a related ISO standard, ISO 10646. Unicode and ISO 10646 were
originally separate efforts, but the specifications were merged with the 1.1
@ -87,9 +87,11 @@ meanings.
The Unicode standard describes how characters are represented by **code
points**. A code point is an integer value, usually denoted in base 16. In the
standard, a code point is written using the notation U+12ca to mean the
character with value 0x12ca (4,810 decimal). The Unicode standard contains a lot
of tables listing characters and their corresponding code points::
standard, a code point is written using the notation ``U+12CA`` to mean the
character with value ``0x12ca`` (4,810 decimal). The Unicode standard contains
a lot of tables listing characters and their corresponding code points:
.. code-block:: none
0061 'a'; LATIN SMALL LETTER A
0062 'b'; LATIN SMALL LETTER B
@ -98,7 +100,7 @@ of tables listing characters and their corresponding code points::
007B '{'; LEFT CURLY BRACKET
Strictly, these definitions imply that it's meaningless to say 'this is
character U+12ca'. U+12ca is a code point, which represents some particular
character ``U+12CA``'. ``U+12CA`` is a code point, which represents some particular
character; in this case, it represents the character 'ETHIOPIC SYLLABLE WI'. In
informal contexts, this distinction between code points and characters will
sometimes be forgotten.
@ -115,13 +117,15 @@ Encodings
---------
To summarize the previous section: a Unicode string is a sequence of code
points, which are numbers from 0 through 0x10ffff (1,114,111 decimal). This
points, which are numbers from 0 through ``0x10FFFF`` (1,114,111 decimal). This
sequence needs to be represented as a set of bytes (meaning, values
from 0 through 255) in memory. The rules for translating a Unicode string
into a sequence of bytes are called an **encoding**.
The first encoding you might think of is an array of 32-bit integers. In this
representation, the string "Python" would look like this::
representation, the string "Python" would look like this:
.. code-block:: none
P y t h o n
0x50 00 00 00 79 00 00 00 74 00 00 00 68 00 00 00 6f 00 00 00 6e 00 00 00
@ -133,10 +137,10 @@ problems.
1. It's not portable; different processors order the bytes differently.
2. It's very wasteful of space. In most texts, the majority of the code points
are less than 127, or less than 255, so a lot of space is occupied by zero
are less than 127, or less than 255, so a lot of space is occupied by ``0x00``
bytes. The above string takes 24 bytes compared to the 6 bytes needed for an
ASCII representation. Increased RAM usage doesn't matter too much (desktop
computers have megabytes of RAM, and strings aren't usually that large), but
computers have gigabytes of RAM, and strings aren't usually that large), but
expanding our usage of disk and network bandwidth by a factor of 4 is
intolerable.
@ -175,14 +179,12 @@ internal detail.
UTF-8 is one of the most commonly used encodings. UTF stands for "Unicode
Transformation Format", and the '8' means that 8-bit numbers are used in the
encoding. (There's also a UTF-16 encoding, but it's less frequently used than
UTF-8.) UTF-8 uses the following rules:
encoding. (There are also a UTF-16 and UTF-32 encodings, but they are less
frequently used than UTF-8.) UTF-8 uses the following rules:
1. If the code point is <128, it's represented by the corresponding byte value.
2. If the code point is between 128 and 0x7ff, it's turned into two byte values
between 128 and 255.
3. Code points >0x7ff are turned into three- or four-byte sequences, where each
byte of the sequence is between 128 and 255.
1. If the code point is < 128, it's represented by the corresponding byte value.
2. If the code point is >= 128, it's turned into a sequence of two, three, or
four bytes, where each byte of the sequence is between 128 and 255.
UTF-8 has several convenient properties:
@ -192,8 +194,8 @@ UTF-8 has several convenient properties:
processed by C functions such as ``strcpy()`` and sent through protocols that
can't handle zero bytes.
3. A string of ASCII text is also valid UTF-8 text.
4. UTF-8 is fairly compact; the majority of code points are turned into two
bytes, and values less than 128 occupy only a single byte.
4. UTF-8 is fairly compact; the majority of commonly used characters can be
represented with one or two bytes.
5. If bytes are corrupted or lost, it's possible to determine the start of the
next UTF-8-encoded code point and resynchronize. It's also unlikely that
random 8-bit data will look like valid UTF-8.
@ -203,25 +205,25 @@ UTF-8 has several convenient properties:
References
----------
The Unicode Consortium site at <http://www.unicode.org> has character charts, a
The `Unicode Consortium site <http://www.unicode.org>`_ has character charts, a
glossary, and PDF versions of the Unicode specification. Be prepared for some
difficult reading. <http://www.unicode.org/history/> is a chronology of the
origin and development of Unicode.
difficult reading. `A chronology <http://www.unicode.org/history/>`_ of the
origin and development of Unicode is also available on the site.
To help understand the standard, Jukka Korpela has written an introductory guide
to reading the Unicode character tables, available at
<http://www.cs.tut.fi/~jkorpela/unicode/guide.html>.
To help understand the standard, Jukka Korpela has written `an introductory
guide <http://www.cs.tut.fi/~jkorpela/unicode/guide.html>`_ to reading the
Unicode character tables.
Another good introductory article was written by Joel Spolsky
<http://www.joelonsoftware.com/articles/Unicode.html>.
Another `good introductory article <http://www.joelonsoftware.com/articles/Unicode.html>`_
was written by Joel Spolsky.
If this introduction didn't make things clear to you, you should try reading this
alternate article before continuing.
.. Jason Orendorff XXX http://www.jorendorff.com/articles/unicode/ is broken
Wikipedia entries are often helpful; see the entries for "character encoding"
<http://en.wikipedia.org/wiki/Character_encoding> and UTF-8
<http://en.wikipedia.org/wiki/UTF-8>, for example.
Wikipedia entries are often helpful; see the entries for "`character encoding
<http://en.wikipedia.org/wiki/Character_encoding>`_" and `UTF-8
<http://en.wikipedia.org/wiki/UTF-8>`_, for example.
Python's Unicode Support
@ -233,11 +235,11 @@ Unicode features.
The String Type
---------------
Since Python 3.0, the language features a ``str`` type that contain Unicode
Since Python 3.0, the language features a :class:`str` type that contain Unicode
characters, meaning any string created using ``"unicode rocks!"``, ``'unicode
rocks!'``, or the triple-quoted string syntax is stored as Unicode.
To insert a Unicode character that is not part ASCII, e.g., any letters with
To insert a non-ASCII Unicode character, e.g., any letters with
accents, one can use escape sequences in their string literals as such::
>>> "\N{GREEK CAPITAL LETTER DELTA}" # Using the character name
@ -247,15 +249,16 @@ accents, one can use escape sequences in their string literals as such::
>>> "\U00000394" # Using a 32-bit hex value
'\u0394'
In addition, one can create a string using the :func:`decode` method of
:class:`bytes`. This method takes an encoding, such as UTF-8, and, optionally,
an *errors* argument.
In addition, one can create a string using the :func:`~bytes.decode` method of
:class:`bytes`. This method takes an *encoding* argument, such as ``UTF-8``,
and optionally, an *errors* argument.
The *errors* argument specifies the response when the input string can't be
converted according to the encoding's rules. Legal values for this argument are
'strict' (raise a :exc:`UnicodeDecodeError` exception), 'replace' (use U+FFFD,
'REPLACEMENT CHARACTER'), or 'ignore' (just leave the character out of the
Unicode result). The following examples show the differences::
``'strict'`` (raise a :exc:`UnicodeDecodeError` exception), ``'replace'`` (use
``U+FFFD``, ``REPLACEMENT CHARACTER``), or ``'ignore'`` (just leave the
character out of the Unicode result).
The following examples show the differences::
>>> b'\x80abc'.decode("utf-8", "strict") #doctest: +NORMALIZE_WHITESPACE
Traceback (most recent call last):
@ -273,8 +276,8 @@ a question mark because it may not be displayed on some systems.)
Encodings are specified as strings containing the encoding's name. Python 3.2
comes with roughly 100 different encodings; see the Python Library Reference at
:ref:`standard-encodings` for a list. Some encodings have multiple names; for
example, 'latin-1', 'iso_8859_1' and '8859' are all synonyms for the same
encoding.
example, ``'latin-1'``, ``'iso_8859_1'`` and ``'8859``' are all synonyms for
the same encoding.
One-character Unicode strings can also be created with the :func:`chr`
built-in function, which takes integers and returns a Unicode string of length 1
@ -290,13 +293,14 @@ returns the code point value::
Converting to Bytes
-------------------
Another important str method is ``.encode([encoding], [errors='strict'])``,
which returns a ``bytes`` representation of the Unicode string, encoded in the
requested encoding. The ``errors`` parameter is the same as the parameter of
the :meth:`decode` method, with one additional possibility; as well as 'strict',
'ignore', and 'replace' (which in this case inserts a question mark instead of
the unencodable character), you can also pass 'xmlcharrefreplace' which uses
XML's character references. The following example shows the different results::
The opposite method of :meth:`bytes.decode` is :meth:`str.encode`,
which returns a :class:`bytes` representation of the Unicode string, encoded in the
requested *encoding*. The *errors* parameter is the same as the parameter of
the :meth:`~bytes.decode` method, with one additional possibility; as well as
``'strict'``, ``'ignore'``, and ``'replace'`` (which in this case inserts a
question mark instead of the unencodable character), you can also pass
``'xmlcharrefreplace'`` which uses XML's character references.
The following example shows the different results::
>>> u = chr(40960) + 'abcd' + chr(1972)
>>> u.encode('utf-8')
@ -313,6 +317,8 @@ XML's character references. The following example shows the different results::
>>> u.encode('ascii', 'xmlcharrefreplace')
b'&#40960;abcd&#1972;'
.. XXX mention the surrogate* error handlers
The low-level routines for registering and accessing the available encodings are
found in the :mod:`codecs` module. However, the encoding and decoding functions
returned by this module are usually more low-level than is comfortable, so I'm
@ -365,14 +371,14 @@ they have no significance to Python but are a convention. Python looks for
``coding: name`` or ``coding=name`` in the comment.
If you don't include such a comment, the default encoding used will be UTF-8 as
already mentioned.
already mentioned. See also :pep:`263` for more information.
Unicode Properties
------------------
The Unicode specification includes a database of information about code points.
For each code point that's defined, the information includes the character's
For each defined code point, the information includes the character's
name, its category, the numeric value if applicable (Unicode has characters
representing the Roman numerals and fractions such as one-third and
four-fifths). There are also properties related to the code point's use in
@ -392,7 +398,9 @@ prints the numeric value of one particular character::
# Get numeric value of second character
print(unicodedata.numeric(u[1]))
When run, this prints::
When run, this prints:
.. code-block:: none
0 00e9 Ll LATIN SMALL LETTER E WITH ACUTE
1 0bf2 No TAMIL NUMBER ONE THOUSAND
@ -413,7 +421,7 @@ list of category codes.
References
----------
The ``str`` type is described in the Python library reference at
The :class:`str` type is described in the Python library reference at
:ref:`typesseq`.
The documentation for the :mod:`unicodedata` module.
@ -443,16 +451,16 @@ columns and can return Unicode values from an SQL query.
Unicode data is usually converted to a particular encoding before it gets
written to disk or sent over a socket. It's possible to do all the work
yourself: open a file, read an 8-bit byte string from it, and convert the string
with ``str(bytes, encoding)``. However, the manual approach is not recommended.
yourself: open a file, read an 8-bit bytes object from it, and convert the string
with ``bytes.decode(encoding)``. However, the manual approach is not recommended.
One problem is the multi-byte nature of encodings; one Unicode character can be
represented by several bytes. If you want to read the file in arbitrary-sized
chunks (say, 1K or 4K), you need to write error-handling code to catch the case
chunks (say, 1k or 4k), you need to write error-handling code to catch the case
where only part of the bytes encoding a single Unicode character are read at the
end of a chunk. One solution would be to read the entire file into memory and
then perform the decoding, but that prevents you from working with files that
are extremely large; if you need to read a 2Gb file, you need 2Gb of RAM.
are extremely large; if you need to read a 2GB file, you need 2GB of RAM.
(More, really, since for at least a moment you'd need to have both the encoded
string and its Unicode version in memory.)
@ -460,9 +468,9 @@ The solution would be to use the low-level decoding interface to catch the case
of partial coding sequences. The work of implementing this has already been
done for you: the built-in :func:`open` function can return a file-like object
that assumes the file's contents are in a specified encoding and accepts Unicode
parameters for methods such as ``.read()`` and ``.write()``. This works through
parameters for methods such as :meth:`read` and :meth:`write`. This works through
:func:`open`\'s *encoding* and *errors* parameters which are interpreted just
like those in string objects' :meth:`encode` and :meth:`decode` methods.
like those in :meth:`str.encode` and :meth:`bytes.decode`.
Reading Unicode from a file is therefore simple::
@ -478,7 +486,7 @@ writing::
f.seek(0)
print(repr(f.readline()[:1]))
The Unicode character U+FEFF is used as a byte-order mark (BOM), and is often
The Unicode character ``U+FEFF`` is used as a byte-order mark (BOM), and is often
written as the first character of a file in order to assist with autodetection
of the file's byte ordering. Some encodings, such as UTF-16, expect a BOM to be
present at the start of a file; when such an encoding is used, the BOM will be
@ -520,12 +528,12 @@ Functions in the :mod:`os` module such as :func:`os.stat` will also accept Unico
filenames.
Function :func:`os.listdir`, which returns filenames, raises an issue: should it return
the Unicode version of filenames, or should it return byte strings containing
the Unicode version of filenames, or should it return bytes containing
the encoded versions? :func:`os.listdir` will do both, depending on whether you
provided the directory path as a byte string or a Unicode string. If you pass a
provided the directory path as bytes or a Unicode string. If you pass a
Unicode string as the path, filenames will be decoded using the filesystem's
encoding and a list of Unicode strings will be returned, while passing a byte
path will return the byte string versions of the filenames. For example,
path will return the bytes versions of the filenames. For example,
assuming the default filesystem encoding is UTF-8, running the following
program::
@ -559,13 +567,13 @@ Unicode.
The most important tip is:
Software should only work with Unicode strings internally, converting to a
particular encoding on output.
Software should only work with Unicode strings internally, decoding the input
data as soon as possible and encoding the output only at the end.
If you attempt to write processing functions that accept both Unicode and byte
strings, you will find your program vulnerable to bugs wherever you combine the
two different kinds of strings. There is no automatic encoding or decoding if
you do e.g. ``str + bytes``, a :exc:`TypeError` is raised for this expression.
two different kinds of strings. There is no automatic encoding or decoding: if
you do e.g. ``str + bytes``, a :exc:`TypeError` will be raised.
When using data coming from a web browser or some other untrusted source, a
common technique is to check for illegal characters in a string before using the

Write Preview
Loading…
Cancel
Save