From Fedora Project Wiki

< User:Toshio

Revision as of 12:29, 29 July 2010 by Toshio (talk | contribs) (Start of page)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

Why python and unicode leads to frustration

In python-2.x, there's two types that deal with text.

1. str is for strings of bytes. These are very similar in nature to how strings are handled in C. 2. unicode for strings of unicode codepoints.

Just what the dickens is "Unicode"?
One mistake that people encountering this issue for the first time make is confusing the unicode type and the encodings of unicode: UTF8, UTF16, UTF32, UCS4, etc. The unicode encodings are byte strings that assign a certain sequence of bytes to each unicode code point.

In the python-2.x world, these are used pretty interchangably but there are several important APIs where only one or the other will do the right thing. When you give the wrong type of string to an API that wants the other one, you may end up with an exception being raised (UnicodeDecodeError or UnicodeEncodeError). However, these exceptions aren't always raised because python implicitly converts between types... sometimes.

Although doing the right thing when possible seems like the right thing to do, it's actually the first source of frustration. A programmer can test out their program with a string like: "The quick brown fox jumped over the lazy dog" and not encounter any issues. But when they release their software into the wild, someone enters the string: <code>"I sat down for coffee at the café" and suddenly an exception is thrown. The reason? The mechanism that converts between the two types is only able to deal with ASCII characters. Once you throw non-ASCII characters into your strings, you have to start dealing with the conversion manually.

So, if I manually convert everything to either bytes or strings, will I be okay?

A few solutions

On the

Use kitchen to convert at the border

In python-2.x I've started work on a library that can help with unicode issues. The part that's especially relevant is the converter functions. They allow you to transform your text from bytes to unicode and unicode to bytes pretty painlessly.

Designing APIs

If you're writing APIs to deal with text, there's a few techniques to use. However, you must be wary in what you do as each method has some drawbacks.



Two functions

Reasons to avoid

One function and make the user convert

What to watch out for