Comments on: Unicode for dummies – just use UTF-8

By: Aditya

Aditya — Mon, 29 Apr 2013 10:50:49 +0000

Nice article,lucid ,concise and to the point.Though a small paragraph on why UTF-16 did come up or why someone/some people felt the need for UTF-16 would make “always using UTF-8″ an even better “informed” decision. Juxtapose them ,I feel it may help. Anyways it’s just a suggestion

By: Steve Ferg

Steve Ferg — Sun, 18 Mar 2012 20:42:07 +0000

You’re absolutely correct. This has been fixed. Thanks!

By: Pim

Pim — Mon, 12 Sep 2011 09:20:13 +0000

I agree with Paul that the BOM is a useful signature check for UTF-8 files.
In fact, it’s more useful for UTF-8 than for UTF-16 or UTF-32. If a computer program has to guess what encoding a BOM-less file is in, checking the first four bytes will provide enough information in the majority of cases. But not for UTF-8! UTF-8 will look just like any 1-byte charset in the absense of a BOM.

By: costy

costy — Wed, 27 Apr 2011 17:07:45 +0000

Hello! I’m at work surfing around your blog from my new iphone 3gs! Just wanted to say I love reading through your blog and look forward to all your posts! Carry on the great work!

By: Walter Dörwald

Walter Dörwald — Thu, 28 Oct 2010 09:36:41 +0000

Also see http://docs.python.org/library/codecs.html#encodings-and-unicode for an explaination of UTF-8 and the BOM

By: Simon Hibbs

Simon Hibbs — Thu, 28 Oct 2010 08:30:21 +0000

Quote: These terms are derived from “Big End In” and “Little End In.”

I’ve seen that theory posited before, but Danny Cohen explicitly refers to Guliver in his article, so it’s pretty clear his usage of the terms derives from Swift and the Lilliputians dispute as to which end of an egg should be eaten first.

By: Paul

Paul — Mon, 07 Jun 2010 16:14:29 +0000

Its a shame that the BOM is frowned upon when a file is encoded UTF8, since it provides a useful signature check.

By: flow

flow — Sat, 05 Jun 2010 15:41:03 +0000

you are perfectly right that character encoding schemes often feel, and act, like ‘brittle ice’. however, i want to promote the idea that especially in an environment like ms windows, with all the legacy stuff lying around, where several layers of the system apply differing heuristics to render text and rely on differing assumptions, trying to ‘max out on utf-8’ is the only thing that will work in the long run. this strategy means (eg in a web application) that when tests or actual usage reveal an encoding error, go through the entire tool stack — os, web server, database, everything — and re-affirm that yes, i want to use utf-8 here. make sure the web page comes out with a meta header stating utf-8, and that the http response header also states utf-8. make sure your source files are all in utf-8, and that your editor / ide also uses that. ah, and for filenames: [-a-z0-8_.], except for tested exceptions that work. this last one is mainly intended for when you want to package your sources in a *.tgz or *.zip and unpack it on another platform; this will often lead to funny results when you go beyond 7-bit us ascii.

By: ΤΖΩΤΖΙΟΥ

ΤΖΩΤΖΙΟΥ — Fri, 04 Jun 2010 23:13:07 +0000

The “Just use UTF-8 and forget about the BOM” is a sound advice, one with which I couldn’t agree more. However, this leaves millions of MS Windows users on brittle ice. Which version of Windows supports correctly UTF-8 as the default “codepage” for the system (and accordingly the “chcp 65001” thing for the command prompt ecosystem)?