I recently rediscovered this strange behaviour in Python’s Unicode handling.—Evan

►

-1; there’s no standard for UTF-8 BOMs—adding it to the codecs module was probably a mistake to begin with.—M.-A.

►

There is a standard for UTF-8 signatures, however.—Stephen

►

With the UTF-8-SIG codec, it would apply to all operation modes of the codec, whether stream-based or from strings.—"Martin

►

I’d suggest to use the same mode of operation as we have in the UTF-16 codec:—M.-A.

►

I’ve actually been confused about this point for quite some time now, but never had a chance to bring it up.—Nicholas

►

The codec writes a BOM in the first call to .write()—it doesn’t write a BOM before reading from the file.—M.-A.

►

Yes, see, I read a lot of UTF-16 that comes from other sources.—Nicholas

►

Ok, but I don’t really follow you here:—M.-A.

►

This is about what we do now—we catch UnicodeError and then add a BOM to the file, and read it again.—Nicholas

►

It should be feasible to implement your own codec for that based on Lib/encodings/utf_16.py. Simply replace the line in StreamReader.decode(): raise UnicodeError,"UTF-16 stream does not start with BOM" with: self.decode = codecs.utf_16_be_decode and you should be done.

Bye,—Walter

Oops, this only works if you have a big endian system. Otherwise you have to redecode the input with: codecs.utf_16_ex_decode(input, errors, 1, False)

Bye,—Walter