I recently rediscovered this strange behaviour in Python’s Unicode handling.—Evan

►

-1; there’s no standard for UTF-8 BOMs—adding it to the codecs module was probably a mistake to begin with.—M.-A.

►

There is a standard for UTF-8 signatures, however.—Stephen

►

With the UTF-8-SIG codec, it would apply to all operation modes of the codec, whether stream-based or from strings.—"Martin

►

I’d suggest to use the same mode of operation as we have in the UTF-16 codec:—M.-A.

►

I’ve started writing such a codec. Making the BOM optional on decoding definitely simplifies the implementation.—Walter

►

OK, here is the patch:—Walter

►

This functionality is provided by a flush() method on similar objects, such as the zlib compression objects.

Evan Jones—Evan

Or by close() on other objects (htmllib, HTMLParser, the SAX incremental parser, etc.).

Too bad there’s more than one way to do it. :-(—Fred

Theoretically the name is unimportant, but read(..., final=True) or flush() or close() should subject the pending bytes to normal error handling and must return the result of decoding these pending bytes just like the other methods do. This would mean that we would have to implement a decodecode(), a readclose() and a readlineclose(). IMHO it would be best to add this argument to decode, read and readline directly. But I’m not sure, what this would mean for iterating through a StreamReader.

Bye,—Walter