I recently rediscovered this strange behaviour in Python’s Unicode handling.—Evan

►

-1; there’s no standard for UTF-8 BOMs—adding it to the codecs module was probably a mistake to begin with.—M.-A.

►

There is a standard for UTF-8 signatures, however.—Stephen

►

With the UTF-8-SIG codec, it would apply to all operation modes of the codec, whether stream-based or from strings.—"Martin

►

I had in mind the ability to treat a string as a stream.—Stephen

►

Hmm. A string is not a stream, but it could be the contents of a stream.—"Martin

►

Of course it must be supported.—Stephen

►

I can’t put these two paragraphs together.—"Martin

►

Because the signature/BOM is not a chunk, it’s a header.—Stephen

►

Not really. In every encoding where a sequence of more than one byte maps to one Unicode character, you will always need some kind of buffering. If we remove the handling of initial BOMs from the codecs (except for UTF-16 where it is required), this wouldn’t change any buffering requirements.—Walter

Sure. My point is that codecs should be stateful only to the extent needed to assemble semantically meaningful units (ie, multioctet coded characters). In particular, they should not need to know about location at the beginning, middle, or end of some stream---because in the context of operating on a string they _can’t_.—Stephen