I recently rediscovered this strange behaviour in Python’s Unicode handling.—Evan

►

-1; there’s no standard for UTF-8 BOMs—adding it to the codecs module was probably a mistake to begin with.—M.-A.

►

There is a standard for UTF-8 signatures, however.—Stephen

►

With the UTF-8-SIG codec, it would apply to all operation modes of the codec, whether stream-based or from strings.—"Martin

►

What I think should be provided is a stateful object encapsulating the codec.—Stephen

►

No. People who want streaming should use cStringIO, i.e.—"Martin

►

Yes!—Stephen

►

However, if there is an utf-8-sig codec for streams, there is currently no way of preventing this codec to also be available for strings. The very same code is used for streams and for strings, and automatically so.

Regards, Martin—"Martin

And of course it should be. But if it’s not possible to move the -sig facility out of the codecs into the streams, that would be a shame. I think we should encourage people to use streams where initialization or finalization semantics are non-trivial, as they are with signatures.

But as long as both utf-8-we-dont-need-no-steenkin-sigs-in-strings and utf-8-sig are available, I can program as I want to (and refer those whose strings get cratered by stray BOMs to you<wink>).—Stephen