I recently rediscovered this strange behaviour in Python’s Unicode handling.—Evan

►

-1; there’s no standard for UTF-8 BOMs—adding it to the codecs module was probably a mistake to begin with.—M.-A.

►

There is a standard for UTF-8 signatures, however.—Stephen

►

With the UTF-8-SIG codec, it would apply to all operation modes of the codec, whether stream-based or from strings.—"Martin

►

I’d suggest to use the same mode of operation as we have in the UTF-16 codec:—M.-A.

►

I’ve started writing such a codec. Making the BOM optional on decoding definitely simplifies the implementation.—Walter

►

OK, here is the patch:—Walter

►

Shouldn’t an empty read from the underlying stream be taken as an EOF?—"Martin

►

There are situations where the byte stream might be temporarily exhausted, e.g.—Walter

►

Yes, but these are not file-like objects.—"Martin

►

Not neccassarily. In the example above the IncrementalParser gets fed a chunk of data, it stuffs this data into the Queue, so that the StreamReader can decode it. Once the data from the Queue is exhausted, there won’t any further data until the user calls feed() on the IncrementalParser again.

Bye,—Walter