xml.etree.ElementTree: file source must be binary for non-UTF-8 encodings

[coproc]

Documentation

In the documentation for xml.etree.ElementTree.parse it says for the first argument source: "... source is a filename or file object containing XML data. ..."

But for file objects this does not work in all (expected) cases. A hint like the following is missing:

For file objects containing XML data with non-ASCII and non-UTF-8 encoding (e.g. ISO 8859-1), the file must have been opened in binary mode.

Otherwise (if opening the file in ASCII mode, regardless of the specified encoding) non-ASCII characters are not read correctly. (see this question on stackoverflow and also the attached files in test_parseXml.zip for reproducing the problem)

Here is an excerpt of the attached test code:

import xml

# ok
with open('test_ISO-8859-1.xml', 'rb') as fileInBinary:
    root = xml.etree.ElementTree.parse(fileInBinary).getroot()
print(root.attrib['attributeWithUmlauts'])

# garbage
with open('test_ISO-8859-1.xml', 'r', encoding='ISO-8859-1') as fileInAscii:
    root = xml.etree.ElementTree.parse(fileInAscii).getroot()
print(root.attrib['attributeWithUmlauts'])

giving the following output:

äöü
äöü

Linked PRs

• gh-99064: Added warnings for file objects with non-UTF8 content #123887
• gh-99064: Ignore the encoding declaration when parsing decoded text #156734
• [3.15] gh-99064: Ignore the encoding declaration when parsing decoded text (GH-156734) #156750
• [3.14] gh-99064: Ignore the encoding declaration when parsing decoded text (GH-156734) #156751
• [3.13] gh-99064: Ignore the encoding declaration when parsing decoded text (GH-156734) #156752

[vstinner]

It's unclear to me which encodings should emit a warning or not. For example:

• No warning: ASCII, UTF-8?
• Warning: ISO-8859-1, ISO-8859-15, ShiftJIS, Big5, UTF-16, etc.?

[serhiy-storchaka]

This is not a documentation issue. ElementTree.parse() is simply wrong here, and only in the C accelerator; every other way of parsing the same text is correct:

>>> doc = '<?xml version="1.0" encoding="ISO-8859-1"?><r a="äöü"/>'
>>> ET.parse(io.StringIO(doc)).getroot().get('a')
'äöü'
>>> ET.parse(io.StringIO(doc), ET.XMLParser()).getroot().get('a')
'äöü'
>>> ET.fromstring(doc).get('a')
'äöü'

XMLParser.feed() encodes str data as UTF-8 and calls XML_SetEncoding(parser, "utf-8"), so the encoding declared in the document, which no longer applies to already decoded text, is ignored. _parse_whole(), the fast path which parse() uses when no parser is given, does the same encoding but not the override. It got its handling of str in 2012 (bpo-14246), and the override was added to feed() in 2013 (bpo-16986) without touching it.

PR #156734 adds the same override there. @vstinner, this answers your question about which encodings should warn: none of them. The text is already decoded by the file object, so its declaration is irrelevant by that point, and there is nothing left to warn about. A text source declaring UTF-16 or UTF-32 also stops raising ParseError, matching feed().