Skip to content

gh-149468: Add option to validate ElementTree during serialization #149469

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
serhiy-storchaka wants to merge 13 commits into
base: main
Choose a base branch
from
Open

gh-149468: Add option to validate ElementTree during serialization #149469

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
e41d61c
gh-149489: Fix ElementTree serialization to HTML
serhiy-storchaka May 7, 2026
a134c0b
gh-149468: Add option to validate ElementTree during serialization
serhiy-storchaka May 6, 2026
c81fe70
Merge branch 'main' into xml-etree-validate
serhiy-storchaka May 29, 2026
ea414fa
Apply suggestions from code review
serhiy-storchaka May 31, 2026
452480d
Merge branch 'main' into xml-etree-validate
serhiy-storchaka May 31, 2026
474411f
Add more tests for processing instructions
serhiy-storchaka May 31, 2026
22e5543
Add more details in exceptions.
serhiy-storchaka May 31, 2026
e12e88d
Check also for surrogates in HTML.
serhiy-storchaka May 31, 2026
8cc34e7
Move the What's New entry to 3.16.
serhiy-storchaka May 31, 2026
3d0fdd2
Update the NEWS entry.
serhiy-storchaka May 31, 2026
0a7c117
Merge remote-tracking branch 'upstream/main' into xml-etree-validate
serhiy-storchaka Sep 3, 2026
2b67e17
Rename the error message to "invalid namespace prefix"
serhiy-storchaka Sep 3, 2026
e2ab82e
Use subTests in the validation tests
serhiy-storchaka Sep 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 28 additions & 7 deletions Doc/library/xml.etree.elementtree.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -710,15 +710,16 @@ Functions

.. function:: tostring(element, encoding="us-ascii", method="xml", *, \
xml_declaration=None, default_namespace=None, \
short_empty_elements=True, standalone=None)
validate=False, short_empty_elements=True, \
standalone=None)

Generates a string representation of an XML element, including all
subelements. *element* is an :class:`Element` instance. *encoding* [1]_ is
the output encoding (default is US-ASCII). Use ``encoding="unicode"`` to
generate a Unicode string (otherwise, a bytestring is generated). *method*
is either ``"xml"``, ``"html"`` or ``"text"`` (default is ``"xml"``).
*xml_declaration*, *default_namespace*, *short_empty_elements* and
*standalone* has the same meaning as in :meth:`ElementTree.write`.
*xml_declaration*, *default_namespace*, *validate*, *short_empty_elements*
and *standalone* have the same meaning as in :meth:`ElementTree.write`.
Returns an (optionally) encoded string containing the XML data.

.. versionchanged:: 3.4
Expand All @@ -734,18 +735,22 @@ Functions
.. versionchanged:: next
Added the *standalone* parameter.

.. versionchanged:: next
Added the *validate* parameter.


.. function:: tostringlist(element, encoding="us-ascii", method="xml", *, \
xml_declaration=None, default_namespace=None, \
short_empty_elements=True, standalone=None)
validate=False, short_empty_elements=True, \
standalone=None)

Generates a string representation of an XML element, including all
subelements. *element* is an :class:`Element` instance. *encoding* [1]_ is
the output encoding (default is US-ASCII). Use ``encoding="unicode"`` to
generate a Unicode string (otherwise, a bytestring is generated). *method*
is either ``"xml"``, ``"html"`` or ``"text"`` (default is ``"xml"``).
*xml_declaration*, *default_namespace*, *short_empty_elements* and
*standalone* has the same meaning as in :meth:`ElementTree.write`.
*xml_declaration*, *default_namespace*, *validate*, *short_empty_elements*
and *standalone* have the same meaning as in :meth:`ElementTree.write`.
Returns a list of (optionally) encoded strings containing the XML data.
It does not guarantee any specific sequence,
except that ``b"".join(tostringlist(element)) == tostring(element)``.
Expand All @@ -765,6 +770,9 @@ Functions
.. versionchanged:: next
Added the *standalone* parameter.

.. versionchanged:: next
Added the *validate* parameter.


.. function:: XML(text, parser=None)

Expand Down Expand Up @@ -1201,7 +1209,8 @@ ElementTree Objects

.. method:: write(file, encoding="us-ascii", xml_declaration=None, \
default_namespace=None, method="xml", *, \
short_empty_elements=True, standalone=None)
validate=False, short_empty_elements=True, \
standalone=None)

Writes the element tree to a file, as XML. *file* is a file name, or a
:term:`file object` opened for writing. *encoding* [1]_ is the output
Expand All @@ -1212,6 +1221,15 @@ ElementTree Objects
*default_namespace* sets the default XML namespace (for "xmlns").
*method* is either ``"xml"``, ``"html"`` or ``"text"`` (default is
``"xml"``).

If *validate* is true, check that all characters are legal,
that element and attribute names are valid, and that the content
of comments, processing instructions and HTML elements
like ``<script>`` do not contain illegal sequences according
to the selected *method* (``"xml"`` or ``"html"``).
Raise :exc:`ValueError` if any check fails.
By default, or if *method* is ``"text"``, no validation is performed.

@vstinner vstinner Jun 1, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand method="text" case. Does it force validate to False? How is it different from the default behavior (validate=False)?

@serhiy-storchaka serhiy-storchaka Sep 3, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

validate has no effect for method="text".


The keyword-only *short_empty_elements* parameter controls the formatting
of elements that contain no content. If ``True`` (the default), they are
emitted as a single self-closed tag, otherwise they are emitted as a pair
Expand Down Expand Up @@ -1241,6 +1259,9 @@ ElementTree Objects
.. versionchanged:: next
Added the *standalone* parameter.

.. versionchanged:: next
Added the *validate* parameter.


This is the XML file that is going to be manipulated::

Expand Down
6 changes: 6 additions & 0 deletions Doc/whatsnew/3.16.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -660,6 +660,12 @@ xml
rather than defaulted from the DTD.
(Contributed by Jason Orendorff and Serhiy Storchaka in :gh:`44871`.)

* Add the *validate* option to :func:`~xml.etree.ElementTree.tostring`,
:func:`~xml.etree.ElementTree.tostringlist` and
:meth:`ElementTree.write <xml.etree.ElementTree.ElementTree.write>`,
which makes them check that the produced document is well-formed.
(Contributed by Serhiy Storchaka in :gh:`149468`.)

zipfile
-------

Expand Down
186 changes: 186 additions & 0 deletions Lib/test/test_xml_etree.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1545,6 +1545,192 @@ def test_attlist_default(self):
{'{http://www.w3.org/XML/1998/namespace}lang': 'eng'})


class XMLValidationTest(unittest.TestCase):

def check(self, elem):
self.assertRaises(ValueError,
ET.tostring, elem, validate=True)
ET.tostring(elem) # no exception

def check_valid(self, elem, expected):
self.assertEqual(ET.tostring(elem, validate=True), expected)

@support.subTests('text', ('a--b', ' B+, B, or B-',
'\x00', '\x01', '\ud8ff', '\ufffe'))
def test_invalid_comment(self, text):
self.check(ET.Comment(text))

def test_invalid_processing_instruction(self):
self.check(ET.PI(''))
self.check(ET.PI('0'))
self.check(ET.PI('a/b'))
self.check(ET.PI('foo\xa0bar'))
self.check(ET.PI('foo\fbar'))
self.check(ET.PI('xml'))
self.check(ET.PI('XML'))
self.check(ET.PI('xml', 'encoding="UTF-8"'))
self.check(ET.PI('foo', 'a?>b'))
self.check(ET.PI('foo', '\x00'))
self.check(ET.PI('foo', '\x01'))
self.check(ET.PI('foo', '\ud8ff'))
self.check(ET.PI('foo', '\ufffe'))

self.check_valid(ET.PI('foo\tbar'), b'<?foo\tbar?>')
self.check_valid(ET.PI('foo\nbar'), b'<?foo\nbar?>')
self.check_valid(ET.PI('foo\rbar'), b'<?foo\rbar?>')

@support.subTests('name', ('', '0', 'a/b'))
def test_invalid_tag(self, name):
self.check(ET.Element(name))
self.check(ET.Element(ET.QName(name)))

@support.subTests('name', ('', '0', 'a/b'))
def test_invalid_attr_name(self, name):
self.check(ET.Element('tag', attrib={name: 'value'}))
self.check(ET.Element('tag', attrib={ET.QName(name): 'value'}))

@support.subTests('value', ('\x00', '\ud8ff', '\ufffe'))
def test_invalid_attr_value(self, value):
self.check(ET.Element('tag', attrib={'key': value}))
self.check(ET.Element('tag', attrib={'key': ET.QName(value)}))

@support.subTests('text', ('\x00', '\ud8ff', '\ufffe'))
def test_invalid_text(self, text):
elem = ET.Element('tag')
elem.text = text
self.check(elem)

@support.subTests('tail', ('\x00', '\ud8ff', '\ufffe'))
def test_invalid_tail(self, tail):
elem = ET.Element('tag')
elem.tail = tail
self.check(elem)

@support.subTests('text', ('\x00', '\ud8ff', '\ufffe'))
def test_invalid_text_without_tag(self, text):
elem = ET.Element(None)
elem.text = text
self.check(elem)

def test_invalid_subelements(self):
elem = ET.Element('tag')
subelem = ET.SubElement(elem, 'subtag')
ET.SubElement(subelem, '\x00')
self.check(elem)
elem.tag = None
self.check(elem)

@support.subTests('uri', ('\x00', '\ud8ff', '\ufffe'))
def test_invalid_namespace_uri(self, uri):
self.check(ET.Element('{%s}tag' % uri))
self.check(ET.Element(ET.QName(uri, 'tag')))


class HTMLValidationTest(unittest.TestCase):

def check(self, elem):
self.assertRaises(ValueError,
ET.tostring, elem, method='html', validate=True)
ET.tostring(elem, method='html') # no exception

@support.subTests('text', ('>', '->', 'a-->b', 'a--!>b',
'a\x00b', 'a\ud8ffb'))
def test_invalid_comment(self, text):
self.check(ET.Comment(text))

@support.subTests('text', ('a>b', 'a\x00b', 'a\ud8ffb'))
def test_invalid_processing_instruction(self, text):
self.check(ET.PI(text))

@support.subTests('name', ('', '0', 'a/b'))
def test_invalid_tag(self, name):
self.check(ET.Element(name))
self.check(ET.Element(ET.QName(name)))

@support.subTests('name', ('?', '!', ' a', 'a b', 'a\nb', 'a>b',
'a\x00b', 'a\ud8ffb'))
def test_invalid_tag_str_only(self, name):
self.check(ET.Element(name))

@support.subTests('name', ('', 'a/b'))
def test_invalid_attr_name(self, name):
self.check(ET.Element('tag', attrib={name: 'value'}))
self.check(ET.Element('tag', attrib={ET.QName(name): 'value'}))

@support.subTests('name', ('\x00', '\ud8ff', 'a=b',
'a\x00b', 'a\ud8ffb'))
def test_invalid_attr_name_str_only(self, name):
self.check(ET.Element('tag', attrib={name: 'value'}))

@support.subTests('value', ('\x00', '\ud8ff'))
def test_invalid_attr_value(self, value):
self.check(ET.Element('tag', attrib={'key': value}))
self.check(ET.Element('tag', attrib={'key': ET.QName(value)}))

@support.subTests('value', ('a"b', 'a&b'))
def test_invalid_attr_value_qname(self, value):
# a string value is escaped, a QName is written as is
self.check(ET.Element('tag', attrib={'key': ET.QName(value)}))

@support.subTests('text', ('\x00', '\ud8ff'))
def test_invalid_text(self, text):
elem = ET.Element('tag')
elem.text = text
self.check(elem)

@support.subTests('tail', ('\x00', '\ud8ff'))
def test_invalid_tail(self, tail):
elem = ET.Element('tag')
elem.tail = tail
self.check(elem)

@support.subTests('text', ('\x00', '\ud8ff'))
def test_invalid_text_without_tag(self, text):
elem = ET.Element(None)
elem.text = text
self.check(elem)

def test_invalid_subelements(self):
elem = ET.Element('tag')
subelem = ET.SubElement(elem, 'subtag')
ET.SubElement(subelem, '\x00')
self.check(elem)
elem.tag = None
self.check(elem)

@support.subTests('uri', ('\x00', '\ud8ff'))
def test_invalid_namespace_uri(self, uri):
self.check(ET.Element('{%s}tag' % uri))
self.check(ET.Element(ET.QName(uri, 'tag')))

@support.subTests('tag', ("script", "style", "xmp", "iframe", "noembed", "noframes"))
def test_invalid_cdata_content(self, tag):
elem = ET.Element(tag.upper())
elem.text = 'a</%s>b' % tag.title()
self.check(elem)
elem.text = 'a</%s b' % tag.title()
self.check(elem)
elem.text = 'a</%s/b' % tag.title()
self.check(elem)
elem.text = 'a\x00b'
self.check(elem)
elem.text = 'a\ud8ffb'
self.check(elem)

@support.subTests('tag', ("script", "style", "xmp", "iframe", "noembed", "noframes"))
def test_cdata_subelements(self, tag):
elem = ET.Element(tag)
ET.SubElement(elem, 'subtag')
self.check(elem)

def test_invalid_plaintext_content(self):
elem = ET.Element('plaintext')
elem.text = 'a\x00b'
self.check(elem)
elem.text = 'a\ud8ffb'
self.check(elem)


class IterparseTest(unittest.TestCase):
Comment thread
serhiy-storchaka marked this conversation as resolved.
# Test iterparse interface.

Expand Down
Loading
Loading