gh-68475: Keep comments and processing instructions outside the root element

[serhiy-storchaka]

ElementTree now has a children attribute: a view of the children of the document, containing the root element and any number of comments and processing instructions around it. Adding a second element is an error.

TreeBuilder no longer discards the comments and processing instructions which occur outside the root element, and returns them, together with the root element, from the new document() method. This only happens when insert_comments or insert_pis is true, so nothing changes for existing code.

ElementTree.iter() now iterates over all children of the document. find(), findall() and iterfind() still search from the root element.

A view which validates its content when modified follows @scoder's suggestion above, rather than the originally proposed ElementTree.append(), whose name @vadmium and @scoder both objected to. It covers the epilog as well as the prolog.

• Issue: Let ElementTree prolog include comments and processing instructions #68475

[read-the-docs-community]

Documentation build overview

📚 cpython-previews | 🛠️ Build #34390603 | 📁 Comparing a5c2b17 against main (d87ee27)

  🔍 Preview build  

22 files changed · ± 22 modified

± Modified

• glossary.html
• howto/curses.html
• library/atexit.html
• library/concurrent.interpreters.html
• library/curses.html
• library/functions.html
• library/inspect.html
• library/pyexpat.html
• library/stdtypes.html
• library/tomllib.html
• and 12 more...

[scoder]

Generally ok. I like the idea of a list-like .children property. I could probably implement the same in lxml.

I'd be more conservative about the interface between parser and that list, though, as noted below.

[scoder]

A parser target can be and can do anything (and a parser object as well, really), and this adds somewhat of a new protocol by interpreting parser.target.document as a callable method. There is no reason to assume that this may not fail for some existing user code somewhere, e.g. if a target implementation stores the document that it builds in a self.document attribute, which is not far-fetched.

I would recommend not adding a protocol but special casing known target (sub-)types only. If users provide their entirely own target implementation, there is really nothing to assume beyond the existing target callback methods.

[serhiy-storchaka]

We have two TreeBuilder implementations, in Python and C. There is also C14NWriterTarget, and I might add yet one or two specialized implementations for iterparse. They do not share common base class. The instance check would be very inconvenient.

We can chose a different method name. My initial implementation changed TreeBuilder.close() to return a list, and ElementTree.parse() then checked the type of the result, but I think that using a special method is better.

[scoder]

My initial implementation changed TreeBuilder.close() to return a list

That would also change an interface that could be used by wrappers. We shouldn't do that.

We can chose a different method name.

Take a long one that is less likely to conflict, e.g. target_top_level_result.

Or maybe add a second entry point for .close() that CAN return a list, like .close_with_multiple_results() in search of a better name. If the parser cannot find it, let it fall back to calling .close(). Implement the new method for the target classes that xml.etree provides by forwarding to a call to .close() internally, to keep the original interface for subclasses.

[serhiy-storchaka]

close_with_multiple_results() will not help, because target.close() is called in XMLParser.close(), but we need the result in ElementTree.parse(). If XMLParser.close() calls target.close_with_multiple_results() and returns its result, then we need to check the result type in ElementTree.parse() -- this does not differ much from simply returning a list in close(). This method is attractive, but I do not like it, it makes the code too rigid -- what if the result is a tuple or other sequence instead of a list? If there were more reliable method to distinguish a sequence of elements from a single root Element (which itself supports some methods of sequence protocol), I would prefer it.