gh-156939: Fix xmlcharrefreplace() buffer overflow

[vstinner]

Write into a temporay buffer to not write the trailing NUL byte.

• Issue: C API: Detect buffer overflow in PyBytesWriter #156939

[vstinner]

I wrote a patch for Python 3.14 to check if it's also affected:

diff --git a/Include/internal/pycore_bytesobject.h b/Include/internal/pycore_bytesobject.h
index 8ea9b3ebb88..7ab96b109a5 100644
--- a/Include/internal/pycore_bytesobject.h
+++ b/Include/internal/pycore_bytesobject.h
@@ -86,6 +86,7 @@ typedef struct {
     /* Stack buffer */
     int use_small_buffer;
     char small_buffer[512];
+    char canary_byte;
 } _PyBytesWriter;
 
 /* Initialize a bytes writer
diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c
index 03245788bb1..be698236843 100644
--- a/Objects/bytesobject.c
+++ b/Objects/bytesobject.c
@@ -3456,6 +3456,7 @@ _PyBytesWriter_Init(_PyBytesWriter *writer)
     memset(writer->small_buffer, PYMEM_CLEANBYTE,
            sizeof(writer->small_buffer));
 #endif
+    writer->canary_byte = 0xAB;
 }
 
 void
@@ -3524,6 +3525,8 @@ _PyBytesWriter_CheckConsistency(_PyBytesWriter *writer, char *str)
     end = start + writer->allocated;
     assert(str != NULL);
     assert(start <= str && str <= end);
+
+    assert(writer->canary_byte == (char)0xAB);
     return 1;
 }
 #endif
@@ -3665,6 +3668,10 @@ _PyBytesWriter_Finish(_PyBytesWriter *writer, void *str)
     PyObject *result;
 
     assert(_PyBytesWriter_CheckConsistency(writer, str));
+    if (writer->canary_byte != (char)0xAB) {
+        fprintf(stderr, "PyBytesWriter: buffer overflow detected! abort\n");
+        abort();
+    }
 
     size = _PyBytesWriter_GetSize(writer, str);
     if (size == 0 && !writer->use_bytearray) {

I wrote a script to check for the buffer overflow in Python 3.14:

writer_small_buffer = 512
ch = '\u20ac'
ch_encoded = ch.encode('latin1', 'xmlcharrefreplace')

repeat = writer_small_buffer - len(ch_encoded)
data = 'x' * repeat + ch
res = data.encode('latin1', 'xmlcharrefreplace')
print(res)
print(len(res))

Output:

$ ./python x.py 
PyBytesWriter: buffer overflow detected! abort
Abandon                    (core dumped)./python x

So yes, Python 3.14, which uses the old internal _PyBytesWriter API, is also affected.

In Python 3.14, _PyBytesWriter has its "small buffer" at the end of the structure. So the NUL byte write is actually a buffer overflow writing in the stack. It's quite bad :-(

[vstinner]

It seems like Python 3.10 to 3.16 are affected. (I didn't check older branches which no longer get security fixes.)

[vstinner]

@serhiy-storchaka: Would you mind to review this change?

[vstinner]

On Python 3.15 and 3.16, the new PyBytesWriter implementation is used. The NUL byte is written into PyBytesWriter.obj which is already a NULL pointer, so the write is harmless.

We should add the Changelog (NEWS) entry on Python 3.14 where the write occurs in the stack (buffer overflow).