Wrong error position for tokenizer errors reporting a range if the line contains non-ASCII characters

[serhiy-storchaka]

The tokenizer errors which report a range point at the wrong characters if the line contains non-ASCII characters before the error.

>>> compile("aaaa = 00010", "<t>", "exec")
  File "<t>", line 1
    aaaa = 00010
           ^^^
SyntaxError: leading zeros in decimal integer literals are not permitted; use an 0o prefix for octal integers
>>> compile("фффф = 00010", "<t>", "exec")
  File "<t>", line 1
    фффф = 00010
               ^
SyntaxError: leading zeros in decimal integer literals are not permitted; use an 0o prefix for octal integers

Both lines should mark 000; the second one is shifted right by one column per preceding non-ASCII character (offset=8, end_offset=11 versus offset=12, end_offset=15).

The same happens with incompatible string prefixes:

>>> compile("фф = ub'a'", "<t>", "exec")
  File "<t>", line 1
    фф = ub'a'
           ^^
SyntaxError: 'u' and 'b' prefixes are incompatible

SyntaxError.offset and end_offset are character columns, but every caller of _PyTokenizer_syntaxerror_known_range() passes byte offsets:

• Parser/lexer/number.c:238 — tok->start + 1 - tok->line_start, zeros_end - tok->line_start
• Parser/lexer/string.c:187 — the same expressions
• Parser/tokenizer/decoder.c:252 — 0, end_col (only for a BOM-with-cookie line, so ASCII in practice)

_PyTokenizer_syntaxerror() (the -1, -1 case) is not affected: it computes the column by decoding the line prefix with PyUnicode_DecodeUTF8(..., "replace").

The same class of bug was fixed for other errors in gh-102310 and gh-102312; these call sites were missed.

The fix could be either to convert in _syntaxerror_range() with _PyPegen_byte_offset_to_character_offset_raw(), which fixes all callers at once, or to require character offsets from the callers.

Linked PRs

• gh-156894: Fix the position of syntax errors which cover a range #156901