gh-156894: Fix the position of syntax errors which cover a range

[serhiy-storchaka]

SyntaxError.offset and end_offset are columns in characters, but all callers of _PyTokenizer_syntaxerror_known_range() pass columns in bytes, so the reported position drifts by one column per preceding non-ASCII character:

>>> compile("abcd = 00010", "<t>", "exec")     # before and after
  File "<t>", line 1
    abcd = 00010
           ^^^
>>> compile("αβγδ = 00010", "<t>", "exec")     # before
  File "<t>", line 1
    αβγδ = 00010
               ^
>>> compile("αβγδ = 00010", "<t>", "exec")     # after
  File "<t>", line 1
    αβγδ = 00010
           ^^^

The conversion is done in _syntaxerror_range(), so all three call sites are fixed at once, and the -1 case is unaffected: it already counts characters.

test_tokenize.test_tolerant_incompatible_prefix_position_after_non_ascii asserted the column in bytes, so its expected value changes from 6 to 5.

• Issue: Wrong error position for tokenizer errors reporting a range if the line contains non-ASCII characters #156894

[grayjk]

include a test with a unicode combining character (e.g. \u0301) which has zero display width

Suggested change

	('\u03b1\u03b2\u03b3\u03b4 = 00010', 8, 11),
	('\u03b1\u03b2\u03b3\u03b4 = 00010', 8, 11),
	('e\u0301'*4 = 00010', 8, 11),

[serhiy-storchaka]

It counts code points, not visible width.

[miss-islington-app]

Thanks @serhiy-storchaka for the PR 🌮🎉.. I'm working now to backport this PR to: 3.13, 3.14, 3.15.
🐍🍒⛏🤖

[miss-islington-app]

Sorry, @serhiy-storchaka, I could not cleanly backport this to 3.15 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 59a691361cb7b1d4fde3b92f98a61490381c62c9 3.15

[miss-islington-app]

Sorry, @serhiy-storchaka, I could not cleanly backport this to 3.14 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 59a691361cb7b1d4fde3b92f98a61490381c62c9 3.14

[miss-islington-app]

Sorry, @serhiy-storchaka, I could not cleanly backport this to 3.13 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker 59a691361cb7b1d4fde3b92f98a61490381c62c9 3.13

[bedevere-app]

GH-156926 is a backport of this pull request to the 3.15 branch.

[bedevere-app]

GH-156927 is a backport of this pull request to the 3.14 branch.

[bedevere-app]

GH-156928 is a backport of this pull request to the 3.13 branch.