Skip to content

Fix precision loss in Newton-Raphson inverse - #552

Open
tompng wants to merge 1 commit into
ruby:masterfrom
tompng:newton_inverse_margin
Open

Fix precision loss in Newton-Raphson inverse#552
tompng wants to merge 1 commit into
ruby:masterfrom
tompng:newton_inverse_margin

Conversation

@tompng

@tompng tompng commented Sep 3, 2026

Copy link
Copy Markdown
Member

The residual precision SIZET2NUM(SIZET2NUM(n / 2)) was a double conversion. It accidentally requested about n digits, and the intended n / 2 is not enough: the digit schedule n = (prec >> i) + 2 leaves only 1-2 margin digits, so rounding errors compound over iterations and the inverse loses many digits. The correction loop in divmod_by_inv_mul then runs once per lost unit of the quotient and a single divmod can take minutes. With ROUND_DOWN or ROUND_FLOOR the same divergence happened even with the accidental n-digit residual, because the iteration converges from below and truncation adds error in the same direction at every step.

  • Use 4 margin digits for the schedule and n / 2 + 4 digits for the residual
  • Calculate with ROUND_HALF_UP in VpDivdNewton and restore the rounding mode together with the precision limit
  • Initial approximation from the leading 18 digits with 64-bit division (about 8 correct digits instead of 2), and skip iterations below that. The previous one could produce frac[0] == BASE for x like 1.001
  • Pass VpDivdNewtonInner arguments as a struct

Additional information:
Initial approximation was 2-digits precision because BASE_FIG was 4(uint64 unavailable) or 9(uint64 available) before. 2-digits is the half of BASE_FIG=4.
BASE_FIG is now always 9, so we can increase the initial approximation and the margin.
(Larger margin requires larger initial approximation precision)

The residual precision `SIZET2NUM(SIZET2NUM(n / 2))` was a double conversion.
It accidentally requested about n digits, and the intended n / 2 is not enough:
the digit schedule `n = (prec >> i) + 2` leaves only 1-2 margin digits, so
rounding errors compound over iterations and the inverse loses many digits.
The correction loop in divmod_by_inv_mul then runs once per lost unit of the
quotient and a single divmod can take minutes. With ROUND_DOWN or ROUND_FLOOR
the same divergence happened even with the accidental n-digit residual,
because the iteration converges from below and truncation adds error in the
same direction at every step.

- Use 4 margin digits for the schedule and n / 2 + 4 digits for the residual
- Calculate with ROUND_HALF_UP in VpDivdNewton and restore the rounding mode
  together with the precision limit
- Initial approximation from the leading 18 digits with 64-bit division
  (about 8 correct digits instead of 2), and skip iterations below that.
  The previous one could produce frac[0] == BASE for x like 1.001
- Pass VpDivdNewtonInner arguments as a struct

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant