Skip to content

MASM: non-AVX VEX fix, x64 home space, vzeroupper on VAES exits - #11290

Merged
SparkiDev merged 2 commits into
wolfSSL:masterfrom
kaleb-himes:PQ-FS-2026-Part3-SecurityReview-nofallback-K
Sep 7, 2026
Merged

SparkiDev merged 2 commits into
wolfSSL:masterfrom
kaleb-himes:PQ-FS-2026-Part3-SecurityReview-nofallback-K

Conversation

@kaleb-himes

@kaleb-himes kaleb-himes commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Description @SparkiDev

  1. vzeroupper: After using the wide 256-bit registers, this instruction wipes their upper halves before returning. Without it, the next ordinary SSE code in the program runs slow. Speed only, nothing computes wrong. (Intel Optimization Reference Manual, "Mixing AVX and SSE code".)

  2. vmovd → movd: Three ECC routines are the versions used on old CPUs that lack AVX, but each had an AVX-only instruction in it. Those CPUs crash. Proved it: under an emulated Nehalem the old library dies with SIGILL, the fixed one passes. (Intel SDM Vol. 2A §2.3.)

  3. sub rsp,32 before calls: Windows requires the caller to set aside 32 bytes of stack for the function it calls. We weren't, so the callee could overwrite our data. Windows only. (Microsoft, "x64 calling convention".)

  4. The big block-copy change: Copies now use plain integer registers instead of vector ones. We didn't need this fix; it came along because you require the committed asm to match the generator exactly, and that's what the generator emits. I can remove item 4 if there is an outstanding PR on the scripts/ repo that fixes the generators and simply hasn't merged yet.

Pairs with: https://github.com/wolfSSL/scripts/pull/671

Testing

Emulation and virtualized.

Checklist

  • added tests
  • updated/added doxygen
  • updated appropriate READMEs
  • Updated manual and documentation

@wolfSSL-Bot

Copy link
Copy Markdown

Can one of the admins verify this patch?

@kaleb-himes kaleb-himes self-assigned this Aug 26, 2026
@kaleb-himes

kaleb-himes commented Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

retest this please. (no logs).

@kaleb-himes
kaleb-himes force-pushed the PQ-FS-2026-Part3-SecurityReview-nofallback-K branch 2 times, most recently from d69c94a to 903bccd Compare August 27, 2026 14:00
@kaleb-himes
kaleb-himes requested a review from SparkiDev August 27, 2026 14:01
@kaleb-himes

kaleb-himes commented Aug 27, 2026

Copy link
Copy Markdown
Contributor Author

retest this please (aborted, no test failure)...

@kaleb-himes

kaleb-himes commented Aug 28, 2026

Copy link
Copy Markdown
Contributor Author

@wolfSSL-Fenrir-bot wolfSSL-Fenrir-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fenrir Automated Review — PR #11290

No scan targets match the changed files in this PR. Review skipped.

@kaleb-himes
kaleb-himes force-pushed the PQ-FS-2026-Part3-SecurityReview-nofallback-K branch from 903bccd to ca806c8 Compare September 1, 2026 16:39
@kaleb-himes

Copy link
Copy Markdown
Contributor Author

retest this please.

@wolfSSL-Fenrir-bot wolfSSL-Fenrir-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fenrir Automated Review — PR #11290

No scan targets match the changed files in this PR. Review skipped.

@SparkiDev
SparkiDev merged commit 38ed458 into wolfSSL:master Sep 7, 2026
385 of 386 checks passed
@kaleb-himes
kaleb-himes deleted the PQ-FS-2026-Part3-SecurityReview-nofallback-K branch September 11, 2026 19:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants