Fix peer key masking

[miyazakh]

Summary(F9972)

RFC 7748 Section 5 requires that a receiver of a 32-byte X25519 u-coordinate mask (clear) the reserved high bit of the final byte rather than reject it. wolfSSL's TLS 1.2/1.3 handshake code instead rejected any peer X25519 public value with that bit set, aborting the handshake with ECC_PEERKEY_ERROR before the key was ever imported.

This affected all three places a peer's X25519 public value is parsed off the wire:

• TLSX_KeyShare_ProcessX25519_ex (src/tls.c) — TLS 1.3 KeyShare
• GetEcDiffieHellmanKea (src/internal.c) — TLS 1.2 ServerKeyExchange
• ImportPeerECCKey (src/internal.c) — TLS 1.2 ClientKeyExchange

Fix

Each site now copies the received 32-byte value into a local buffer, clears bit 7 of the last byte (&= 0x7f), and passes the masked copy to both wc_curve25519_check_public() and the import call. Behavior is unchanged when WOLFSSL_X25519_NO_MASK_PEER is defined (existing opt-in strict rejection is preserved).

wc_curve25519_check_public() itself is untouched — masking is a TLS-layer concern per RFC 7748; the primitive's existing reject-on-high-bit behavior and its unit tests (tests/api/test_curve25519.c) remain correct as-is.

Testing

Added test_tls13_x25519_keyshare_masks_reserved_bit (tests/api/test_tls13.c): drives a real TLS 1.3 memio handshake, tampers the client's ClientHello key_share entry to set the reserved bit, and confirms the server now masks it and proceeds (WOLFSSL_ERROR_WANT_READ) instead of aborting with ECC_PEERKEY_ERROR.

Verified the test fails with the expected -352 (ECC_PEERKEY_ERROR) when run against the code without this fix, confirming it catches the regression.

TLS 1.2 paths (GetEcDiffieHellmanKea, ImportPeerECCKey) share the same masking logic but are not yet covered by a dedicated test.

Checklist

• added tests
• updated/added doxygen
• updated appropriate READMEs
• Updated manual and documentation

[wolfSSL-Bot]

Can one of the admins verify this patch?

[Copilot]

Pull request overview

This PR updates wolfSSL’s TLS handshake processing to comply with RFC 7748 §5 by masking (clearing) the reserved high bit in received 32-byte X25519 u-coordinates instead of rejecting them, while preserving the existing opt-in strict behavior when WOLFSSL_X25519_NO_MASK_PEER is defined. It also adds a TLS 1.3 regression test to ensure a tampered key_share with the reserved bit set no longer fails the handshake with ECC_PEERKEY_ERROR.

Changes:

• Mask the reserved high bit on received X25519 peer public values before calling wc_curve25519_check_public() and before importing the key (TLS 1.3 KeyShare and TLS 1.2 key exchange paths).
• Add a TLS 1.3 memio regression test that flips the reserved bit in the ClientHello X25519 key_share and confirms the server continues the handshake.
• Register the new test in the TLS 1.3 API test declarations.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File	Description
tests/api/test_tls13.h	Declares and registers the new TLS 1.3 regression test.
tests/api/test_tls13.c	Adds a memio-based TLS 1.3 test that tampers the X25519 key_share reserved bit and asserts the server proceeds.
src/tls.c	Masks the reserved high bit of the peer’s X25519 key_share value before validation/import in the TLS 1.3 path (unless WOLFSSL_X25519_NO_MASK_PEER).
src/internal.c	Masks the reserved high bit before validation/import in the TLS 1.2 peer public key parsing/import paths (unless WOLFSSL_X25519_NO_MASK_PEER).

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

MemBrowse Memory Report

gcc-arm-cortex-m4-dtls13

• FLASH: .text +64 B (+0.0%, 186,300 B / 1,048,576 B, total: 18% used)

gcc-arm-cortex-m4-openssl-compat

• FLASH: .text +128 B (+0.0%, 780,412 B / 1,048,576 B, total: 74% used)

gcc-arm-cortex-m4-pq

• FLASH: .text +64 B (+0.0%, 301,084 B / 1,048,576 B, total: 29% used)

gcc-arm-cortex-m4-tls13

• FLASH: .text +64 B (+0.0%, 241,697 B / 262,144 B, total: 92% used)

gcc-arm-cortex-m7-tls13

• FLASH: .text +64 B (+0.0%, 241,761 B / 262,144 B, total: 92% used)

linuxkm-standard

• Data: __patchable_function_entries +8 B (+0.0%, 49,672 B)
  No memory changes detected for:
• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m3
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-crypto-only
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-pkcs7
• gcc-arm-cortex-m4-rsa-only
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m4-tls12
• gcc-arm-cortex-m7
• gcc-arm-cortex-m7-pq
• linuxkm-pie
• stm32-sim-stm32h753

[wolfSSL-Fenrir-bot]

Fenrir Automated Review — PR #11283

Scan targets checked: wolfcrypt-rs-bugs, wolfssl-bugs, wolfssl-src

Findings: 1
1 finding(s) posted as inline comments (see file-level comments below)

This review was generated automatically by Fenrir. Reported findings require changes before merge.

Findings dealt with