How buffer overflow in P-256 key decompression happens in C with mbedTLS and how to fix it

Summary

A critical buffer overflow vulnerability was discovered in helpers/src/edhoc_cipher_suite_2.c within the EDHOC cipher suite 2 implementation. The mbedtls_ecp_decompress() function used raw_key_len to copy a compressed peer public key into a fixed-size buffer without first verifying that the key length fit within the destination. An attacker sending a crafted EDHOC message with an oversized compressed key could exploit this to corrupt adjacent memory, potentially achieving remote code execution.

Introduction

The helpers/src/edhoc_cipher_suite_2.c file implements the P-256 elliptic curve key agreement for the EDHOC (Ephemeral Diffie-Hellman Over COSE) protocol — a lightweight authenticated key exchange designed for constrained IoT devices. Because EDHOC runs over untrusted networks and processes messages from potentially hostile peers, the key decompression logic is a high-value target.

Inside the static function mbedtls_ecp_decompress(), a compressed peer public key arrives as a byte array with an attacker-controlled length (raw_key_len). The function is responsible for expanding that compressed point into its uncompressed form — a 65-byte 0x04 || X || Y structure — and storing it in decomp_key. The problem: before this fix, the code trusted raw_key_len unconditionally and passed it straight to memcpy().

The Vulnerability Explained

The vulnerable code (before the fix)

At line 72 of helpers/src/edhoc_cipher_suite_2.c, the decompression function looked like this:

/* decomp_key will consist of 0x04|X|Y */
(void)memcpy(&decomp_key[1], raw_key, raw_key_len);
decomp_key[0] = 0x04;

The destination decomp_key is sized to hold exactly one prefix byte (0x04) plus the P-256 field element — 32 bytes for the X coordinate, making the buffer 33 bytes in total. The function already checked whether *decomp_key_len was large enough for the output (the full uncompressed point), but it never checked whether raw_key_len — the input length — was small enough to fit in the data portion of decomp_key.

For P-256, raw_key_len should always be exactly 32 bytes (the X coordinate of the compressed point). But nothing in the code enforced this. An attacker could craft an EDHOC message where the compressed key field claims to be 128 bytes long, and the memcpy would obediently write 128 bytes starting at &decomp_key[1] — blowing 96 bytes past the end of a 33-byte buffer.

How an attacker exploits this

The attack surface is straightforward: EDHOC processes messages from unauthenticated peers during the key exchange handshake. Before any authentication is established, the responder parses the initiator's ephemeral public key from the EDHOC message. This key is passed through the cipher suite 2 key agreement path, which calls mbedtls_ecp_decompress() with a raw_key_len derived directly from the parsed message.

Step-by-step exploitation:

1. Attacker crafts an EDHOC MESSAGE_1 with a compressed public key field whose CBOR-encoded length claims 128 bytes.
2. The EDHOC parser extracts the key bytes and length, setting raw_key_len = 128.
3. mbedtls_ecp_decompress() is called with this attacker-controlled length.
4. memcpy(&decomp_key[1], raw_key, 128) writes 96 bytes past the end of the 33-byte decomp_key buffer.
5. Depending on the memory layout, this overwrites adjacent stack variables, return addresses, or heap metadata — enabling memory corruption, denial of service, or potentially arbitrary code execution.

The PR description notes that similar unchecked patterns exist at line 331 (pub_key_size without validation) and in helpers/src/edhoc_helpers.c:316 (connection ID length not validated against bstr_value), making this a systemic pattern in the EDHOC helper library.

Real-world impact

This vulnerability is in production library code that processes untrusted network input. EDHOC is specifically designed for IoT and embedded systems — environments where memory corruption vulnerabilities are particularly dangerous because:

• Devices often lack ASLR and stack canaries
• A single compromised device can pivot to attack the broader IoT network
• Remote exploitation without authentication is possible since the overflow occurs before the handshake completes

The Fix

The fix is surgical and precise. A single bounds check was inserted immediately before the memcpy call:

Before

/* decomp_key will consist of 0x04|X|Y */
(void)memcpy(&decomp_key[1], raw_key, raw_key_len);
decomp_key[0] = 0x04;

After

if (raw_key_len > p_len) {
    return MBEDTLS_ERR_ECP_BUFFER_TOO_SMALL;
}

/* decomp_key will consist of 0x04|X|Y */
(void)memcpy(&decomp_key[1], raw_key, raw_key_len);
decomp_key[0] = 0x04;

Here, p_len is the byte length of the P-256 prime field — exactly 32 bytes for P-256. This is the correct upper bound because the X coordinate of a P-256 point can never exceed the field size. If raw_key_len is larger than p_len, the input is cryptographically invalid and the function returns MBEDTLS_ERR_ECP_BUFFER_TOO_SMALL immediately, before any bytes are copied.

Why this fix is correct

The choice of p_len as the bound is semantically meaningful, not just a buffer-size guard. In P-256 key decompression:

• The input is the compressed X coordinate, which must be exactly p_len bytes (32 for P-256)
• decomp_key is allocated as 1 + 2 * p_len bytes (prefix + X + Y)
• Writing raw_key_len bytes at offset 1 is safe if and only if raw_key_len <= p_len

By checking raw_key_len > p_len, the fix simultaneously:
1. Prevents the buffer overflow — no write can exceed decomp_key's data portion
2. Rejects cryptographically invalid input — keys longer than the field size are mathematically nonsensical for P-256
3. Propagates a meaningful error code — callers receive MBEDTLS_ERR_ECP_BUFFER_TOO_SMALL, which is the correct mbedTLS error for this condition

The regression test

The PR also adds a dedicated test case in tests/unit/test_crypto_suite2.c:

/**
 * @scenario  key_agreement rejects a compressed peer key longer than p_len (33 bytes).
 * @action    Call key_agreement with peer_pub_key_len = 33 (one byte over P-256 field size).
 * @expected  Returns EDHOC_ERROR_CRYPTO_FAILURE; mbedtls_ecp_decompress bounds
 *            check prevents the overlong key from being copied.
 */
TEST(crypto_suite2, key_agreement_peer_key_oversized)

This test specifically exercises the boundary condition — a key that is exactly one byte over the P-256 field size — ensuring the fix holds and cannot regress silently.

Prevention & Best Practices

1. Treat all length fields from network input as untrusted

Any length value parsed from a network message — even from a well-structured protocol like EDHOC — must be validated against the actual destination buffer before use in memcpy, memmove, or similar functions. This is true even if the protocol spec defines valid ranges, because attackers do not follow specs.

/* Pattern: always validate before copy */
if (src_len > dest_capacity) {
    return ERROR_INVALID_INPUT;
}
memcpy(dest, src, src_len);

2. Use memcpy_s() where available (C11 Annex K)

The C11 standard introduced bounds-checking versions of memory functions:

errno_t err = memcpy_s(&decomp_key[1], dest_capacity, raw_key, raw_key_len);
if (err != 0) {
    return MBEDTLS_ERR_ECP_BUFFER_TOO_SMALL;
}

memcpy_s() will return a non-zero error code if raw_key_len > dest_capacity, providing a safety net even if the explicit check is missed.

3. Apply defense-in-depth with compiler hardening

For production builds processing network input, enable:

• Stack canaries: -fstack-protector-strong (GCC/Clang)
• FORTIFY_SOURCE: -D_FORTIFY_SOURCE=2 — catches some unsafe memcpy calls at compile time
• AddressSanitizer in CI: -fsanitize=address — would have caught this overflow immediately in testing
• Control Flow Integrity: -fsanitize=cfi on supported platforms

4. Audit all memcpy calls in protocol parsing code

The PR notes that similar patterns exist at edhoc_cipher_suite_2.c:331 and edhoc_helpers.c:316. When fixing one instance of a vulnerability class, always search for structurally similar patterns in the same codebase. A targeted grep or Semgrep rule can surface all memcpy calls where the length argument is derived from parsed data:

# Find memcpy calls with potentially unchecked length variables
semgrep --pattern 'memcpy($DST, $SRC, $LEN)' --lang c helpers/src/

5. Relevant standards and references

• CWE-120: Buffer Copy without Checking Size of Input — https://cwe.mitre.org/data/definitions/120.html
• OWASP: Buffer Overflow prevention — https://cheatsheetseries.owasp.org/cheatsheets/C-Based_Toolchain_Hardening_Cheat_Sheet.html
• CERT C: MEM35-C — Allocate sufficient memory for an object — https://wiki.sei.cmu.edu/confluence/display/c/MEM35-C

Key Takeaways

• raw_key_len from a parsed EDHOC message is attacker-controlled — it must never be used in memcpy() without first checking it against the P-256 field size (p_len = 32).
• The overflow in mbedtls_ecp_decompress() occurred before authentication — meaning any unauthenticated network peer could trigger it during the EDHOC handshake.
• The fix is a single comparison (raw_key_len > p_len) — a tiny change with a large security impact, illustrating how one missing bounds check can create a critical vulnerability.
• Similar unchecked patterns existed at lines 331 and edhoc_helpers.c:316 — finding one instance of this pattern should always trigger a broader audit of the same file and related helpers.
• Regression tests should encode the exact boundary condition — the new test uses peer_pub_key_len = 33 (one byte over the field size) to precisely guard the boundary that was previously unprotected.

How Orbis AppSec Detected This

• Source: raw_key_len — a length field parsed directly from an untrusted EDHOC network message
• Sink: memcpy(&decomp_key[1], raw_key, raw_key_len) in helpers/src/edhoc_cipher_suite_2.c:72
• Missing control: No validation of raw_key_len against p_len (the P-256 field size, 32 bytes) before the copy; the destination buffer decomp_key can only safely hold p_len bytes at offset 1
• CWE: CWE-120 — Buffer Copy without Checking Size of Input ('Classic Buffer Overflow')
• Fix: Added if (raw_key_len > p_len) return MBEDTLS_ERR_ECP_BUFFER_TOO_SMALL; immediately before the memcpy call, rejecting any key longer than the P-256 field size before any bytes are written

Orbis AppSec automatically detected this vulnerability and opened a pull request with the fix. Try Orbis AppSec on your repositories to find and fix issues like this automatically.

Conclusion

This vulnerability is a textbook example of why every length value from the network must be treated as adversarial input, even in well-specified cryptographic protocols. The mbedtls_ecp_decompress() function in the EDHOC cipher suite 2 helper had all the right structure — it checked output buffer sizes, used standard mbedTLS error codes, and followed the protocol spec — but it missed the one check that mattered most: whether the input key length was small enough to fit in the destination buffer.

The fix is three lines. The potential impact without it — memory corruption in an IoT key exchange library processing unauthenticated peer messages — was critical. For developers writing C code that handles cryptographic protocol messages, the lesson is clear: validate lengths before copying, use p_len (or equivalent field-size constants) as semantic upper bounds, and run AddressSanitizer in CI to catch these issues before they reach production.

References

• CWE-120: Buffer Copy without Checking Size of Input
• CWE-122: Heap-Based Buffer Overflow
• OWASP C-Based Toolchain Hardening Cheat Sheet
• mbedTLS ECP API Documentation
• Semgrep rules for unsafe memcpy patterns
• fix: add bounds check before memcpy in edhoc_cipher_suite_2.c