How integer overflow in TLS KDF buffer allocation happens in C with OpenSSL and how to fix it

Summary

A critical integer overflow vulnerability was discovered in OpenSSL's tls1_export_keying_material() function inside ssl/t1_enc.c, where attacker-influenced length values could wrap around during arithmetic, causing the vallen buffer to be allocated far smaller than needed. The four subsequent memcpy calls would then write beyond the heap buffer boundary, enabling potential remote code execution. The fix adds two targeted overflow checks before the arithmetic operations, preventing the allocation from proceeding with a corrupted size.

Introduction

The ssl/t1_enc.c file is one of the most security-critical files in the OpenSSL codebase — it implements the TLS encryption and key derivation machinery that protects billions of HTTPS connections. A flaw in tls1_export_keying_material(), specifically in how the function computes the size of its KDF value buffer at line 409, created a path to heap corruption that could be triggered by a remote attacker.

The root of the problem is deceptively simple: two attacker-influenced length fields, llen (the label length) and contextlen (the context length), are summed together with constants to produce vallen, which is then passed directly to a heap allocator. No overflow check was performed before the arithmetic. In C, when a size_t addition wraps around, the result is a legally valid but dangerously small number — and the allocator has no way to know the intent was different.

If you write C code that computes allocation sizes from external inputs, this vulnerability is a textbook example of why every addition in that path deserves an explicit overflow guard.

The Vulnerability Explained

What the code does

tls1_export_keying_material() implements the TLS Keying Material Exporter (RFC 5705 / RFC 8446). It allows TLS peers to derive additional keying material from the master secret, which is used by protocols layered on top of TLS (like EAP-TLS). The function accepts a label, a context buffer, and their respective lengths as parameters.

Before calling the underlying PRF, the function constructs a single concatenated value buffer. The size of this buffer is computed at line 409:

/* VULNERABLE CODE (before fix) */
vallen = llen + SSL3_RANDOM_SIZE * 2;
if (use_context) {
    vallen += 2 + contextlen;
}

Here, llen is the length of the caller-supplied label, and contextlen is the length of the caller-supplied context. Both values originate from attacker-influenced TLS message fields.

The overflow scenario

vallen is a size_t. On a 64-bit system, SIZE_MAX is 0xFFFFFFFFFFFFFFFF. Consider what happens if an attacker supplies:

• llen = SIZE_MAX - SSL3_RANDOM_SIZE * 2 + 1 (i.e., just large enough to wrap)

Then:

vallen = (SIZE_MAX - 64 + 1) + 64;
// = SIZE_MAX + 1
// = 0 (wraps around to zero on size_t)

The heap allocator receives a request for 0 bytes (or some tiny value). It succeeds and returns a valid pointer to a near-empty buffer. The function then proceeds to execute four memcpy calls at lines 413, 415, 417, and 426, each writing the full intended data into what is effectively a zero-byte allocation. The result is a classic heap buffer overflow.

Why this is exploitable

Heap corruption from controlled overflow is one of the most powerful primitives available to an attacker. By carefully crafting the overflow and the subsequent writes, an attacker can:

1. Overwrite heap metadata — corrupting allocator bookkeeping to redirect future allocations
2. Overwrite adjacent heap objects — for example, function pointers or SSL session state stored nearby
3. Achieve remote code execution — by redirecting control flow to attacker-controlled data

Because tls1_export_keying_material() is reachable during a TLS handshake from a remote peer (any TLS client or server that sends a crafted label/context in a keying material export request), this is a remotely exploitable, pre-authentication vulnerability in a critical cryptographic path.

The Fix

The fix adds two explicit overflow checks — one before each addition that contributes to vallen — using the standard C idiom for safe size_t arithmetic.

Before the fix

vallen = llen + SSL3_RANDOM_SIZE * 2;
if (use_context) {
    vallen += 2 + contextlen;
}

No bounds checking. Both additions silently wrap if inputs are large enough.

After the fix

if (llen > SIZE_MAX - SSL3_RANDOM_SIZE * 2) {
    ERR_raise(ERR_LIB_SSL, ERR_R_PASSED_INVALID_ARGUMENT);
    return 0;
}
vallen = llen + SSL3_RANDOM_SIZE * 2;
if (use_context) {
    if (vallen > SIZE_MAX - 2 - contextlen) {
        ERR_raise(ERR_LIB_SSL, ERR_R_PASSED_INVALID_ARGUMENT);
        return 0;
    }
    vallen += 2 + contextlen;
}

Why this specific approach works

Check 1 — llen > SIZE_MAX - SSL3_RANDOM_SIZE * 2

This is the standard safe-addition idiom in C: instead of checking llen + constant > SIZE_MAX (which itself could overflow!), we rearrange to llen > SIZE_MAX - constant. Since SSL3_RANDOM_SIZE * 2 is a compile-time constant (64 bytes), the right-hand side is a fixed, safe value. If llen exceeds it, the addition would overflow, so we abort immediately with ERR_raise() and return 0.

Check 2 — vallen > SIZE_MAX - 2 - contextlen

After the first addition, vallen holds llen + 64. The second check guards the conditional addition of 2 + contextlen. Note the check uses the current value of vallen (already partially computed), so it correctly accounts for the accumulated total, not just the new addend.

ERR_raise() and early return

Rather than silently truncating or clamping the value, the fix uses OpenSSL's error reporting mechanism to signal that an invalid argument was passed, and returns 0 to indicate failure. This is the correct behavior: callers can inspect the error queue and handle the failure gracefully, rather than proceeding with a corrupted state.

Prevention & Best Practices

1. Always guard size arithmetic before heap allocation in C

Any time you write code of the form:

size_t n = a + b + c;
buf = OPENSSL_malloc(n);

...you must verify that a + b + c cannot overflow size_t. The correct idiom:

if (a > SIZE_MAX - b) { /* handle error */ }
size_t ab = a + b;
if (ab > SIZE_MAX - c) { /* handle error */ }
size_t n = ab + c;

2. Use safe integer libraries where available

Languages and runtimes that support checked arithmetic (Rust's checked_add(), Swift's &+ overflow operators, Java's Math.addExact()) eliminate this class of bug entirely. For C, consider using helper macros or functions like those in safe_iop or compiler built-ins:

// GCC/Clang built-in
size_t result;
if (__builtin_add_overflow(llen, SSL3_RANDOM_SIZE * 2, &result)) {
    // overflow detected
}

3. Treat all length fields from external sources as attacker-controlled

In TLS, QUIC, and similar protocols, every length field in a message can be crafted by a remote peer. Length fields should be validated against both minimum and maximum bounds before being used in any arithmetic.

4. Enable compiler and sanitizer warnings

• -fsanitize=undefined (UBSan): catches integer overflow at runtime during testing
• -Wall -Wextra: enables many useful warnings
• AddressSanitizer (-fsanitize=address): detects heap buffer overflows when they occur
• Static analysis: tools like CodeQL, Coverity, and Semgrep can flag unchecked arithmetic in allocation paths

5. Reference standards

• CWE-190: Integer Overflow or Wraparound — https://cwe.mitre.org/data/definitions/190.html
• CWE-122: Heap-based Buffer Overflow — https://cwe.mitre.org/data/definitions/122.html
• OWASP: Input Validation Cheat Sheet — https://cheatsheetseries.owasp.org/cheatsheets/Input_Validation_Cheat_Sheet.html

Key Takeaways

• llen and contextlen are attacker-controlled inputs in TLS keying material export — any arithmetic on them must be guarded before use in OPENSSL_malloc() or similar calls.
• The safe-subtraction idiom (a > SIZE_MAX - b) is the correct way to check size_t overflow in C — never check a + b > SIZE_MAX because that addition itself can overflow.
• Two separate checks are needed because vallen is built up in two distinct steps in tls1_export_keying_material(), and each step introduces a new overflow risk.
• ERR_raise() + return 0 is the correct OpenSSL failure pattern — it gives callers actionable error information rather than leaving the library in a corrupted state.
• Heap corruption from integer overflow in cryptographic code is remotely exploitable — the four memcpy calls that follow the allocation make this a write-what-where primitive, not merely a crash.

How Orbis AppSec Detected This

• Source: Attacker-supplied llen (label length) and contextlen (context length) fields from a TLS keying material export request
• Sink: OPENSSL_malloc(vallen) followed by four memcpy calls at lines 413, 415, 417, and 426 in ssl/t1_enc.c, where vallen is computed from the tainted inputs without overflow protection
• Missing control: No bounds check on the llen + SSL3_RANDOM_SIZE * 2 addition or the vallen + 2 + contextlen addition before the allocation size is finalized
• CWE: CWE-190 — Integer Overflow or Wraparound
• Fix: Two SIZE_MAX-based pre-addition overflow guards were inserted at lines 401–408 in ssl/t1_enc.c, causing the function to raise an SSL error and return 0 if either addition would overflow

Orbis AppSec automatically detected this vulnerability and opened a pull request with the fix. Try Orbis AppSec on your repositories to find and fix issues like this automatically.

Conclusion

Integer overflow in allocation size arithmetic is one of the oldest and most dangerous vulnerability classes in C, and it remains relevant in 2024 precisely because it appears in critical infrastructure code like OpenSSL. The tls1_export_keying_material() function is a perfect case study: the vulnerable code looks completely ordinary at a glance — just two additions — but those additions operate on attacker-controlled values in a security-critical path.

The fix is equally instructive: two lines of overflow checking, using the correct SIZE_MAX-subtraction idiom, completely eliminate the risk. There is no performance cost, no behavioral change for valid inputs, and no complexity added to the calling code. This is what good security patching looks like — minimal, targeted, and correct.

If you maintain C code that constructs allocation sizes from external inputs, audit every addition in those paths today.

References

• CWE-190: Integer Overflow or Wraparound
• CWE-122: Heap-based Buffer Overflow
• OWASP Input Validation Cheat Sheet
• OpenSSL ERR_raise() Documentation
• Semgrep rules for integer overflow
• fix: add integer overflow check in t1_enc.c