Heap Buffer Overflow in SSL/TLS: When Proto Length Goes Wrong

Severity: Critical | CVE Class: Heap Buffer Overflow | Component: SSL/TLS Protocol Negotiation (src/ssl.c)

Introduction

Memory safety bugs in SSL/TLS libraries are among the most dangerous vulnerabilities in existence. They sit at the intersection of two uncomfortable truths: cryptographic code is notoriously difficult to get right, and memory corruption bugs in C can have catastrophic consequences. The infamous Heartbleed vulnerability (CVE-2014-0160) reminded the entire industry just how devastating a single bounds-check omission in OpenSSL could be — exposing private keys, session tokens, and sensitive data at scale.

The vulnerability we're examining today follows a similar pattern: a missing bounds check during protocol list construction in an SSL implementation's ALPN/NPN negotiation code. Combined with an integer overflow in size arithmetic, this creates a classic "allocate too small, write too much" scenario that attackers have exploited for decades.

If you write C code that handles network protocols, cryptographic libraries, or any user-controlled binary data, this post is required reading.

The Vulnerability Explained

Background: ALPN and NPN Protocol Negotiation

Application-Layer Protocol Negotiation (ALPN) and its predecessor Next Protocol Negotiation (NPN) are TLS extensions that allow a client and server to agree on which application protocol to use (e.g., HTTP/2, HTTP/1.1, gRPC) during the TLS handshake — without adding extra round trips.

When configuring an SSL context in code (for example, via a Lua binding or a configuration API), a developer can specify a list of supported protocols. Internally, these protocols are packed into a binary buffer using a length-prefixed format:

[length_byte][protocol_string][length_byte][protocol_string]...

For example, the protocol list for ["h2", "http/1.1"] would be encoded as:

\x02h2\x08http/1.1

This buffer is then passed to the SSL library. The vulnerability lives in the code that builds this buffer.

The Vulnerable Code Pattern

The vulnerability exists at src/ssl.c:1021 and involves two distinct but related flaws:

Flaw 1: Missing Bounds Check Before memcpy

// VULNERABLE CODE (illustrative reconstruction)
static int ssl_set_protocols(SSL_CTX *ctx, const char **protocols, int count) {
    unsigned char proto_list[MAX_PROTO_LIST_LEN];
    size_t proto_list_len = 0;

    for (int i = 0; i < count; i++) {
        size_t proto_len = strlen(protocols[i]);

        proto_list[proto_list_len] = (unsigned char)proto_len;
        proto_list_len++;

        // ❌ NO CHECK: Is there enough space remaining in proto_list?
        memcpy(proto_list + proto_list_len, protocols[i], proto_len);
        proto_list_len += proto_len;
    }

    return SSL_CTX_set_alpn_protos(ctx, proto_list, proto_list_len);
}

The critical omission: there is no check that proto_list_len + proto_len is within the bounds of proto_list before calling memcpy. If an attacker supplies a crafted protocol name that is long enough, or enough protocol names in aggregate, the memcpy will write past the end of the buffer into adjacent heap memory.

Flaw 2: Integer Overflow in Size Arithmetic

In a dynamically allocated variant of the same pattern, the allocation size is computed as:

// VULNERABLE CODE (illustrative reconstruction)
size_t total_size = proto_list_len + proto_len; // ❌ Can overflow!
unsigned char *proto_list = realloc(proto_list, total_size);

If proto_list_len is near SIZE_MAX (the maximum value of a size_t), adding proto_len causes the result to wrap around to a small number. The realloc call then allocates a tiny buffer, but the subsequent memcpy writes the full, large amount of data — a classic integer overflow → heap overflow chain.

How Could This Be Exploited?

Attack Surface

The vulnerability is reachable through the Lua ALPN/NPN SSL configuration API, meaning any user or application that can pass protocol names through this interface could trigger the overflow. Depending on the deployment:

• A malicious configuration file specifying crafted protocol names
• A Lua script with attacker-controlled input passed to the SSL API
• A network-facing service that allows dynamic SSL configuration

Exploitation Scenario

Here's a realistic attack chain:

1. Attacker identifies a service using this SSL library with Lua bindings
2. Attacker crafts a protocol name that is 255+ bytes long, or supplies
   many short protocol names totaling > MAX_PROTO_LIST_LEN bytes
3. The memcpy overflows the heap buffer, overwriting adjacent heap metadata
   or other security-sensitive objects (function pointers, SSL session data)
4. Depending on heap layout:
   - Crash (Denial of Service)
   - Heap metadata corruption → arbitrary write primitive
   - Control flow hijack → Remote Code Execution

The Integer Overflow Path

proto_list_len = SIZE_MAX - 5  (e.g., 18446744073709551610 on 64-bit)
proto_len      = 10

total_size = (SIZE_MAX - 5) + 10
           = SIZE_MAX + 5
           = 4  (after 64-bit wraparound!)  ← allocates 4 bytes

memcpy(proto_list, data, 10)  ← writes 10 bytes into a 4-byte buffer

This is textbook CWE-190 (Integer Overflow) feeding directly into CWE-122 (Heap-Based Buffer Overflow).

Real-World Impact

The CVSS score for a vulnerability of this class in a network-facing SSL component routinely reaches 9.0–10.0 (Critical).

The Fix

The patch addresses both root causes: the missing capacity check and the integer overflow in size arithmetic.

Fixed Code Pattern

// FIXED CODE (illustrative)
#define MAX_PROTO_LIST_LEN 512  // or appropriate constant

static int ssl_set_protocols(SSL_CTX *ctx, const char **protocols, int count) {
    unsigned char proto_list[MAX_PROTO_LIST_LEN];
    size_t proto_list_len = 0;

    for (int i = 0; i < count; i++) {
        size_t proto_len = strlen(protocols[i]);

        // ✅ FIX 1: Validate proto_len is a valid single-byte length prefix value
        if (proto_len == 0 || proto_len > 255) {
            return -1;  // Protocol names must be 1-255 bytes per TLS spec
        }

        // ✅ FIX 2: Check for integer overflow before arithmetic
        if (proto_len > MAX_PROTO_LIST_LEN - proto_list_len - 1) {
            return -1;  // Not enough space remaining
        }

        // ✅ FIX 3: Bounds are now verified — safe to write
        proto_list[proto_list_len] = (unsigned char)proto_len;
        proto_list_len++;

        memcpy(proto_list + proto_list_len, protocols[i], proto_len);
        proto_list_len += proto_len;
    }

    return SSL_CTX_set_alpn_protos(ctx, proto_list, proto_list_len);
}

For the dynamic allocation variant:

// FIXED dynamic allocation
// ✅ Check for integer overflow BEFORE computing allocation size
if (proto_len > SIZE_MAX - proto_list_len) {
    return -1;  // Would overflow
}

size_t new_size = proto_list_len + proto_len;
unsigned char *new_list = realloc(proto_list, new_size);
if (new_list == NULL) {
    return -1;  // Allocation failure
}
proto_list = new_list;

What the Fix Actually Does

The fix follows the "validate inputs before use" principle — ensuring that all size values are within expected ranges before any memory operations take place.

Prevention & Best Practices

1. Always Validate Lengths Before Memory Operations

Never trust a length value — even one that came from your own code earlier in the call stack. Before any memcpy, memmove, or strcpy:

// Pattern: always check remaining capacity
if (src_len > dst_capacity - dst_used) {
    handle_error();
}
memcpy(dst + dst_used, src, src_len);

2. Use Overflow-Safe Arithmetic

In C, unsigned integer overflow is defined behavior (it wraps), but it's almost never what you want in security-sensitive code. Use explicit overflow checks:

// Safe addition check
if (b > SIZE_MAX - a) {
    // Would overflow
    return -1;
}
size_t result = a + b;

On modern compilers, you can also use built-in overflow-checking functions:

// GCC/Clang built-ins (preferred in new code)
size_t result;
if (__builtin_add_overflow(a, b, &result)) {
    return -1;
}

3. Prefer Safe Abstractions

When possible, avoid raw buffer management entirely:

• Use dynamic strings (e.g., strbuf, custom growable buffers with built-in bounds checking)
• Use memory-safe languages (Rust, Go) for new SSL/TLS tooling where feasible
• Use existing well-tested libraries (OpenSSL, BoringSSL, mbedTLS) rather than rolling your own protocol negotiation code

4. Enable Compiler and Runtime Protections

These won't prevent all exploits, but they raise the bar significantly:

# Compiler flags for security hardening
CFLAGS += -fstack-protector-strong
CFLAGS += -D_FORTIFY_SOURCE=2
CFLAGS += -fsanitize=address,undefined  # During development/testing
LDFLAGS += -Wl,-z,relro,-z,now

AddressSanitizer (ASan) would have caught this vulnerability immediately in testing:

==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x...
WRITE of size 200 at 0x... thread T0
    #0 memcpy
    #1 ssl_set_protocols  src/ssl.c:1021

5. Fuzz Your SSL Code

Protocol parsing and negotiation code is an ideal target for fuzzing:

# Using AFL++ or libFuzzer on the protocol list construction
# Provide a corpus of valid protocol name lists
# The fuzzer will quickly find length boundary conditions
afl-fuzz -i corpus/ -o findings/ -- ./ssl_proto_fuzz_target @@

6. Code Review Checklist for C Memory Operations

When reviewing C code involving buffers, ask:

• [ ] Is every memcpy/memmove preceded by a bounds check?
• [ ] Is every size arithmetic operation checked for integer overflow?
• [ ] Are all malloc/realloc return values checked for NULL?
• [ ] Are length values validated against protocol/spec constraints?
• [ ] Is the buffer size clearly documented and enforced?

Relevant Security Standards and References

Conclusion

This vulnerability is a textbook example of how two individually subtle bugs — a missing bounds check and an integer overflow — can combine to create a critical, potentially exploitable memory corruption issue in security-critical code. The fact that it lives in SSL/TLS protocol negotiation code makes it especially dangerous, since this component handles the very infrastructure we rely on to protect sensitive communications.

Key Takeaways

1. Length values are attacker-controlled data — treat them with the same skepticism as any other user input
2. Integer overflow + heap overflow is a classic chain — always check arithmetic before allocation
3. AddressSanitizer and fuzzing would catch this — integrate them into your CI pipeline
4. Validate against the spec — TLS protocol names are 1–255 bytes; enforce that constraint at the boundary
5. Defense in depth matters — compiler hardening flags won't prevent the bug, but they make exploitation significantly harder

Memory safety in C is hard, but it's not magic. The patterns for safe buffer handling are well-established and well-documented. The challenge is discipline: applying these patterns consistently, especially in the complex, performance-sensitive code paths where SSL implementations live.

The next time you write a memcpy, ask yourself: have I actually proven that there's enough space in the destination? That one question, asked consistently, prevents an entire class of critical vulnerabilities.

This vulnerability was identified and patched by the OrbisAI Security automated scanning system. Automated security scanning, combined with human review, is a powerful combination for catching memory safety issues before they reach production.

Further Reading
- Heartbleed: The Full Story
- CERT C Coding Standard: Memory Management
- Google Project Zero: Exploiting the Linux kernel via packet sockets
- Chromium's Rule of 2: Memory Safety