Heap Buffer Overflow in SSL/TLS: When Proto Length Goes Wrong

Severity: Critical | CVE Class: Heap Buffer Overflow | Component: SSL/TLS Protocol Negotiation (src/ssl.c)

Introduction

Memory safety bugs in SSL/TLS libraries are among the most dangerous vulnerabilities in existence. They sit at the intersection of two uncomfortable truths: cryptographic code is notoriously difficult to get right, and memory corruption bugs in C can have catastrophic consequences. The infamous Heartbleed vulnerability (CVE-2014-0160) reminded the entire industry just how devastating a single bounds-check omission in OpenSSL could be — exposing private keys, session tokens, and sensitive data at scale.

The vulnerability we're examining today follows a similar pattern: a missing bounds check during protocol list construction in an SSL implementation's ALPN/NPN negotiation code. Combined with an integer overflow in size arithmetic, this creates a classic "allocate too small, write too much" scenario that attackers have exploited for decades.

If you write C code that handles network protocols, cryptographic libraries, or any user-controlled binary data, this post is required reading.

The Vulnerability Explained

Background: ALPN and NPN Protocol Negotiation

Application-Layer Protocol Negotiation (ALPN) and its predecessor Next Protocol Negotiation (NPN) are TLS extensions that allow a client and server to agree on which application protocol to use (e.g., HTTP/2, HTTP/1.1, gRPC) during the TLS handshake — without adding extra round trips.

When configuring an SSL context in code (for example, via a Lua binding or a configuration API), a developer can specify a list of supported protocols. Internally, these protocols are packed into a binary buffer using a length-prefixed format:

[length_byte][protocol_string][length_byte][protocol_string]...

For example, the protocol list for ["h2", "http/1.1"] would be encoded as:

\x02h2\x08http/1.1

This buffer is then passed to the SSL library. The vulnerability lives in the code that builds this buffer.

The Vulnerable Code Pattern

The vulnerability exists at src/ssl.c:1021 and involves two distinct but related flaws:

Flaw 1: Missing Bounds Check Before memcpy

// VULNERABLE CODE (illustrative reconstruction)
static int ssl_set_protocols(SSL_CTX *ctx, const char **protocols, int count) {
    unsigned char proto_list[MAX_PROTO_LIST_LEN];
    size_t proto_list_len = 0;

    for (int i = 0; i < count; i++) {
        size_t proto_len = strlen(protocols[i]);

        proto_list[proto_list_len] = (unsigned char)proto_len;
        proto_list_len++;

        // ❌ NO CHECK: Is there enough space remaining in proto_list?
        memcpy(proto_list + proto_list_len, protocols[i], proto_len);
        proto_list_len += proto_len;
    }

    return SSL_CTX_set_alpn_protos(ctx, proto_list, proto_list_len);
}

The critical omission: there is no check that proto_list_len + proto_len is within the bounds of proto_list before calling memcpy. If an attacker supplies a crafted protocol name that is long enough, or enough protocol names in aggregate, the memcpy will write past the end of the buffer into adjacent heap memory.

Flaw 2: Integer Overflow in Size Arithmetic

In a dynamically allocated variant of the same pattern, the allocation size is computed as:

// VULNERABLE CODE (illustrative reconstruction)
size_t total_size = proto_list_len + proto_len; // ❌ Can overflow!
unsigned char *proto_list = realloc(proto_list, total_size);

If proto_list_len is near SIZE_MAX (the maximum value of a size_t), adding proto_len causes the result to wrap around to a small number. The realloc call then allocates a tiny buffer, but the subsequent memcpy writes the full, large amount of data — a classic integer overflow → heap overflow chain.

How Could This Be Exploited?

Attack Surface

The vulnerability is reachable through the Lua ALPN/NPN SSL configuration API, meaning any user or application that can pass protocol names through this interface could trigger the overflow. Depending on the deployment:

• A malicious configuration file specifying crafted protocol names
• A Lua script with attacker-controlled input passed to the SSL API
• A network-facing service that allows dynamic SSL configuration

Exploitation Scenario

Here's a realistic attack chain:

1. Attacker identifies a service using this SSL library with Lua bindings
2. Attacker crafts a protocol name that is 255+ bytes long, or supplies
   many short protocol names totaling > MAX_PROTO_LIST_LEN bytes
3. The memcpy overflows the heap buffer, overwriting adjacent heap metadata
   or other security-sensitive objects (function pointers, SSL session data)
4. Depending on heap layout:
   - Crash (Denial of Service)
   - Heap metadata corruption → arbitrary write primitive
   - Control flow hijack → Remote Code Execution

The Integer Overflow Path

proto_list_len = SIZE_MAX - 5  (e.g., 18446744073709551610 on 64-bit)
proto_len      = 10

total_size = (SIZE_MAX - 5) + 10
           = SIZE_MAX + 5
           = 4  (after 64-bit wraparound!)  ← allocates 4 bytes

memcpy(proto_list, data, 10)  ← writes 10 bytes into a 4-byte buffer

This is textbook CWE-190 (Integer Overflow) feeding directly into CWE-122 (Heap-Based Buffer Overflow).

Real-World Impact

The CVSS score for a vulnerability of this class in a network-facing SSL component routinely reaches 9.0–10.0 (Critical).

The Fix

The patch addresses both root causes: the missing capacity check and the integer overflow in size arithmetic.

Fixed Code Pattern

// FIXED CODE (illustrative)
#define MAX_PROTO_LIST_LEN 512  // or appropriate constant

static int ssl_set_protocols(SSL_CTX *ctx, const char **protocols, int count) {
    unsigned char proto_list[MAX_PROTO_LIST_LEN];
    size_t proto_list_len = 0;

    for (int i = 0; i < count; i++) {
        size_t proto_len = strlen(protocols[i]);

        // ✅ FIX 1: Validate proto_len is a valid single-byte length prefix value
        if (proto_len == 0 || proto_len > 255) {
            return -1;  // Protocol names must be 1-255 bytes per TLS spec
        }

        // ✅ FIX 2: Check for integer overflow before arithmetic
        if (proto_len > MAX_PROTO_LIST_LEN - proto_list_len - 1) {
            return -1;  // Not enough space remaining
        }

        // ✅ FIX 3: Bounds are now verified — safe to write
        proto_list[proto_list_len] = (unsigned char)proto_len;
        proto_list_len++;

        memcpy(proto_list + proto_list_len, protocols[i], proto_len);
        proto_list_len += proto_len;
    }

    return SSL_CTX_set_alpn_protos(ctx, proto_list, proto_list_len);
}

For the dynamic allocation variant:

// FIXED dynamic allocation
// ✅ Check for integer overflow BEFORE computing allocation size
if (proto_len > SIZE_MAX - proto_list_len) {
    return -1;  // Would overflow
}

size_t new_size = proto_list_len + proto_len;
unsigned char *new_list = realloc(proto_list, new_size);
if (new_list == NULL) {
    return -1;  // Allocation failure
}
proto_list = new_list;

What the Fix Actually Does

The fix follows the "validate inputs before use" principle — ensuring that all size values are within expected ranges before any memory operations take place.

Conclusion

This vulnerability is a textbook example of how two individually subtle bugs — a missing bounds check and an integer overflow — can combine to create a critical, potentially exploitable memory corruption issue in security-critical code. The fact that it lives in SSL/TLS protocol negotiation code makes it especially dangerous, since this component handles the very infrastructure we rely on to protect sensitive communications.

Key Takeaways

1. Length values are attacker-controlled data — treat them with the same skepticism as any other user input
2. Integer overflow + heap overflow is a classic chain — always check arithmetic before allocation
3. AddressSanitizer and fuzzing would catch this — integrate them into your CI pipeline
4. Validate against the spec — TLS protocol names are 1–255 bytes; enforce that constraint at the boundary
5. Defense in depth matters — compiler hardening flags won't prevent the bug, but they make exploitation significantly harder

Memory safety in C is hard, but it's not magic. The patterns for safe buffer handling are well-established and well-documented. The challenge is discipline: applying these patterns consistently, especially in the complex, performance-sensitive code paths where SSL implementations live.

The next time you write a memcpy, ask yourself: have I actually proven that there's enough space in the destination? That one question, asked consistently, prevents an entire class of critical vulnerabilities.

This vulnerability was identified and patched by the OrbisAI Security automated scanning system. Automated security scanning, combined with human review, is a powerful combination for catching memory safety issues before they reach production.

Further Reading
- Heartbleed: The Full Story
- CERT C Coding Standard: Memory Management
- Google Project Zero: Exploiting the Linux kernel via packet sockets
- Chromium's Rule of 2: Memory Safety