Critical Integer Sign Bug in runtime_malloc(): How a Missing Check Enables Heap Corruption

Introduction

There is a category of security bug that looks almost embarrassingly small on paper — a single missing if statement — yet carries consequences that can range from a denial-of-service crash all the way to full process compromise. The vulnerability patched in this post belongs squarely in that category.

The culprit is runtime_malloc(), a thin wrapper around the standard C malloc() function in runtime/zenith_runtime.c. The function accepted a signed long parameter and cast it directly to the unsigned size_t type that malloc() expects — with no check to confirm the value was positive. That single omission opened the door to heap corruption.

If you write C or C++, maintain a runtime or plugin host, or simply want to understand why integer type mismatches are a perennial source of critical CVEs, read on.

The Vulnerability Explained

What Went Wrong

Here is the original function in its entirety:

// VULNERABLE — runtime/zenith_runtime.c (before fix)
void* runtime_malloc(long size) { return malloc((size_t)size); }

The problem lives in the implicit contract between long and size_t:

When a negative long value is cast to size_t, C's unsigned integer wrap-around rules apply. The result is a very large positive number — often near the maximum value of the type.

Two Distinct Attack Paths

Path 1 — Enormous allocation → NULL dereference

Pass -1 as the size. After casting, size_t receives 0xFFFFFFFFFFFFFFFF (18.4 exabytes on a 64-bit system). malloc() cannot satisfy that request and returns NULL. Because the return value is never checked, any caller that immediately dereferences the pointer triggers undefined behavior — typically a segmentation fault and process crash.

// Attacker-controlled call
void* buf = runtime_malloc(-1);
memcpy(buf, data, len);  // CRASH: NULL dereference

Path 2 — Undersized allocation → heap buffer overflow

This path is subtler and more dangerous. A carefully chosen small negative value wraps around to a small positive size_t:

-65528 (long)  →  0xFFFFFFFFFFFF0008  …  but on many platforms
-65528 cast to size_t (64-bit) = 18446744073709486088

// However on 32-bit or with truncation scenarios:
(uint32_t)(long)-65528  →  0xFFFF0008  = 4294901768 — still large

// The classic 16-bit-era variant (still relevant in embedded):
(uint16_t)(int16_t)-8  →  0xFFF8 = 65528

More practically, consider a plugin host that reads a size field from an untrusted message and forwards it to runtime_malloc(). An attacker submits a size of -8. After the cast, malloc(8) succeeds and returns a valid 8-byte buffer. The caller, trusting the original -8 value (or a derived expected-size), then writes far more data into that buffer — overflowing the heap allocation and corrupting adjacent heap metadata or live objects.

Heap metadata corruption is one of the most powerful primitives available to an attacker: it can be escalated to arbitrary read/write and, under the right conditions, to arbitrary code execution.

CWE Classification

This vulnerability maps to several Common Weakness Enumeration entries:

• CWE-195: Signed to Unsigned Conversion Error
• CWE-122: Heap-based Buffer Overflow (the downstream consequence)
• CWE-476: NULL Pointer Dereference (the other downstream consequence)
• CWE-190: Integer Overflow or Wraparound

The Fix

The patch is a single, focused guard clause inserted before the cast ever happens:

// BEFORE — vulnerable
void* runtime_malloc(long size) { return malloc((size_t)size); }

// AFTER — fixed
void* runtime_malloc(long size) { if (size <= 0) return NULL; return malloc((size_t)size); }

Why This Works

The guard if (size <= 0) return NULL enforces a precondition at the only entry point into the allocation path:

1. Zero is rejected. A zero-byte allocation is technically implementation-defined in C (it may return a valid unique pointer or NULL). Returning NULL for zero is the safest and most predictable behavior for a runtime wrapper.
2. Negative values are rejected. No negative long can ever reach the (size_t) cast, eliminating the wrap-around entirely.
3. Callers already handle NULL. Any well-written caller of runtime_malloc() must already handle a NULL return (e.g., out-of-memory). Returning NULL for invalid sizes slots cleanly into that existing error path without requiring call-site changes.

A Note on Robustness

For a production runtime, you might go one step further and add an upper-bound sanity check:

// Defensive production version
#define RUNTIME_MAX_ALLOC (1UL << 30)  // 1 GiB — tune to your use case

void* runtime_malloc(long size) {
    if (size <= 0 || (unsigned long)size > RUNTIME_MAX_ALLOC) {
        return NULL;
    }
    return malloc((size_t)size);
}

This caps runaway allocations and makes the function's contract explicit in the code itself.

Prevention & Best Practices

1. Prefer Unsigned Types for Sizes at API Boundaries

If a parameter represents a memory size, it should almost never be signed. Use size_t or uint32_t/uint64_t from the start:

// Better API design — sign confusion is impossible
void* runtime_malloc(size_t size) {
    if (size == 0) return NULL;
    return malloc(size);
}

Callers with a long value are then forced to perform the conversion themselves, making the potential sign issue visible at the call site rather than hidden inside the wrapper.

2. Enable Compiler Warnings

Modern compilers can catch signed/unsigned mismatches at compile time:

# GCC / Clang
-Wall -Wextra -Wsign-conversion -Wconversion

# MSVC
/W4 /analyze

-Wsign-conversion in particular will flag implicit conversions between signed and unsigned integer types — exactly the class of bug seen here.

3. Use Static Analysis

Tools that would have flagged this before it shipped:

4. Fuzz the Allocation Path

Any function that accepts an externally influenced size and calls malloc() is a prime fuzzing target. Add it to your fuzzing harness and include negative values, zero, and boundary values like LONG_MIN, -1, 0, 1, and LONG_MAX in your seed corpus.

// Example libFuzzer entry point
int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
    if (size < sizeof(long)) return 0;
    long alloc_size;
    memcpy(&alloc_size, data, sizeof(long));
    void* p = runtime_malloc(alloc_size);
    if (p) runtime_free(p);
    return 0;
}

5. Always Check malloc() Return Values

Even with the sign guard in place, malloc() can still return NULL on legitimate out-of-memory conditions. Every call site should handle this:

void* buf = runtime_malloc(requested_size);
if (buf == NULL) {
    // handle allocation failure gracefully
    return ERROR_OUT_OF_MEMORY;
}

6. Consult the Standards

• CERT C Coding Standard — INT31-C: Ensure integer conversions do not result in lost or misinterpreted data.
• OWASP — Integer Overflow: https://owasp.org/www-community/vulnerabilities/Integer_overflow
• CWE-195: https://cwe.mitre.org/data/definitions/195.html

Conclusion

The runtime_malloc() vulnerability is a textbook example of how a type mismatch between a signed input and an unsigned allocation function can cascade from "looks fine" to "critical severity" in a single step. The fix is minimal — one guard clause — but the protection it provides is substantial: it closes off both the NULL-dereference crash path and the heap buffer overflow path simultaneously.

Key takeaways for every C and C++ developer:

• Match your types. If a value represents a memory size, use an unsigned type.
• Validate before you cast. Never pass an externally influenced value through a sign-changing cast without first confirming it is in a safe range.
• Turn on conversion warnings. Let the compiler do the first pass of sign-mismatch detection for free.
• Fuzz your allocation wrappers. Negative and boundary values are exactly what attackers will try.

A single if (size <= 0) return NULL; prevented what could have been a heap corruption exploit. That is the power — and the responsibility — of secure coding at the systems level.