Integer Overflow in malloc: How a Silent Bug Becomes a Heap Overflow

Introduction

Some of the most dangerous security vulnerabilities don't announce themselves with loud crashes or obvious errors. Instead, they hide in plain sight — tucked inside a single arithmetic expression that looks completely harmless at first glance. Integer overflow vulnerabilities in memory allocation are exactly this kind of silent killer.

This post covers a high-severity integer overflow vulnerability found and fixed in src/coredump/_UCD_create.c, a C source file responsible for creating core dump analysis contexts. The bug was caught by a Semgrep static analysis rule (utils.custom.integer-overflow-malloc) and has since been patched.

If you write C or C++, or if you maintain any codebase that performs manual memory management, this is a vulnerability class you absolutely need to understand.

The Vulnerability Explained

What Is an Integer Overflow in malloc?

In C, when you allocate memory dynamically, you typically compute the size of the buffer you need and pass it to malloc() (or a similar allocator). A common pattern looks like this:

size_t count = get_item_count();
void *buf = malloc(count * sizeof(some_struct));

This looks reasonable. But here's the problem: count * sizeof(some_struct) is just arithmetic on integers. If count is large enough, the multiplication can overflow — wrapping around to a small number due to the fixed width of integer types in C.

For example, on a 32-bit or even 64-bit system:

size_t count = 0x40000001; // A large attacker-controlled value
size_t size  = count * 4;  // Wraps around! Result: 4 (not 4GB)

void *buf = malloc(4);     // Only 4 bytes allocated!
// But code proceeds to write count * 4 bytes into buf...
// → Heap overflow!

The allocator happily returns a tiny buffer, but the rest of the code assumes it's large enough to hold all the data. Writes beyond the buffer's end corrupt adjacent heap memory.

The Vulnerable Code Pattern

In src/coredump/_UCD_create.c at line 124, arithmetic multiplication was used to compute an allocation size without any overflow check:

// VULNERABLE: No overflow check before multiplication
size_t alloc_size = n_threads * sizeof(coredump_thread_t);
threads = malloc(alloc_size);

If n_threads is attacker-influenced (e.g., read from a malformed core dump file), a crafted value could cause the multiplication to wrap, resulting in a drastically undersized allocation.

How Could This Be Exploited?

Here's a realistic attack scenario:

1. Attacker crafts a malicious core dump file with a manipulated thread count field — say, n_threads = 0x20000000 (536 million).
2. The vulnerable code computes: 0x20000000 * 8 = 0x100000000, which on a 32-bit size_t wraps to 0. On a 64-bit system with smaller struct sizes, similar wraps are achievable.
3. malloc(0) or malloc(small_number) succeeds, returning a tiny (or implementation-defined) buffer.
4. The code then iterates over all n_threads entries, writing structured data into what it believes is a large buffer — but is actually tiny.
5. Heap memory is corrupted, potentially overwriting heap metadata, function pointers, or other sensitive allocations.
6. With enough control, an attacker may achieve arbitrary code execution.

Real-World Impact

This vulnerability is particularly dangerous in tools that process untrusted input files — like core dump analyzers, which by their very nature consume externally-supplied binary data.

The Fix

What Changed?

The fix introduces an overflow check before the multiplication is used to compute the allocation size. The corrected pattern validates that the multiplication won't exceed the maximum representable value before proceeding.

Here's the general structure of a safe fix:

// SAFE: Check for overflow before multiplying
if (n_threads > SIZE_MAX / sizeof(coredump_thread_t)) {
    // Handle error: would overflow
    return NULL;
}

size_t alloc_size = n_threads * sizeof(coredump_thread_t);
threads = malloc(alloc_size);
if (threads == NULL) {
    // Handle allocation failure
    return NULL;
}

Why This Works

The key insight is the pre-division check:

if (n_threads > SIZE_MAX / sizeof(coredump_thread_t))

By dividing SIZE_MAX by the element size first, we compute the maximum safe value of n_threads. If n_threads exceeds this threshold, we know the multiplication would overflow — and we bail out before any damage is done.

This approach:
- ✅ Works entirely with integer arithmetic (no floating point needed)
- ✅ Is portable across 32-bit and 64-bit platforms
- ✅ Handles edge cases like sizeof() == 0 (though that's rare in practice)
- ✅ Adds negligible performance overhead

Alternative: Use calloc()

Another clean approach is to use calloc() instead of malloc() for array allocations:

// calloc performs the overflow check internally (on most implementations)
threads = calloc(n_threads, sizeof(coredump_thread_t));

calloc(nmemb, size) takes the count and element size as separate arguments, allowing the implementation to perform its own overflow check. It also zero-initializes the memory, which can prevent some classes of uninitialized-read bugs. However, relying solely on calloc without understanding your platform's guarantees is not a substitute for explicit validation of untrusted input values.

Prevention & Best Practices

1. Always Validate Sizes Before Allocation

Never trust externally-supplied size values directly. Validate bounds before use:

#define MAX_REASONABLE_THREADS 65536

if (n_threads == 0 || n_threads > MAX_REASONABLE_THREADS) {
    return -EINVAL;
}

2. Use Safe Multiplication Helpers

Consider wrapping your size calculations in a helper:

#include <stdint.h>

static inline int safe_mul_size(size_t a, size_t b, size_t *result) {
    if (b != 0 && a > SIZE_MAX / b) {
        return -1; // Would overflow
    }
    *result = a * b;
    return 0;
}

// Usage:
size_t alloc_size;
if (safe_mul_size(n_threads, sizeof(coredump_thread_t), &alloc_size) != 0) {
    return -EOVERFLOW;
}
void *buf = malloc(alloc_size);

3. Leverage Static Analysis Tools

The vulnerability in this case was caught by Semgrep with a custom rule targeting unsafe multiplication in allocation contexts. Add similar tools to your CI pipeline:

Run these tools on every pull request — don't wait for a security audit.

4. Use Compiler Flags

Enable compiler-level overflow detection during development and testing:

# GCC/Clang: Enable undefined behavior sanitizer
gcc -fsanitize=undefined,address -o myprogram myprogram.c

# Enable overflow warnings
gcc -Wall -Wextra -Woverflow myprogram.c

5. Consider Safer Abstractions

Where possible, use higher-level abstractions that handle size arithmetic safely:

• In C++: Use std::vector, std::array, or smart pointers
• In C: Consider libraries like safe-iop for safe integer operations
• Use OS/platform APIs that perform bounds checking internally

6. Relevant Security Standards

This vulnerability class is well-documented in the security community:

• CWE-190: Integer Overflow or Wraparound
• CWE-122: Heap-based Buffer Overflow
• CWE-680: Integer Overflow to Buffer Overflow
• OWASP: Memory Management Cheat Sheet
• SEI CERT C: Rule INT30-C — Ensure that unsigned integer operations do not wrap
• SEI CERT C: Rule MEM35-C — Allocate sufficient memory for an object

A Note on Core Dump Parsers and Attack Surface

It's worth pausing to think about where this vulnerability lives: a core dump parser. Core dump analyzers are often run by developers and security researchers on files from production systems, crash reports, or — in adversarial contexts — files submitted by external parties.

This makes the attack surface non-trivial. A malicious actor who can influence the core dump file being analyzed (e.g., by submitting a crash report, or providing a "test" core file) could potentially exploit this vulnerability to compromise the machine running the analyzer. Parsing untrusted binary data is inherently high-risk, and every size field, count field, and offset in such files must be treated as adversarial input.

Conclusion

Integer overflow vulnerabilities in memory allocation are a classic, well-understood bug class — yet they continue to appear in production code. The pattern is deceptively simple: a multiplication that looks correct under normal inputs silently wraps under adversarial ones, producing a buffer that's far too small and setting the stage for heap corruption.

The key takeaways from this fix:

• 🔍 Always check for overflow before using multiplication to compute allocation sizes
• 🛡️ Treat all externally-supplied size values as untrusted — validate and bound them
• 🤖 Use static analysis tools like Semgrep, CodeQL, or Coverity in your CI pipeline to catch these patterns automatically
• 🧪 Run sanitizers (ASan, UBSan) during testing to catch runtime overflows that static analysis misses
• 📚 Know your CWEs — CWE-190 and CWE-680 are the canonical references for this vulnerability class

Security is a discipline, not a feature. Catching and fixing bugs like this one — before they reach production — is exactly what a healthy security practice looks like. Kudos to the automated scanner and the team for addressing this promptly.

This vulnerability was identified and fixed as part of an automated security scanning process. The fix was verified by re-scan and code review before merging.