Integer Overflow to Heap Corruption: Fixing a Critical Buffer Overflow in C Memory Allocation

Introduction

Memory safety bugs in C remain one of the most dangerous classes of vulnerabilities in software today. Despite decades of awareness, integer overflows leading to heap buffer overflows continue to appear in real-world codebases — including in test utilities and language bindings that developers often treat as "low-risk" code.

This post examines a critical severity integer overflow vulnerability (CWE-190) found in bindings/ruby/test/jfk_reader/jfk_reader.c, explains how it could be exploited, and walks through the fix that eliminates the risk. Whether you're a seasoned C developer or someone just learning about memory safety, this is a pattern you need to recognize.

The Vulnerability Explained

What Is an Integer Overflow?

An integer overflow occurs when an arithmetic operation produces a result that exceeds the maximum value a data type can hold. In C, when this happens with unsigned integers, the value wraps around to zero or a small number — silently, without any error.

This becomes dangerous when the overflowed value is used to determine how much memory to allocate.

The Vulnerable Code

Here's the original code in question:

const int n_samples = 176000;
float *data    = (float *)malloc(n_samples * sizeof(float));
short *samples = (short *)malloc(n_samples * sizeof(short));

At first glance, this looks harmless — n_samples is a constant 176000. But the vulnerability pattern here is structural: the code uses raw malloc() with a multiplication and no overflow check, and it's embedded in a function that processes external data (an audio file path and, potentially, externally influenced sample counts).

The specific risk identified (CWE-190) is:

If n_samples is attacker-controlled and close to SIZE_MAX / sizeof(float), the multiplication n_samples * sizeof(float) wraps around to a small value. malloc() then allocates an undersized buffer. When the code subsequently writes the full sample data into this buffer, it overflows the heap.

How Could This Be Exploited?

Let's make this concrete. On a 64-bit system, SIZE_MAX is 18446744073709551615. sizeof(float) is 4.

If an attacker can influence n_samples to be, say, 4611686018427387905 (which is SIZE_MAX / 4 + 1), then:

n_samples * sizeof(float)
= 4611686018427387905 * 4
= 18446744073709551620

This overflows a 64-bit unsigned integer and wraps around to 4 — so malloc(4) allocates just 4 bytes. But the code then tries to write n_samples * sizeof(float) bytes of actual audio data into that 4-byte buffer, corrupting heap memory far beyond the allocation.

Real-World Impact

Heap buffer overflows are among the most severe memory corruption bugs because they can lead to:

• Arbitrary code execution — an attacker can overwrite heap metadata or adjacent objects to redirect program control flow
• Denial of service — corrupted heap structures cause crashes or undefined behavior
• Data leakage — adjacent heap allocations may contain sensitive data that gets exposed
• Security bypass — heap spray techniques can turn overflow primitives into full exploits

Even in a "test utility," this code runs on a developer's machine or CI/CD pipeline — environments where exploitation can have serious consequences.

Additional Issue: No NULL Check

The original code also had another problem:

FILE *file = fopen(audio_path_str, "rb");
// No NULL check before using `file`!
fseek(file, 78, SEEK_SET);
fread(samples, sizeof(short), n_samples, file);

If fopen() fails (e.g., file not found, permission denied), file is NULL, and the subsequent fseek and fread calls dereference a null pointer — causing an immediate crash or undefined behavior.

The Fix

What Changed

The fix takes a multi-layered approach to address both the allocation safety and the error handling:

1. Using Ruby's ALLOC_N Instead of Raw malloc()

The raw malloc() calls are replaced with Ruby's ALLOC_N macro:

// BEFORE (unsafe):
float *data    = (float *)malloc(n_samples * sizeof(float));
short *samples = (short *)malloc(n_samples * sizeof(short));

// AFTER (safe):
a->data    = ALLOC_N(float, a->n_samples);
a->samples = ALLOC_N(short, a->n_samples);

ALLOC_N is Ruby's safe memory allocation macro. It internally uses ruby_xmalloc2(), which performs overflow checking before multiplying the element count by the element size. If an overflow would occur, it raises a Ruby exception rather than proceeding with a corrupted size. This directly eliminates the CWE-190 integer overflow risk.

2. Exception-Safe Allocation with rb_protect()

The allocations are wrapped in a new helper function and called via rb_protect():

typedef struct {
    VALUE audio_path;
    int   n_samples;
    const char *audio_path_str;
    float      *data;
    short      *samples;
} jfk_alloc_args;

static VALUE
jfk_reader_alloc_resources(VALUE arg)
{
    jfk_alloc_args *a = (jfk_alloc_args *)arg;
    a->audio_path_str = StringValueCStr(a->audio_path);
    a->data    = ALLOC_N(float, a->n_samples);
    a->samples = ALLOC_N(short, a->n_samples);
    return Qnil;
}

And called with:

int state;
rb_protect(jfk_reader_alloc_resources, (VALUE)&args, &state);
if (state) {
    if (args.samples) xfree(args.samples);
    if (args.data)    xfree(args.data);
    return false;
}

rb_protect() catches any Ruby exceptions that occur during allocation (including the overflow exception that ALLOC_N would raise) and returns a non-zero state value. The cleanup block then safely frees any partially allocated memory, preventing memory leaks.

3. NULL Check for fopen()

The fix adds a proper null check after fopen():

FILE *file = fopen(args.audio_path_str, "rb");
if (file == NULL) {
    xfree(args.samples);
    xfree(args.data);
    return false;
}

This prevents the null pointer dereference and ensures resources are cleaned up on failure.

Before and After: Side-by-Side

Prevention & Best Practices

1. Never Use Raw malloc() with Unchecked Multiplications in C

Always validate that the multiplication won't overflow before passing it to malloc(). A safe pattern:

#include <stdint.h>
#include <stdlib.h>

// Safe allocation helper
void *safe_malloc_array(size_t count, size_t elem_size) {
    if (count == 0 || elem_size == 0) return NULL;
    if (count > SIZE_MAX / elem_size) {
        // Overflow would occur
        return NULL;
    }
    return malloc(count * elem_size);
}

Or use calloc(), which takes separate count and size arguments and handles overflow checking internally on most modern implementations:

float *data = calloc(n_samples, sizeof(float));  // Safer than malloc

2. Use Language/Framework Allocation APIs When Available

When writing C extensions for higher-level languages (Ruby, Python, etc.), always prefer the framework's allocation functions:

• Ruby: ALLOC_N(type, n), ALLOC(type), ruby_xmalloc2()
• Python: PyMem_New(type, n), PyMem_Malloc()
• Rust: Use Vec — overflow is checked by default

These APIs exist precisely because they handle the overflow and exception-safety concerns that raw malloc() does not.

3. Validate All Externally Influenced Sizes

Any value that could be influenced by external input (files, network, user input) must be validated before use in memory allocation:

#define MAX_SAMPLES 10000000  // Define a reasonable upper bound

if (n_samples <= 0 || n_samples > MAX_SAMPLES) {
    // Reject invalid input
    return false;
}

4. Use Static Analysis Tools

Several tools can detect integer overflow and unsafe allocation patterns automatically:

• Clang Static Analyzer (scan-build) — catches many integer overflow paths
• AddressSanitizer (ASan) — detects heap overflows at runtime during testing
• UndefinedBehaviorSanitizer (UBSan) — catches integer overflows at runtime
• Coverity / CodeQL — commercial/open-source static analysis with CWE-190 detection
• Valgrind — detects memory errors including heap overflows

Enable sanitizers during development and testing:

# Compile with sanitizers
gcc -fsanitize=address,undefined -g -o your_program your_program.c

5. Follow the CWE-190 Guidance

The MITRE CWE-190 (Integer Overflow or Wraparound) entry provides detailed guidance on this vulnerability class. Key mitigations include:

• Use languages with built-in overflow protection where possible
• Perform pre-multiplication validation
• Use safe integer libraries (e.g., safe_iop for C)
• Enable compiler warnings: -Wall -Wextra -Woverflow

6. Don't Neglect "Test" and "Binding" Code

This vulnerability was in a test utility — code that developers often treat as lower-stakes than production code. But test code runs on developer machines, CI/CD systems, and build infrastructure. A compromise there can be just as damaging as a production exploit. Apply the same security standards everywhere.

Conclusion

This vulnerability is a textbook example of why C memory management requires constant vigilance. An integer overflow in a malloc() call — just a few characters of code — can create a critical heap corruption vulnerability that leads to arbitrary code execution. The fix is elegant: use the framework's safe allocation primitives, wrap operations in exception-safe constructs, and always check for failure.

The key takeaways from this fix:

• Raw malloc() with multiplication is dangerous — always check for overflow or use safer alternatives
• Framework allocation APIs exist for a reason — ALLOC_N, calloc(), and similar functions provide safety guarantees that malloc() does not
• Error paths matter — every allocation failure and every fopen() failure needs to be handled, with resources properly freed
• Test code is production code from a security perspective — treat it accordingly

Security is built one safe allocation at a time. Use the tools and patterns available to you, enable your sanitizers, and run static analysis as part of your CI/CD pipeline.

This vulnerability was automatically detected and fixed by OrbisAI Security. Automated security scanning helps catch memory safety issues before they reach production.