How integer overflow in _MALLOC() happens in C emulator memory allocation and how to fix it

How Integer Overflow in _MALLOC() Happens in C Emulator Memory Allocation and How to Fix It

Introduction

In the i286c/i286c.c file — part of an i286 CPU emulator's memory management subsystem — we discovered a medium-severity integer overflow vulnerability at line 216. The function i286c_setextsize() allocates extended memory for the emulated CPU using the expression _MALLOC(size + 16, "EXTMEM"). When the size parameter (a UINT32 derived from configuration) approaches UINT32_MAX (0xFFFFFFFF), adding 16 wraps the value around to as little as 15 bytes. The emulator then proceeds to write size bytes into this tiny allocation — a catastrophic heap buffer overflow.

This is the kind of bug that sits silently in codebases for years. The allocation "succeeds" (malloc happily returns a pointer to 15 bytes), no error is raised, and the subsequent memory operations corrupt the heap with potentially gigabytes of data flowing into a handful of bytes.

The Vulnerability Explained

Here's the vulnerable code in i286c/i286c.c, inside the i286c_setextsize() function:

void i286c_setextsize(UINT32 size) {
    // ... existing cleanup code ...
    if (size != 0) {
        extmem = (UINT8 *)_MALLOC(size + 16, "EXTMEM");
    }
    if (extmem != NULL) {
        CPU_EXTMEM = extmem;
        // ... writes 'size' bytes into extmem ...
    }
}

The problem is on the _MALLOC line. Let's trace through what happens with a malicious size value:

1. Attacker sets size = 0xFFFFFFF0 (UINT32_MAX - 15)
2. Arithmetic: size + 16 = 0xFFFFFFF0 + 0x10 = 0x100000000
3. Truncation: Since this is a 32-bit unsigned integer, it wraps to 0x00000000
4. Result: _MALLOC(0, "EXTMEM") — allocates either 0 bytes or a minimal block
5. Exploitation: The emulator then writes ~4GB of data into this allocation

Even more dangerous is the case where size = 0xFFFFFFFF:
- size + 16 = 0xFFFFFFFF + 0x10 = 0x0000000F (wraps to 15)
- _MALLOC(15, "EXTMEM") succeeds, returning a valid 15-byte buffer
- The emulator writes 4,294,967,295 bytes into those 15 bytes

Attack Scenario

An attacker who can influence the emulator's memory size configuration — through a config file, command-line parameter, or import file — can set the extended memory size to a value near UINT32_MAX. The emulator will:

1. Allocate a tiny buffer (due to overflow)
2. Write massive amounts of data past the end of that buffer
3. Corrupt heap metadata, potentially enabling arbitrary code execution
4. At minimum, crash the application; at worst, achieve remote code execution

This is particularly dangerous because the size value comes from user-controlled configuration (the emulator memory size setting), making it directly exploitable without any complex setup.

The Fix

The fix adds an overflow guard using a ternary expression that checks whether size + 16 would exceed UINT32_MAX before performing the addition:

Before:

extmem = (UINT8 *)_MALLOC(size + 16, "EXTMEM");

After:

extmem = (size <= (UINT32)(~0) - 16) ? (UINT8 *)_MALLOC(size + 16, "EXTMEM") : NULL;

Let's break down this fix:

• (UINT32)(~0) evaluates to 0xFFFFFFFF — the maximum value for a 32-bit unsigned integer
• (UINT32)(~0) - 16 equals 0xFFFFFFEF — the largest size that can safely have 16 added
• If size > 0xFFFFFFEF, the addition would overflow, so we return NULL instead
• If size <= 0xFFFFFFEF, the addition is safe and we proceed with _MALLOC

The existing code already handles extmem == NULL gracefully (it checks if (extmem != NULL) before using the pointer), so setting extmem to NULL on overflow integrates cleanly with the existing error path.

This is elegant because it:
1. Requires only a single line change
2. Leverages existing NULL-check error handling
3. Has zero performance impact on valid inputs
4. Completely eliminates the overflow regardless of the size value

Prevention & Best Practices

1. Always Check Arithmetic Before Allocation

Any time you compute an allocation size, verify the arithmetic won't overflow:

// Pattern: safe allocation with overflow check
if (size > SIZE_MAX - padding) {
    return NULL;  // or handle error
}
void *ptr = malloc(size + padding);

2. Use Safe Arithmetic Helpers

Consider creating or using overflow-checking arithmetic functions:

// GCC/Clang built-in
size_t alloc_size;
if (__builtin_add_overflow(size, 16, &alloc_size)) {
    return NULL;
}
void *ptr = malloc(alloc_size);

3. Validate Configuration Inputs

Before values reach allocation code, validate them against reasonable bounds:

#define MAX_EXTMEM_SIZE (256 * 1024 * 1024)  // 256MB max
if (size > MAX_EXTMEM_SIZE) {
    log_error("Extended memory size exceeds maximum");
    return;
}

4. Enable Compiler Warnings

Use -ftrapv (GCC) or UBSan (-fsanitize=unsigned-integer-overflow) during testing to catch overflow at runtime.

5. Static Analysis

Tools like Coverity, PVS-Studio, and Semgrep can detect patterns where user-influenced arithmetic feeds into allocation functions.

Key Takeaways

• Never add padding to a size value without checking for overflow first — the size + 16 pattern in i286c_setextsize() is a textbook example of how a simple addition becomes exploitable.
• Configuration-derived values are attacker-controlled — the emulator memory size setting flows directly into the allocation, making this trivially exploitable.
• Existing NULL checks provide a natural error path — the fix leverages the existing if (extmem != NULL) guard, requiring zero architectural changes.
• (UINT32)(~0) - N is the canonical overflow threshold — memorize this pattern for any code that adds a constant to a user-influenced size before allocation.
• A single line of code prevented a heap corruption vulnerability — security fixes don't need to be complex to be effective.

How Orbis AppSec Detected This

• Source: User-controlled emulator memory size configuration parameter passed to i286c_setextsize()
• Sink: _MALLOC(size + 16, "EXTMEM") in i286c/i286c.c:216
• Missing control: No validation that size + 16 would not overflow a UINT32, allowing integer wraparound to produce an undersized allocation
• CWE: CWE-120 (Buffer Copy without Checking Size of Input)
• Fix: Added a ternary overflow guard that returns NULL when size > UINT32_MAX - 16, preventing the undersized allocation entirely

Orbis AppSec automatically detected this vulnerability and opened a pull request with the fix. Try Orbis AppSec on your repositories to find and fix issues like this automatically.

Conclusion

This vulnerability in i286c/i286c.c demonstrates how a seemingly innocent expression — size + 16 — can become a critical security flaw when the input isn't bounded. The fix is a single ternary expression that costs nothing at runtime but completely eliminates the integer overflow. When writing C code that performs arithmetic on sizes before allocation, always ask: "What happens when this value is near its maximum?" If the answer involves wraparound, you need a guard.

References

• CWE-120: Buffer Copy without Checking Size of Input
• CWE-190: Integer Overflow or Wraparound
• OWASP Buffer Overflow
• SEI CERT C Coding Standard - INT30-C: Ensure that unsigned integer operations do not wrap
• Semgrep rules for integer overflow
• fix: add buffer-length check in i286c.c