Heap Buffer Overflow in drawgfx.c: How ROM Dimensions Can Lead to Code Execution

Introduction

When a program trusts external data to control how much memory it reads or writes, it hands attackers a powerful weapon. This is exactly what happened in src/emu/drawgfx.c, where graphics layout dimensions sourced directly from ROM files were used, without validation, to drive a memcpy operation. The result: a classic heap buffer overflow (CWE-120) that could allow an attacker to overwrite adjacent heap objects — including function pointers — and achieve arbitrary code execution.

This vulnerability serves as a textbook example of why every size or dimension value derived from untrusted input must be validated before it influences memory operations. Whether you're writing an emulator, a file parser, an image decoder, or a network protocol handler, the lesson here applies directly to your code.

The Vulnerability Explained

What Is a Heap Buffer Overflow?

A heap buffer overflow occurs when a program writes more data into a heap-allocated buffer than the buffer was sized to hold. Unlike stack overflows (which famously corrupt return addresses), heap overflows corrupt adjacent heap metadata or objects. Modern heaps contain allocator metadata, vtable pointers, and function pointers — all of which become attacker-controlled write targets when a heap overflow is triggered.

CWE-120 specifically describes the "classic" buffer copy without checking the size of the input — precisely what was happening here.

The Vulnerable Code Pattern

In drawgfx.c around line 117, graphics offset data was being copied using memcpy, with the size of the copy derived from glcopy.width and glcopy.height:

// VULNERABLE (conceptual representation)
// glcopy.width and glcopy.height come from ROM file data — untrusted input

size_t copy_size = glcopy.width * glcopy.height * sizeof(pixel_t);

// No validation that copy_size fits within the destination buffer!
memcpy(dest_buffer, src_data, copy_size);

The critical flaw: glcopy.width and glcopy.height originate from graphics layout definitions embedded in ROM files. ROM files are external, user-supplied data. An attacker can craft a ROM with arbitrarily large width and height values.

How the Math Becomes a Weapon

Consider a destination buffer allocated for a standard tile size, say 8×8 pixels:

pixel_t *dest_buffer = malloc(8 * 8 * sizeof(pixel_t));  // 64 bytes allocated

Now an attacker crafts a ROM where glcopy.width = 0xFFFF and glcopy.height = 0xFFFF. The computed copy size becomes:

0xFFFF × 0xFFFF × sizeof(pixel_t) = ~4 GB

Even without the integer overflow angle, if the attacker supplies width = 256 and height = 256 against an 8×8 destination, memcpy will happily write 65,536 bytes starting at dest_buffer — obliterating everything the heap allocator placed after it.

Integer Overflow Makes It Worse

There's a secondary hazard: the multiplication width * height * sizeof(pixel_t) can itself integer-overflow on 32-bit size types. An attacker can pick values that wrap the multiplication result to a small number, causing malloc to allocate a tiny buffer while memcpy later uses the un-wrapped large value — a classic "allocate small, write large" pattern.

// Example of integer overflow leading to under-allocation:
uint32_t w = 0x10000, h = 0x10000;
size_t sz = w * h;  // Overflows to 0 on 32-bit!
void *buf = malloc(sz);          // malloc(0) — returns a tiny valid pointer
memcpy(buf, src, w * h);         // Writes 4GB from that tiny pointer

Real-World Attack Scenario

1. Attacker crafts a malicious ROM file with a graphics layout entry containing oversized width/height values.
2. Victim loads the ROM into the emulator (or a ROM is distributed via a ROM-sharing site, bundled in a pack, etc.).
3. The emulator processes the graphics layout, calls into drawgfx.c, and triggers the overflow.
4. Heap corruption occurs, overwriting an adjacent heap object — potentially a C++ vtable pointer, a function pointer stored in a struct, or allocator freelist metadata.
5. On the next virtual function call or callback, execution is redirected to attacker-controlled data.
6. Arbitrary code executes in the context of the emulator process — with whatever privileges the user is running.

This is a remote code execution (RCE) primitive delivered through a data file, a particularly dangerous class of vulnerability because users routinely download and open ROM files from the internet.

The Fix

What Changed

The fix, applied to src/emu/drawgfx.c, introduces explicit buffer-length validation before the memcpy call. The core principle: compute the required copy size first, compare it against the allocated destination buffer size, and abort (or clamp) if the source dimensions would exceed the destination.

// FIXED (conceptual representation)

size_t required_size = (size_t)glcopy.width * (size_t)glcopy.height * sizeof(pixel_t);

// Validate before copying
if (glcopy.width == 0 || glcopy.height == 0) {
    return;  // Nothing to copy
}

if (required_size > dest_buffer_size) {
    // Log the anomaly and refuse to process this graphics entry
    logerror("drawgfx: ROM-supplied dimensions (%u x %u) exceed buffer size %zu, skipping\n",
             glcopy.width, glcopy.height, dest_buffer_size);
    return;
}

// Safe to copy — dimensions are validated
memcpy(dest_buffer, src_data, required_size);

Key Security Improvements

Why Casting to size_t Matters

Notice the cast (size_t)glcopy.width * (size_t)glcopy.height in the fix. If width and height are stored as uint16_t or uint32_t, performing the multiplication in their native type risks integer overflow before the result is ever compared. Casting to size_t (which is 64-bit on modern platforms) ensures the multiplication is performed in a wider type, producing the true mathematical result and allowing the bounds check to catch even extreme values.

Prevention & Best Practices

1. Treat All External File Data as Untrusted Input

ROM files, PDFs, images, audio files, network packets — any data that crosses a trust boundary is attacker-controlled. Dimensions, lengths, counts, and offsets embedded in file formats must be validated against:
- Reasonable format-defined maximums
- The actual size of allocated destination buffers
- Arithmetic overflow before use in allocation or copy size calculations

2. Use Safe Integer Arithmetic Libraries

In C, use helpers that detect overflow:

#include <stdint.h>

size_t width = glcopy.width;
size_t height = glcopy.height;
size_t elem_size = sizeof(pixel_t);

// Check for multiplication overflow before proceeding
if (width != 0 && height > SIZE_MAX / width) {
    /* overflow! */
    return ERROR_INVALID_DIMENSIONS;
}
size_t required = width * height;

if (required != 0 && elem_size > SIZE_MAX / required) {
    /* overflow! */
    return ERROR_INVALID_DIMENSIONS;
}
required *= elem_size;

In C++, consider std::numeric_limits checks or a dedicated safe-math library. In Rust, integer overflow is caught in debug builds and can be handled explicitly with checked_mul.

3. Prefer Bounded Copy Functions

Where possible, replace unbounded memcpy with size-aware alternatives or wrappers:

// Instead of:
memcpy(dest, src, untrusted_size);

// Use a wrapper that enforces the bound:
safe_memcpy(dest, dest_capacity, src, untrusted_size);
// where safe_memcpy asserts/returns error if untrusted_size > dest_capacity

POSIX memcpy_s (from C11 Annex K) and Microsoft's memcpy_s serve this purpose on supported platforms.

4. Fuzz Your File Parsers

Heap buffer overflows in file parsers are a prime target for fuzzing. Tools to integrate into your pipeline:

• AFL++ / libFuzzer — mutation-based fuzzers that excel at finding parser bugs
• AddressSanitizer (ASan) — compile with -fsanitize=address to turn silent heap overflows into loud, immediate crashes during testing
• Valgrind / Memcheck — slower but thorough memory error detection

# Compile with AddressSanitizer to catch this class of bug immediately:
gcc -fsanitize=address -fsanitize=undefined -g -o emulator drawgfx.c ...

5. Apply the Principle of Least Privilege

Even if exploitation succeeds, its impact is limited when the process runs with minimal privileges. Emulators and file-processing applications should:
- Drop unnecessary OS privileges at startup
- Use OS sandboxing (seccomp on Linux, App Sandbox on macOS, AppContainer on Windows)
- Avoid running as root/Administrator

6. Reference Security Standards

This vulnerability maps to several well-documented weakness categories:

• CWE-120: Buffer Copy without Checking Size of Input ("Classic Buffer Overflow")
• CWE-190: Integer Overflow or Wraparound
• CWE-129: Improper Validation of Array Index
• OWASP: A03:2021 – Injection (data controlling program behavior)
• SEI CERT C: Rule ARR38-C — Guarantee that library functions do not form invalid pointers

Conclusion

The heap buffer overflow in drawgfx.c is a sharp reminder that data files are attack surfaces. When an emulator, image viewer, document renderer, or any file-processing application lets external data dictate the size of a memory operation, it must validate those values with the same suspicion it would apply to network input or user form data.

The fix is conceptually simple — check that the computed size fits within the allocated buffer before copying — but the consequences of omitting that check are severe: heap corruption, function pointer overwrite, and arbitrary code execution.

Key Takeaways

• ✅ Validate all externally-supplied dimensions before using them in memory operations
• ✅ Use widened integer types when computing sizes to prevent overflow before bounds checks
• ✅ Compile with AddressSanitizer during development and testing to catch these bugs early
• ✅ Fuzz your file parsers — automated fuzzing finds these bugs faster than manual review
• ✅ Apply least privilege so that even successful exploits have limited blast radius

Security is not a feature you add at the end — it's a discipline woven into every line of code that touches untrusted data. When in doubt, validate first, copy second.

This post is part of our ongoing series on real-world security fixes. Vulnerability remediation powered by OrbisAI Security.