Integer Overflow in Graphics Blit: When Bit Shifts Go Dangerously Wrong

Introduction

There's a particular class of bug in C programming that is deceptively simple to write, notoriously difficult to spot in code review, and potentially catastrophic in its consequences: the integer overflow. It doesn't announce itself with a crash (at least, not always). It doesn't leave an obvious trail. Sometimes it silently corrupts data. Other times, it hands an attacker the keys to your memory.

This is the story of exactly that kind of vulnerability — catalogued as V-007 / CWE-190 — discovered and patched in rtg/mntgfx-gcc.c, a C source file responsible for graphics pattern blitting in a retargetable graphics subsystem. The vulnerability centers on a deceptively small piece of arithmetic: computing the size of a memory copy operation without ever validating the input that drives it.

If you write C or C++, work with low-level graphics or hardware interfaces, or simply care about writing robust systems code, this one is worth understanding deeply.

The Vulnerability Explained

What Is a "Pattern Blit"?

In computer graphics — particularly in classic and retro-computing contexts like AmigaOS — a blit (Block Image Transfer) is a hardware- or software-accelerated operation that copies a rectangular block of pixel data from one memory region to another. A pattern blit applies a repeating graphical pattern during this operation.

The pattern itself is described by a data structure — in this case, an AmigaOS Pattern struct — which includes a Size field. This Size field encodes the height of the pattern as a power of two: a Size of 0 means 1 row, 1 means 2 rows, 2 means 4 rows, and so on.

The Dangerous Calculation

The vulnerable code computed the number of bytes to copy like this:

// VULNERABLE CODE (simplified)
memcpy(dest, src, 2 * (1 << pat->Size));

Let's unpack why this is dangerous.

The expression 1 << pat->Size performs a left bit-shift on the integer 1. On a 32-bit system, int is typically a signed 32-bit type. Here's what happens as pat->Size grows:

When pat->Size is 31, the expression 1 << 31 sets the sign bit of a signed 32-bit integer. In C, this is undefined behavior (UB) — the C standard explicitly states that left-shifting into or past the sign bit of a signed integer is undefined. The compiler is free to do anything here: produce a negative number, wrap around, optimize the entire branch away, or generate code that behaves unpredictably across platforms or optimization levels.

Then multiplying by 2 compounds the problem:
- If 1 << 31 produces 0x80000000 (the most common outcome), then 2 * 0x80000000 overflows a 32-bit signed integer, wrapping to 0 on many implementations.
- A memcpy of 0 bytes copies nothing — causing silent data corruption where the destination buffer is left uninitialized or stale.
- On other platforms or with unsigned arithmetic, the value could become an enormous positive number, causing memcpy to perform a massive out-of-bounds write, scribbling over arbitrary memory well beyond the destination buffer.

Where Does pat->Size Come From?

This is the critical detail that elevates this from a theoretical bug to a real security concern: pat->Size is sourced directly from a caller-supplied AmigaOS graphics Pattern structure, with no bounds checking performed before use.

In other words, any caller — including untrusted code, a malformed file, or a network-supplied graphics command — can provide a Pattern struct with an arbitrary Size value, and the code will blindly use it in this dangerous calculation.

Real-World Impact

The consequences fall into two dangerous categories:

1. Silent Data Corruption (memcpy of 0 bytes)
When the overflow resolves to zero, the destination buffer is never written. If downstream code assumes the blit succeeded and reads from the destination, it reads uninitialized or stale memory. This can cause:
- Rendering artifacts or display corruption
- Logic errors in graphics-dependent code paths
- Information disclosure if stale buffer contents are exposed

2. Out-of-Bounds Memory Write (heap/stack smashing)
When the overflow produces a very large value, memcpy writes far beyond the intended destination buffer. Depending on the memory layout, this can:
- Overwrite adjacent heap metadata, enabling heap exploitation
- Corrupt stack frames, potentially enabling code execution
- Crash the process (denial of service)
- In the worst case, enable an attacker to achieve arbitrary code execution

Example Attack Scenario

Imagine a graphics application that renders user-supplied pattern data — perhaps from a loaded file format, a network stream, or an emulation layer. An attacker crafts a malicious Pattern structure with Size = 31. The application passes this structure to the blit function without validation. The integer overflow fires, memcpy is called with a computed size of 0 or 4294967296, and the attacker has either silently corrupted the graphics state or triggered an out-of-bounds write they may be able to leverage further.

In emulation contexts (like AmigaOS emulators), where guest code directly influences host memory operations, the attack surface is particularly significant.

The Fix

What Changed

The fix introduces explicit bounds validation on pat->Size before it is used in the shift-and-multiply expression. The corrected logic ensures that pat->Size cannot exceed a safe maximum value — specifically, a value that keeps the computed copy size within the bounds of a valid int (or size_t) and within the bounds of the destination buffer.

A safe implementation looks like this:

// SAFE CODE (illustrative fix)
#define MAX_PATTERN_SIZE 15  // 1 << 15 = 32768 rows, well within safe range

if (pat->Size > MAX_PATTERN_SIZE) {
    // Handle error: invalid pattern size
    return;  // or set an error code
}

size_t copy_size = (size_t)2 * (1u << pat->Size);
memcpy(dest, src, copy_size);

Key improvements in this approach:

1. Bounds check before use: pat->Size is validated against a safe maximum before any arithmetic is performed.
2. Use of unsigned arithmetic: Casting to unsigned (1u) or size_t before the shift avoids signed integer overflow entirely. Unsigned left-shift is well-defined in C as long as the result fits in the type.
3. Explicit size_t for the copy size: Using size_t (the natural type for memory sizes) rather than int avoids sign-related issues when passing the value to memcpy.
4. Early return on invalid input: Rather than attempting to proceed with corrupted state, the function bails out cleanly.

Why This Solves the Problem

By capping pat->Size at a known-safe maximum, the shift operation can never reach the sign bit or overflow. By using unsigned arithmetic, we eliminate the undefined behavior class entirely. And by validating at the point of ingestion — before the value influences any memory operation — we break the attacker's ability to supply a malicious Size value and have it acted upon.

Conclusion

The vulnerability patched here is a reminder that security bugs don't always look dramatic in source code. A single line — 2 * (1 << pat->Size) — with no apparent malice, hides a critical flaw that can lead to silent data corruption or a full out-of-bounds memory write. The root cause is simple: an external value was trusted and used in arithmetic without validation.

The fix is equally simple in principle: check the input before you use it. But the discipline to apply that principle consistently, especially in performance-sensitive graphics and systems code where "we control the inputs" feels like a reasonable assumption, is what separates robust code from vulnerable code.

Key takeaways for developers:

• Treat all externally-sourced values as untrusted, even in internal APIs
• Use unsigned arithmetic for sizes and bit-shift operations
• Validate bounds before arithmetic, not after
• Enable sanitizers (-fsanitize=undefined,address) in your development and CI builds — they would have caught this immediately
• Know your CWEs: CWE-190 (Integer Overflow) is perennially in the top vulnerability categories for a reason

Security is built one careful validation check at a time. Make yours count.

This post is part of an ongoing series on real-world security vulnerabilities and their fixes. Vulnerability identified and remediated by OrbisAI Security.