Integer Overflow in Graphics Blit: When Bit Shifts Go Dangerously Wrong

Introduction

There's a particular class of bug in C programming that is deceptively simple to write, notoriously difficult to spot in code review, and potentially catastrophic in its consequences: the integer overflow. It doesn't announce itself with a crash (at least, not always). It doesn't leave an obvious trail. Sometimes it silently corrupts data. Other times, it hands an attacker the keys to your memory.

This is the story of exactly that kind of vulnerability — catalogued as V-007 / CWE-190 — discovered and patched in rtg/mntgfx-gcc.c, a C source file responsible for graphics pattern blitting in a retargetable graphics subsystem. The vulnerability centers on a deceptively small piece of arithmetic: computing the size of a memory copy operation without ever validating the input that drives it.

If you write C or C++, work with low-level graphics or hardware interfaces, or simply care about writing robust systems code, this one is worth understanding deeply.

The Vulnerability Explained

What Is a "Pattern Blit"?

In computer graphics — particularly in classic and retro-computing contexts like AmigaOS — a blit (Block Image Transfer) is a hardware- or software-accelerated operation that copies a rectangular block of pixel data from one memory region to another. A pattern blit applies a repeating graphical pattern during this operation.

The pattern itself is described by a data structure — in this case, an AmigaOS Pattern struct — which includes a Size field. This Size field encodes the height of the pattern as a power of two: a Size of 0 means 1 row, 1 means 2 rows, 2 means 4 rows, and so on.

The Dangerous Calculation

The vulnerable code computed the number of bytes to copy like this:

// VULNERABLE CODE (simplified)
memcpy(dest, src, 2 * (1 << pat->Size));

Let's unpack why this is dangerous.

The expression 1 << pat->Size performs a left bit-shift on the integer 1. On a 32-bit system, int is typically a signed 32-bit type. Here's what happens as pat->Size grows:

When pat->Size is 31, the expression 1 << 31 sets the sign bit of a signed 32-bit integer. In C, this is undefined behavior (UB) — the C standard explicitly states that left-shifting into or past the sign bit of a signed integer is undefined. The compiler is free to do anything here: produce a negative number, wrap around, optimize the entire branch away, or generate code that behaves unpredictably across platforms or optimization levels.

Then multiplying by 2 compounds the problem:
- If 1 << 31 produces 0x80000000 (the most common outcome), then 2 * 0x80000000 overflows a 32-bit signed integer, wrapping to 0 on many implementations.
- A memcpy of 0 bytes copies nothing — causing silent data corruption where the destination buffer is left uninitialized or stale.
- On other platforms or with unsigned arithmetic, the value could become an enormous positive number, causing memcpy to perform a massive out-of-bounds write, scribbling over arbitrary memory well beyond the destination buffer.

Where Does pat->Size Come From?

This is the critical detail that elevates this from a theoretical bug to a real security concern: pat->Size is sourced directly from a caller-supplied AmigaOS graphics Pattern structure, with no bounds checking performed before use.

In other words, any caller — including untrusted code, a malformed file, or a network-supplied graphics command — can provide a Pattern struct with an arbitrary Size value, and the code will blindly use it in this dangerous calculation.

Real-World Impact

The consequences fall into two dangerous categories:

1. Silent Data Corruption (memcpy of 0 bytes)
When the overflow resolves to zero, the destination buffer is never written. If downstream code assumes the blit succeeded and reads from the destination, it reads uninitialized or stale memory. This can cause:
- Rendering artifacts or display corruption
- Logic errors in graphics-dependent code paths
- Information disclosure if stale buffer contents are exposed

2. Out-of-Bounds Memory Write (heap/stack smashing)
When the overflow produces a very large value, memcpy writes far beyond the intended destination buffer. Depending on the memory layout, this can:
- Overwrite adjacent heap metadata, enabling heap exploitation
- Corrupt stack frames, potentially enabling code execution
- Crash the process (denial of service)
- In the worst case, enable an attacker to achieve arbitrary code execution

Example Attack Scenario

Imagine a graphics application that renders user-supplied pattern data — perhaps from a loaded file format, a network stream, or an emulation layer. An attacker crafts a malicious Pattern structure with Size = 31. The application passes this structure to the blit function without validation. The integer overflow fires, memcpy is called with a computed size of 0 or 4294967296, and the attacker has either silently corrupted the graphics state or triggered an out-of-bounds write they may be able to leverage further.

In emulation contexts (like AmigaOS emulators), where guest code directly influences host memory operations, the attack surface is particularly significant.

The Fix

What Changed

The fix introduces explicit bounds validation on pat->Size before it is used in the shift-and-multiply expression. The corrected logic ensures that pat->Size cannot exceed a safe maximum value — specifically, a value that keeps the computed copy size within the bounds of a valid int (or size_t) and within the bounds of the destination buffer.

A safe implementation looks like this:

// SAFE CODE (illustrative fix)
#define MAX_PATTERN_SIZE 15  // 1 << 15 = 32768 rows, well within safe range

if (pat->Size > MAX_PATTERN_SIZE) {
    // Handle error: invalid pattern size
    return;  // or set an error code
}

size_t copy_size = (size_t)2 * (1u << pat->Size);
memcpy(dest, src, copy_size);

Key improvements in this approach:

1. Bounds check before use: pat->Size is validated against a safe maximum before any arithmetic is performed.
2. Use of unsigned arithmetic: Casting to unsigned (1u) or size_t before the shift avoids signed integer overflow entirely. Unsigned left-shift is well-defined in C as long as the result fits in the type.
3. Explicit size_t for the copy size: Using size_t (the natural type for memory sizes) rather than int avoids sign-related issues when passing the value to memcpy.
4. Early return on invalid input: Rather than attempting to proceed with corrupted state, the function bails out cleanly.

Why This Solves the Problem

By capping pat->Size at a known-safe maximum, the shift operation can never reach the sign bit or overflow. By using unsigned arithmetic, we eliminate the undefined behavior class entirely. And by validating at the point of ingestion — before the value influences any memory operation — we break the attacker's ability to supply a malicious Size value and have it acted upon.

Prevention & Best Practices

This vulnerability is a textbook example of CWE-190: Integer Overflow or Wraparound, and it's far more common than most developers realize. Here's how to systematically avoid it:

1. Always Validate External Input Before Arithmetic

Any value that originates from user input, file data, network packets, or inter-process communication must be treated as untrusted. Validate ranges before using values in arithmetic expressions, especially those that feed into memory operations.

// Bad: use first, validate never
memcpy(dst, src, compute_size(external_value));

// Good: validate first, use second
if (external_value > SAFE_MAX) { return ERROR; }
memcpy(dst, src, compute_size(external_value));

2. Prefer Unsigned Types for Sizes and Counts

When computing sizes, counts, or indices, use size_t or unsigned types. This eliminates sign-bit overflow UB from left shifts and makes arithmetic wrapping behavior well-defined (though still potentially incorrect — pair with bounds checks).

// Risky: signed shift
int size = 2 * (1 << n);

// Safer: unsigned shift
size_t size = (size_t)2 * ((size_t)1 << n);

3. Use Safe Arithmetic Helpers

For critical calculations, consider using overflow-safe arithmetic wrappers. GCC and Clang provide built-in overflow-checking functions:

int result;
if (__builtin_mul_overflow(2, (1 << pat->Size), &result)) {
    // Handle overflow
}

Alternatively, libraries like safe-iop or C23's <stdckdint.h> provide portable checked arithmetic.

4. Enable Compiler Warnings and Sanitizers

Modern compilers can catch many of these issues at compile time or runtime:

• -Wall -Wextra: Enables common warning classes
• -fsanitize=undefined: UBSan catches undefined behavior including signed overflow at runtime
• -fsanitize=address: ASan detects out-of-bounds memory accesses
• -fwrapv: Makes signed overflow wrap (defined behavior), though this doesn't eliminate logic errors

# Build with sanitizers during development/testing
gcc -fsanitize=undefined,address -Wall -Wextra -o myprogram myprogram.c

5. Use Static Analysis Tools

Static analyzers can identify integer overflow risks without even running the code:

• Coverity: Industry-standard static analysis with strong integer overflow detection
• CodeQL: GitHub's semantic code analysis engine (free for open source)
• Clang Static Analyzer: Built into the LLVM toolchain
• PVS-Studio: Commercial analyzer with excellent C/C++ coverage

6. Follow the Principle of Least Surprise in APIs

When designing functions that accept size or count parameters, document valid ranges explicitly and enforce them at the API boundary. Don't rely on callers to validate — validate defensively inside your function.

7. Reference Security Standards

This vulnerability maps to well-established security standards:

• CWE-190: Integer Overflow or Wraparound
• CWE-787: Out-of-bounds Write (the consequence)
• CERT C Rule INT32-C: Ensure that operations on signed integers do not result in overflow
• OWASP Input Validation: Validate all inputs

Conclusion

The vulnerability patched here is a reminder that security bugs don't always look dramatic in source code. A single line — 2 * (1 << pat->Size) — with no apparent malice, hides a critical flaw that can lead to silent data corruption or a full out-of-bounds memory write. The root cause is simple: an external value was trusted and used in arithmetic without validation.

The fix is equally simple in principle: check the input before you use it. But the discipline to apply that principle consistently, especially in performance-sensitive graphics and systems code where "we control the inputs" feels like a reasonable assumption, is what separates robust code from vulnerable code.

Key takeaways for developers:

• Treat all externally-sourced values as untrusted, even in internal APIs
• Use unsigned arithmetic for sizes and bit-shift operations
• Validate bounds before arithmetic, not after
• Enable sanitizers (-fsanitize=undefined,address) in your development and CI builds — they would have caught this immediately
• Know your CWEs: CWE-190 (Integer Overflow) is perennially in the top vulnerability categories for a reason

Security is built one careful validation check at a time. Make yours count.

This post is part of an ongoing series on real-world security vulnerabilities and their fixes. Vulnerability identified and remediated by OrbisAI Security.