Integer Overflow to Heap Corruption: Fixing a Critical q3asm Vulnerability

Severity: Critical | File: ratoa_gamecode/code/tools/asm/q3asm.c | CWE: CWE-190 (Integer Overflow), CWE-122 (Heap-Based Buffer Overflow)

Introduction

Build tools are the unsung workhorses of software development. They compile, assemble, and link millions of lines of code every day — often running with elevated privileges inside CI/CD pipelines, trusted implicitly by the systems that invoke them. That trust makes them a high-value target.

This post covers a critical security vulnerability discovered and patched in q3asm.c, the assembler component of the Quake 3 Arena game engine toolchain (as found in the ratoa_gamecode project). The bug is a textbook integer overflow leading to heap corruption — a class of vulnerability that has been responsible for some of the most impactful exploits in computing history.

If you write C or C++, work on build tooling, or care about supply-chain security, read on. This one's a masterclass in how a single missing bounds check can cascade into full code execution.

The Vulnerability Explained

What Is an Integer Overflow?

An integer overflow occurs when an arithmetic operation produces a result that exceeds the maximum value a given integer type can hold, causing it to "wrap around" to a small (or even zero) value. In C, this behavior is well-defined for unsigned integers but undefined for signed ones — and in both cases, it can be catastrophic when the result is used to size a memory allocation.

The Specific Bug in q3asm.c

The vulnerability lives in the hash table initialization logic of the assembler. When q3asm processes an assembly source file, it reads configuration values — including H->buckets (the number of hash buckets) and elems (element counts) — directly from the input file. These attacker-controlled values are then used in a calloc() call to allocate a hash table:

// VULNERABLE CODE (simplified illustration)
// H->buckets is read directly from attacker-controlled assembly source
hashtable_t *hashtable_init(size_t buckets) {
    hashtable_t *H = malloc(sizeof(hashtable_t));
    H->buckets = buckets;

    // DANGER: No overflow check before multiplication!
    // If buckets = (SIZE_MAX / sizeof(ptr)) + 1,
    // then buckets * sizeof(ptr) wraps to a tiny value (e.g., 8 bytes)
    H->table = calloc(H->buckets, sizeof(symbol_t *));

    return H;
}

// Later: symbol insertions use H->buckets as the table size bound,
// but the actual allocation is only 8 bytes — writes go far out of bounds
void hashtable_insert(hashtable_t *H, symbol_t *sym) {
    size_t idx = hash(sym->name) % H->buckets; // idx can be >> actual alloc
    H->table[idx] = sym;  // HEAP CORRUPTION
}

The Math of the Attack

Here's the clever (and dangerous) part. calloc(nmemb, size) is supposed to be overflow-safe — but only if the underlying implementation checks for it. The real danger is when the overflow happens before calloc is called, or when the programmer manually computes nmemb * size first.

Consider:
- sizeof(symbol_t *) = 8 bytes on a 64-bit system
- SIZE_MAX = 18,446,744,073,709,551,615 on a 64-bit system
- If an attacker sets H->buckets = (SIZE_MAX / 8) + 1 = 2,305,843,009,213,693,953

Then: H->buckets * sizeof(symbol_t *) wraps around to 8 bytes.

calloc allocates a buffer of just 8 bytes. But the code believes it has a table with over 2 quintillion slots. Every subsequent hash lookup computes an index that can be billions of entries beyond the actual allocation, and writes a pointer there — directly into heap metadata, other allocations, or mapped memory.

Exploitation Path

This isn't just a crash. A skilled attacker can turn this into controlled code execution:

1. Craft a malicious .asm source file with a carefully chosen H->buckets value
2. The overflow causes calloc() to return a tiny 8-byte buffer
3. Attacker controls symbol names and values inserted into the "table"
4. Writes land at predictable offsets beyond the buffer
5. A heap-resident function pointer (e.g., in a vtable or malloc hook) gets overwritten
6. Next time that function is called → attacker controls RIP/EIP
7. In a CI/CD context running as root or a privileged service account → full compromise

Real-World Impact

The scenario that makes this truly dangerous isn't a developer running the assembler locally — it's automated build pipelines:

• 🏭 CI/CD runners often execute with elevated privileges or access to secrets
• 🔑 Build artifacts produced by a compromised assembler could contain backdoors
• 🔗 Supply-chain poisoning: downstream consumers of the built artifacts inherit the compromise
• 💣 Blast radius: a single malicious .asm file committed to a repository could compromise every machine that builds it

This is the same class of attack that made the XZ Utils backdoor so alarming — trusted build-time tools are an attacker's dream.

The Fix

What Was Changed

The fix in q3asm.c addresses the root cause: unvalidated, attacker-controlled values being used directly in allocation size arithmetic.

The remediation involves two complementary defenses:

1. Explicit Overflow Check Before Allocation

// FIXED CODE (illustrative)
hashtable_t *hashtable_init(size_t buckets) {
    hashtable_t *H = malloc(sizeof(hashtable_t));

    // Guard against integer overflow in size calculation
    if (buckets == 0 || buckets > (SIZE_MAX / sizeof(symbol_t *))) {
        // Reject unreasonable values — fail safe
        Com_Error(ERR_FATAL, 
            "hashtable_init: invalid bucket count %zu (possible overflow attack)", 
            buckets);
        return NULL;
    }

    H->buckets = buckets;
    H->table = calloc(H->buckets, sizeof(symbol_t *));

    if (!H->table) {
        Com_Error(ERR_FATAL, "hashtable_init: allocation failed");
        return NULL;
    }

    return H;
}

2. Reasonable Upper Bound Validation

Even if the math doesn't overflow, a legitimate assembler source file has no business requesting billions of hash buckets:

// Sanity cap: no legitimate .asm file needs more than this
#define MAX_HASH_BUCKETS  (1 << 20)  // 1,048,576 — generous but bounded

if (buckets > MAX_HASH_BUCKETS) {
    Com_Error(ERR_FATAL, 
        "hashtable_init: bucket count %zu exceeds maximum %d", 
        buckets, MAX_HASH_BUCKETS);
}

Why This Fix Works

The fix follows the fail-safe defaults principle: when input is outside expected bounds, reject it loudly rather than proceeding with corrupted state.

Prevention & Best Practices

This vulnerability is representative of an entire family of C/C++ bugs. Here's how to systematically prevent them:

1. Always Validate Before Multiplying for Allocation Sizes

// ❌ DANGEROUS
ptr = malloc(count * element_size);

// ✅ SAFE: Use a helper that checks for overflow
#include <stdint.h>

static inline int size_mul_overflow(size_t a, size_t b, size_t *result) {
    if (a != 0 && b > SIZE_MAX / a) return 1; // overflow
    *result = a * b;
    return 0;
}

size_t alloc_size;
if (size_mul_overflow(count, element_size, &alloc_size)) {
    // Handle error
}
ptr = malloc(alloc_size);

2. Prefer calloc Over malloc + Manual Size Math

calloc(nmemb, size) is required by the C standard to detect and handle internal overflow (returning NULL). Using it directly is safer than computing nmemb * size yourself:

// ✅ Better: calloc handles the overflow internally per C11 standard
ptr = calloc(count, element_size);
if (!ptr) {
    // Either OOM or overflow — both are errors
}

3. Treat All File-Derived Values as Untrusted Input

Any value that originates from a file, network, or environment variable is attacker-controlled. Apply the same validation you'd use for web input:

// Values from source files need the same scrutiny as HTTP parameters
size_t buckets = parse_buckets_from_source(source_file);

// Validate range BEFORE use
assert(buckets > 0);
assert(buckets <= REASONABLE_MAX);

4. Use Safe Integer Libraries

For C projects, consider using established safe-integer libraries:

• safe-iop — Safe integer operations for C
• IntegerLib — CERT-inspired safe arithmetic
• Microsoft's intsafe.h — Windows SDK safe integer functions
• Rust — If you can, use a memory-safe language where overflow is caught at runtime in debug mode and wraps predictably in release mode

5. Enable Compiler Sanitizers During Development

# AddressSanitizer catches heap overflows at runtime
gcc -fsanitize=address,undefined -g q3asm.c -o q3asm

# UndefinedBehaviorSanitizer catches signed integer overflow
gcc -fsanitize=undefined -g q3asm.c -o q3asm

# In CMake:
target_compile_options(q3asm PRIVATE -fsanitize=address,undefined)
target_link_options(q3asm PRIVATE -fsanitize=address,undefined)

6. Static Analysis in CI/CD

Add static analysis to your pipeline to catch these issues before they ship:

# Example GitHub Actions step
- name: Static Analysis (CodeQL)
  uses: github/codeql-action/analyze@v3
  with:
    languages: cpp
    queries: security-extended

# Or with cppcheck
- name: cppcheck
  run: cppcheck --enable=all --error-exitcode=1 ratoa_gamecode/code/tools/

Tools to consider:
- CodeQL — GitHub's semantic code analysis (free for open source)
- Coverity — Industry-standard static analysis
- cppcheck — Open-source C/C++ static analyzer
- Clang Static Analyzer — Built into LLVM toolchain
- Semgrep — Fast, customizable pattern-based analysis

7. Relevant Security Standards

This vulnerability maps to several well-documented weaknesses:

A Note on Build Tool Security

This vulnerability highlights a broader truth that the security community is increasingly vocal about: your build toolchain is part of your attack surface.

The 2020 SolarWinds attack, the 2024 XZ Utils backdoor, and countless other supply-chain incidents have demonstrated that attackers don't always target your production code directly — they target the tools that build your production code.

Recommendations for build tool security:

• 🔒 Run build tools with least privilege — don't build as root
• 🧪 Fuzz your build tools — if they parse files, they can be attacked through files
• 📦 Pin and verify tool versions — use checksums and sigstore/SLSA attestation
• 🔍 Audit toolchain dependencies — treat them like production dependencies
• 🚧 Sandbox build environments — use containers with restricted capabilities

Conclusion

The integer overflow in q3asm.c is a reminder that security vulnerabilities don't always look dramatic from the outside. A missing bounds check on a hash table size calculation — something that might look like an innocuous omission — can cascade into heap corruption, function pointer hijacking, and full system compromise when the right attacker-controlled inputs are provided.

The key takeaways from this fix:

1. Attacker-controlled values must be validated before use in arithmetic — especially when that arithmetic determines allocation sizes
2. Integer overflow in allocation size calculations is a critical vulnerability class, not a theoretical concern
3. Build tools deserve the same security scrutiny as production code — perhaps more, given their elevated privileges in CI/CD
4. Failing safely is better than proceeding with corrupted state — reject bad input loudly and early
5. Compiler sanitizers and static analysis are your safety net — use them in development and CI/CD

Security is a discipline of attention to detail. The developers who fixed this vulnerability did the right thing: they identified the root cause, validated the input at the source, and added clear, auditable bounds checks. That's the model for secure C development.

Write safe code. Validate your inputs. And remember: the tools that build your software are just as important to secure as the software itself.

This vulnerability was identified and fixed as part of an automated security scanning and remediation process. The fix was verified by build testing, automated re-scanning, and LLM-assisted code review.

Have a vulnerability you'd like us to cover? Found a security issue in your codebase? Check out OrbisAI Security for automated vulnerability detection and remediation.