Severity: Critical | File:
ratoa_gamecode/code/tools/asm/q3asm.c| CWE: CWE-190 (Integer Overflow), CWE-122 (Heap-Based Buffer Overflow)
Introduction
Build tools are the unsung workhorses of software development. They compile, assemble, and link millions of lines of code every day — often running with elevated privileges inside CI/CD pipelines, trusted implicitly by the systems that invoke them. That trust makes them a high-value target.
This post covers a critical security vulnerability discovered and patched in q3asm.c, the assembler component of the Quake 3 Arena game engine toolchain (as found in the ratoa_gamecode project). The bug is a textbook integer overflow leading to heap corruption — a class of vulnerability that has been responsible for some of the most impactful exploits in computing history.
If you write C or C++, work on build tooling, or care about supply-chain security, read on. This one's a masterclass in how a single missing bounds check can cascade into full code execution.
The Vulnerability Explained
What Is an Integer Overflow?
An integer overflow occurs when an arithmetic operation produces a result that exceeds the maximum value a given integer type can hold, causing it to "wrap around" to a small (or even zero) value. In C, this behavior is well-defined for unsigned integers but undefined for signed ones — and in both cases, it can be catastrophic when the result is used to size a memory allocation.
The Specific Bug in q3asm.c
The vulnerability lives in the hash table initialization logic of the assembler. When q3asm processes an assembly source file, it reads configuration values — including H->buckets (the number of hash buckets) and elems (element counts) — directly from the input file. These attacker-controlled values are then used in a calloc() call to allocate a hash table:
// VULNERABLE CODE (simplified illustration)
// H->buckets is read directly from attacker-controlled assembly source
hashtable_t *hashtable_init(size_t buckets) {
hashtable_t *H = malloc(sizeof(hashtable_t));
H->buckets = buckets;
// DANGER: No overflow check before multiplication!
// If buckets = (SIZE_MAX / sizeof(ptr)) + 1,
// then buckets * sizeof(ptr) wraps to a tiny value (e.g., 8 bytes)
H->table = calloc(H->buckets, sizeof(symbol_t *));
return H;
}
// Later: symbol insertions use H->buckets as the table size bound,
// but the actual allocation is only 8 bytes — writes go far out of bounds
void hashtable_insert(hashtable_t *H, symbol_t *sym) {
size_t idx = hash(sym->name) % H->buckets; // idx can be >> actual alloc
H->table[idx] = sym; // HEAP CORRUPTION
}
The Math of the Attack
Here's the clever (and dangerous) part. calloc(nmemb, size) is supposed to be overflow-safe — but only if the underlying implementation checks for it. The real danger is when the overflow happens before calloc is called, or when the programmer manually computes nmemb * size first.
Consider:
- sizeof(symbol_t *) = 8 bytes on a 64-bit system
- SIZE_MAX = 18,446,744,073,709,551,615 on a 64-bit system
- If an attacker sets H->buckets = (SIZE_MAX / 8) + 1 = 2,305,843,009,213,693,953
Then: H->buckets * sizeof(symbol_t *) wraps around to 8 bytes.
calloc allocates a buffer of just 8 bytes. But the code believes it has a table with over 2 quintillion slots. Every subsequent hash lookup computes an index that can be billions of entries beyond the actual allocation, and writes a pointer there — directly into heap metadata, other allocations, or mapped memory.
Exploitation Path
This isn't just a crash. A skilled attacker can turn this into controlled code execution:
1. Craft a malicious .asm source file with a carefully chosen H->buckets value
2. The overflow causes calloc() to return a tiny 8-byte buffer
3. Attacker controls symbol names and values inserted into the "table"
4. Writes land at predictable offsets beyond the buffer
5. A heap-resident function pointer (e.g., in a vtable or malloc hook) gets overwritten
6. Next time that function is called → attacker controls RIP/EIP
7. In a CI/CD context running as root or a privileged service account → full compromise
Real-World Impact
The scenario that makes this truly dangerous isn't a developer running the assembler locally — it's automated build pipelines:
- 🏭 CI/CD runners often execute with elevated privileges or access to secrets
- 🔑 Build artifacts produced by a compromised assembler could contain backdoors
- 🔗 Supply-chain poisoning: downstream consumers of the built artifacts inherit the compromise
- 💣 Blast radius: a single malicious
.asmfile committed to a repository could compromise every machine that builds it
This is the same class of attack that made the XZ Utils backdoor so alarming — trusted build-time tools are an attacker's dream.
The Fix
What Was Changed
The fix in q3asm.c addresses the root cause: unvalidated, attacker-controlled values being used directly in allocation size arithmetic.
The remediation involves two complementary defenses:
1. Explicit Overflow Check Before Allocation
// FIXED CODE (illustrative)
hashtable_t *hashtable_init(size_t buckets) {
hashtable_t *H = malloc(sizeof(hashtable_t));
// Guard against integer overflow in size calculation
if (buckets == 0 || buckets > (SIZE_MAX / sizeof(symbol_t *))) {
// Reject unreasonable values — fail safe
Com_Error(ERR_FATAL,
"hashtable_init: invalid bucket count %zu (possible overflow attack)",
buckets);
return NULL;
}
H->buckets = buckets;
H->table = calloc(H->buckets, sizeof(symbol_t *));
if (!H->table) {
Com_Error(ERR_FATAL, "hashtable_init: allocation failed");
return NULL;
}
return H;
}
2. Reasonable Upper Bound Validation
Even if the math doesn't overflow, a legitimate assembler source file has no business requesting billions of hash buckets:
// Sanity cap: no legitimate .asm file needs more than this
#define MAX_HASH_BUCKETS (1 << 20) // 1,048,576 — generous but bounded
if (buckets > MAX_HASH_BUCKETS) {
Com_Error(ERR_FATAL,
"hashtable_init: bucket count %zu exceeds maximum %d",
buckets, MAX_HASH_BUCKETS);
}
Why This Fix Works
| Attack Step | Before Fix | After Fix |
|---|---|---|
Attacker sets buckets to overflow value |
✅ Accepted | ❌ Rejected with fatal error |
calloc receives wrapped-around tiny size |
✅ Proceeds | ❌ Never reached |
| Out-of-bounds writes to heap | ✅ Occurs | ❌ Prevented |
| Function pointer overwrite | ✅ Possible | ❌ Impossible |
The fix follows the fail-safe defaults principle: when input is outside expected bounds, reject it loudly rather than proceeding with corrupted state.
A Note on Build Tool Security
This vulnerability highlights a broader truth that the security community is increasingly vocal about: your build toolchain is part of your attack surface.
The 2020 SolarWinds attack, the 2024 XZ Utils backdoor, and countless other supply-chain incidents have demonstrated that attackers don't always target your production code directly — they target the tools that build your production code.
Recommendations for build tool security:
- 🔒 Run build tools with least privilege — don't build as root
- 🧪 Fuzz your build tools — if they parse files, they can be attacked through files
- 📦 Pin and verify tool versions — use checksums and sigstore/SLSA attestation
- 🔍 Audit toolchain dependencies — treat them like production dependencies
- 🚧 Sandbox build environments — use containers with restricted capabilities
Conclusion
The integer overflow in q3asm.c is a reminder that security vulnerabilities don't always look dramatic from the outside. A missing bounds check on a hash table size calculation — something that might look like an innocuous omission — can cascade into heap corruption, function pointer hijacking, and full system compromise when the right attacker-controlled inputs are provided.
The key takeaways from this fix:
- Attacker-controlled values must be validated before use in arithmetic — especially when that arithmetic determines allocation sizes
- Integer overflow in allocation size calculations is a critical vulnerability class, not a theoretical concern
- Build tools deserve the same security scrutiny as production code — perhaps more, given their elevated privileges in CI/CD
- Failing safely is better than proceeding with corrupted state — reject bad input loudly and early
- Compiler sanitizers and static analysis are your safety net — use them in development and CI/CD
Security is a discipline of attention to detail. The developers who fixed this vulnerability did the right thing: they identified the root cause, validated the input at the source, and added clear, auditable bounds checks. That's the model for secure C development.
Write safe code. Validate your inputs. And remember: the tools that build your software are just as important to secure as the software itself.
This vulnerability was identified and fixed as part of an automated security scanning and remediation process. The fix was verified by build testing, automated re-scanning, and LLM-assisted code review.
Have a vulnerability you'd like us to cover? Found a security issue in your codebase? Check out OrbisAI Security for automated vulnerability detection and remediation.