How integer overflow happens in C reliable.c and how to fix it

How Integer Overflow Happens in C reliable.c and How to Fix It

Introduction

The reliable.c file is the heart of a reliable UDP packet-delivery library. It handles packet fragmentation, reassembly, and sequencing — the kind of low-level networking code where every byte of every buffer matters. At line 1299, a single line of arithmetic was responsible for calculating how large a reassembly buffer needed to be before writing received fragments into it:

int packet_buffer_size = RELIABLE_MAX_PACKET_HEADER_BYTES + num_fragments * endpoint->config.fragment_size;

This looks innocent. It looks like every other buffer-size calculation you have ever written in C. But it contains a critical integer overflow vulnerability (CWE-190) that could allow an attacker with control over endpoint configuration to trigger an undersized heap allocation and corrupt memory beyond the buffer's boundary.

The Vulnerability Explained

What Actually Goes Wrong

In C, int is a signed 32-bit integer on virtually all modern platforms, with a maximum value of 2,147,483,647 (roughly 2.1 billion). The expression:

num_fragments * endpoint->config.fragment_size

multiplies two int values. If an attacker can influence fragment_size — for example, by controlling endpoint configuration through an API — they can supply a value large enough that the product overflows INT_MAX.

Signed integer overflow in C is undefined behavior per the C standard, but in practice on x86/x64 with common compilers, the value wraps around to a large negative number. Adding RELIABLE_MAX_PACKET_HEADER_BYTES to a large negative number still produces a small or negative result.

The Vulnerable Line

// reliable.c:1299 — BEFORE the fix
int packet_buffer_size = RELIABLE_MAX_PACKET_HEADER_BYTES + num_fragments * endpoint->config.fragment_size;

When this value is subsequently used as an allocation size (e.g., passed to malloc or used to size a stack/heap buffer), the allocator receives a far smaller size than the actual data requires. The reassembly logic then writes full-sized fragment data into a buffer that is far too small, producing a heap buffer overflow.

Concrete Attack Scenario

Consider the following:

• RELIABLE_MAX_PACKET_HEADER_BYTES = 16 (a small constant)
• num_fragments = 2
• endpoint->config.fragment_size = 1,073,741,824 (1 GB, i.e., INT_MAX / 2 + 1)

The multiplication 2 * 1,073,741,824 = 2,147,483,648, which is exactly INT_MAX + 1. On a 32-bit signed integer, this wraps to -2,147,483,648. Adding 16 gives -2,147,483,632.

If this negative value is cast to size_t for a malloc call, it becomes an enormous allocation that will fail. But if it is used directly as a signed size in a comparison or stack allocation, the program may allocate a tiny buffer and then proceed to write gigabytes of fragment data into it — a textbook heap overflow.

The PR's threat model notes that an attacker can influence fragment_size through an API to set large values, making this a realistic exploitation path in networked deployments.

Real-World Impact

• Heap corruption: Fragment data written past the buffer boundary corrupts adjacent heap metadata or application data.
• Potential remote code execution: On systems without heap hardening, controlled heap corruption is a well-understood path to code execution.
• Denial of service: Even without code execution, the overflow causes a crash, disrupting packet delivery for all connected endpoints.

The Fix

What Changed

The fix is surgical and precise — one line, but every detail matters:

Before:

int packet_buffer_size = RELIABLE_MAX_PACKET_HEADER_BYTES + num_fragments * endpoint->config.fragment_size;

After:

size_t packet_buffer_size = (size_t) RELIABLE_MAX_PACKET_HEADER_BYTES + (size_t) num_fragments * (size_t) endpoint->config.fragment_size;

Why This Works

Three specific changes were made:

1. Type changed from int to size_t: size_t is an unsigned type guaranteed to be wide enough to represent the size of any object in memory. On 64-bit platforms it is 64 bits wide, making overflow astronomically unlikely for any realistic fragment_size.

2. Each operand individually cast to size_t: This is critical. In C, the cast must happen before the multiplication, not after. If you wrote (size_t)(num_fragments * fragment_size), the multiplication would still occur in int arithmetic and overflow before the cast. By casting each operand first — (size_t) num_fragments * (size_t) endpoint->config.fragment_size — the multiplication is performed in size_t arithmetic from the start.

3. RELIABLE_MAX_PACKET_HEADER_BYTES also cast: The header bytes constant is cast to size_t as well, ensuring the entire addition is performed in unsigned arithmetic and no implicit signed/unsigned conversion warnings remain.

Before/After Comparison

// BEFORE — signed int arithmetic, overflow is undefined behavior
int packet_buffer_size = RELIABLE_MAX_PACKET_HEADER_BYTES + num_fragments * endpoint->config.fragment_size;

// AFTER — size_t arithmetic, no overflow possible for any realistic input
size_t packet_buffer_size = (size_t) RELIABLE_MAX_PACKET_HEADER_BYTES
                          + (size_t) num_fragments * (size_t) endpoint->config.fragment_size;

The fix ensures that any downstream use of packet_buffer_size — whether as an argument to malloc, a loop bound, or a comparison — operates on a correct, non-overflowed value.

Prevention & Best Practices

Use size_t for All Buffer-Size Arithmetic

Any variable that represents a memory size, buffer length, or byte count should be declared as size_t. This is not just a style preference — it is a correctness requirement. The C standard library itself uses size_t for all allocation and size parameters (malloc, memcpy, strlen, etc.) precisely to avoid this class of bug.

// Prefer this pattern for all buffer-size calculations
size_t buffer_size = (size_t) count * (size_t) element_size;

Add Explicit Overflow Guards for Untrusted Inputs

When fragment_size or num_fragments can be influenced by external input, add a pre-multiplication range check:

// Guard against multiplication overflow before using the result
if (num_fragments > 0 && fragment_size > (SIZE_MAX - RELIABLE_MAX_PACKET_HEADER_BYTES) / num_fragments) {
    // Handle error: inputs would cause overflow
    return RELIABLE_ERROR_INVALID_CONFIG;
}
size_t packet_buffer_size = (size_t) RELIABLE_MAX_PACKET_HEADER_BYTES
                          + (size_t) num_fragments * (size_t) fragment_size;

Enable Compiler Sanitizers During Development

Clang and GCC both support runtime integer overflow detection:

# Catch signed integer overflow at runtime during testing
clang -fsanitize=signed-integer-overflow -fsanitize=unsigned-integer-overflow reliable.c

Running your test suite with these flags will catch overflow bugs before they reach production.

Use Safe Integer Libraries

For code that performs extensive arithmetic on potentially untrusted values, consider using a safe integer library such as:

• SafeInt (C++)
• IntegerLib (C)
• Or CERT C's recommended checked arithmetic macros

Static Analysis

Configure your CI pipeline to run static analysis tools that flag CWE-190 patterns:

• Semgrep: Rules for integer overflow feeding into allocation calls
• Coverity / Klocwork: Commercial tools with strong integer overflow detection
• cppcheck: Open-source, catches many arithmetic overflow patterns
• Compiler warnings: -Wconversion and -Wsign-conversion in GCC/Clang will warn on many implicit type conversions that can lead to this issue

Relevant Standards

• CWE-190: Integer Overflow or Wraparound — https://cwe.mitre.org/data/definitions/190.html
• CWE-122: Heap-based Buffer Overflow — https://cwe.mitre.org/data/definitions/122.html
• CWE-131: Incorrect Calculation of Buffer Size — https://cwe.mitre.org/data/definitions/131.html
• CERT C Rule INT30-C: Ensure that unsigned integer operations do not wrap
• CERT C Rule INT32-C: Ensure that operations on signed integers do not result in overflow

Key Takeaways

• Cast operands before multiplication, not after: (size_t) a * (size_t) b is safe; (size_t)(a * b) is not — the overflow happens before the cast.
• packet_buffer_size in reliable.c was the exact variable at risk: any downstream malloc(packet_buffer_size) call would have received a wrapped, dangerously small value.
• fragment_size coming from endpoint configuration is an attacker-controlled surface: any field that flows from API input into a size calculation must be treated as untrusted.
• Signed int overflow in C is undefined behavior, meaning the compiler is free to optimize it in ways that make the bug even harder to predict — size_t eliminates this entirely.
• The regression test in the PR explicitly tests INT_MAX / 2 + 1 as a fragment_size: this is the exact boundary value that triggered the overflow, and it should remain in the test suite permanently.

How Orbis AppSec Detected This

• Source: The endpoint->config.fragment_size value, which flows from externally controlled endpoint configuration via API
• Sink: The arithmetic expression num_fragments * endpoint->config.fragment_size assigned to int packet_buffer_size at reliable.c:1299, subsequently used as a buffer allocation size
• Missing control: No type-width promotion, no overflow guard, and no range validation on fragment_size before it was used in multiplication with a signed int result
• CWE: CWE-190 — Integer Overflow or Wraparound
• Fix: All operands in the packet_buffer_size calculation were cast to size_t before arithmetic, and the result variable type was changed from int to size_t

Orbis AppSec automatically detected this vulnerability and opened a pull request with the fix. Try Orbis AppSec on your repositories to find and fix issues like this automatically.

Conclusion

Integer overflow vulnerabilities in buffer-size calculations are one of the oldest and most dangerous classes of bugs in C — and one of the easiest to overlook precisely because the arithmetic looks correct. The one-line change in reliable.c — switching from int to size_t and casting each operand individually — is a textbook example of how a minimal, targeted fix can eliminate an entire class of memory-safety risk.

For developers writing packet-processing, fragmentation, or any other code that computes buffer sizes from external inputs: make size_t your default type for size arithmetic, add explicit overflow guards when inputs are untrusted, and let your compiler and static analysis tools catch the cases your eyes miss.

References

• CWE-190: Integer Overflow or Wraparound
• CWE-122: Heap-based Buffer Overflow
• CWE-131: Incorrect Calculation of Buffer Size
• OWASP Memory Management Cheat Sheet
• CERT C Rule INT32-C: Ensure that operations on signed integers do not result in overflow
• Semgrep rules for integer overflow
• fix: add integer overflow check in reliable.c