Back to Blog
critical SEVERITY8 min read

How integer overflow happens in C reliable.c and how to fix it

A critical integer overflow vulnerability was discovered in `reliable.c` at line 1299, where the `packet_buffer_size` calculation used signed `int` arithmetic that could wrap to a negative or undersized value when large `fragment_size` values were involved. By casting each operand to `size_t` before multiplication, the fix eliminates the overflow risk entirely and ensures the allocated buffer is always large enough to hold the reassembled packet data.

O
By Orbis AppSec
•Technically reviewed by Anupam Mediratta•Published July 9, 2026•Reviewed July 9, 2026
#integer-overflow#c#cwe-190#buffer-overflow#network-security#memory-safety#secure-coding

Answer Summary

This is an integer overflow vulnerability (CWE-190) in `reliable.c`, a C networking library, where the `packet_buffer_size` variable was calculated using signed `int` arithmetic — allowing large `fragment_size` or `num_fragments` values to overflow and produce an undersized buffer. The fix promotes all operands to `size_t` before the multiplication, ensuring the result can never wrap around on 32- or 64-bit platforms. Anyone writing packet reassembly logic in C should use `size_t` for all buffer-size calculations and add explicit overflow guards before calling `malloc` or equivalent allocators.

Vulnerability at a Glance

cweCWE-190
fixCast all operands to `size_t` before multiplication to prevent signed integer overflow
riskUndersized heap buffer allocation leading to potential heap corruption or remote code execution
languageC
root cause`packet_buffer_size` computed with signed `int` arithmetic, allowing wrap-around on large inputs
vulnerabilityInteger Overflow in Buffer Size Calculation

How Integer Overflow Happens in C reliable.c and How to Fix It

Introduction

The reliable.c file is the heart of a reliable UDP packet-delivery library. It handles packet fragmentation, reassembly, and sequencing — the kind of low-level networking code where every byte of every buffer matters. At line 1299, a single line of arithmetic was responsible for calculating how large a reassembly buffer needed to be before writing received fragments into it:

int packet_buffer_size = RELIABLE_MAX_PACKET_HEADER_BYTES + num_fragments * endpoint->config.fragment_size;

This looks innocent. It looks like every other buffer-size calculation you have ever written in C. But it contains a critical integer overflow vulnerability (CWE-190) that could allow an attacker with control over endpoint configuration to trigger an undersized heap allocation and corrupt memory beyond the buffer's boundary.

The Vulnerability Explained

What Actually Goes Wrong

In C, int is a signed 32-bit integer on virtually all modern platforms, with a maximum value of 2,147,483,647 (roughly 2.1 billion). The expression:

num_fragments * endpoint->config.fragment_size

multiplies two int values. If an attacker can influence fragment_size — for example, by controlling endpoint configuration through an API — they can supply a value large enough that the product overflows INT_MAX.

Signed integer overflow in C is undefined behavior per the C standard, but in practice on x86/x64 with common compilers, the value wraps around to a large negative number. Adding RELIABLE_MAX_PACKET_HEADER_BYTES to a large negative number still produces a small or negative result.

The Vulnerable Line

// reliable.c:1299 — BEFORE the fix
int packet_buffer_size = RELIABLE_MAX_PACKET_HEADER_BYTES + num_fragments * endpoint->config.fragment_size;

When this value is subsequently used as an allocation size (e.g., passed to malloc or used to size a stack/heap buffer), the allocator receives a far smaller size than the actual data requires. The reassembly logic then writes full-sized fragment data into a buffer that is far too small, producing a heap buffer overflow.

Concrete Attack Scenario

Consider the following:

  • RELIABLE_MAX_PACKET_HEADER_BYTES = 16 (a small constant)
  • num_fragments = 2
  • endpoint->config.fragment_size = 1,073,741,824 (1 GB, i.e., INT_MAX / 2 + 1)

The multiplication 2 * 1,073,741,824 = 2,147,483,648, which is exactly INT_MAX + 1. On a 32-bit signed integer, this wraps to -2,147,483,648. Adding 16 gives -2,147,483,632.

If this negative value is cast to size_t for a malloc call, it becomes an enormous allocation that will fail. But if it is used directly as a signed size in a comparison or stack allocation, the program may allocate a tiny buffer and then proceed to write gigabytes of fragment data into it — a textbook heap overflow.

The PR's threat model notes that an attacker can influence fragment_size through an API to set large values, making this a realistic exploitation path in networked deployments.

Real-World Impact

  • Heap corruption: Fragment data written past the buffer boundary corrupts adjacent heap metadata or application data.
  • Potential remote code execution: On systems without heap hardening, controlled heap corruption is a well-understood path to code execution.
  • Denial of service: Even without code execution, the overflow causes a crash, disrupting packet delivery for all connected endpoints.

The Fix

What Changed

The fix is surgical and precise — one line, but every detail matters:

Before:

int packet_buffer_size = RELIABLE_MAX_PACKET_HEADER_BYTES + num_fragments * endpoint->config.fragment_size;

After:

size_t packet_buffer_size = (size_t) RELIABLE_MAX_PACKET_HEADER_BYTES + (size_t) num_fragments * (size_t) endpoint->config.fragment_size;

Why This Works

Three specific changes were made:

  1. Type changed from int to size_t: size_t is an unsigned type guaranteed to be wide enough to represent the size of any object in memory. On 64-bit platforms it is 64 bits wide, making overflow astronomically unlikely for any realistic fragment_size.

  2. Each operand individually cast to size_t: This is critical. In C, the cast must happen before the multiplication, not after. If you wrote (size_t)(num_fragments * fragment_size), the multiplication would still occur in int arithmetic and overflow before the cast. By casting each operand first — (size_t) num_fragments * (size_t) endpoint->config.fragment_size — the multiplication is performed in size_t arithmetic from the start.

  3. RELIABLE_MAX_PACKET_HEADER_BYTES also cast: The header bytes constant is cast to size_t as well, ensuring the entire addition is performed in unsigned arithmetic and no implicit signed/unsigned conversion warnings remain.

Before/After Comparison

// BEFORE — signed int arithmetic, overflow is undefined behavior
int packet_buffer_size = RELIABLE_MAX_PACKET_HEADER_BYTES + num_fragments * endpoint->config.fragment_size;

// AFTER — size_t arithmetic, no overflow possible for any realistic input
size_t packet_buffer_size = (size_t) RELIABLE_MAX_PACKET_HEADER_BYTES
                          + (size_t) num_fragments * (size_t) endpoint->config.fragment_size;

The fix ensures that any downstream use of packet_buffer_size — whether as an argument to malloc, a loop bound, or a comparison — operates on a correct, non-overflowed value.

Key Takeaways

  • Cast operands before multiplication, not after: (size_t) a * (size_t) b is safe; (size_t)(a * b) is not — the overflow happens before the cast.
  • packet_buffer_size in reliable.c was the exact variable at risk: any downstream malloc(packet_buffer_size) call would have received a wrapped, dangerously small value.
  • fragment_size coming from endpoint configuration is an attacker-controlled surface: any field that flows from API input into a size calculation must be treated as untrusted.
  • Signed int overflow in C is undefined behavior, meaning the compiler is free to optimize it in ways that make the bug even harder to predict — size_t eliminates this entirely.
  • The regression test in the PR explicitly tests INT_MAX / 2 + 1 as a fragment_size: this is the exact boundary value that triggered the overflow, and it should remain in the test suite permanently.

How Orbis AppSec Detected This

  • Source: The endpoint->config.fragment_size value, which flows from externally controlled endpoint configuration via API
  • Sink: The arithmetic expression num_fragments * endpoint->config.fragment_size assigned to int packet_buffer_size at reliable.c:1299, subsequently used as a buffer allocation size
  • Missing control: No type-width promotion, no overflow guard, and no range validation on fragment_size before it was used in multiplication with a signed int result
  • CWE: CWE-190 — Integer Overflow or Wraparound
  • Fix: All operands in the packet_buffer_size calculation were cast to size_t before arithmetic, and the result variable type was changed from int to size_t

Orbis AppSec automatically detected this vulnerability and opened a pull request with the fix. Try Orbis AppSec on your repositories to find and fix issues like this automatically.

Conclusion

Integer overflow vulnerabilities in buffer-size calculations are one of the oldest and most dangerous classes of bugs in C — and one of the easiest to overlook precisely because the arithmetic looks correct. The one-line change in reliable.c — switching from int to size_t and casting each operand individually — is a textbook example of how a minimal, targeted fix can eliminate an entire class of memory-safety risk.

For developers writing packet-processing, fragmentation, or any other code that computes buffer sizes from external inputs: make size_t your default type for size arithmetic, add explicit overflow guards when inputs are untrusted, and let your compiler and static analysis tools catch the cases your eyes miss.

Prevention and further reading

View the Security Fix

Check out the pull request that fixed this vulnerability

View PR #40
View Repository →

Related Articles

high

nanoid 3.3.11 Integer Overflow: Predictable ID Generation

An integer overflow in nanoid 3.3.11's internal randomness generation causes the library to fall back to predictable ID sequences, undermining the cryptographic guarantees of its supposedly unguessable identifiers. The fix upgrades the dependency tree to patched versions 3.3.12 or 5.1.11.

medium

How Integer Overflow happens in C++ image processing and how to fix it

A signed integer overflow in OpenCV's `bilateralFilter.cpp` allowed the buffer size calculation `cal_width * cal_height * cn` to wrap around to a small or negative value, causing `padding.resize()` to allocate far less memory than needed. Subsequent `memcpy` operations would then write beyond the allocated buffer, creating a heap corruption primitive. The fix is a single targeted cast to `size_t` that promotes the multiplication to unsigned 64-bit arithmetic before any overflow can occur.

critical

How integer overflow in CsoundMYFLTArray constructor happens in C++ and how to fix it

A critical integer overflow vulnerability was discovered in `Java/cs_glue.cpp` at line 324, where the `CsoundMYFLTArray` constructor multiplied a user-controlled integer `n` by `sizeof(MYFLT)` without checking for overflow before passing the result to `malloc`. An attacker supplying a value near `INT_MAX` could trigger the overflow, causing an undersized heap allocation that subsequent writes would overflow. The fix adds an explicit `SIZE_MAX / sizeof(MYFLT)` guard and replaces `malloc` with `ca

high

How integer overflow in malloc happens in C bipartite matching and how to fix it

A high-severity integer overflow vulnerability was discovered in the bipartite matching algorithm implementation where unchecked multiplication operations for memory allocation could wrap around, causing undersized buffer allocations and subsequent heap overflow. The fix replaces vulnerable `malloc(sizeof(int) * V)` patterns with safe `calloc(V, sizeof(int))` calls and adds proper bounds validation to prevent exploitation.

high

How integer truncation heap overflow happens in C++ UEFI ACPI parsing and how to fix it

A high-severity integer truncation vulnerability was discovered in `Mobility.Uefi.Acpi.cpp` where heap allocation sizes were stored in a 16-bit integer (`MO_UINT16`), causing silent truncation when the computed size exceeded 65535 bytes. This led to undersized heap allocations followed by out-of-bounds writes, exploitable by an attacker who can influence ACPI SRAT table contents in virtualized environments. The fix promotes the size variable to `MO_UINTN` (platform-native width) to prevent trunc

critical

How integer overflow in buffer size calculation happens in C and how to fix it

A critical integer overflow vulnerability was discovered in the `nsh_setvar()` function in `nshlib/nsh_vars.c`, where the buffer size calculation `newsize = pstate->varsz + varlen` could wrap around, causing a heap buffer overflow. The fix adds overflow checking before the addition, preventing attackers with shell access from corrupting memory by setting variables with crafted names and values.