How integer overflow in buffer size calculation happens in C++ and how to fix it

Introduction

In the OpenCV Hardware Abstraction Layer (HAL), we discovered a critical integer overflow vulnerability in hal/ndsrvp/src/filter.cpp at line 140. The filter() function processes image data through convolution operations, but a flaw in how it calculates buffer sizes created a severe security risk that could lead to heap corruption.

The vulnerable code multiplies three integers—cal_width, cal_height, and cnes (channels)—to determine how much memory to allocate for padding data. When an attacker supplies image dimensions large enough to overflow a 32-bit signed integer, the calculation wraps around to a small value, allocating a tiny buffer that subsequent operations then overflow.

This matters for any developer working with image processing, embedded systems, or any code that calculates buffer sizes from external input. The fix is deceptively simple—a single cast—but understanding why it works requires grasping how integer arithmetic interacts with memory allocation.

The Vulnerability Explained

What Happens in the Code

The vulnerable code in filter.cpp calculates the size of a padding buffer used during image convolution:

// calculate source border
ctx->padding.resize(cal_width * cal_height * cnes);

Here's the problem: cal_width, cal_height, and cnes are all int types (32-bit signed integers). When you multiply three int values in C++, the result is also an int. The maximum value a signed 32-bit integer can hold is 2,147,483,647 (approximately 2.1 billion).

Consider what happens with crafted dimensions:
- cal_width = 65538 (65536 + kernel_width - 1)
- cal_height = 65538
- cnes = 4 (RGBA channels)

The true product is: 65538 × 65538 × 4 = 17,180,917,776

But this exceeds INT_MAX by a factor of 8. The 32-bit multiplication overflows, and the result wraps around to a much smaller (possibly negative) value. When this corrupted value is passed to resize(), the vector allocates far less memory than needed.

The Attack Scenario

An attacker targeting this embedded firmware could:

1. Craft malicious image data with dimensions specifically chosen to trigger overflow (e.g., width=65536, height=65536)
2. Submit the image through whatever interface feeds the filter function—this could be a network peer on the same LAN segment or physical access to the device
3. Trigger the overflow causing ctx->padding to allocate perhaps only a few kilobytes
4. Corrupt the heap when the filter writes cal_width * cnes bytes per row into the undersized buffer

The subsequent write operations don't know the buffer is too small—they write based on the original dimensions, overwriting adjacent heap memory. This can:
- Crash the device (denial of service)
- Corrupt function pointers for code execution
- Overwrite security-critical data structures

Why This Is Critical

This vulnerability exists in production embedded firmware code. While exploitation requires physical access or a compromised network peer, embedded systems often operate in environments where such access is plausible—industrial control systems, IoT devices, or automotive systems. A heap corruption vulnerability in image processing code has historically been a vector for sophisticated attacks.

The Fix

The fix is elegant in its simplicity—a single type cast that changes everything:

Before (Vulnerable)

ctx->padding.resize(cal_width * cal_height * cnes);

After (Fixed)

ctx->padding.resize((size_t)cal_width * cal_height * cnes);

Why This Works

By casting cal_width to size_t (an unsigned 64-bit type on most platforms) before the multiplication, C++ promotion rules kick in:

1. (size_t)cal_width converts the first operand to 64-bit
2. When multiplying size_t by int, the int is promoted to size_t
3. The entire calculation now happens in 64-bit arithmetic
4. 65538 × 65538 × 4 = 17,180,917,776 fits comfortably in a 64-bit value

The resize() method of std::vector accepts a size_t parameter, so the correct large value is passed through without truncation.

The Critical Detail

The cast must be on the first operand. If you wrote:

ctx->padding.resize(cal_width * cal_height * (size_t)cnes);  // STILL VULNERABLE!

The overflow would still occur because cal_width * cal_height is evaluated first as 32-bit arithmetic, overflows, and only then is the corrupted result promoted to 64-bit for the final multiplication.

Prevention & Best Practices

1. Always Use size_t for Size Calculations

When calculating buffer sizes, use size_t from the start:

size_t buffer_size = static_cast<size_t>(width) * height * channels;

2. Add Explicit Overflow Checks

For critical code, validate before allocation:

if (cal_width > SIZE_MAX / cal_height) {
    return ERROR_OVERFLOW;
}
size_t partial = static_cast<size_t>(cal_width) * cal_height;
if (partial > SIZE_MAX / cnes) {
    return ERROR_OVERFLOW;
}
size_t total = partial * cnes;

3. Use Safe Integer Libraries

Consider libraries like SafeInt (Microsoft) or safe_math utilities:

#include <safeint.h>
size_t total;
if (!SafeMultiply(cal_width, cal_height, total) || 
    !SafeMultiply(total, cnes, total)) {
    return ERROR_OVERFLOW;
}

4. Validate Input Dimensions Early

Reject unreasonable dimensions before they reach calculation code:

const int MAX_DIMENSION = 32768;
if (width > MAX_DIMENSION || height > MAX_DIMENSION) {
    return ERROR_INVALID_DIMENSIONS;
}

5. Enable Compiler Warnings

Use -ftrapv (GCC/Clang) during testing to trap on signed overflow, or -fsanitize=integer for runtime detection.

Key Takeaways

• The cal_width * cal_height * cnes pattern is dangerous when any operand comes from external input—always cast to size_t before multiplication
• Cast position matters: (size_t)a * b * c is safe; a * b * (size_t)c is not
• Image dimensions are attacker-controlled input—treat width, height, and channel count as untrusted
• Embedded firmware requires the same security rigor as web applications—physical access attacks are real
• A single-character fix can prevent heap corruption—but only if you understand the underlying integer promotion rules

How Orbis AppSec Detected This

• Source: Image dimension parameters (width, height) passed to the filter() function from external image data
• Sink: ctx->padding.resize(cal_width * cal_height * cnes) at filter.cpp:140
• Missing control: No overflow check or 64-bit promotion before the multiplication that determines allocation size
• CWE: CWE-190 (Integer Overflow or Wraparound)
• Fix: Cast the first multiplication operand to size_t to ensure 64-bit arithmetic: (size_t)cal_width * cal_height * cnes

Orbis AppSec automatically detected this vulnerability and opened a pull request with the fix. Try Orbis AppSec on your repositories to find and fix issues like this automatically.

Conclusion

This integer overflow vulnerability in filter.cpp demonstrates how a seemingly innocuous multiplication can become a critical security flaw. The pattern of multiplying dimensions to calculate buffer sizes is ubiquitous in image processing, audio processing, and data handling code—making this class of vulnerability particularly widespread.

The fix—casting to size_t before multiplication—is a pattern every C and C++ developer should internalize. When you see width * height * channels or similar calculations feeding into memory allocation, immediately ask: "What happens if these values are large?" If the answer involves 32-bit overflow, you've found a bug.

Remember: in security-critical code, the smallest details matter. A single (size_t) cast was the difference between a heap corruption vulnerability and safe code.

References

• CWE-190: Integer Overflow or Wraparound
• CWE-122: Heap-based Buffer Overflow
• OWASP Integer Overflow
• SEI CERT C Coding Standard - INT32-C
• Semgrep Integer Overflow Rules
• fix: in filter in filter.cpp (PR link)