Introduction
In the OpenCV Hardware Abstraction Layer (HAL), we discovered a critical integer overflow vulnerability in hal/ndsrvp/src/filter.cpp at line 140. The filter() function processes image data through convolution operations, but a flaw in how it calculates buffer sizes created a severe security risk that could lead to heap corruption.
The vulnerable code multiplies three integers—cal_width, cal_height, and cnes (channels)—to determine how much memory to allocate for padding data. When an attacker supplies image dimensions large enough to overflow a 32-bit signed integer, the calculation wraps around to a small value, allocating a tiny buffer that subsequent operations then overflow.
This matters for any developer working with image processing, embedded systems, or any code that calculates buffer sizes from external input. The fix is deceptively simple—a single cast—but understanding why it works requires grasping how integer arithmetic interacts with memory allocation.
The Vulnerability Explained
What Happens in the Code
The vulnerable code in filter.cpp calculates the size of a padding buffer used during image convolution:
// calculate source border
ctx->padding.resize(cal_width * cal_height * cnes);
Here's the problem: cal_width, cal_height, and cnes are all int types (32-bit signed integers). When you multiply three int values in C++, the result is also an int. The maximum value a signed 32-bit integer can hold is 2,147,483,647 (approximately 2.1 billion).
Consider what happens with crafted dimensions:
- cal_width = 65538 (65536 + kernel_width - 1)
- cal_height = 65538
- cnes = 4 (RGBA channels)
The true product is: 65538 × 65538 × 4 = 17,180,917,776
But this exceeds INT_MAX by a factor of 8. The 32-bit multiplication overflows, and the result wraps around to a much smaller (possibly negative) value. When this corrupted value is passed to resize(), the vector allocates far less memory than needed.
The Attack Scenario
An attacker targeting this embedded firmware could:
- Craft malicious image data with dimensions specifically chosen to trigger overflow (e.g., width=65536, height=65536)
- Submit the image through whatever interface feeds the filter function—this could be a network peer on the same LAN segment or physical access to the device
- Trigger the overflow causing
ctx->paddingto allocate perhaps only a few kilobytes - Corrupt the heap when the filter writes
cal_width * cnesbytes per row into the undersized buffer
The subsequent write operations don't know the buffer is too small—they write based on the original dimensions, overwriting adjacent heap memory. This can:
- Crash the device (denial of service)
- Corrupt function pointers for code execution
- Overwrite security-critical data structures
Why This Is Critical
This vulnerability exists in production embedded firmware code. While exploitation requires physical access or a compromised network peer, embedded systems often operate in environments where such access is plausible—industrial control systems, IoT devices, or automotive systems. A heap corruption vulnerability in image processing code has historically been a vector for sophisticated attacks.
The Fix
The fix is elegant in its simplicity—a single type cast that changes everything:
Before (Vulnerable)
ctx->padding.resize(cal_width * cal_height * cnes);
After (Fixed)
ctx->padding.resize((size_t)cal_width * cal_height * cnes);
Why This Works
By casting cal_width to size_t (an unsigned 64-bit type on most platforms) before the multiplication, C++ promotion rules kick in:
(size_t)cal_widthconverts the first operand to 64-bit- When multiplying
size_tbyint, theintis promoted tosize_t - The entire calculation now happens in 64-bit arithmetic
- 65538 × 65538 × 4 = 17,180,917,776 fits comfortably in a 64-bit value
The resize() method of std::vector accepts a size_t parameter, so the correct large value is passed through without truncation.
The Critical Detail
The cast must be on the first operand. If you wrote:
ctx->padding.resize(cal_width * cal_height * (size_t)cnes); // STILL VULNERABLE!
The overflow would still occur because cal_width * cal_height is evaluated first as 32-bit arithmetic, overflows, and only then is the corrupted result promoted to 64-bit for the final multiplication.
Prevention & Best Practices
1. Always Use size_t for Size Calculations
When calculating buffer sizes, use size_t from the start:
size_t buffer_size = static_cast<size_t>(width) * height * channels;
2. Add Explicit Overflow Checks
For critical code, validate before allocation:
if (cal_width > SIZE_MAX / cal_height) {
return ERROR_OVERFLOW;
}
size_t partial = static_cast<size_t>(cal_width) * cal_height;
if (partial > SIZE_MAX / cnes) {
return ERROR_OVERFLOW;
}
size_t total = partial * cnes;
3. Use Safe Integer Libraries
Consider libraries like SafeInt (Microsoft) or safe_math utilities:
#include <safeint.h>
size_t total;
if (!SafeMultiply(cal_width, cal_height, total) ||
!SafeMultiply(total, cnes, total)) {
return ERROR_OVERFLOW;
}
4. Validate Input Dimensions Early
Reject unreasonable dimensions before they reach calculation code:
const int MAX_DIMENSION = 32768;
if (width > MAX_DIMENSION || height > MAX_DIMENSION) {
return ERROR_INVALID_DIMENSIONS;
}
5. Enable Compiler Warnings
Use -ftrapv (GCC/Clang) during testing to trap on signed overflow, or -fsanitize=integer for runtime detection.
Key Takeaways
- The
cal_width * cal_height * cnespattern is dangerous when any operand comes from external input—always cast tosize_tbefore multiplication - Cast position matters:
(size_t)a * b * cis safe;a * b * (size_t)cis not - Image dimensions are attacker-controlled input—treat width, height, and channel count as untrusted
- Embedded firmware requires the same security rigor as web applications—physical access attacks are real
- A single-character fix can prevent heap corruption—but only if you understand the underlying integer promotion rules
How Orbis AppSec Detected This
- Source: Image dimension parameters (
width,height) passed to thefilter()function from external image data - Sink:
ctx->padding.resize(cal_width * cal_height * cnes)atfilter.cpp:140 - Missing control: No overflow check or 64-bit promotion before the multiplication that determines allocation size
- CWE: CWE-190 (Integer Overflow or Wraparound)
- Fix: Cast the first multiplication operand to
size_tto ensure 64-bit arithmetic:(size_t)cal_width * cal_height * cnes
Orbis AppSec automatically detected this vulnerability and opened a pull request with the fix. Try Orbis AppSec on your repositories to find and fix issues like this automatically.
Conclusion
This integer overflow vulnerability in filter.cpp demonstrates how a seemingly innocuous multiplication can become a critical security flaw. The pattern of multiplying dimensions to calculate buffer sizes is ubiquitous in image processing, audio processing, and data handling code—making this class of vulnerability particularly widespread.
The fix—casting to size_t before multiplication—is a pattern every C and C++ developer should internalize. When you see width * height * channels or similar calculations feeding into memory allocation, immediately ask: "What happens if these values are large?" If the answer involves 32-bit overflow, you've found a bug.
Remember: in security-critical code, the smallest details matter. A single (size_t) cast was the difference between a heap corruption vulnerability and safe code.