How Integer Overflow happens in C++ image processing and how to fix it

How Integer Overflow Happens in C++ Image Processing and How to Fix It

Introduction

The hal/ndsrvp/src/bilateralFilter.cpp file is part of the hardware-accelerated image-processing layer responsible for applying bilateral filtering — a noise-reduction technique that preserves edges. Deep inside this implementation, a single arithmetic expression quietly contained a time-bomb: a signed integer multiplication used directly as a memory allocation size.

At line 193, the code calculated how much memory to allocate for a padded image buffer:

padding.resize(cal_width * cal_height * cn);

All three variables — cal_width, cal_height, and cn (channel count) — are signed int values. For typical small images this works fine. But for large images, the product can silently overflow the 32-bit signed integer range, wrapping around to a small or even negative value. The std::vector::resize() call then allocates a tiny buffer, and every subsequent memcpy into that buffer writes far beyond its end.

This matters to any developer working with image dimensions in C++ — it's one of the most common and underestimated sources of heap corruption in media-processing code.

The Vulnerability Explained

What Goes Wrong

In C++, integer arithmetic is performed in the type of the operands. When cal_width, cal_height, and cn are all int (32-bit signed), the expression:

cal_width * cal_height * cn

is evaluated entirely in 32-bit signed arithmetic. The maximum value of a 32-bit signed integer is 2,147,483,647 (~2 GB of pixels). For a large image — say, 32768 × 32768 with 3 channels — the true product is 3,221,225,472, which exceeds INT_MAX. The result wraps around to -1,073,741,824 (a negative number), or to some small positive value depending on the exact dimensions.

When this wrapped value is passed to padding.resize(), the vector allocates a buffer of that tiny (or zero, or even implementation-defined-on-negative) size. The pointer pad_data then points to a drastically undersized allocation.

The Vulnerable Code (Before Fix)

// Line 192-195 — BEFORE fix
std::vector<uchar> padding;
padding.resize(cal_width * cal_height * cn);   // ← overflow here
uchar* pad_data = &padding[0];
int pad_step = cal_width * cn;

The overflow happens silently. There is no error, no exception, no warning at runtime (unless you compile with -fsanitize=undefined). The vector is just too small.

How It Can Be Exploited

In the context of this embedded firmware HAL, an attacker who can supply image data to the bilateral filter (via a compromised network peer on the same bus/LAN, or with physical access) could craft an image with dimensions chosen to produce a specific overflow value. For example:

1. Attacker supplies an image where cal_width * cal_height * cn overflows to, say, 256 bytes.
2. padding.resize(256) allocates 256 bytes on the heap.
3. The filter proceeds to copy the full padded image (potentially megabytes) into pad_data.
4. Every byte beyond offset 256 overwrites adjacent heap metadata or other allocations.

This is a classic heap buffer overflow. Depending on what lives adjacent to the allocation, this primitive can be used to corrupt function pointers, vtable entries, or other security-critical data structures. The PR notes correctly identify it as an exploit primitive — even if not independently exploitable today, it can be chained with other weaknesses by automated exploit-development tooling.

The PR also flags similar patterns at lines 164, 217, 224, and 236 in the same file, suggesting this is a systemic pattern worth auditing across the entire filter implementation.

The Fix

The One-Line Change

The fix is elegantly minimal:

-    padding.resize(cal_width * cal_height * cn);
+    padding.resize((size_t)cal_width * cal_height * cn);

Why This Works

In C++, when you mix types in a multiplication, the operands are implicitly promoted to the wider type. By casting cal_width to size_t (an unsigned 64-bit type on 64-bit platforms), the entire chain of multiplications is performed in 64-bit unsigned arithmetic:

(size_t)cal_width * cal_height * cn
  ↑ size_t          ↑ int          ↑ int

Because the first operand is size_t, cal_height is promoted to size_t before multiplication, and then cn is also promoted before the second multiplication. The result is a size_t that can represent values up to 18,446,744,073,709,551,615 — far beyond any realistic image size.

Before vs. After

The fix is strictly additive in safety: it only changes behavior for inputs that would have previously overflowed, and for those inputs it now correctly allocates the required memory (or lets the system throw std::bad_alloc if the system genuinely cannot satisfy the request — which is the correct and safe behavior).

Prevention & Best Practices

1. Always Use size_t for Memory Sizes

Any expression that will be used as an allocation size should be computed in size_t from the start. A common idiom:

// Prefer this pattern for all allocation sizes
size_t buffer_size = (size_t)width * height * channels;
std::vector<uchar> buf(buffer_size);

2. Validate Dimensions Before Arithmetic

Add explicit bounds checks before performing dimension arithmetic:

// Guard against unreasonable dimensions
constexpr int MAX_DIMENSION = 65536;
constexpr int MAX_CHANNELS  = 4;
if (cal_width <= 0 || cal_height <= 0 || cn <= 0 ||
    cal_width > MAX_DIMENSION || cal_height > MAX_DIMENSION || cn > MAX_CHANNELS) {
    return CV_HAL_ERROR_UNKNOWN;
}

3. Use Compiler Sanitizers During Development

Enable UBSan (Undefined Behavior Sanitizer) in your CI pipeline:

# GCC / Clang
-fsanitize=undefined,integer
-fsanitize-undefined-trap-on-error

UBSan will catch signed integer overflows at runtime during testing, before they reach production.

4. Apply the Pattern Consistently

The PR notes that lines 164, 217, 224, and 236 in the same file use similar patterns. Whenever you fix one instance of this pattern, search the entire file (and related files) for the same idiom:

# Find similar patterns in C++ files
grep -n 'resize([a-z_]* \* [a-z_]*' hal/ndsrvp/src/*.cpp

5. Use Static Analysis

Tools that can catch this class of issue:
- Semgrep: Rules for integer-to-size conversions (semgrep.dev/r?q=integer-overflow)
- Coverity / CodeQL: Taint-tracking from image dimensions to allocation sizes
- MSVC /analyze or Clang-Tidy bugprone-implicit-widening-of-multiplication-result: Flags exactly this pattern

Relevant Standards

• CWE-190: Integer Overflow or Wraparound
• CWE-131: Incorrect Calculation of Buffer Size
• CERT C++ Rule INT30-C: Ensure that unsigned integer operations do not wrap
• OWASP: A05:2021 – Security Misconfiguration (improper resource limits)

Key Takeaways

• cal_width * cal_height * cn at line 193 was the exact overflow site — three signed int values multiplied together before being passed to resize(), with no widening cast anywhere in the chain.
• A single (size_t) cast on the first operand is sufficient to promote the entire expression to 64-bit unsigned arithmetic in C++, eliminating the overflow.
• Heap buffer overflows from integer overflow are silent — the program does not crash at the overflow; it crashes (or worse, silently corrupts memory) later during the memcpy, making root-cause analysis difficult without sanitizers.
• Similar patterns at lines 164, 217, 224, and 236 in the same file should be reviewed and hardened with the same (size_t) cast pattern.
• Embedded firmware is not immune — even when exploitation requires physical access or a compromised peer, removing exploit primitives proactively raises the cost of attack chains for increasingly capable automated tools.

How Orbis AppSec Detected This

• Source: Image dimension parameters (cal_width, cal_height, cn) derived from externally supplied image data passed into the bilateral filter HAL entry point.
• Sink: padding.resize(cal_width * cal_height * cn) at hal/ndsrvp/src/bilateralFilter.cpp:193 — a memory allocation size computed from a signed integer multiplication with no overflow guard.
• Missing control: No cast to size_t (or other unsigned 64-bit type) before the multiplication, and no bounds validation on the dimension values prior to the arithmetic.
• CWE: CWE-190 — Integer Overflow or Wraparound.
• Fix: Added a (size_t) cast to cal_width so that (size_t)cal_width * cal_height * cn is evaluated in 64-bit unsigned arithmetic, preventing overflow before the result is passed to resize().

Orbis AppSec automatically detected this vulnerability and opened a pull request with the fix. Try Orbis AppSec on your repositories to find and fix issues like this automatically.

Conclusion

Integer overflow in buffer size calculations is one of the oldest and most persistent vulnerability classes in C and C++ code, and it remains dangerous precisely because it is invisible at the point of failure. The overflow happens silently during arithmetic; the program only misbehaves later, during memory access, making it hard to connect cause and effect without the right tooling.

In bilateralFilter.cpp, the fix required changing exactly one token — adding (size_t) before cal_width — but the security impact is significant: it eliminates a heap corruption primitive that could otherwise be chained into a more serious exploit. The lesson for all C++ developers working with image dimensions, file sizes, or any externally influenced numeric values: always compute allocation sizes in size_t, and validate bounds before you do arithmetic, not after.

References

• CWE-190: Integer Overflow or Wraparound
• CWE-131: Incorrect Calculation of Buffer Size
• OWASP Input Validation Cheat Sheet
• CERT C++ INT30-C: Ensure unsigned integer operations do not wrap
• Clang-Tidy: bugprone-implicit-widening-of-multiplication-result
• Semgrep rules: integer-overflow
• harden: add integer overflow check in bilateralFilter.cpp