Integer Overflow in edge_detect.c Heap Allocation Enables Camera-Based Exploit

Introduction

The C/filters/edge_detect.c file handles image processing for camera input — specifically, it applies an unsharp mask edge-boost filter to raw pixel data. But a subtle arithmetic flaw in do_edge_boost() turned routine memory allocation into a heap corruption vector. An attacker controlling a virtual camera device (via v4l2loopback) could report fraudulent image dimensions, triggering an integer overflow that causes malloc() to allocate a near-zero-byte buffer — while the rest of the code blissfully writes megabytes of pixel data into it.

This is a textbook example of how a single missing cast can elevate a low-level C arithmetic quirk into a critical, remotely triggerable memory corruption vulnerability.

The Vulnerability Explained

The Dangerous Code Pattern

At line 11 of C/filters/edge_detect.c, the do_edge_boost() function allocates a temporary working buffer for its unsharp mask computation:

// BEFORE (vulnerable)
uint8_t *tmp = malloc(w * h);

Here, w and h are int values representing the camera frame's width and height. The multiplication w * h is performed as a 32-bit signed integer operation. When the result exceeds INT_MAX (or wraps UINT32_MAX for unsigned types), the value silently overflows.

The Overflow Math

Consider the exact exploit scenario documented in the PR: a crafted V4L2 device reports dimensions of width = 65536, height = 65536.

65536 * 65536 = 4,294,967,296

In 32-bit arithmetic, this is exactly 2^32, which wraps to 0 mod 2^32. So malloc(0) is called. On most platforms, malloc(0) returns a non-NULL pointer to a valid-but-zero-length allocation. The if (!tmp) return; NULL check passes successfully, and the code proceeds to write an entire camera frame's worth of pixel data — potentially 12 GB for a 65536×65536 frame — into a buffer with zero usable bytes.

Even with more modest overflow scenarios, like 65535 × 65535 = 4,294,836,225, the 32-bit result wraps to a tiny value (4,294,836,225 mod 2^32 = 4,294,836,225 - 4,294,967,296 = -131,071, or as unsigned: 4,294,836,225), potentially still causing a massively undersized allocation.

Attack Scenario: Crafted V4L2 Device

V4L2 (Video4Linux2) is the Linux kernel framework for camera devices. Using the v4l2loopback kernel module, an attacker can create a virtual camera that reports arbitrary VIDIOC_G_FMT metadata — including fraudulent width and height values — without any kernel-level validation of whether those dimensions are physically plausible.

The attack chain looks like this:

1. Attacker loads v4l2loopback and creates a virtual /dev/video device
2. The device advertises width=65536, height=65536 in its format descriptor
3. The application opens the device and passes these dimensions to do_edge_boost()
4. malloc(65536 * 65536) → malloc(0) → returns a tiny valid pointer
5. The unsharp mask computation writes pixel data far beyond the allocation boundary
6. Heap corruption — potentially leading to arbitrary code execution

This doesn't require kernel exploits, privileged access, or network access. Any process that can create a v4l2loopback device (often available to regular users) can trigger this.

The Fix

The fix is precise and addresses two distinct failure modes: negative/zero dimensions, and dimensions whose product overflows.

Before vs. After

// BEFORE (vulnerable)
uint8_t *tmp = malloc(w * h);
if (!tmp)
    return;

// AFTER (fixed)
if (h <= 0 || w <= 0)
    return;
uint8_t *tmp = calloc((size_t)w, (size_t)h);
if (!tmp)
    return;

Why Each Change Matters

1. Explicit dimension validation (if (h <= 0 || w <= 0))

Before even touching memory allocation, the fix rejects nonsensical dimensions. Zero or negative width/height values are physically impossible for a camera frame and should never reach the allocation logic. This guard eliminates an entire class of degenerate inputs before they can cause harm.

2. calloc((size_t)w, (size_t)h) instead of malloc(w * h)

This is the critical change. calloc(nmemb, size) takes two separate arguments and is defined by the C standard to perform the multiplication internally with overflow detection. Specifically, a conforming calloc implementation must return NULL if nmemb * size would overflow SIZE_MAX, rather than silently allocating a tiny buffer.

Additionally, casting both arguments to (size_t) before passing them ensures the multiplication happens in the address-space-native unsigned integer width (64 bits on a 64-bit system), rather than in 32-bit signed arithmetic where overflow is undefined behavior.

The switch from malloc to calloc also zero-initializes the buffer, which eliminates any potential information disclosure from uninitialized heap memory being read as part of the filter output.

The Regression Test

The PR includes a new test file tests/test_invariant_edge_detect.c that directly encodes the security invariant:

dims payloads[] = {
    /* Exact exploit: w*h overflows 32-bit (65536*65536 = 0 mod 2^32) */
    { 65536, 65536 },
    /* Boundary: just over 16-bit max */
    { 65535, 65535 },
    /* Valid small input: should succeed */
    { 64, 48 },
};

For overflow-triggering inputs, the test asserts that edge_detect() must return NULL rather than a valid pointer backed by a corrupt allocation:

if ((uint64_t)w * (uint64_t)h > (uint64_t)UINT32_MAX) {
    ck_assert_msg(result == NULL,
        "edge_detect must reject overflowing dimensions safely");
}

Note the use of (uint64_t) casts in the test's own overflow check — a demonstration of the exact safe pattern the fix applies to production code.

Prevention & Best Practices

1. Always Use calloc() for Array-Shaped Allocations

When allocating n elements of size bytes each, use calloc(n, size) rather than malloc(n * size). The standard mandates that calloc handles the overflow check for you.

// Risky
uint8_t *buf = malloc(width * height * channels);

// Safe
uint8_t *buf = calloc(width * height, channels);  // still risky if w*h overflows first
// Even safer: validate each dimension first, then calloc

2. Validate External Inputs Before Arithmetic

Any dimension, count, or size value that originates from an external source — a file, a network packet, a device driver, user input — must be validated against realistic bounds before being used in arithmetic:

// Validate before use
if (w == 0 || h == 0 || w > MAX_CAMERA_WIDTH || h > MAX_CAMERA_HEIGHT)
    return ERROR_INVALID_DIMENSIONS;

3. Use Safe Integer Libraries

For complex arithmetic on untrusted inputs, consider using safe integer libraries:
- C: <stdckdint.h> (C23) provides ckd_mul() for checked multiplication
- C: The safe_iop or IntegerLib libraries for pre-C23 codebases
- Rust: Integer overflow is checked by default in debug builds; use checked_mul() in production

// C23 checked multiplication
size_t total;
if (ckd_mul(&total, (size_t)w, (size_t)h)) {
    return NULL; // overflow detected
}
uint8_t *buf = malloc(total);

4. Relevant Standards and References

• CWE-190: Integer Overflow or Wraparound
• CWE-122: Heap-based Buffer Overflow
• CERT C Rule INT30-C: Ensure that unsigned integer operations do not wrap
• CERT C Rule MEM35-C: Allocate sufficient memory for an object
• OWASP: Input Validation Cheat Sheet

Key Takeaways

• malloc(w * h) with int operands is never safe for camera or image dimensions — the product of two 16-bit-range values can silently overflow a 32-bit int, and malloc(0) succeeds without returning NULL.

• calloc(w, h) is the correct idiom for 2D buffer allocation — it separates the element count from the element size, and a conforming implementation handles overflow detection internally.

• V4L2 device metadata is attacker-controlled — any application that reads frame dimensions from a V4L2 device and passes them directly to memory allocation functions without bounds checking is potentially vulnerable to this class of attack.

• The fix required only 3 lines of code, but catching it required understanding the interaction between C integer promotion rules, malloc(0) behavior, and the V4L2 threat model — a reminder that security bugs often hide in plain sight.

• Zero-initialization via calloc is a defense-in-depth bonus — even if a future code change introduces a read-before-write bug in the filter logic, calloc's zero-initialization prevents uninitialized heap data from leaking into filter output.

Conclusion

The integer overflow in do_edge_boost() within C/filters/edge_detect.c is a clear example of how a single arithmetic expression — w * h — can become a critical security boundary when the inputs are externally controlled. The V4L2 attack surface makes this particularly dangerous: virtual camera devices are a realistic, low-privilege attack vector that many developers don't consider when writing image processing code.

The fix is minimal but complete: validate dimensions before allocation, and use calloc()'s two-argument form to let the standard library handle overflow detection. The accompanying regression test encodes the security invariant in an executable, permanent form, ensuring this class of bug cannot silently regress.

If your codebase processes image or video data from external devices, audit every allocation that uses frame dimensions in its size calculation. The pattern malloc(w * h) or malloc(width * height * bpp) deserves the same scrutiny as strcpy or gets.

This vulnerability was identified and fixed by OrbisAI Security. Automated security scanning, triage, and patch generation — so your team can ship with confidence.