How integer overflow in _wopendir() happens in C Windows dirent and how to fix it

Summary

A critical integer overflow vulnerability in include/compat/dirent_msvc.h allowed an attacker-controlled directory path length to wrap the sizeof(wchar_t) * n + 16 allocation calculation, resulting in a dangerously undersized heap buffer. Subsequent writes to that buffer caused a heap overflow, enabling potential memory corruption or code execution on Windows systems. The fix adds a pre-allocation bounds check and proper errno signaling to safely reject overflow-inducing inputs.

Introduction

The include/compat/dirent_msvc.h file is a POSIX compatibility shim that brings opendir/readdir/closedir semantics to Windows MSVC builds. Deep inside its _wopendir() function — the wide-character implementation that handles Windows-native wchar_t directory paths — a single unguarded arithmetic expression created a textbook heap buffer overflow condition.

The vulnerable line, at line 168 of the original file, reads:

dirp->patt = (wchar_t*) malloc(sizeof(wchar_t) * n + 16);

Here, n is the return value of GetFullPathNameW(dirname, 0, NULL, NULL) — the number of wide characters needed to represent the full path. On its own this looks reasonable. But n is ultimately derived from the dirname argument passed by the caller, and if an attacker can influence that string, they can supply a path length that causes sizeof(wchar_t) * n to silently overflow SIZE_MAX, wrapping around to a tiny number. malloc() then happily allocates that tiny buffer, and the code proceeds to write a full-length path into it — corrupting the heap.

The Vulnerability Explained

What actually overflows

sizeof(wchar_t) is 2 on Windows. The multiplication sizeof(wchar_t) * n is performed in size_t arithmetic. On a 32-bit system, SIZE_MAX is 0xFFFFFFFF. If n equals (SIZE_MAX / 2) + 1 — which is 0x80000000 — the product becomes:

2 * 0x80000000 = 0x100000000

In 32-bit size_t arithmetic that wraps to 0x00000000. Add 16 and you get an allocation of exactly 16 bytes. On 64-bit Windows the threshold is larger but equally reachable given that GetFullPathNameW accepts arbitrarily long input strings.

The vulnerable code, verbatim from the diff:

/* VULNERABLE - no overflow check */
dirp->patt = (wchar_t*) malloc(sizeof(wchar_t) * n + 16);
if (dirp->patt) {
    /*
     * ... code that writes sizeof(wchar_t) * n bytes into dirp->patt ...
     */
}

The if (dirp->patt) check only guards against malloc() returning NULL. It does not guard against malloc() succeeding with a dangerously small allocation — which is exactly what happens after an overflow.

How an attacker exploits this

On Windows, directory paths can be supplied through many channels: file open dialogs, network share paths (\\server\share\...), environment variables, or application-level path configuration. An attacker who can influence the dirname argument to _wopendir() — for example, by supplying a crafted UNC path or by setting a working directory through an application's configuration — can trigger the following sequence:

1. Supply a path string long enough that GetFullPathNameW returns n ≈ SIZE_MAX / sizeof(wchar_t).
2. sizeof(wchar_t) * n + 16 overflows to a small value, say 16 or 32 bytes.
3. malloc(16) succeeds and returns a valid 16-byte heap chunk.
4. The subsequent GetFullPathNameW call writes the full multi-kilobyte path into that 16-byte buffer.
5. Adjacent heap metadata and allocations are overwritten — classic heap corruption.

Heap corruption of this kind can be leveraged for denial of service (crash), data leakage from adjacent heap regions, or, with sufficient heap-grooming effort, arbitrary code execution.

Why this file is particularly sensitive

dirent_msvc.h is a production compatibility header included in Windows builds. It is not test scaffolding. Any code path on Windows that calls opendir() with a user-influenced path goes through _wopendir(). This is a broad attack surface in applications that enumerate files or directories based on external input.

The Fix

The patch introduces two targeted changes, both visible in the diff:

Change 1: Pre-allocation overflow guard (line 166–167)

Before:

/* Allocate room for absolute directory name and search pattern */
dirp->patt = (wchar_t*) malloc(sizeof(wchar_t) * n + 16);

After:

/* Allocate room for absolute directory name and search pattern */
if (n <= (SIZE_MAX - 16) / sizeof(wchar_t))
    dirp->patt = (wchar_t*) malloc(sizeof(wchar_t) * n + 16);

The guard n <= (SIZE_MAX - 16) / sizeof(wchar_t) performs the overflow check using only safe division — no multiplication that could itself overflow. Breaking it down:

• SIZE_MAX - 16 computes the largest value that can still have 16 added to it without overflow.
• Dividing by sizeof(wchar_t) gives the maximum n for which sizeof(wchar_t) * n + 16 fits in size_t.
• If n exceeds that threshold, the malloc call is skipped entirely, leaving dirp->patt as NULL.

The existing if (dirp->patt) check immediately below then correctly handles the NULL case and falls through to the error path.

Change 2: Explicit errno on allocation failure (line 219)

Before:

} else {
    /* Cannot allocate memory for search pattern */
    error = 1;
}

After:

} else {
    /* Cannot allocate memory for search pattern */
    _set_errno(ENOMEM);
    error = 1;
}

Previously, when dirp->patt was NULL (whether from a real OOM or from the new overflow guard), the error path set error = 1 but left errno undefined. Callers checking errno after a failed opendir() would get stale or misleading values. Adding _set_errno(ENOMEM) ensures that the POSIX contract is honored: a failed allocation signals ENOMEM, allowing callers to distinguish "directory not found" from "allocation failure."

Why stdint.h was added

The diff also adds #include <stdint.h> at the top of the file. SIZE_MAX is defined in <stdint.h> (or <stddef.h> depending on the platform). Without this include, the guard expression (SIZE_MAX - 16) / sizeof(wchar_t) would fail to compile on toolchains that do not implicitly provide SIZE_MAX. This is a necessary portability companion to the fix.

Prevention & Best Practices

1. Always check multiplication before passing to malloc()

The canonical safe-allocation pattern in C is:

/* Safe: check before multiply */
if (count > 0 && element_size > 0 && count <= (SIZE_MAX - overhead) / element_size) {
    ptr = malloc(element_size * count + overhead);
}

Never write malloc(a * b + c) when any of a, b, or c come from external input without this kind of guard.

2. Prefer calloc() for array allocations

calloc(nmemb, size) performs the overflow check internally on conforming implementations:

/* calloc checks nmemb * size overflow internally */
ptr = calloc(n, sizeof(wchar_t));

This does not cover the + 16 overhead pattern, but for pure array allocations it is safer by default.

3. Use compiler-assisted overflow detection during development

On GCC/Clang, -fsanitize=integer (UBSan) detects integer overflow at runtime during testing. On MSVC, /RTC1 catches some arithmetic issues. These are not substitutes for correct code but catch regressions quickly.

4. Treat all lengths from OS APIs as untrusted

GetFullPathNameW returns a length derived from caller-supplied input. Any value returned from an API that processes external data — file names, network paths, environment variables — must be treated as potentially adversarial before use in arithmetic.

5. Reference standards

• CWE-120: Buffer Copy without Checking Size of Input — https://cwe.mitre.org/data/definitions/120.html
• CWE-190: Integer Overflow or Wraparound — https://cwe.mitre.org/data/definitions/190.html
• CWE-122: Heap-Based Buffer Overflow — https://cwe.mitre.org/data/definitions/122.html
• OWASP: Memory Management Cheat Sheet

Key Takeaways

• sizeof(wchar_t) * n is not safe without an overflow check when n comes from GetFullPathNameW() or any other API that processes caller-supplied paths.
• A successful malloc() return does not mean the buffer is large enough — if the size argument overflowed, the buffer is silently undersized.
• The fix pattern n <= (SIZE_MAX - overhead) / element_size is the correct idiom for pre-multiplication overflow detection in C without introducing further overflow risk.
• Always set errno explicitly in error paths — leaving it undefined after a failure violates POSIX contracts and makes callers unable to distinguish error types.
• Production compatibility headers like dirent_msvc.h are high-value targets because they sit below application logic and handle OS-level resources with attacker-influenced inputs.

How Orbis AppSec Detected This

• Source: The dirname argument to _wopendir(), which originates from caller-supplied directory path strings (e.g., user input, environment variables, network paths on Windows).
• Sink: malloc(sizeof(wchar_t) * n + 16) at include/compat/dirent_msvc.h:168, where n is the unvalidated return value of GetFullPathNameW(dirname, 0, NULL, NULL).
• Missing control: No bounds check on n before the multiplication. The expression sizeof(wchar_t) * n + 16 was passed directly to malloc() without verifying it would not overflow size_t.
• CWE: CWE-120 — Buffer Copy without Checking Size of Input (also related: CWE-190 Integer Overflow, CWE-122 Heap-Based Buffer Overflow).
• Fix: Added the guard if (n <= (SIZE_MAX - 16) / sizeof(wchar_t)) immediately before the malloc() call to reject any n that would cause arithmetic overflow, and added _set_errno(ENOMEM) to the failure branch for correct error signaling.

Orbis AppSec automatically detected this vulnerability and opened a pull request with the fix. Try Orbis AppSec on your repositories to find and fix issues like this automatically.

Conclusion

Integer overflow in heap allocation size calculations is one of the most subtle and dangerous classes of C vulnerability. The bug in _wopendir() is a perfect illustration: the code looks correct at a glance — it allocates memory, checks the pointer, and proceeds. But the allocation size itself was never validated, and an attacker with control over path length could silently reduce a multi-kilobyte allocation to 16 bytes.

The fix is small — a single conditional and an errno assignment — but its impact is significant. It closes a potential heap corruption vector in every Windows build that calls opendir() with external input. If you maintain C code on Windows that processes file paths, audit every malloc() call where the size involves multiplication by a caller-influenced value. The pattern if (n <= (SIZE_MAX - overhead) / element_size) is your friend.

References

• CWE-120: Buffer Copy without Checking Size of Input
• CWE-190: Integer Overflow or Wraparound
• CWE-122: Heap-Based Buffer Overflow
• OWASP Memory Management Cheat Sheet
• Microsoft Docs: GetFullPathNameW
• Semgrep rules: integer overflow
• fix: add buffer-length check in dirent_msvc.h