How NULL pointer dereference happens in C gotcha_malloc() and how to fix it

Summary

A critical NULL pointer dereference vulnerability was discovered in src/gotcha_utils.c at line 84, where the add_library() function called gotcha_malloc() without checking whether the allocation succeeded before dereferencing the returned pointer. Because gotcha_malloc uses mmap internally, it can return NULL or MAP_FAILED under memory pressure, causing a segmentation fault that crashes the host application. The fix adds a single, targeted null check that returns early if allocation fails, preventing the crash entirely.

Introduction

The src/gotcha_utils.c file is responsible for managing library tracking in the GOTCHA interposition library — a system that wraps shared library function calls at runtime. A flaw in the add_library() function created a reliability and security risk: the function allocated memory using gotcha_malloc() but proceeded to dereference the result unconditionally, without ever checking whether the allocation actually succeeded.

Here is the vulnerable code pattern, exactly as it existed before the fix:

struct library_t *add_library(struct link_map *map) {
  library_t *newlib = gotcha_malloc(sizeof(library_t));
  newlib->map = map;       // ← dereference with no NULL check
  newlib->flags = 0;
  newlib->generation = 0;
  ...
}

The moment gotcha_malloc() returns NULL — which it can, because it uses mmap internally — the very next line writes through a null pointer. On any POSIX system, that is an immediate segmentation fault.

This matters for any developer writing C code that uses custom allocators, memory pools, or wrappers around mmap. The pattern of "allocate, then immediately use without checking" is one of the most common sources of crashes in production C software.

The Vulnerability Explained

What is gotcha_malloc and why can it fail?

gotcha_malloc is a custom allocator used internally by the GOTCHA library. Unlike a simple malloc backed by the heap, gotcha_malloc uses mmap to request memory directly from the operating system. The mmap system call can fail in several real-world conditions:

• Memory pressure: The system is running low on virtual address space or physical memory.
• Resource limits: The process has hit RLIMIT_AS (address space limit) or RLIMIT_DATA.
• OOM conditions: The OS memory overcommit policy rejects the request.

When mmap fails, it returns MAP_FAILED (which is (void *) -1), and a correctly written wrapper would propagate NULL to indicate failure. If callers don't check for this, they proceed as if they have a valid pointer — and crash.

The vulnerable code at line 84

// BEFORE FIX — src/gotcha_utils.c line 84
library_t *newlib = gotcha_malloc(sizeof(library_t));
newlib->map = map;   // ← if newlib is NULL, this is UB and a segfault

The problem is not subtle. newlib is used on the very next line with no intervening check. In C, dereferencing a null pointer is undefined behavior, and on virtually every real platform it produces SIGSEGV, killing the process immediately.

How this could be exploited

While this is primarily a reliability vulnerability, it has security implications in the context of GOTCHA's use case. GOTCHA is a function interposition library — it is used to wrap and replace functions in loaded shared libraries at runtime. It is commonly embedded in larger applications (including HPC tools and security instrumentation frameworks).

An attacker or a malicious shared library that can induce memory pressure at a critical moment — for example, by exhausting virtual address space before GOTCHA processes a new library — could trigger this crash path in add_library() deliberately. The result is a denial of service: the entire host process crashes.

In environments where GOTCHA is used as part of a security monitoring or auditing layer, crashing the interposition library would also disable the security instrumentation itself, potentially allowing subsequent malicious activity to go undetected.

Real-world impact

• Crash on memory pressure: Any production system under load that hits the allocation failure path will segfault without warning.
• No graceful degradation: Because there is no null check, there is no opportunity to log the failure, alert operators, or fall back to a safe state.
• Cascading failures: Since add_library() is called during library tracking setup, a crash here can corrupt the entire interposition state.

The Fix

The fix is minimal, surgical, and correct. A single line was added immediately after the gotcha_malloc() call:

Before

struct library_t *add_library(struct link_map *map) {
  library_t *newlib = gotcha_malloc(sizeof(library_t));
  newlib->map = map;
  newlib->flags = 0;
  newlib->generation = 0;

After

struct library_t *add_library(struct link_map *map) {
  library_t *newlib = gotcha_malloc(sizeof(library_t));
  if (!newlib) return NULL;   // ← added: safe early exit on allocation failure
  newlib->map = map;
  newlib->flags = 0;
  newlib->generation = 0;

Why this fix works

The guard if (!newlib) return NULL; intercepts the failure case before any pointer dereference occurs. By returning NULL, the function signals to its callers that library registration failed. This follows the established C convention for communicating allocation failure through the return value.

Key properties of this fix:

1. No undefined behavior: The null pointer is never dereferenced.
2. Caller notification: Returning NULL allows upstream code to detect and handle the failure gracefully.
3. Zero overhead on the success path: The check is a single branch that is almost always false in practice.
4. Consistent with C idioms: The pattern if (!ptr) return NULL; is universally understood by C developers.

Note for maintainers: The PR description notes that multiple callers of gotcha_malloc may have this issue. This fix addresses the specific instance at line 84 in add_library(), but a thorough audit of all gotcha_malloc call sites is recommended to ensure no other locations skip the null check.

Prevention & Best Practices

1. Always check allocation return values in C

Every call to malloc, calloc, realloc, mmap, or any custom allocator must be followed by a null check before the pointer is used. This is a non-negotiable rule in production C code.

// Always do this:
void *ptr = malloc(size);
if (!ptr) {
    // handle error
    return -1;
}
// safe to use ptr here

2. Use a wrapper that enforces checking

Some codebases use a xmalloc() pattern — a wrapper that calls abort() or exit() on allocation failure. While this doesn't prevent crashes, it at least makes the failure explicit and avoids undefined behavior:

void *xmalloc(size_t size) {
    void *ptr = malloc(size);
    if (!ptr) {
        fprintf(stderr, "fatal: memory allocation failed\n");
        abort();
    }
    return ptr;
}

This is not a substitute for proper error handling, but it is safer than silent null dereference.

3. Enable compiler and sanitizer warnings

Modern compilers and sanitizers can catch many of these issues:

• GCC/Clang: Compile with -Wall -Wextra. Clang's -Weverything includes -Wnull-dereference.
• AddressSanitizer (ASan): Catches null dereferences at runtime: clang -fsanitize=address.
• UBSan: Catches undefined behavior: clang -fsanitize=undefined.
• Static analysis: Use clang --analyze or cppcheck as part of CI.

4. Audit all custom allocator call sites

When a codebase uses a custom allocator like gotcha_malloc, run a targeted search for all call sites and verify each one checks the return value:

grep -n "gotcha_malloc" src/*.c | grep -v "if ("

This quick grep can surface allocation calls that lack an immediate conditional check.

5. Reference standards

• CWE-476: NULL Pointer Dereference
• CERT C Coding Standard: MEM32-C — Detect and handle memory allocation errors
• OWASP: While OWASP focuses primarily on web vulnerabilities, the Memory Management Cheat Sheet covers allocation safety

Key Takeaways

• gotcha_malloc() can return NULL because it wraps mmap, which fails under memory pressure — never assume a custom allocator is infallible.
• add_library() in gotcha_utils.c dereferenced newlib at line 84 without any null check, making it one missed memory allocation away from crashing the host process.
• The fix is a single line: if (!newlib) return NULL; — proving that critical security and reliability bugs do not always require complex solutions.
• Multiple callers of gotcha_malloc may share this flaw — any codebase using a custom allocator should audit all call sites, not just the one that was reported.
• In function interposition libraries like GOTCHA, a crash in library tracking doesn't just cause downtime — it can silently disable security instrumentation, making this a security issue, not just a reliability one.

How Orbis AppSec Detected This

• Source: The gotcha_malloc(sizeof(library_t)) call in add_library() in src/gotcha_utils.c, which wraps mmap and can return NULL on allocation failure.
• Sink: The immediate dereference newlib->map = map; at line 84, which writes through the pointer without any null check.
• Missing control: No null/failure check on the return value of gotcha_malloc() before the pointer was used.
• CWE: CWE-476: NULL Pointer Dereference
• Fix: Added if (!newlib) return NULL; on the line immediately following the gotcha_malloc() call, preventing any dereference of a potentially null pointer.

Orbis AppSec automatically detected this vulnerability and opened a pull request with the fix. Try Orbis AppSec on your repositories to find and fix issues like this automatically.

Conclusion

The NULL pointer dereference in add_library() is a textbook example of a vulnerability that is trivial to fix but easy to miss in code review — especially when the allocator is a custom wrapper whose failure semantics aren't immediately obvious to every contributor. A single missing if (!newlib) return NULL; was all it took to turn a memory allocation failure into a guaranteed crash.

For C developers, the lesson is clear: treat every allocation as a potential failure. Custom allocators built on mmap, memory pools, or arena allocators can all fail, and their callers must be written defensively. Static analysis, sanitizers, and automated security scanning are your safety nets for catching the cases that slip through code review.

Secure, reliable C code doesn't assume success — it handles failure at every step.

References

• CWE-476: NULL Pointer Dereference
• OWASP Memory Management Cheat Sheet
• CERT C Coding Standard: MEM32-C
• Semgrep rules for null dereference in C
• fix: multiple callers of gotcha_malloc do not check ... in...