How kernel stack buffer overflow happens in C vsprintf() and how to fix it

How kernel stack buffer overflow happens in C vsprintf() and how to fix it

Summary

A critical stack buffer overflow vulnerability was discovered in sys/kern/debug.c where the kernel's printf() function called a custom vsprintf() implementation without any length constraint on the output buffer db_msg. By replacing the unbounded vsprintf() call with a size-aware vsnprintf() implementation, the fix prevents crafted format strings or oversized arguments from overwriting kernel stack memory, closing a path to arbitrary kernel code execution.

Introduction

The sys/kern/debug.c file handles the kernel's own diagnostic output — the low-level printf() used by the operating system itself to emit debug messages. It's easy to overlook security in a debug subsystem; after all, who's formatting kernel log messages maliciously? But a flaw in how printf() delegated its formatting work to a custom vsprintf() in sys/lib/vsprintf.c created one of the most dangerous classes of vulnerability in systems programming: an unbounded write onto the kernel stack.

At line 98 of sys/kern/debug.c, the call was straightforward:

vsprintf(db_msg, fmt, args);

db_msg is a fixed-size buffer. vsprintf() had no idea how large it was. This is the classic setup for a stack buffer overflow — and in the kernel, the consequences are far worse than in userspace.

The Vulnerability Explained

What went wrong at sys/kern/debug.c:98

The custom vsprintf() implementation in sys/lib/vsprintf.c was written to iterate over a format string and write formatted output into a destination buffer buf:

int vsprintf(char* buf, const char* fmt, va_list args)
{
    char *p, *str;
    // ...
    for (p = buf; *fmt; fmt++) {
        if (*fmt != '%') {
            *p++ = *fmt;
            continue;
        }
        // ... format specifier handling
    }
}

Notice what's missing: there is no size parameter, no end pointer, and no bounds check on p. Every character written advances p unconditionally. The loop condition *fmt only stops when the format string is exhausted — not when the buffer is full.

At the call site in debug.c, db_msg is a fixed-size buffer on the kernel stack:

vsprintf(db_msg, fmt, args);

If the formatted output exceeds DBGMSGSZ bytes, vsprintf() will happily keep writing past the end of db_msg, overwriting whatever lives next on the kernel stack — which includes saved frame pointers, return addresses, and function pointers.

How this could be exploited

In a kernel context, overwriting a return address or function pointer with attacker-controlled data means arbitrary kernel code execution — the highest possible privilege level on the system. An attacker who can influence the format string arguments passed to the kernel printf() (for example, through a device driver interface, a syscall path that triggers debug output, or a crafted filesystem or network packet that causes a kernel log message with attacker-controlled content) could:

1. Pass a %s argument pointing to a very long string, causing the for (; *str && width != 0; str++) loop to write hundreds or thousands of bytes into db_msg.
2. Overflow db_msg on the kernel stack, overwriting the return address of printf().
3. Redirect execution to attacker-controlled shellcode or a ROP chain within the kernel address space.

The %s handler in the original code made this especially straightforward:

case 's':
    str = va_arg(args, char*);
    if (str == NULL)
        str = "<NULL>";
    for (; *str && width != 0; str++) {
        // writes to p++ with NO bounds check
    }

A string argument with no width specifier (width defaults to 0, which the loop treats as "no limit") and a length exceeding DBGMSGSZ would overflow the buffer entirely.

Real-world impact

This is a critical vulnerability because:
- It lives in the kernel itself, not a userspace application
- Exploitation leads to full system compromise (ring 0 / kernel privilege)
- The debug subsystem is often active in development and staging builds, and sometimes in production kernels
- The overflow is deterministic and reproducible, not probabilistic

The Fix

The fix required changes across three files: sys/kern/debug.c, sys/lib/vsprintf.c, and sys/include/libkern.h. Each change was necessary.

1. sys/kern/debug.c — Pass the buffer size

The call site was updated to use the new vsnprintf() function, passing DBGMSGSZ as the maximum number of bytes to write:

Before:

vsprintf(db_msg, fmt, args);

After:

vsnprintf(db_msg, DBGMSGSZ, fmt, args);

This single character change at the call site is the most visible part of the fix, but it only works because the underlying implementation was also corrected.

2. sys/lib/vsprintf.c — Implement size-bounded formatting

The custom vsprintf() was refactored into vsnprintf() with a size parameter. The key additions:

int vsnprintf(char* buf, size_t size, const char* fmt, va_list args)
{
    char *p, *end, *str;
    // ...

    if (size == 0)
        return 0;
    end = buf + size - 1;          // one byte reserved for null terminator
    for (p = buf; *fmt && p < end; fmt++) {

The end pointer is computed once and used as the hard boundary throughout. Every write path now checks p < end before advancing:

case 'c':
    if (p < end) *p++ = (char)va_arg(args, int);
    continue;
case 's':
    str = va_arg(args, char*);
    if (str == NULL)
        str = "<NULL>";
    for (; *str && width != 0 && p < end;   // <-- bound check added

The p < end guard appears in the main loop condition and in every format specifier handler. No matter how long the input, p can never advance past end, which is always at least one byte before the end of db_msg.

3. sys/include/libkern.h — Declare the new function

The header was updated to expose vsnprintf() to the rest of the kernel:

int vsprintf(char*, const char*, va_list);
+int vsnprintf(char*, size_t, const char*, va_list);

This ensures any other kernel code that needs bounded string formatting can use vsnprintf() rather than reaching for the unsafe vsprintf().

Before and after at a glance

Prevention & Best Practices

Never use vsprintf() or sprintf() in new code

The C standard itself acknowledges that sprintf() and vsprintf() are unsafe. Their replacements — snprintf() and vsnprintf() — have been standardized since C99. In kernel code where you're writing your own implementations, always include the size parameter from the start.

Treat fixed-size stack buffers as a red flag

Any time you see a pattern like:

char buf[SOME_SIZE];
vsprintf(buf, fmt, args);

treat it as a potential overflow. Ask: "What is the maximum possible output of this format call? Can it exceed SOME_SIZE?"

Apply the end pointer pattern consistently

The fix uses end = buf + size - 1 to compute a hard boundary once and check it everywhere. This is more reliable than tracking remaining capacity (size - (p - buf)) because it avoids integer arithmetic that can itself be a source of bugs.

Use static analysis to catch unsafe vsprintf() calls

Tools that can flag this pattern:
- Semgrep: Write a rule matching vsprintf($BUF, $FMT, $ARGS) where $BUF is a fixed-size array
- Clang Static Analyzer: The alpha.security.ArrayBound and security.insecureAPI.DeprecatedOrUnsafeBufferHandling checkers
- Coverity: Flags unbounded sprintf/vsprintf calls automatically
- GCC/Clang -Wformat-overflow: Catches some cases at compile time when buffer sizes are statically known

Relevant standards

• CWE-121: Stack-based Buffer Overflow — this vulnerability is a textbook example
• CWE-119: Improper Restriction of Operations within the Bounds of a Memory Buffer — the parent class
• OWASP: Buffer Overflow — general guidance
• CERT C Coding Standard: STR07-C — Use the bounds-checking interfaces for string manipulation

Key Takeaways

• vsprintf() in custom kernel implementations is inherently unsafe — the missing size parameter in sys/lib/vsprintf.c meant any caller could silently overflow their buffer.
• The db_msg buffer in sys/kern/debug.c was one oversized %s argument away from a kernel takeover — debug paths deserve the same security scrutiny as production code paths.
• The fix required changes at both the call site and the implementation — changing only debug.c without fixing vsprintf.c would have been incomplete; the unsafe function would remain available to other callers.
• Computing end = buf + size - 1 once and checking p < end everywhere is the correct pattern for bounded string formatting in C without standard library support.
• Adding vsnprintf() to libkern.h propagates the safe API — future kernel developers now have the right tool available and documented, making it less likely the unsafe vsprintf() gets used again.

How Orbis AppSec Detected This

• Source: Format string fmt and variadic args passed into printf() in sys/kern/debug.c, potentially carrying attacker-influenced content from kernel subsystems
• Sink: vsprintf(db_msg, fmt, args) at sys/kern/debug.c:98, calling the custom unbounded implementation in sys/lib/vsprintf.c:68
• Missing control: No length parameter passed to vsprintf(); the custom implementation had no size argument, no end pointer, and no bounds check on any write to the output buffer
• CWE: CWE-121 — Stack-based Buffer Overflow
• Fix: Introduced vsnprintf() with a size_t size parameter and end = buf + size - 1 boundary enforcement, replacing the unsafe vsprintf() call at the kernel printf() site

Orbis AppSec automatically detected this vulnerability and opened a pull request with the fix. Try Orbis AppSec on your repositories to find and fix issues like this automatically.

Conclusion

A single missing parameter — the buffer size — turned the kernel's own debug output function into a potential root exploit. The vsprintf(db_msg, fmt, args) call at sys/kern/debug.c:98 trusted that formatted output would never exceed the size of db_msg, a trust that was never enforced in code. The fix is elegant in its simplicity: introduce vsnprintf(), compute a hard end boundary, and check p < end at every write. Three files changed, one class of vulnerability eliminated.

The lesson for systems programmers is enduring: in C, the buffer knows its size, but the function doesn't — unless you tell it. Always use the n-bounded variants of string functions, always pass the correct size, and always verify that custom implementations honor that bound throughout their entire execution path.

References

• CWE-121: Stack-based Buffer Overflow
• CWE-119: Improper Restriction of Operations within the Bounds of a Memory Buffer
• OWASP Buffer Overflow
• OWASP Input Validation Cheat Sheet
• C11 Standard — vsnprintf specification (cppreference)
• Semgrep rules — vsprintf unsafe usage
• fix: remove unsafe exec() in debug.c