How buffer overflow happens in C kernel PTY subsystem (tty_ptmx.c) and how to fix it

Summary

A stack buffer overflow vulnerability was discovered in tty_ptmx.c, the kernel-level pseudo-terminal multiplexer component, where an unchecked sprintf() call at line 293 could overflow the device_name buffer by combining root_path and dev_rel_path without bounds validation. Because this code executes in kernel context during PTY device creation, successful exploitation could lead to kernel memory corruption, privilege escalation, or system crashes. The fix replaces the unbounded sprintf() with a properly bounded snprintf() call that explicitly limits output to the allocated buffer size.

Introduction

The components/lwp/terminal/tty_ptmx.c file is responsible for initializing pseudo-terminal multiplexer (PTY) devices in the RT-Thread Smart kernel. It handles the creation and registration of /dev/ptmx-style devices — the entry point for every terminal session spawned by the system. A flaw in the lwp_ptmx_init() function, specifically the sprintf() call at line 293, created a classic but dangerous stack buffer overflow condition.

The vulnerable line looks innocent at first glance:

sprintf(device_name, "%s%s", root_path, dev_rel_path);

But device_name is a fixed-size stack buffer, and neither root_path nor dev_rel_path are validated for length before this call. If an attacker or a misconfigured caller supplies path components whose combined length exceeds the buffer, sprintf() will happily write past the end of device_name and into adjacent kernel stack memory.

This matters enormously because lwp_ptmx_init() runs in kernel context. There is no userspace sandbox to contain the damage.

The Vulnerability Explained

What's happening at line 293

Inside lwp_ptmx_init(), the code allocates a buffer for the device name and then formats it using sprintf():

// VULNERABLE CODE (before fix) — tty_ptmx.c line 293
if (device_name)
{
    /* Register device */
    sprintf(device_name, "%s%s", root_path, dev_rel_path);
    rt_device_register(ptmx_device, device_name, 0);
    ...
}

The device_name buffer has a known, bounded size — it's allocated based on root_len + sizeof("/ptmx"). However, sprintf() is completely unaware of that size. It writes characters until the format string is exhausted, regardless of how much space remains. If root_path or dev_rel_path are longer than expected (due to a bug, misconfiguration, or deliberate manipulation), the write overflows the buffer.

Why this is especially dangerous in kernel context

In userspace, a stack buffer overflow typically corrupts the local stack frame and might be mitigated by ASLR, stack canaries, or NX bits. In kernel context, the stakes are higher:

• Overwriting the kernel stack can corrupt return addresses, redirecting execution to attacker-controlled code.
• It can corrupt adjacent kernel data structures, leading to privilege escalation.
• Even without code execution, it can cause a kernel panic, taking down the entire system.

The companion issue at line 321

The PR also flagged a second unsafe string operation at line 321:

// VULNERABLE CODE (before fix) — tty_ptmx.c line 321
strncpy(buf, "pts/ptmx", len);

While strncpy() does accept a length argument, it has a subtle hazard: it does not guarantee null-termination if the source string exactly fills the destination buffer. Using snprintf() here is cleaner and unambiguously safe.

Attack scenario

Consider a scenario where root_path is supplied through a configuration interface or a mount namespace operation that allows longer-than-expected path strings. An attacker with the ability to influence root_path — for example, through a crafted filesystem mount point — could supply a string like:

/dev/pts/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...

Combined with dev_rel_path, the total length exceeds the device_name buffer. The sprintf() call writes beyond the buffer, overwriting the kernel stack. Depending on what lies adjacent in memory, this could overwrite a saved return address, enabling kernel-level code execution.

The Fix

Change 1: Replace sprintf() with snprintf() at line 293

The fix is precise and surgical:

// BEFORE (vulnerable)
sprintf(device_name, "%s%s", root_path, dev_rel_path);

// AFTER (fixed)
snprintf(device_name, root_len + sizeof("/ptmx"), "%s%s", root_path, dev_rel_path);

The second argument to snprintf() — root_len + sizeof("/ptmx") — is exactly the size of the allocated device_name buffer. This means:

• snprintf() will write at most root_len + sizeof("/ptmx") - 1 characters.
• The output is always null-terminated.
• Any excess input is silently truncated rather than overflowing the buffer.

This is the ideal fix because the size argument directly mirrors the allocation size, leaving no gap between what was allocated and what is written.

Change 2: Replace strncpy() with snprintf() at line 321

// BEFORE
strncpy(buf, "pts/ptmx", len);

// AFTER
snprintf(buf, len, "pts/ptmx");

snprintf() is strictly safer here: it always null-terminates, and the intent of the code (write a bounded string into a caller-supplied buffer) is more clearly expressed. The strncpy() function's non-termination behavior when len == strlen("pts/ptmx") is a known footgun that snprintf() eliminates entirely.

Change 3: Regression test infrastructure

The PR also adds a dedicated Kconfig option (RT_UTEST_LWP_TTY_PTMX) and a test case file (tty_ptmx_tc.c) to the utest framework:

config RT_UTEST_LWP_TTY_PTMX
    bool "Enable Utest for tty_ptmx buffer overflow regression (V-004)"
    depends on RT_USING_SMART
    default n

This ensures that future changes to tty_ptmx.c can be validated against a regression suite that explicitly tests buffer boundary conditions — a critical addition for kernel-level code where subtle regressions can be catastrophic.

Prevention & Best Practices

1. Treat sprintf() as deprecated in C kernel code

Every use of sprintf() in kernel or systems code should be considered a code smell. Modern C development guidelines — including the Linux kernel's own coding standards — strongly prefer snprintf() for all string formatting operations. Many static analysis tools can be configured to flag sprintf() as an error.

2. Pair allocation size with write size

When you allocate a buffer with a calculated size (e.g., rt_malloc(root_len + sizeof("/ptmx"))), the corresponding write operation should use that exact same size expression as the length limit. This is what the fix does:

// Allocation and write use the same size expression — easy to audit
char *device_name = rt_malloc(root_len + sizeof("/ptmx"));
snprintf(device_name, root_len + sizeof("/ptmx"), "%s%s", root_path, dev_rel_path);

3. Validate path lengths at the entry point

While the snprintf() fix prevents the overflow, it silently truncates oversized input. For correctness, consider adding an explicit length check before the format call:

if (strlen(root_path) + strlen(dev_rel_path) >= root_len + sizeof("/ptmx")) {
    // Log an error and return early
    return -RT_EINVAL;
}

This makes the failure mode explicit and auditable rather than silent.

4. Use compiler and linker hardening

Enable stack canaries (-fstack-protector-strong), FORTIFY_SOURCE (-D_FORTIFY_SOURCE=2), and address sanitization (-fsanitize=address) during development and testing. These mechanisms can catch overflow conditions that slip past code review.

5. Apply consistent safe-string wrappers

Consider introducing a project-wide safe_snprintf() wrapper that asserts on truncation during debug builds:

static inline int safe_snprintf(char *buf, size_t size, const char *fmt, ...) {
    va_list args;
    va_start(args, fmt);
    int n = vsnprintf(buf, size, fmt, args);
    va_end(args);
    RT_ASSERT(n >= 0 && (size_t)n < size);  // Catch truncation in debug builds
    return n;
}

Relevant standards

• CWE-120: Buffer Copy without Checking Size of Input ("Classic Buffer Overflow")
• CWE-121: Stack-based Buffer Overflow
• OWASP: Buffer Overflow Prevention Cheat Sheet
• SEI CERT C: STR07-C — Use the bounds-checking interfaces for string manipulation

Key Takeaways

• sprintf() in lwp_ptmx_init() had no idea how large device_name was — the fix ties the write limit directly to the allocation size using root_len + sizeof("/ptmx").
• Kernel-context overflows are categorically more dangerous than userspace ones — there is no process isolation or OS-level containment when the kernel stack is corrupted.
• strncpy() is not a safe replacement for sprintf() — it doesn't guarantee null-termination; snprintf() does.
• Regression tests belong in the build system — the new RT_UTEST_LWP_TTY_PTMX Kconfig entry ensures this specific overflow scenario is permanently guarded against future regressions.
• The size expression used in rt_malloc() should be reused verbatim in the corresponding snprintf() call — this makes allocation/write pairs easy to audit and keeps the invariant obvious.

How Orbis AppSec Detected This

• Source: The root_path parameter passed into lwp_ptmx_init(), which can be influenced by mount namespace configuration or device initialization paths.
• Sink: sprintf(device_name, "%s%s", root_path, dev_rel_path) at components/lwp/terminal/tty_ptmx.c:293 — an unbounded write into a stack-allocated kernel buffer.
• Missing control: No length check or bounded write function was used; sprintf() was called with two variable-length string arguments and a fixed-size destination with no size argument.
• CWE: CWE-120 — Buffer Copy without Checking Size of Input ("Classic Buffer Overflow").
• Fix: Replaced sprintf() with snprintf(device_name, root_len + sizeof("/ptmx"), "%s%s", root_path, dev_rel_path), capping the write to the exact allocation size.

Orbis AppSec automatically detected this vulnerability and opened a pull request with the fix. Try Orbis AppSec on your repositories to find and fix issues like this automatically.

Conclusion

The sprintf() → snprintf() change in tty_ptmx.c is a small diff with significant security consequences. In kernel-level C code, the difference between a bounded and unbounded string write is the difference between a safe device registration and a potential kernel compromise. This vulnerability is a reminder that even mature, well-reviewed systems code can harbor classic C pitfalls — and that automated static analysis is essential for catching them at scale.

The key principle to take away: every string write in C must be paired with an explicit size limit, and that limit must match the allocation. When working in kernel context, there is no safety net below you.

References

• CWE-120: Buffer Copy without Checking Size of Input
• CWE-121: Stack-based Buffer Overflow
• OWASP C-Based Toolchain Hardening Cheat Sheet
• SEI CERT C Coding Standard: STR07-C
• snprintf() — Linux man page
• Semgrep rules for unsafe sprintf patterns
• fix: add buffer-length check in tty_ptmx.c