Back to Blog
medium SEVERITY8 min read

How buffer overflow happens in C kernel PTY subsystem (tty_ptmx.c) and how to fix it

A stack buffer overflow vulnerability was discovered in `tty_ptmx.c`, the kernel-level pseudo-terminal multiplexer component, where an unchecked `sprintf()` call at line 293 could overflow the `device_name` buffer by combining `root_path` and `dev_rel_path` without bounds validation. Because this code executes in kernel context during PTY device creation, successful exploitation could lead to kernel memory corruption, privilege escalation, or system crashes. The fix replaces the unbounded `sprin

O
By Orbis AppSec
•Published June 16, 2026•Reviewed June 16, 2026
#buffer-overflow#C#kernel-security#sprintf#snprintf#CWE-120#embedded-systems#PTY

Answer Summary

This is a stack buffer overflow vulnerability (CWE-120) in the C kernel PTY subsystem (`components/lwp/terminal/tty_ptmx.c`), where `sprintf(device_name, "%s%s", root_path, dev_rel_path)` writes a formatted string into a fixed-size buffer without any length check. If the combined length of `root_path` and `dev_rel_path` exceeds the buffer, adjacent stack memory is overwritten. The fix replaces `sprintf()` with `snprintf(device_name, root_len + sizeof("/ptmx"), "%s%s", root_path, dev_rel_path)`, explicitly capping the write to the known buffer size and preventing overflow.

Vulnerability at a Glance

cweCWE-120
fixReplaced sprintf() with snprintf() using a precisely calculated size limit (root_len + sizeof("/ptmx"))
riskKernel memory corruption, privilege escalation, or system crash
languageC (kernel/embedded)
root causesprintf() used without length limit when concatenating two path strings into a fixed-size buffer
vulnerabilityStack Buffer Overflow via unbounded sprintf()

How buffer overflow happens in C kernel PTY subsystem (tty_ptmx.c) and how to fix it

Summary

A stack buffer overflow vulnerability was discovered in tty_ptmx.c, the kernel-level pseudo-terminal multiplexer component, where an unchecked sprintf() call at line 293 could overflow the device_name buffer by combining root_path and dev_rel_path without bounds validation. Because this code executes in kernel context during PTY device creation, successful exploitation could lead to kernel memory corruption, privilege escalation, or system crashes. The fix replaces the unbounded sprintf() with a properly bounded snprintf() call that explicitly limits output to the allocated buffer size.

Introduction

The components/lwp/terminal/tty_ptmx.c file is responsible for initializing pseudo-terminal multiplexer (PTY) devices in the RT-Thread Smart kernel. It handles the creation and registration of /dev/ptmx-style devices — the entry point for every terminal session spawned by the system. A flaw in the lwp_ptmx_init() function, specifically the sprintf() call at line 293, created a classic but dangerous stack buffer overflow condition.

The vulnerable line looks innocent at first glance:

sprintf(device_name, "%s%s", root_path, dev_rel_path);

But device_name is a fixed-size stack buffer, and neither root_path nor dev_rel_path are validated for length before this call. If an attacker or a misconfigured caller supplies path components whose combined length exceeds the buffer, sprintf() will happily write past the end of device_name and into adjacent kernel stack memory.

This matters enormously because lwp_ptmx_init() runs in kernel context. There is no userspace sandbox to contain the damage.

The Vulnerability Explained

What's happening at line 293

Inside lwp_ptmx_init(), the code allocates a buffer for the device name and then formats it using sprintf():

// VULNERABLE CODE (before fix) — tty_ptmx.c line 293
if (device_name)
{
    /* Register device */
    sprintf(device_name, "%s%s", root_path, dev_rel_path);
    rt_device_register(ptmx_device, device_name, 0);
    ...
}

The device_name buffer has a known, bounded size — it's allocated based on root_len + sizeof("/ptmx"). However, sprintf() is completely unaware of that size. It writes characters until the format string is exhausted, regardless of how much space remains. If root_path or dev_rel_path are longer than expected (due to a bug, misconfiguration, or deliberate manipulation), the write overflows the buffer.

Why this is especially dangerous in kernel context

In userspace, a stack buffer overflow typically corrupts the local stack frame and might be mitigated by ASLR, stack canaries, or NX bits. In kernel context, the stakes are higher:

  • Overwriting the kernel stack can corrupt return addresses, redirecting execution to attacker-controlled code.
  • It can corrupt adjacent kernel data structures, leading to privilege escalation.
  • Even without code execution, it can cause a kernel panic, taking down the entire system.

The companion issue at line 321

The PR also flagged a second unsafe string operation at line 321:

// VULNERABLE CODE (before fix) — tty_ptmx.c line 321
strncpy(buf, "pts/ptmx", len);

While strncpy() does accept a length argument, it has a subtle hazard: it does not guarantee null-termination if the source string exactly fills the destination buffer. Using snprintf() here is cleaner and unambiguously safe.

Attack scenario

Consider a scenario where root_path is supplied through a configuration interface or a mount namespace operation that allows longer-than-expected path strings. An attacker with the ability to influence root_path — for example, through a crafted filesystem mount point — could supply a string like:

/dev/pts/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa...

Combined with dev_rel_path, the total length exceeds the device_name buffer. The sprintf() call writes beyond the buffer, overwriting the kernel stack. Depending on what lies adjacent in memory, this could overwrite a saved return address, enabling kernel-level code execution.

The Fix

Change 1: Replace sprintf() with snprintf() at line 293

The fix is precise and surgical:

// BEFORE (vulnerable)
sprintf(device_name, "%s%s", root_path, dev_rel_path);

// AFTER (fixed)
snprintf(device_name, root_len + sizeof("/ptmx"), "%s%s", root_path, dev_rel_path);

The second argument to snprintf() — root_len + sizeof("/ptmx") — is exactly the size of the allocated device_name buffer. This means:

  • snprintf() will write at most root_len + sizeof("/ptmx") - 1 characters.
  • The output is always null-terminated.
  • Any excess input is silently truncated rather than overflowing the buffer.

This is the ideal fix because the size argument directly mirrors the allocation size, leaving no gap between what was allocated and what is written.

Change 2: Replace strncpy() with snprintf() at line 321

// BEFORE
strncpy(buf, "pts/ptmx", len);

// AFTER
snprintf(buf, len, "pts/ptmx");

snprintf() is strictly safer here: it always null-terminates, and the intent of the code (write a bounded string into a caller-supplied buffer) is more clearly expressed. The strncpy() function's non-termination behavior when len == strlen("pts/ptmx") is a known footgun that snprintf() eliminates entirely.

Change 3: Regression test infrastructure

The PR also adds a dedicated Kconfig option (RT_UTEST_LWP_TTY_PTMX) and a test case file (tty_ptmx_tc.c) to the utest framework:

config RT_UTEST_LWP_TTY_PTMX
    bool "Enable Utest for tty_ptmx buffer overflow regression (V-004)"
    depends on RT_USING_SMART
    default n

This ensures that future changes to tty_ptmx.c can be validated against a regression suite that explicitly tests buffer boundary conditions — a critical addition for kernel-level code where subtle regressions can be catastrophic.

Key Takeaways

  • sprintf() in lwp_ptmx_init() had no idea how large device_name was — the fix ties the write limit directly to the allocation size using root_len + sizeof("/ptmx").
  • Kernel-context overflows are categorically more dangerous than userspace ones — there is no process isolation or OS-level containment when the kernel stack is corrupted.
  • strncpy() is not a safe replacement for sprintf() — it doesn't guarantee null-termination; snprintf() does.
  • Regression tests belong in the build system — the new RT_UTEST_LWP_TTY_PTMX Kconfig entry ensures this specific overflow scenario is permanently guarded against future regressions.
  • The size expression used in rt_malloc() should be reused verbatim in the corresponding snprintf() call — this makes allocation/write pairs easy to audit and keeps the invariant obvious.

How Orbis AppSec Detected This

  • Source: The root_path parameter passed into lwp_ptmx_init(), which can be influenced by mount namespace configuration or device initialization paths.
  • Sink: sprintf(device_name, "%s%s", root_path, dev_rel_path) at components/lwp/terminal/tty_ptmx.c:293 — an unbounded write into a stack-allocated kernel buffer.
  • Missing control: No length check or bounded write function was used; sprintf() was called with two variable-length string arguments and a fixed-size destination with no size argument.
  • CWE: CWE-120 — Buffer Copy without Checking Size of Input ("Classic Buffer Overflow").
  • Fix: Replaced sprintf() with snprintf(device_name, root_len + sizeof("/ptmx"), "%s%s", root_path, dev_rel_path), capping the write to the exact allocation size.

Orbis AppSec automatically detected this vulnerability and opened a pull request with the fix. Try Orbis AppSec on your repositories to find and fix issues like this automatically.

Conclusion

The sprintf() → snprintf() change in tty_ptmx.c is a small diff with significant security consequences. In kernel-level C code, the difference between a bounded and unbounded string write is the difference between a safe device registration and a potential kernel compromise. This vulnerability is a reminder that even mature, well-reviewed systems code can harbor classic C pitfalls — and that automated static analysis is essential for catching them at scale.

The key principle to take away: every string write in C must be paired with an explicit size limit, and that limit must match the allocation. When working in kernel context, there is no safety net below you.

Prevention and further reading

View the Security Fix

Check out the pull request that fixed this vulnerability

View PR #11447
View Repository →

Related Articles

critical

How buffer overflow happens in C++ and how to fix it

A critical buffer overflow in `create_hex_string()` within `hmlangw.cpp` let an unconditional 16-iteration loop write past the bounds of a 100-byte `hex` buffer using unchecked `sprintf` calls. The fix replaces `sprintf` with `snprintf` and caps the loop iterations based on the actual destination buffer size, closing off a memory corruption path reachable from serial or network input.

critical

How Buffer Overflow via strcpy() Happens in C++ XML Parsers and How to Fix It

A critical buffer overflow vulnerability was discovered in `buildroot-external/package/libxmlparser/xmlParser.cpp`, where the `toXMLString` function used `_tcscpy()` to write XML escape sequences into a destination buffer without any bounds checking. An attacker supplying a crafted XML document could overflow the buffer and potentially execute arbitrary code. The fix replaces all five unsafe `_tcscpy()` calls with `memcpy()` calls that copy only the exact number of bytes required for each escape

critical

How Heap Buffer Overflows Happen in C++ ZIP Extraction and How to Fix Them

A critical heap buffer overflow vulnerability was discovered in `TKLiveSync/unzip.cpp`, where ZIP archive entry names were copied into a `PATH_MAX`-sized heap buffer using `strcpy()` without any length validation. Since the ZIP specification allows entry names up to 65,535 bytes — far exceeding typical `PATH_MAX` values of 1,024 to 4,096 bytes — a crafted archive could overflow the buffer and corrupt heap memory. The fix replaces the unsafe `strcpy`/`dirname` pattern with `std::string` operation

medium

How Integer Overflow happens in C++ image processing and how to fix it

A signed integer overflow in OpenCV's `bilateralFilter.cpp` allowed the buffer size calculation `cal_width * cal_height * cn` to wrap around to a small or negative value, causing `padding.resize()` to allocate far less memory than needed. Subsequent `memcpy` operations would then write beyond the allocated buffer, creating a heap corruption primitive. The fix is a single targeted cast to `size_t` that promotes the multiplication to unsigned 64-bit arithmetic before any overflow can occur.

critical

How Stack Buffer Overflows Happen in C with sprintf() and How to Fix Them

A critical stack buffer overflow was discovered in `libuv/Learn-libuv/docs/code/tty-gravity/main.c` where `sprintf()` wrote ANSI escape sequences and user-controlled variables into a fixed 500-byte buffer without any bounds checking. An attacker controlling the `pos`, `width`, or `message` variables could overflow the stack, overwrite return addresses, and potentially achieve arbitrary code execution. The fix replaces `sprintf()` with `snprintf()` and adds explicit length validation to ensure wr

high

How insecure string copy functions happen in C and how to fix them

A high-severity buffer overflow risk was discovered in `login/main.c` where `strcpy()` was used to copy the `HOME` environment variable into a fixed-size 512-byte buffer without any bounds checking. An attacker controlling the `HOME` environment variable could overflow `pwd_file_name`, potentially corrupting memory or hijacking execution. The fix replaces the two-step `strcpy`/`strcat` pattern with a single, bounds-safe `snprintf` call.