Buffer Overflow in RS-232 Serial Input: How a Missing Length Check Put Embedded Systems at Risk

Introduction

In the world of embedded systems and low-level OS development, a single missing bounds check can be the difference between a stable system and a fully compromised one. This post examines a critical buffer overflow vulnerability discovered in archive/yx/os/ram/serial.c — specifically in how the system handles RS-232 serial input — and walks through exactly how it was fixed.

Buffer overflows are among the oldest and most well-understood vulnerability classes in software security, yet they continue to appear in production code, especially in systems-level C programming where manual memory management is the norm. Understanding how they arise — and how to prevent them — is essential knowledge for any developer working close to the metal.

The Vulnerability Explained

What Happened?

The vulnerable code lives in the rs232_getb function, which reads a byte from a serial input buffer. When the buffer is empty, it refills it by calling rs232_buffered_input(rs232_ibuff):

// BEFORE (vulnerable)
if (rs232_ib_beg == rs232_ib_end) {
    rs232_ib_beg = 0;
    rs232_ib_end = rs232_buffered_input(rs232_ibuff);
}

The problem is deceptively simple: rs232_buffered_input accepts no maximum size parameter. It reads serial data into whatever buffer you hand it, but the caller has no way to tell it how large that buffer actually is.

The buffer rs232_ibuff has a fixed size — RS232_IBUFF_SIZE bytes. But if the serial port delivers more bytes than that, rs232_buffered_input will happily keep writing, marching right past the end of rs232_ibuff and into adjacent memory.

Technical Breakdown

This is a classic stack or heap buffer overflow (CWE-121 / CWE-122, depending on where rs232_ibuff lives). Here's the chain of events:

1. An attacker sends a carefully crafted stream of bytes over the RS-232 serial connection.
2. rs232_buffered_input reads the incoming data into rs232_ibuff with no length enforcement.
3. Once the buffer is full, writes continue into adjacent memory — potentially overwriting:
   - Return addresses on the stack (enabling code execution hijacking)
   - OS data structures (corrupting system state)
   - Other variables in memory (causing unpredictable behavior)
4. The return value rs232_ib_end reflects the total bytes read — a number larger than RS232_IBUFF_SIZE — which then drives further out-of-bounds reads downstream.

Why Is This Particularly Dangerous?

In embedded and OS-level code, memory protections that desktop developers take for granted (ASLR, stack canaries, NX bits) are often absent or limited. This means a buffer overflow in this context can be significantly easier to exploit reliably.

Physical access to a serial port might sound like a high bar, but consider:

• Industrial control systems where RS-232 is a standard interface
• Maintenance ports left accessible on deployed hardware
• Networked serial adapters that expose the port remotely
• Malicious peripherals or compromised upstream devices

Attack Scenario

An attacker connects to an exposed RS-232 maintenance port on an embedded device. They craft a serial payload that exceeds RS232_IBUFF_SIZE bytes — say, 512 bytes when the buffer only holds 256. The overflow overwrites the return address of the calling function with an address pointing to attacker-controlled data. On the next function return, execution jumps to the attacker's shellcode, giving them full control of the device.

The Fix

The fix is elegant in its simplicity — a two-line bounds check added immediately after rs232_buffered_input returns:

// AFTER (fixed)
if (rs232_ib_beg == rs232_ib_end) {
    rs232_ib_beg = 0;
    rs232_ib_end = rs232_buffered_input(rs232_ibuff);
    if (rs232_ib_end > RS232_IBUFF_SIZE)       // ← NEW
        rs232_ib_end = RS232_IBUFF_SIZE;        // ← NEW
}

How Does This Solve the Problem?

The check clamps rs232_ib_end to RS232_IBUFF_SIZE after the fact. Let's be precise about what this achieves and what it doesn't:

What it does:
- Prevents rs232_ib_end from reflecting an out-of-bounds value, stopping downstream code from reading past the buffer boundary.
- Limits the visible effect of any overflow — excess bytes are silently discarded from the index perspective.

Important nuance:
This fix is a defensive clamp at the consumer side. The underlying rs232_buffered_input function may still write beyond the buffer if it truly has no length awareness — the fix prevents the consequences from propagating, but the ideal long-term solution (discussed below) is to fix rs232_buffered_input itself to accept a maximum size parameter.

Think of it like this: the fix puts a fence at the edge of the cliff. The deeper fix is to move the road away from the cliff entirely.

Before and After

Prevention & Best Practices

1. Always Pass Buffer Sizes to Input Functions

The root cause here is an API design flaw: rs232_buffered_input should never have been written to accept a buffer without a corresponding size. The correct signature should be:

// Unsafe — no size limit
int rs232_buffered_input(byte *buf);

// Safe — size-bounded
int rs232_buffered_input(byte *buf, size_t max_len);

This is the same principle behind why strncpy exists alongside strcpy, and why fgets takes a size parameter while gets (now removed from C11) did not.

2. Use sizeof at the Call Site

When you do have size-aware functions, always use sizeof rather than hardcoded numbers:

// Fragile — hardcoded size can drift from actual buffer size
rs232_ib_end = rs232_buffered_input(rs232_ibuff, 256);

// Robust — always matches the actual buffer
rs232_ib_end = rs232_buffered_input(rs232_ibuff, sizeof(rs232_ibuff));

3. Validate All Return Values from I/O Functions

Any function that reads external data and returns a count should have its return value validated against known bounds before that count is used to index memory.

size_t bytes_read = read_input(buffer, sizeof(buffer));
if (bytes_read > sizeof(buffer)) {
    // This shouldn't happen with a correct implementation,
    // but defensive programming catches bugs in dependencies
    bytes_read = sizeof(buffer);
    log_error("Input truncated: possible overflow in read_input()");
}

4. Consider Safer Alternatives

For new code, consider:

• Rust for embedded development where memory safety is non-negotiable
• C++ with std::array or std::vector which carry their own size information
• Static analysis tools like Coverity, CodeQL, or clang-tidy to catch unbounded writes at compile time
• AddressSanitizer (ASan) during development and testing to catch overflows at runtime

5. Threat Model Your Physical Interfaces

RS-232, JTAG, I2C, SPI — physical interfaces are often treated as "trusted" because they require physical access. But in deployed systems, this assumption can fail. Apply the same input validation to serial data as you would to network data.

Relevant Standards and References

• CWE-121: Stack-based Buffer Overflow
• CWE-122: Heap-based Buffer Overflow
• CWE-119: Improper Restriction of Operations within the Bounds of a Memory Buffer
• OWASP: Buffer Overflow
• SEI CERT C Coding Standard: ARR38-C — Guarantee that library functions do not form invalid pointers
• MISRA C:2012: Rule 1.3 — There shall be no occurrence of undefined or critical unspecified behaviour

Conclusion

This vulnerability is a textbook example of how API design decisions made early in development can create security problems that are hard to fix later. The rs232_buffered_input function was structurally incapable of enforcing a read limit — not because of a bug in its logic, but because it was never given the information it needed to be safe.

The immediate fix — clamping rs232_ib_end after the fact — is a solid defensive measure that stops the vulnerability from being exploited through normal code paths. The deeper lesson is to design I/O APIs with size parameters from the start, validate all externally-derived counts before using them as indices, and treat every external interface, including physical ones like RS-232, as a potential attack surface.

Key takeaways:
- 🔴 Never write an input function that accepts a buffer but not its size
- 🟡 Always clamp or validate return values from I/O operations before using them as indices
- 🟢 Use sizeof at call sites to keep size arguments in sync with actual buffer sizes
- 🟢 Apply static analysis and sanitizers to catch these issues before they reach production

Security in embedded and systems code isn't just about firewalls and encryption — it's about the discipline of writing every function as if the data it receives is adversarial. Because sometimes, it is.

This vulnerability was identified and fixed by automated security scanning. Automated tools are a force multiplier for security — but they work best when developers understand the underlying principles well enough to write safe code in the first place.