How buffer overflow happens in C netlink attribute handling and how to fix it

How Buffer Overflow Happens in C Netlink Attribute Handling and How to Fix It

Introduction

The libopencas/libopencas.c file implements low-level netlink socket communication, including the nl_resolve_family() function that resolves a Generic Netlink family name to its numeric ID. This is foundational kernel-userspace IPC code — the kind of code where a single missing bounds check can silently corrupt memory and destabilize the entire system.

That's exactly what happened here. At line 170, a memcpy() call copied a caller-supplied name string into a netlink attribute (NLA) buffer with no validation that the string actually fits. The result: an attacker or a misbehaving caller providing a name longer than the available buffer space would overflow into adjacent memory.

This post walks through exactly how the vulnerability works, what the fix does, and how to write safe netlink attribute code going forward.

The Vulnerability Explained

The Vulnerable Code Pattern

Inside nl_resolve_family(), the code constructs a Netlink message to query the kernel's Generic Netlink controller. Here's the relevant section before the fix:

nla = (struct nlattr *)((char *)buf + nlh->nlmsg_len);
nla->nla_type = CTRL_ATTR_FAMILY_NAME;
nla->nla_len = NLA_HDRLEN + strlen(name) + 1;
memcpy(nla_data(nla), name, strlen(name) + 1);  // ← vulnerable line 170

The problem is on the last line. memcpy() copies strlen(name) + 1 bytes (the string plus its null terminator) into the NLA data region. The size of the copy is derived entirely from the name parameter — but there is no check that this size fits within the space remaining in buf after the existing message headers.

How the Buffer Layout Works

To understand the overflow, picture the buffer layout:

[ buf ]
|-- nlmsghdr (nlh->nlmsg_len bytes) --|-- genlmsghdr --|-- NLA header --|-- NLA data --|-- ??? -->
                                                                          ^
                                                                    nla_data(nla)
                                                                    memcpy writes here

The available space for NLA data is:

sizeof(buf) - nlh->nlmsg_len - NLA_HDRLEN

If strlen(name) + 1 exceeds this value, memcpy() writes past the end of buf, corrupting whatever memory follows it — potentially other stack variables, return addresses, or kernel-visible data structures.

What an Attacker Can Do

If name is derived from any external or untrusted source — a configuration file, a network packet, a user-supplied argument — an attacker can craft a name string longer than the remaining buffer space. The memcpy() will dutifully write every byte of it, overflowing into adjacent memory.

Depending on the memory layout and the privilege context in which this code runs:

• Stack corruption: Overwriting a return address enables control-flow hijacking.
• Heap corruption: Overwriting heap metadata can lead to arbitrary write primitives.
• Denial of service: Even without code execution, corrupted kernel-userspace message structures can crash the process or cause undefined behavior in subsequent netlink operations.

In a library like libopencas that may be called from privileged system services, the impact of exploitation is amplified significantly.

The Fix

The fix is concise and surgical — exactly two lines added immediately before the memcpy():

Before (Vulnerable)

nla = (struct nlattr *)((char *)buf + nlh->nlmsg_len);
nla->nla_type = CTRL_ATTR_FAMILY_NAME;
nla->nla_len = NLA_HDRLEN + strlen(name) + 1;
memcpy(nla_data(nla), name, strlen(name) + 1);

After (Fixed)

nla = (struct nlattr *)((char *)buf + nlh->nlmsg_len);
if (strlen(name) + 1 > sizeof(buf) - nlh->nlmsg_len - NLA_HDRLEN)
    return -EINVAL;
nla->nla_type = CTRL_ATTR_FAMILY_NAME;
nla->nla_len = NLA_HDRLEN + strlen(name) + 1;
memcpy(nla_data(nla), name, strlen(name) + 1);

Why This Fix Works

The guard expression precisely computes the available space in the buffer for NLA data:

sizeof(buf)           // total buffer size
- nlh->nlmsg_len      // bytes already consumed by the netlink message header
- NLA_HDRLEN          // bytes consumed by the netlink attribute header itself

If the name (including its null terminator) would exceed that space, the function returns -EINVAL immediately — before any write occurs. The memcpy() is only reached when the copy is guaranteed to be safe.

This approach follows the fail-fast, fail-explicitly principle: invalid input is rejected at the earliest possible point with a meaningful error code, rather than silently proceeding into undefined behavior.

Prevention & Best Practices

1. Always Validate Before Copying Into Fixed Buffers

In C, any time you use memcpy(), strcpy(), or similar functions with a destination of known size, the copy length must be validated first. The pattern is:

size_t available = sizeof(dest_buf) - offset;
size_t needed    = strlen(src) + 1;

if (needed > available)
    return -EINVAL;  // or handle the error appropriately

memcpy(dest_buf + offset, src, needed);

Never trust that a caller-supplied string is short enough to fit.

2. Prefer Safe Alternatives Where Possible

For simple string copies, strlcpy() (BSD/glibc extension) or snprintf() can truncate safely. However, for netlink attribute construction, truncation is often semantically wrong — a truncated family name won't match the kernel's registered name. Explicit rejection is better than silent truncation in protocol code.

3. Use Compiler and OS Mitigations as Defense-in-Depth

Enable stack protection (-fstack-protector-strong), address space layout randomization (ASLR), and fortify source (-D_FORTIFY_SOURCE=2) in your build flags. These won't prevent the overflow, but they make exploitation significantly harder.

4. Apply Static Analysis to Netlink and IPC Code

Netlink message construction is a well-known source of buffer overflows in Linux systems programming. Run static analysis tools as part of CI:

• Semgrep: Rules for unsafe memcpy patterns
• Coverity / CodeChecker: Taint analysis from input to unsafe copy
• AddressSanitizer (ASan): Runtime detection during testing

5. Reference Standards

• CWE-120: Buffer Copy without Checking Size of Input
• CWE-787: Out-of-bounds Write
• OWASP: Buffer Overflow Prevention Cheat Sheet

Key Takeaways

• The memcpy() at line 170 of libopencas.c used strlen(name) + 1 as the copy size without ever checking whether that many bytes were available in the destination buffer — a textbook CWE-120.
• The available space in an NLA buffer is not just sizeof(buf) — it's sizeof(buf) - nlh->nlmsg_len - NLA_HDRLEN, and failing to account for the already-consumed header bytes is exactly how this class of bug arises.
• Returning -EINVAL on oversized input is the correct behavior for kernel-facing protocol code; truncation would produce a silently broken netlink message.
• Two lines of validation code — a single if statement — were sufficient to close this vulnerability entirely.
• Netlink attribute construction code deserves the same scrutiny as network packet parsing because it sits at the userspace-kernel boundary and is often called with partially-trusted input.

How Orbis AppSec Detected This

• Source: The name parameter of nl_resolve_family(), which is caller-supplied and not length-validated before use.
• Sink: memcpy(nla_data(nla), name, strlen(name) + 1) at libopencas/libopencas.c:170, where the copy size is derived directly from the unsanitized input.
• Missing control: No bounds check comparing strlen(name) + 1 against the remaining buffer space (sizeof(buf) - nlh->nlmsg_len - NLA_HDRLEN) before the copy.
• CWE: CWE-120 — Buffer Copy without Checking Size of Input ("Classic Buffer Overflow").
• Fix: Added a two-line guard that computes available NLA data space and returns -EINVAL if the name would overflow it, placed immediately before the memcpy() call.

Orbis AppSec automatically detected this vulnerability and opened a pull request with the fix. Try Orbis AppSec on your repositories to find and fix issues like this automatically.

Conclusion

Buffer overflows in C remain one of the most consequential vulnerability classes in systems programming — not because they're exotic, but because they're easy to miss in low-level protocol code where developers are focused on correctness of the message structure rather than bounds of the buffer. The nl_resolve_family() bug in libopencas.c is a perfect example: the nla_len field was set correctly, the nla_type was set correctly, but the actual copy operation was never guarded.

The lesson is straightforward: every memcpy() call that writes into a fixed-size buffer must be preceded by a check that the write fits. In netlink code, that means accounting for all the headers that have already consumed space in the buffer, not just the nominal buffer size.

If you're writing or reviewing Linux netlink code, treat every string-to-NLA-buffer copy as a potential overflow site until proven otherwise.

References

• CWE-120: Buffer Copy without Checking Size of Input
• CWE-787: Out-of-bounds Write
• OWASP C-Based Toolchain Hardening Cheat Sheet
• Linux Kernel Netlink Documentation
• Semgrep rules for unsafe memcpy patterns
• fix: at line 170, a name string is copied into a net... in libopencas.c