How path traversal happens in Python file handling and how to fix it

How Path Traversal Happens in Python File Handling and How to Fix It

The Specific Incident

The scripts/merge_m3u.py file is responsible for discovering and merging custom M3U playlist files from a custom/ directory. Inside the find_and_sort_custom_files() function, a glob.glob() call collected all files matching the pattern custom/custom*.m3u. Those file paths were then passed directly to open() — without ever verifying that the resolved, real path of each file actually lived inside the custom/ directory.

Semgrep flagged this at line 84 under the rule utils.custom.path-traversal-open: user-controlled input used in a file path for open() without sanitization. The fix, introduced in this PR, adds a os.path.realpath() boundary check that filters out any path resolving outside the intended directory.

The Vulnerability Explained

What the Vulnerable Code Looked Like

Before the fix, find_and_sort_custom_files() looked like this:

def find_and_sort_custom_files():
    # 使用 glob 查找所有匹配的文件
    pattern = os.path.join(custom_dir, 'custom*.m3u')
    custom_files = glob.glob(pattern)

    # 对文件列表进行自然排序
    custom_files.sort(key=natural_sort_key)
    # ... files later passed to open()

The glob pattern custom/custom*.m3u looks safe at first glance — it matches only filenames beginning with custom and ending in .m3u inside the custom/ directory. However, it says nothing about where those files actually point on disk.

The Attack Surface: Symbolic Links

Here is the subtle danger. If an attacker can place a symbolic link inside the custom/ directory — for example, through a misconfigured upload endpoint, a shared filesystem, or another vulnerability in the application — they can create a file that matches the glob pattern but resolves to an entirely different location:

# Attacker creates a symlink that matches the glob pattern
ln -s /etc/passwd custom/custom_evil.m3u

Now glob.glob('custom/custom*.m3u') happily returns custom/custom_evil.m3u. The code then opens it, reads it, and processes /etc/passwd as if it were a playlist file. The glob pattern was matched, but the boundary was never enforced.

Why This Is a Real Risk for This Application

M3U playlist merging scripts are often run in environments where:

• The custom/ directory is writable by a service account or even a web-facing upload feature.
• The script runs with elevated privileges (e.g., as part of a media server's automation pipeline).
• Output from the merge is served to users or logged, potentially leaking the contents of the traversed file.

In this context, an attacker who can write a symlink into custom/ can exfiltrate any file readable by the process — /etc/shadow, private keys, application secrets, or database configuration files.

The Fix

What Changed

The fix adds a real-path boundary check immediately after the glob call, inside find_and_sort_custom_files():

Before:

pattern = os.path.join(custom_dir, 'custom*.m3u')
custom_files = glob.glob(pattern)

# 对文件列表进行自然排序
custom_files.sort(key=natural_sort_key)

After:

pattern = os.path.join(custom_dir, 'custom*.m3u')
custom_files = glob.glob(pattern)

# 防止路径穿越：过滤掉解析到 custom 目录之外的文件（如符号链接攻击）
custom_dir_abs = os.path.realpath(custom_dir)
custom_files = [
    f for f in custom_files
    if os.path.realpath(f).startswith(custom_dir_abs + os.sep)
]

# 对文件列表进行自然排序
custom_files.sort(key=natural_sort_key)

Why This Fix Works

The key is os.path.realpath(). Unlike os.path.abspath(), realpath() resolves symbolic links and returns the true canonical path on the filesystem. By:

1. Resolving custom_dir to its absolute real path (custom_dir_abs)
2. Resolving each glob result to its real path
3. Asserting that the resolved path starts with custom_dir_abs + os.sep

...the code ensures that even a symlink pointing to /etc/passwd will be filtered out, because os.path.realpath('custom/custom_evil.m3u') returns /etc/passwd, which does not start with /absolute/path/to/custom/.

The + os.sep Detail

Notice the fix appends os.sep (a / on Unix) to custom_dir_abs before the startswith() check. This prevents a subtle bypass: without it, a directory named custom_sibling/ would pass the check because its absolute path starts with the same characters as custom/. Appending the separator ensures only true children of the directory are accepted.

Prevention & Best Practices

Always Resolve Before You Restrict

Whenever you accept a file path — whether from user input, environment variables, a database, or even the filesystem itself via glob — resolve it to its canonical form before applying any boundary check:

import os

def safe_open(base_dir: str, user_path: str):
    base = os.path.realpath(base_dir)
    target = os.path.realpath(os.path.join(base_dir, user_path))
    if not target.startswith(base + os.sep):
        raise ValueError(f"Path traversal detected: {user_path!r}")
    return open(target)

Don't Trust Glob Results Blindly

glob.glob() returns whatever the filesystem reports. In environments where the directory can contain symlinks, treat glob results as untrusted input and validate them the same way you would validate user-supplied filenames.

Use pathlib for Cleaner Checks (Python 3.6+)

Python's pathlib module offers an expressive alternative:

from pathlib import Path

base = Path(custom_dir).resolve()
custom_files = [
    f for f in base.glob('custom*.m3u')
    if f.resolve().is_relative_to(base)
]

Path.is_relative_to() (Python 3.9+) handles the separator edge case automatically.

Relevant Security Standards

• OWASP: Path Traversal — recommends canonicalization before boundary checks.
• CWE-22: Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
• CWE-61: UNIX Symbolic Link Following — specifically covers the symlink attack vector exploited here.

Key Takeaways

• glob.glob() results are not safe by default — a matching filename does not mean the file lives where you expect it to. Symlinks can redirect any path to an arbitrary location.
• os.path.realpath() is the correct tool for resolving symlinks before boundary validation; os.path.abspath() alone is insufficient because it does not follow symlinks.
• Appending os.sep to the base path in a startswith() check is a small but critical detail that prevents sibling-directory bypass attacks.
• The custom/ directory boundary in merge_m3u.py was never enforced at the code level — only at the glob pattern level, which is not a security control.
• Exploit primitives matter even without a direct exploit path — a path traversal primitive in a media-processing script can be chained with a file-upload weakness or a race condition to achieve serious impact.

How Orbis AppSec Detected This

• Source: File paths returned by glob.glob(pattern) inside find_and_sort_custom_files(), where pattern is constructed from the custom_dir variable — a directory that may contain attacker-influenced symlinks.
• Sink: open() called on unsanitized glob results at scripts/merge_m3u.py:84, with no intervening path validation.
• Missing control: No call to os.path.realpath() or equivalent canonicalization; no assertion that the resolved path falls within custom_dir.
• CWE: CWE-22 — Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
• Fix: Glob results are now filtered through os.path.realpath() and compared against the resolved absolute path of custom_dir, discarding any entry that resolves outside the intended directory.

Orbis AppSec automatically detected this vulnerability and opened a pull request with the fix. Try Orbis AppSec on your repositories to find and fix issues like this automatically.

Conclusion

Path traversal vulnerabilities in file-handling scripts are easy to overlook precisely because the dangerous code looks so mundane — a glob pattern, a sort, an open. The merge_m3u.py case is a textbook example of how a pattern that appears safe (a restricted glob) can be undermined by filesystem features (symbolic links) that the code never considered.

The fix is equally straightforward: one call to os.path.realpath() and one list comprehension. The cost is negligible; the security improvement is significant. Whether you are writing a media automation script, a file upload handler, or a log processor, the principle is the same — resolve before you restrict, and never assume a matching filename means a safe file.

References

• CWE-22: Improper Limitation of a Pathname to a Restricted Directory
• CWE-61: UNIX Symbolic Link Following
• OWASP Path Traversal Attack
• OWASP File Upload Cheat Sheet
• Python os.path.realpath() documentation
• Python pathlib.Path.resolve() documentation
• Semgrep path traversal rules
• harden: add path validation in merge_m3u.py...