How URL-Encoded Path Traversal happens in Python nltk.data.load() and how to fix it

How URL-Encoded Path Traversal happens in Python nltk.data.load() and how to fix it

Introduction

The uv.lock file in this project locked NLTK at version 3.9.4 — a version that contains a high-severity path traversal flaw in nltk.data.load(). Because NLTK arrived as a transitive dependency (pulled in by rouge-score, which is pulled in by lm-eval), the vulnerability was invisible to a casual audit of pyproject.toml. It took automated scanning with Trivy to surface CVE-2026-54293 and trace it back to the resolved package version.

What makes this especially dangerous is context: this is a web service. Any vulnerability reachable through a request handler is directly exploitable by an unauthenticated remote attacker. A path traversal in a data-loading utility might sound academic, but in a live API it becomes a remote file-read primitive.

The Vulnerability Explained

What nltk.data.load() does

nltk.data.load() is NLTK's general-purpose resource loader. You give it a resource identifier — a string like corpora/stopwords or a full file: URI — and it resolves that identifier to a path on disk, then reads the file. Internally, NLTK maintains a list of data paths (e.g., ~/nltk_data) and resolves identifiers relative to those directories.

The vulnerability is in how NLTK 3.9.4 and earlier handle the identifier string before resolving it. Consider this call:

# Simplified illustration of the vulnerable pattern in NLTK <= 3.9.4
import nltk.data

# An attacker-influenced resource identifier reaches nltk.data.load()
resource_id = "corpora%2F..%2F..%2Fetc%2Fpasswd"
data = nltk.data.load(resource_id)

NLTK validates the path before URL-decoding the identifier. The encoded %2F characters pass the directory-boundary check as literal percent signs, but when the underlying OS resolves the path, %2F becomes /. The result: the function reads /etc/passwd (or any other file the process has permission to read) and returns its contents.

The specific flaw

The root cause is a classic decode-then-validate inversion. The validation logic in the vulnerable version checks the raw identifier string for .. sequences, but does not first apply urllib.parse.unquote() (or equivalent) to normalize percent-encoded characters. An attacker can therefore represent every / in a traversal sequence as %2F, and every . as %2E, producing payloads like:

corpora%2F%2E%2E%2F%2E%2E%2Fetc%2Fpasswd

This bypasses the naive check entirely.

Real-world attack scenario for this application

In this project, nltk.data.load() is reachable indirectly via the evaluation harness (lm-eval → rouge-score → nltk). If any API endpoint accepts a parameter that eventually flows into NLTK's resource resolution — for example, a benchmark configuration payload that names a corpus — an attacker could supply a crafted identifier and receive the contents of sensitive files (SSH keys, environment files containing API secrets, /proc/self/environ, etc.) in the response.

Even if no current endpoint passes user input directly to nltk.data.load(), the presence of the vulnerable version in the resolved dependency graph means the risk exists for any future code that uses NLTK, and any existing code path that an attacker can reach through indirect means.

The Fix

What changed

Two files were modified: pyproject.toml and uv.lock.

pyproject.toml — adding a constraint dependency

# Before (no constraint on nltk)
constraint-dependencies = [
    "gitpython>=3.1.50",
    "langsmith>=0.9.0",
    # CVE-2026-49825 (High, XSS) — transitive via lxml[html-clean]; fix at 0.4.5
    "lxml-html-clean>=0.4.5",
]

# After (nltk pinned to the patched version)
constraint-dependencies = [
    "gitpython>=3.1.50",
    "langsmith>=0.9.0",
    # CVE-2026-49825 (High, XSS) — transitive via lxml[html-clean]; fix at 0.4.5
    "lxml-html-clean>=0.4.5",
    # CVE-2026-54293 (High) — transitive via rouge-score/lm-eval; fix at 3.10.0
    "nltk>=3.10.0",
]

The constraint-dependencies section in uv's pyproject.toml is the right tool here: it does not add NLTK as a direct dependency of the project, but it does force the resolver to select a version that satisfies >=3.10.0 whenever NLTK appears anywhere in the dependency graph. This means rouge-score and lm-eval will both receive NLTK 3.10.0+, regardless of what their own install_requires specifies.

uv.lock — the resolved constraint

# Before
(no nltk constraint in the constraints block)

# After
{ name = "nltk", specifier = ">=3.10.0" },

The lock file records the constraint so that uv sync on any machine — CI, developer laptop, production container — resolves to the same patched version without re-running the solver.

Comment cleanup in the all extra

The PR also cleaned up the comment above the all extra, removing the mention of nltk CVE-2026-54293 now that it is fixed, while retaining the note about sqlitedict CVE-2024-35515 which remains unpatched upstream:

# Before
# lm-eval pulls two transitive deps with unpatchable High CVEs
# (sqlitedict CVE-2024-35515, nltk CVE-2026-54293 via rouge-score),
# neither of which has an upstream fix.

# After
# lm-eval pulls sqlitedict CVE-2024-35515, which has no upstream fix.

This is good hygiene: keeping the comment accurate prevents future developers from assuming the NLTK issue is still open.

Why this approach is correct

A simpler fix would be to add nltk>=3.10.0 to the project's direct dependencies. That would work, but it would pollute the dependency graph with a package that is not actually used by the core application — it's only needed by the optional benchmark extra. The constraint-dependencies mechanism enforces the minimum version without changing what gets installed in a standard pip install headroom-ai or pip install headroom-ai[all].

Prevention & Best Practices

1. Treat transitive dependencies as first-class security concerns

CVE-2026-54293 was not in a package listed in dependencies or optional-dependencies — it was three levels deep. Use pip-audit, trivy, or safety in CI to scan the full resolved graph, not just direct dependencies.

# Scan the entire resolved environment with pip-audit
pip-audit --requirement requirements.txt

# Or scan a uv.lock file with trivy
trivy fs --scanners vuln uv.lock

2. Use constraint dependencies to enforce minimum versions for transitive packages

When a vulnerability is in a transitive dependency that you do not control directly, constraint-dependencies (uv) or constraints.txt (pip) let you force a safe version without changing your public API surface.

# pyproject.toml (uv)
[tool.uv]
constraint-dependencies = [
    "some-transitive-package>=safe.version",
]

3. Never pass user-controlled strings to nltk.data.load() without canonicalization

If your application code calls nltk.data.load() with any string that originates from user input, apply strict allowlisting:

import re
import nltk.data

ALLOWED_CORPUS_PATTERN = re.compile(r'^[a-zA-Z0-9_\-]+(/[a-zA-Z0-9_\-]+)*$')

def safe_load_corpus(resource_id: str):
    if not ALLOWED_CORPUS_PATTERN.match(resource_id):
        raise ValueError(f"Invalid resource identifier: {resource_id!r}")
    return nltk.data.load(resource_id)

This defense-in-depth measure is valuable even on NLTK 3.10.0+.

4. Pin lock files in production

The uv.lock file ensures deterministic installs. Commit it to version control and require CI to fail if the lock file is out of date. This prevents a scenario where a developer's local install silently resolves to a vulnerable version.

5. Security standards references

• OWASP Path Traversal: https://owasp.org/www-community/attacks/Path_Traversal
• CWE-22: Improper Limitation of a Pathname to a Restricted Directory
• OWASP Input Validation Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/Input_Validation_Cheat_Sheet.html

Key Takeaways

• The vulnerable function was nltk.data.load(), which failed to URL-decode resource identifiers before validating path boundaries — a classic decode-then-validate inversion.
• The attack vector was URL-encoding: substituting %2F for / and %2E for . bypassed NLTK 3.9.4's path traversal checks entirely.
• The vulnerability arrived transitively via rouge-score → lm-eval, not as a direct dependency — underscoring why full-graph dependency scanning is essential.
• The fix used constraint-dependencies rather than a direct dependency addition, keeping the production install surface clean while still enforcing nltk>=3.10.0 everywhere in the graph.
• Removing the stale comment about the now-fixed CVE from the all extra documentation is as important as the code change — misleading comments cause future developers to make incorrect security assumptions.

How Orbis AppSec Detected This

• Source: The resource identifier string passed to nltk.data.load(), which can be influenced by application-layer inputs in web service request handlers.
• Sink: nltk.data.load() in NLTK ≤ 3.9.4, which resolves the identifier to a filesystem path without first decoding URL-encoded characters.
• Missing control: URL-decoding (e.g., urllib.parse.unquote()) was not applied to the resource identifier before the path-boundary validation check, allowing encoded traversal sequences to bypass the guard.
• CWE: CWE-22 — Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
• Fix: Added nltk>=3.10.0 to constraint-dependencies in pyproject.toml, forcing the resolver to select the patched version across all transitive consumers.

Orbis AppSec automatically detected this vulnerability and opened a pull request with the fix. Try Orbis AppSec on your repositories to find and fix issues like this automatically.

Conclusion

CVE-2026-54293 is a sharp reminder that path traversal vulnerabilities are not limited to hand-written file-handling code — they can arrive pre-packaged inside well-known NLP libraries. The specific failure mode here, processing URL-encoded path components without first normalizing the encoding, is subtle enough to slip through manual code review but well within the detection capability of automated scanners.

The fix is minimal and surgical: a single constraint line in pyproject.toml and its reflection in uv.lock. But the lesson is broader. In a web service, every dependency — direct or transitive — is part of your attack surface. Treat your lock file as a security artifact, scan it on every CI run, and use your package manager's constraint mechanisms to enforce safe versions when upstream control is not available.

References

• CWE-22: Improper Limitation of a Pathname to a Restricted Directory
• OWASP Path Traversal Attack
• OWASP Input Validation Cheat Sheet
• NLTK 3.10.0 Release / PyPI
• uv Constraint Dependencies Documentation
• Semgrep rules for path traversal
• fix: patch nltk vulnerability (CVE-2026-54293)