esearch() SSRF: requests.get() Trusted Any Host in the URL

Summary

A high-severity server-side request forgery (SSRF) path existed in the esearch() helper of a first-party academic citation format-conversion CLI. The helper assembled a PubMed E-utilities URL from caller-supplied search terms and handed it directly to requests.get(). Nothing in the code verified that the string still pointed at eutils.ncbi.nlm.nih.gov by the time the request was made. The fix adds an explicit host allowlist and a _is_allowed_url() check that runs before every outbound fetch in that function.

Introduction

This tool is a CLI that converts academic citations between formats. It accepts a PMID, a DOI, an arXiv ID, or a free-text query, and resolves that input against one of three public APIs: NCBI E-utilities, Crossref, and the arXiv Atom API. The resolution step is plain requests code — build a URL string, fetch it, parse the response.

The problem was that the "build a URL string" and "fetch it" steps had no boundary between them. In esearch():

def esearch(query, max_results=5):
    params = {"db": "pubmed", "term": query, "retmax": max_results, "retmode": "xml"}
    url = f"{EUTILS_BASE}/esearch.fcgi?{urlencode(params)}"
    try:
        resp = requests.get(url, timeout=30)

url is a str built by interpolation, and requests.get() will fetch whatever str you give it: http://, https://, an IP literal, a link-local address, localhost:8080. The function's intent is "fetch PubMed," but its contract is "fetch an arbitrary URL." That gap is CWE-918, and it is one of the most common ways SSRF gets introduced into otherwise-boring API client code.

This matters more than usual here because the script is packaged as a skill invoked by an AI agent. The arguments it receives — query, identifiers, and anything else a caller decides to pass — can originate from document text the agent just read. Prompt-injected content in a paper abstract, an issue comment, or a scraped page becomes CLI input, and CLI input becomes an outbound HTTP request made from inside your network with your instance's credentials attached to the network path.

Affected Versions

There is no package version to pin here. If your copy of this converter calls requests.get() on an interpolated URL without a host check, you are exposed regardless of when you copied it.

The Vulnerability Explained

The vulnerable pattern

Three module-level constants define the intended hosts:

EUTILS_BASE = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils"
CROSSREF_BASE = "https://api.crossref.org/works"
ARXIV_BASE = "https://export.arxiv.org/api/query"

Those constants look like a security control, but they are not one. They are a default. The line that actually decides where the packet goes is requests.get(url, timeout=30), and it accepts any string. Every guarantee about the destination lives in the assumption that nothing between the constant and the call site changed the host — an assumption the code never checks and no test can enforce.

Why urlencode() is not the mitigation

It is tempting to look at urlencode(params) and conclude the input is already safe. urlencode() percent-encodes query and max_results, so a search term cannot introduce a raw &, #, or / into the query string. That is real protection — against query string injection.

It does nothing about the authority component. The host, port, scheme, and userinfo of the final URL are all decided by EUTILS_BASE, which sits entirely outside urlencode()'s reach. Sanitizing the terms does not constrain the destination, and the destination is what SSRF is about.

How the host becomes attacker-influenced

There are several realistic routes, and none of them require an exotic bug:

1. A configurable base. The moment someone adds a mirror or proxy knob — an --api-base flag, a NCBI_BASE_URL environment variable, a settings file for an air-gapped deployment — the destination is caller-controlled and the fetch is an open proxy. This is the single most common way latent CWE-918 becomes live.
2. Redirect following. requests.get() follows redirects by default. Any hop the script is willing to make can point somewhere the script would never have chosen itself.
3. A refactor to a URL-taking helper. Fetch helpers in scripts like this drift toward a shared fetch(url). Once the parameter is a URL, the caller owns the host, and callers here include an LLM relaying untrusted text.
4. Agent-supplied arguments. The skill's arguments are chosen by a model reading untrusted content. Treating them as trusted developer input is the same mistake as trusting a query parameter.

Attack scenario

Suppose the converter runs on an EC2 instance as part of an agent workflow, and a "use this mirror" argument is threaded into EUTILS_BASE. Injected text in a document the agent ingests causes it to invoke the converter with a base of http://169.254.169.254/latest/meta-data/iam/security-credentials. esearch() appends /esearch.fcgi?db=pubmed&term=..., requests.get() fetches it, and the response text is handed to the XML parser. Even if the parse fails, the raw body frequently ends up in agent-visible error output or logs — and a leaked IMDS role credential is a full cloud compromise, not a missing citation.

Softer but equally useful variants against the same call:

• http://127.0.0.1:5000/admin/shutdown — reach a service that only trusts loopback.
• http://10.0.4.19:8500/v1/kv/?recurse — read an internal Consul KV store.
• Iterating http://10.0.0.0/24 ports and measuring which requests time out at 30 seconds versus fail immediately — a port scanner with the script's network identity, the classic "blind SSRF" primitive.

The timeout=30 on the call is a performance guardrail, not a security one; it is precisely the signal that makes timing-based internal scanning legible.

The Fix

The change adds a positive allowlist and enforces it at the call site.

Before — the destination is whatever the interpolated string says:

url = f"{EUTILS_BASE}/esearch.fcgi?{urlencode(params)}"
try:
    resp = requests.get(url, timeout=30)

After — the destination is checked against a closed set first:

ALLOWED_HOSTS = {"eutils.ncbi.nlm.nih.gov", "api.crossref.org", "export.arxiv.org"}


def _is_allowed_url(url):
    """Only allow requests to the known, hardcoded API hosts (mitigates SSRF)."""
    return urlparse(url).hostname in ALLOWED_HOSTS

and, inside esearch():

if not _is_allowed_url(url):
    print("  ESearch error: URL host not in allowlist")
    return []
resp = requests.get(url, timeout=30)

Three details are worth calling out, because they are what make this fix correct rather than merely present:

• hostname, not netloc. urlparse(url).hostname lowercases the host and strips both userinfo and port. That means https://eutils.ncbi.nlm.nih.gov@evil.example/esearch.fcgi resolves to evil.example and is rejected, and https://EUTILS.NCBI.NLM.NIH.GOV/... is accepted. A netloc comparison, or a url.startswith(...) check, would have been bypassed by the @ form — this is the single most common broken SSRF allowlist.
• An allowlist, not a denylist. The alternative approach — blocking 127.0.0.1, 169.254.169.254, and RFC1918 ranges — is a game of whack-a-mole against [::1], 0.0.0.0, decimal-encoded IPs, localtest.me-style DNS tricks, and every new metadata endpoint. Three hostnames are known, finite, and auditable. Matching them exactly is the strongest form of this control.
• urlencode and urlparse imported together. The import line changed from from urllib.parse import urlencode to also pull in urlparse, which is the only supporting change required — the validation needs no new dependency.

The failure mode is also deliberately boring: the guard prints ESearch error: URL host not in allowlist and returns [], matching the function's existing behaviour on network or parse failure. Callers already handle an empty result list, so the fix cannot turn an SSRF attempt into an unhandled exception or a stack trace in agent output.

What this change does not cover

Two gaps remain, and both should be closed in follow-up work:

1. Only esearch() is guarded. ALLOWED_HOSTS already contains api.crossref.org and export.arxiv.org, but the DOI and arXiv fetch helpers do not yet call _is_allowed_url(). They need the same two lines before their requests.get().
2. Redirects are still followed. _is_allowed_url() validates the initial URL only. A 302 from an allowed host to an internal address would still be fetched. Passing allow_redirects=False, or re-running _is_allowed_url() on each hop in resp.history, makes the invariant hold for the whole request chain.

Key Takeaways

• A hardcoded base URL constant is a default, not a control. EUTILS_BASE being a literal in the module says nothing about the string that re