How gitlab.bandit.B501 happens in Python and how to fix it

Introduction

The proverbia-scraper.py file handles a simple but common task: fetching pages from https://proverbia.net/ and parsing them with BeautifulSoup to pull out proverbs. The heavy lifting happens in a single helper function, get_soup(url), which wraps every outbound HTTP call the scraper makes. A flaw in that one function, however, meant that every single request the scraper issued was vulnerable to interception.

The problem lived on what used to be line 33:

page = requests.get(url, headers=HEADERS, timeout=30, verify=False)

That verify=False argument tells the requests library to skip TLS certificate validation entirely. Combined with a urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) call earlier in the file, the code wasn't just insecure — it was insecure silently, with no warning ever surfacing in logs or console output.

The Vulnerability Explained

Here's the vulnerable code as it existed before the fix:

import urllib3

urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

BASE_URL = "https://proverbia.net/"
NOW = datetime.now()

...

def get_soup(url: str) -> BeautifulSoup:
    """Download a web page and return it as a BeautifulSoup object."""
    page = requests.get(url, headers=HEADERS, timeout=30, verify=False)
    page.raise_for_status()
    return BeautifulSoup(page.content, "html.parser")

Two things are happening here, and both matter:

1. verify=False disables TLS certificate validation. Normally, when requests connects to https://proverbia.net/, it checks the server's certificate against a trusted certificate authority chain and verifies the hostname matches. With verify=False, none of that happens — requests will happily complete the TLS handshake with any server presenting any certificate, including a self-signed one from an attacker.

2. urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) suppresses the InsecureRequestWarning that urllib3 would normally print to stderr every time an unverified HTTPS request is made. This is the giveaway that someone added verify=False deliberately and then silenced the tool trying to warn them about it — a pattern that removes the last line of defense (a visible warning in logs) against the misconfiguration going unnoticed.

Attack scenario: Imagine this scraper running on a schedule inside a CI job, a cron container, or a shared network (a coffee shop Wi-Fi, a compromised corporate proxy, a poisoned DNS cache). An attacker positioned on the network path between the scraper and proverbia.net can perform a classic man-in-the-middle attack:

• The attacker intercepts the DNS resolution or ARP-spoofs the local network so traffic to proverbia.net routes through their machine.
• They present their own TLS certificate — self-signed, expired, or issued for a completely different domain.
• Because get_soup() calls requests.get(url, ..., verify=False), requests accepts this bogus certificate without complaint.
• The attacker now sits inside the encrypted tunnel, able to read every byte of the response (page.content) before it's handed to BeautifulSoup, and — more dangerously — can rewrite the HTML the scraper parses.

Since get_soup()'s return value drives whatever logic consumes the parsed proverbs (storage, display, further processing), an attacker who controls the response content controls what the scraper "sees." Depending on how downstream code trusts the parsed data, this could range from serving fake/misleading content to injecting malicious payloads if the scraped text is ever rendered, executed, or stored without further sanitization.

The Fix

The fix is minimal but precise — it removes the insecure argument and the warning suppression that was hiding it:

Before:

import requests
from random import choice
from datetime import datetime
import urllib3

urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

BASE_URL = "https://proverbia.net/"
NOW = datetime.now()

def get_soup(url: str) -> BeautifulSoup:
    """Download a web page and return it as a BeautifulSoup object."""
    page = requests.get(url, headers=HEADERS, timeout=30, verify=False)
    page.raise_for_status()
    return BeautifulSoup(page.content, "html.parser")

After:

import requests
from random import choice
from datetime import datetime
BASE_URL = "https://proverbia.net/"
NOW = datetime.now()

def get_soup(url: str) -> BeautifulSoup:
    """Download a web page and return it as a BeautifulSoup object."""
    page = requests.get(url, headers=HEADERS, timeout=30)
    page.raise_for_status()
    return BeautifulSoup(page.content, "html.parser")

Two coordinated changes made this work:

1. verify=False was dropped entirely from the requests.get() call in get_soup(). Since requests defaults to verify=True, simply omitting the argument restores full certificate chain and hostname validation. requests will now reject any connection to proverbia.net that doesn't present a valid, trusted certificate for that hostname.

2. The urllib3 import and disable_warnings() call were deleted. Once certificate verification is back on, there's no InsecureRequestWarning to suppress — but removing this code also eliminates the risk that a future contributor copies the verify=False pattern elsewhere in the file without noticing the warning had been silenced project-wide.

Notably, the fix preserved timeout=30, which was already correctly set — this wasn't a timeout issue, it was purely a certificate-validation issue. The PR is scoped to exactly one file and one function, so no behavioral change occurs for legitimate traffic: valid HTTPS connections to proverbia.net continue to work exactly as before, while spoofed or attacker-controlled certificates are now correctly rejected.

Key Takeaways

• The get_soup() function in proverbia-scraper.py was making every HTTPS request to proverbia.net without validating the server's certificate — a single verify=False argument undermined TLS for the entire scraper.
• The accompanying urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) call was actively hiding the security warning that would have flagged this misconfiguration.
• The fix required no new logic — deleting verify=False and the warning suppression was sufficient, since requests defaults to secure behavior (verify=True).
• The existing timeout=30 was correctly preserved, showing that not every part of a flagged pattern needs to change — only the actual insecure argument did.
• Any future requests calls added to this scraper should be checked for the same pattern before merging.

How Orbis AppSec Detected This

• Source: The url parameter passed into get_soup(url), ultimately derived from BASE_URL and scraper-constructed page URLs targeting https://proverbia.net/.
• Sink: requests.get(url, headers=HEADERS, timeout=30, verify=False) in proverbia-scraper.py:33.
• Missing control: TLS certificate validation was explicitly disabled via verify=False, and the resulting InsecureRequestWarning was suppressed via urllib3.disable_warnings(), removing both the protection and the visible warning of its absence.
• CWE: CWE-295 — Improper Certificate Validation.
• Fix: Removed verify=False from the requests.get() call and deleted the urllib3.disable_warnings() call, restoring default certificate validation for all outbound requests.

Orbis AppSec automatically detected this vulnerability and opened a pull request with the fix. Try Orbis AppSec on your repositories to find and fix issues like this automatically.

Conclusion

A single argument — verify=False — was enough to strip TLS protection from every outbound request in proverbia-scraper.py, and a companion call to urllib3.disable_warnings() made sure no one would notice. The fix demonstrates that hardening doesn't always require complex logic changes: sometimes the most secure code is the code you remove. Going forward, treat verify=False and warning suppression calls as immediate red flags in code review, and let static analysis tools like Bandit and Semgrep catch these patterns automatically before they ship.