How to use a proxy with Python Requests: HTTP, SOCKS5, sessions and retries

Python Requests takes a proxy through the proxies argument, a dictionary that maps the scheme of the URL you are requesting to the proxy that should carry it. The short version:

import requests

proxy = "http://USERNAME:[email protected]:8080"
proxies = {"http": proxy, "https": proxy}

r = requests.get("https://api.ipify.org?format=json", proxies=proxies, timeout=30)
print(r.json())

That works. What follows is everything that breaks once the script leaves your laptop: the dictionary keys, passwords with special characters, SOCKS5 and DNS, the environment variable trap with Session, rotation, retries and threads. Every snippet was run against Requests 2.34 on Python 3.14.

The proxies dictionary: both keys, every time

The key is the scheme of the target URL. The value is the proxy URL. So "https": proxy means "send requests for https:// URLs through this proxy".

The value starts with http:// even for HTTPS sites. Requests asks the HTTP proxy to open a tunnel with CONNECT, then runs TLS through it, end to end between you and the site. Writing https:// in front of the proxy tells Requests to speak TLS to the proxy itself, which a plain HTTP proxy port does not do, and you get this:

ProxyError('Unable to connect to proxy. Your proxy appears to only use HTTP and not HTTPS, try changing your proxy URL to be HTTP. ...')

The more dangerous mistake is setting only the "http" key. Requests then sends every https:// request directly, from your own IP, with no error and no warning. In our test the proxy saw nothing and the request still returned 200. Since almost every site is HTTPS, a dictionary with only "http" in it means you are not using a proxy at all.

You can also route a single host through the proxy with the scheme://hostname form as the key:

proxies = {"https://api.example.com": "http://USERNAME:[email protected]:8080"}

Credentials with special characters

Username and password go into the proxy URL. If the password contains @, :, /, # or %, the URL parser splits it in the wrong place. A password of p@ss:w/rd#1 pasted raw produced this:

requests.exceptions.InvalidURL: Failed to parse: 'ss:w' is not a valid host or port

Percent-encode the credentials with quote(..., safe=""). Requests decodes them again before sending:

from urllib.parse import quote

user = "USERNAME-country-us"
password = quote("p@ss:w/rd#1", safe="")
proxy = f"http://{user}:{password}@geo.gw.nyxon.sh:8080"

Better still, keep credentials out of the code:

import os
from urllib.parse import quote

user = os.environ["PROXY_USER"]
password = quote(os.environ["PROXY_PASS"], safe="")
proxy = f"http://{user}:{password}@geo.gw.nyxon.sh:8080"
proxies = {"http": proxy, "https": proxy}

SOCKS5, and why the h in socks5h matters

SOCKS support is an optional extra:

pip install "requests[socks]"

Without it, a SOCKS proxy URL fails with InvalidSchema: Missing dependencies for SOCKS support. With it installed, change the scheme and use the SOCKS5 port of your product:

proxy = "socks5h://USERNAME:[email protected]:1080"
proxies = {"http": proxy, "https": proxy}

socks5:// and socks5h:// differ in who resolves the hostname. With socks5://, your machine does the DNS lookup and sends the proxy an IP address. With socks5h://, the hostname goes to the proxy and the lookup happens on the exit side. We watched this on a test proxy: the socks5:// request arrived as a bare IP, the socks5h:// request as api.ipify.org.

Use socks5h. Local lookups tell your own DNS resolver which sites you visit, and they can point you at a CDN server close to you instead of close to the exit, so you get content localized for the wrong country. An HTTP proxy always receives the hostname, so it never has this problem.

Session objects and the environment variable trap

A Session reuses connections and keeps cookies, which is what you want for anything with a login. Most tutorials set the proxy like this:

s = requests.Session()
s.proxies.update(proxies)

The Requests documentation warns that this "may behave differently than expected": proxies from the environment override session.proxies. We tested it. With HTTPS_PROXY set in the environment, a session with session.proxies configured ignored them and used the environment proxy. Passing proxies= on each request won, and so did turning environment lookups off:

s = requests.Session()
s.trust_env = False   # ignore HTTP_PROXY, HTTPS_PROXY and system settings
s.proxies.update(proxies)

On Windows and macOS there is a second source: when no proxy environment variables are set, Python falls back to the proxy configured in the system settings. trust_env = False turns that off too.

You can also use the environment on purpose. Requests reads HTTP_PROXY, HTTPS_PROXY, ALL_PROXY and NO_PROXY without any code changes:

export HTTPS_PROXY="http://USERNAME:[email protected]:8080"
export HTTP_PROXY="http://USERNAME:[email protected]:8080"
export NO_PROXY="localhost,127.0.0.1"
python scraper.py

The documentation also notes that keeping credentials in environment variables or version-controlled files is a security risk, so treat this as a convenience for your own machine.

Rotating and sticky IPs

With a rotating gateway, the provider picks the exit, not your code. A username without a session token gets a fresh exit for each new connection; add a session token and you keep one exit. The sessions guide covers the syntax.

The detail most people miss is the word "connection". A Session keeps connections open, and through an HTTP proxy a connection to an HTTPS site is a tunnel to one exit. We made three requests to the same site through a rotating gateway: with one Session they used a single tunnel, so a single exit, while three plain requests.get() calls opened three tunnels. If you want a new IP on every request, use separate calls, or send Connection: close:

for _ in range(3):
    r = requests.get("https://api.ipify.org?format=json", proxies=proxies, headers={"Connection": "close"}, timeout=30)
    print(r.json()["ip"])

For logins, carts and anything else that must look like one visitor, do the opposite: pin a session id and reuse one Session, so cookies and IP stay together:

import secrets
import requests

def sticky_session(country="us", minutes=10):
    sid = secrets.token_hex(4)
    proxy = f"http://USERNAME-country-{country}-session-{sid}-time-{minutes}:[email protected]:8080"
    s = requests.Session()
    s.trust_env = False
    s.proxies.update({"http": proxy, "https": proxy})
    s.headers["User-Agent"] = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/155.0.0.0 Safari/537.36"
    return s

s = sticky_session()
print(s.get("https://api.ipify.org?format=json", timeout=30).json())
print(s.get("https://api.ipify.org?format=json", timeout=30).json())   # same IP

The -time- value is in minutes, from 1 to 120. The Generate page builds the same string for you if you would rather copy it.

If you have a fixed list of proxies instead of a gateway, cycle through it:

from itertools import cycle

pool = cycle([
    "http://USERNAME:[email protected]:8000",
    "http://USERNAME:[email protected]:8000",
])

for url in ["https://api.ipify.org?format=json"] * 4:
    p = next(pool)
    r = requests.get(url, proxies={"http": p, "https": p}, timeout=30)

Timeouts and retries

Requests has no default timeout. Without one, a stalled exit can hang a worker for minutes. Always pass timeout, ideally as a (connect, read) tuple:

r = requests.get(url, proxies=proxies, timeout=(10, 30))

A residential exit adds a few hundred milliseconds of latency, so give the read timeout some room. When it runs out you get ReadTimeout: ... Read timed out. (read timeout=30).

For retries, mount urllib3's Retry on a session. This retries the status codes that are usually temporary, with exponential backoff:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util import Retry

retries = Retry(
    total=4,
    backoff_factor=0.5,
    status_forcelist=[429, 500, 502, 503, 504],
    allowed_methods=["GET", "HEAD"],
)

s = requests.Session()
s.trust_env = False
s.proxies.update(proxies)
s.mount("http://", HTTPAdapter(max_retries=retries))
s.mount("https://", HTTPAdapter(max_retries=retries))

r = s.get("https://api.ipify.org?format=json", timeout=(10, 30))

Retry also honours the Retry-After header on 429 and 503 responses by default. In our test a 429 with Retry-After: 2 was retried after two seconds and then succeeded. When the retries run out, Requests raises RetryError ... too many 503 error responses.

Two notes. On a rotating session, a retry that opens a new connection can leave through a different exit, which is often exactly what fixes a 502. And no number of retries fixes a 407: that is a credentials problem.

Threads: one Session per thread

Requests does not promise that a Session is thread-safe, and sharing one also shares its cookies and, with a session token, its single exit. Give each thread its own:

import threading
from concurrent.futures import ThreadPoolExecutor

local = threading.local()

def get_session():
    if not hasattr(local, "s"):
        local.s = sticky_session()
    return local.s

def fetch(url):
    return get_session().get(url, timeout=(10, 30)).json()["ip"]

with ThreadPoolExecutor(max_workers=4) as ex:
    print(list(ex.map(fetch, ["https://api.ipify.org?format=json"] * 8)))

Each worker gets its own sticky session, so its own IP, and keeps it for every request it makes.

Check what the site sees

An IP echo only proves the proxy is in the path. To see where the exit is, ask a geolocation service:

r = requests.get("https://ipinfo.io/json", proxies=proxies, timeout=30)
print(r.json())   # ip, city, region, country, org

The org field shows the network: a consumer ISP for residential exits, a hosting company for datacenter ones.

Common errors

• ProxyError ... Tunnel connection failed: 407 Proxy Authentication Required: wrong or badly encoded credentials.
• ProxyError ... Failed to establish a new connection: wrong host or port, or a firewall blocks the port.
• Your proxy appears to only use HTTP and not HTTPS: change https:// to http:// at the start of the proxy URL.
• InvalidSchema: Missing dependencies for SOCKS support.: run pip install "requests[socks]".
• SOCKS5 authentication failed: wrong credentials on the SOCKS5 port.
• InvalidURL: Failed to parse: a special character in the password; percent-encode it.
• ReadTimeout: the exit or the site is slow; raise the read timeout and retry.

Errors from the site itself, such as 403 and 429, come back as normal responses. The proxy error codes guide explains each one.

FAQ

Does Requests support SOCKS5 proxies? Yes, with pip install "requests[socks]". Use socks5h:// so DNS resolves on the proxy side.

Why does my request still show my real IP? Either the dictionary has no "https" key, or an environment proxy overrode session.proxies. Set both keys, and set trust_env = False on sessions.

How do I rotate proxies in Requests? Use a rotating gateway username and open a new connection per request, or cycle through a list of proxies yourself.

Does httpx work the same way? Mostly. httpx takes a single proxy= argument on the client; the curl, Python and Node.js quickstart has an async example.

Try it

Every example here works on any NyxonProxies plan; Residential Budget is the cheapest place to start. Create an account, link Discord under Freebies for 250MB free, and copy your string from the Generate page.