Scrapy proxy setup: meta['proxy'], a proxy middleware and sticky sessions

Scrapy has proxy support built in. Its HttpProxyMiddleware is enabled by default and routes a request through whatever proxy you give it, either from environment variables or from the request's proxy meta key. You do not need a plugin. You do need a few lines of middleware if you want credentials kept out of your spiders, country targeting per request, or sticky sessions that stay paired with cookies. Everything below was tested on Scrapy 2.19.

Per request: the proxy meta key

import scrapy

PROXY = "http://USERNAME-country-us:[email protected]:8080"


class IpSpider(scrapy.Spider):
    name = "ip"

    async def start(self):
        yield scrapy.Request("https://api.ipify.org?format=json", meta={"proxy": PROXY})

    def parse(self, response):
        yield {"ip": response.json()["ip"]}

Run it without a project: scrapy runspider ip_spider.py -O ip.json.

A few things happen under the hood. HttpProxyMiddleware takes the username and password out of the URL and sends them to the proxy as a Proxy-Authorization header, encoded with the HTTPPROXY_AUTH_ENCODING setting (latin-1 by default). The meta value takes precedence over environment variables. And setting meta["proxy"] = None sends that one request directly.

async def start() is the entry point since Scrapy 2.13. On older versions, use def start_requests(self): with the same body.

For every request: environment variables

Scrapy also reads http_proxy, https_proxy and no_proxy. That is the quickest way to check a proxy without writing a spider:

export https_proxy="http://USERNAME:[email protected]:8080"
scrapy fetch --nolog "https://api.ipify.org?format=json"

If the IP printed is not yours, Scrapy is going through the proxy. For a real project, though, a middleware gives you more control.

A middleware for credentials, targeting and sticky sessions

This middleware adds the proxy to every request that does not set one, reads the credentials from settings, takes the country from the request's meta, and turns Scrapy's cookiejar key into a proxy session id:

# middlewares.py
from urllib.parse import quote


class ProxyMiddleware:
    def __init__(self, user, password, host):
        self.user = user
        self.password = quote(password, safe="")
        self.host = host

    @classmethod
    def from_crawler(cls, crawler):
        s = crawler.settings
        return cls(s["PROXY_USER"], s["PROXY_PASS"], s.get("PROXY_HOST", "geo.gw.nyxon.sh:8080"))

    # spider is optional: recent Scrapy versions no longer pass it
    def process_request(self, request, spider=None):
        if "proxy" in request.meta:
            return None
        user = self.user
        if country := request.meta.get("country"):
            user += f"-country-{country}"
        if (jar := request.meta.get("cookiejar")) is not None:
            user += f"-session-{jar}-time-30"
        request.meta["proxy"] = f"http://{user}:{self.password}@{self.host}"
        return None

Enable it in settings.py, with the credentials coming from the environment instead of the repository:

# settings.py
import os

PROXY_USER = os.environ["PROXY_USER"]
PROXY_PASS = os.environ["PROXY_PASS"]

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.ProxyMiddleware": 350,
}

The number matters. Downloader middlewares run in ascending order on the way out, and HttpProxyMiddleware sits at 750. At 350, ours sets meta["proxy"] first, and the built-in middleware then turns the credentials into the header. Put it after 750 and the credentials are never sent.

Sticky sessions that follow the cookies

Scrapy can keep several cookie sessions apart with the cookiejar meta key. The middleware above uses the same key as the proxy session id, so each cookie jar always leaves through the same exit IP: one visitor, one IP, one set of cookies.

One catch from the Scrapy documentation: the cookiejar key is not sticky. Follow-up requests do not inherit it, so pass it along yourself. Here each session crawls its own tag listing:

import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"
    tags = ["love", "inspirational", "life", "humor"]

    async def start(self):
        for i, tag in enumerate(self.tags):
            yield scrapy.Request(f"https://quotes.toscrape.com/tag/{tag}/", meta={"cookiejar": f"s{i}", "country": "us"})

    def parse(self, response):
        for q in response.css("div.quote"):
            yield {"text": q.css("span.text::text").get(), "jar": response.meta["cookiejar"]}
        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, meta={"cookiejar": response.meta["cookiejar"], "country": "us"})

A second catch: Scrapy's duplicate filter looks at the URL, method and body of a request, not at its meta, so the same page requested by two cookie jars counts as a duplicate. When we pointed two jars at the same listing, the first crawled all ten pages and the second stopped after page one: its request for page two was dropped as a duplicate. Give each session its own pages, as above, or pass dont_filter=True on requests that really should repeat in every session.

Keep jar keys short and alphanumeric, since they become part of the proxy username. -time-30 holds the exit for 30 minutes; the value is in minutes, up to 120.

Leave cookiejar off for pages that need no session. Those requests get no session token and rotate, which is what you want for broad crawling.

Retries, timeouts and concurrency

Scrapy's defaults were chosen for direct connections. These are the values in Scrapy 2.19:

• RETRY_TIMES = 2: the first try plus two retries.
• RETRY_HTTP_CODES = [500, 502, 503, 504, 522, 524, 408, 429].
• DOWNLOAD_TIMEOUT = 180: three minutes per request.
• CONCURRENT_REQUESTS = 16 and CONCURRENT_REQUESTS_PER_DOMAIN = 8.
• AUTOTHROTTLE_ENABLED = False and DOWNLOAD_DELAY = 0.

For proxy work, a few changes help:

RETRY_TIMES = 4
DOWNLOAD_TIMEOUT = 45
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_TARGET_CONCURRENCY = 4.0

More retries make sense on a rotating pool, because a retry opens a new connection that can leave through a different exit, which is often exactly what fixes a 502. Three minutes is too long to wait on a stalled exit; 30 to 60 seconds covers slow residential routes. AutoThrottle adjusts the delay to the latency it measures, which helps you stay under rate limits instead of collecting 429s.

Two caveats. A retry of a sticky request keeps its cookiejar, so it goes back to the same exit; if that exit went offline, all its retries fail too. And Scrapy retries tunnel errors, including a 407, which no retry can fix. If your log shows TunnelError: Could not open CONNECT tunnel with proxy ... [{'status': 407, 'reason': b'Proxy Authentication Required'}], fix the credentials.

SOCKS proxies

The default download handler does not speak SOCKS, and it does not warn you either. With socks5h:// in meta["proxy"], Scrapy 2.19 sent a plain HTTP CONNECT to the SOCKS port, got nothing useful back, and gave up after DOWNLOAD_TIMEOUT with DownloadTimeoutError, three times over. Use the HTTP port of the same product instead; it carries HTTPS through an encrypted tunnel, so there is no downside for scraping.

Recent Scrapy versions ship an experimental httpx-based download handler that can use SOCKS proxies, but its own source marks it as not recommended for production use.

Checking it works

Point a spider, or scrapy fetch, at https://ipinfo.io/json and check three fields: ip is not yours, country matches the targeting, and org is a consumer ISP for residential exits. Then run a few pages of the real target with LOG_LEVEL = "INFO" and watch the retry counters in the stats Scrapy prints at the end: a high retry/count against a target that works in a browser usually means headers or pace, not the proxy. The ten proxy mistakes cover the usual suspects, and the proxy error codes guide explains each status.

FAQ

Does Scrapy rotate proxies on its own? No. It uses whatever proxy each request carries. Rotation comes from the gateway, where a username without a session token can get a new exit on each new connection, or from a middleware that picks a different proxy per request.

Do I need scrapy-rotating-proxies or similar packages? Only if you manage a list of individual proxies yourself. With a rotating gateway, one proxy URL is enough.

Can I use a SOCKS5 proxy with Scrapy? Not with the default handler. Use the HTTP port.

Where do I set a different country for some requests? In the request meta, as with the country key in the middleware above.

Try it

Scrapy pairs well with Residential Budget for broad crawls and Residential for ASN, US city or sticky work. Create an account, link Discord under Freebies for 250MB free, and copy your username and password from the Generate page.