Scrapy

Scrapy sends a request through the proxy named in its proxy meta key. The built-in HttpProxyMiddleware takes the username and password out of that URL and sends them to the gateway, so you need no extra package.

Install#

pip install scrapy

One spider#

import scrapy

PROXY = "http://USERNAME:[email protected]:8080"


class IpSpider(scrapy.Spider):
    name = "ip"

    async def start(self):
        yield scrapy.Request("https://api.ipify.org?format=json", meta={"proxy": PROXY})

    def parse(self, response):
        yield {"ip": response.json()["ip"]}

Save it as ip_spider.py and run it without a project:

scrapy runspider ip_spider.py -O ip.json

async def start() is the entry point since Scrapy 2.13. On older versions, use def start_requests(self): with the same body.

The meta value takes priority over environment variables, and meta={"proxy": None} sends one request directly.

Environment variables#

Scrapy also reads http_proxy, https_proxy and no_proxy. That is the quickest way to check the proxy without writing a spider:

export https_proxy="http://USERNAME:[email protected]:8080"
scrapy fetch --nolog "https://api.ipify.org?format=json"

A middleware for every request#

For a project, a downloader middleware gives you more control. This one adds the proxy to every request that has none, reads the credentials from the settings, takes the country from the request's meta, and turns Scrapy's cookiejar key into a proxy session:

# myproject/middlewares.py
from urllib.parse import quote


class ProxyMiddleware:
    def __init__(self, user, password, host):
        self.user = user
        self.password = quote(password, safe="")
        self.host = host

    @classmethod
    def from_crawler(cls, crawler):
        s = crawler.settings
        return cls(s["PROXY_USER"], s["PROXY_PASS"], s.get("PROXY_HOST", "geo.gw.nyxon.sh:8080"))

    def process_request(self, request, spider=None):
        if "proxy" in request.meta:
            return None
        user = self.user
        if country := request.meta.get("country"):
            user += f"-country-{country}"
        if (jar := request.meta.get("cookiejar")) is not None:
            user += f"-session-{jar}-time-30"
        request.meta["proxy"] = f"http://{user}:{self.password}@{self.host}"
        return None

Enable it in the settings, with the credentials coming from the environment instead of the repository:

# myproject/settings.py
import os

PROXY_USER = os.environ["PROXY_USER"]
PROXY_PASS = os.environ["PROXY_PASS"]

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.ProxyMiddleware": 350,
}

Sticky sessions with cookie jars#

Scrapy keeps separate cookie sessions with the cookiejar meta key. With the middleware above, each jar also gets its own proxy session, so one set of cookies always leaves from one IP:

import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"
    tags = ["love", "inspirational", "life", "humor"]

    async def start(self):
        for i, tag in enumerate(self.tags):
            yield scrapy.Request(f"https://quotes.toscrape.com/tag/{tag}/", meta={"cookiejar": f"s{i}", "country": "us"})

    def parse(self, response):
        for q in response.css("div.quote"):
            yield {"text": q.css("span.text::text").get(), "jar": response.meta["cookiejar"]}
        next_page = response.css("li.next a::attr(href)").get()
        if next_page:
            yield response.follow(next_page, meta={"cookiejar": response.meta["cookiejar"], "country": "us"})

Two things to know:

• Follow-up requests do not inherit cookiejar. Pass it on yourself, as parse does above.
• The duplicate filter ignores meta. The same URL requested by two jars counts as a duplicate, and the second request is dropped. Give each session its own pages, or pass dont_filter=True on requests that must repeat.

Keep jar keys short, with letters and digits only, since they become part of the username. Requests without a cookiejar get no session and rotate, which suits broad crawling.

Retries and timeouts#

Scrapy's defaults suit direct connections: two retries and a three-minute timeout. For proxy work, these values help:

RETRY_TIMES = 4
DOWNLOAD_TIMEOUT = 45
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_TARGET_CONCURRENCY = 4.0

On a rotating username, a retry opens a new connection and can leave from a different IP, which often fixes a 502. AutoThrottle adapts the pace to the response times it measures and helps you stay under rate limits.

A retry of a sticky request goes back to the same session. If that IP went offline, its retries fail too, so start a new jar for the work instead.

SOCKS5#

Scrapy's default download handler does not speak SOCKS, and it does not warn you. A socks5h:// URL in meta["proxy"] ends in a timeout. Use the HTTP port of your product: it carries HTTPS sites through an encrypted tunnel, so you lose nothing.

Errors#

More in Error codes and Troubleshooting.