Configure a proxy in Scrapy

Scrapy has proxy support built in, so a Scrapy proxy setup needs no extra package: put the netdash proxy URL, credentials included, in a request's proxy meta key, and Scrapy's own HttpProxyMiddleware handles the authentication. Setting the meta key by hand suits a single spider or a quick test. For a project, a short downloader middleware sets it on every request, including the ones Scrapy creates itself. The examples need Scrapy 2.13 or later (Scrapy 2.14 and later need Python 3.10) and use the Scale plan on the Americas gateway, scale-us.netdash.io:2000. Other plans differ only in host and port, listed in the proxy format reference.

Scrapy proxy quick start: the proxy request meta key

This spider requests one page through netdash and prints the exit IP. It carries the same proxy_url() helper as the Python guide, which reads your credentials from NETDASH_USERNAME and NETDASH_PASSWORD. Run it with scrapy runspider ip_spider.py:

import os
import secrets
from typing import Optional

import scrapy

NETDASH_USERNAME = os.environ["NETDASH_USERNAME"]
NETDASH_PASSWORD = os.environ["NETDASH_PASSWORD"]


def proxy_url(country: str = "US", session_id: Optional[str] = None,
              ttl_minutes: int = 30, gateway: str = "scale-us.netdash.io:2000") -> str:
    """netdash proxy URL: rotating without session_id, sticky with one."""
    cc = country.upper()
    cc = "GB" if cc == "UK" else cc
    username = f"{NETDASH_USERNAME}-cc-{cc}"
    if session_id is not None:
        username += f"-session-{session_id}-ttl-{ttl_minutes}"
    return f"http://{username}:{NETDASH_PASSWORD}@{gateway}"


def new_session_id() -> str:
    return secrets.token_hex(4)  # 8 characters, letters and digits


class IpSpider(scrapy.Spider):
    name = "ip"
    custom_settings = {"ROBOTSTXT_OBEY": True, "DOWNLOAD_TIMEOUT": 30}

    async def start(self):
        yield scrapy.Request("https://api.ipify.org?format=json",
                             meta={"proxy": proxy_url("US")})

    def parse(self, response):
        yield response.json()  # {"ip": "203.0.113.10"}

How it works, according to the Scrapy documentation:

• The proxy URL may contain credentials (http://username:password@host:port); HttpProxyMiddleware sends them to the gateway as proxy authentication.
• A proxy meta value takes precedence over the http_proxy and https_proxy environment variables, and ignores no_proxy.

The proxy URL starts with http://, also for HTTPS websites: netdash is an HTTP proxy, and HTTPS requests go through a CONNECT tunnel.

Meta applies only to the requests you set it on. Requests that Scrapy creates itself, such as the robots.txt check, don't carry your meta, so they go out without the proxy. The middleware below closes that gap.

A downloader middleware for every request

The middleware reads your credentials once and builds the proxy URL for each request from three optional meta keys, netdash_country, netdash_session and netdash_ttl, falling back to spider attributes of the same name without the prefix. It leaves alone any request that already has a proxy key, so meta={"proxy": None} still sends a request directly. Enable it at order 350, so it runs before the built-in HttpProxyMiddleware at 750.

import os


class NetdashProxyMiddleware:
    """Sets a netdash proxy URL on every request that doesn't already have one."""

    def __init__(self, crawler, username, password, gateway):
        self.crawler = crawler
        self.username = username
        self.password = password
        self.gateway = gateway

    @classmethod
    def from_crawler(cls, crawler):
        return cls(
            crawler,
            os.environ["NETDASH_USERNAME"],
            os.environ["NETDASH_PASSWORD"],
            crawler.settings.get("NETDASH_GATEWAY", "scale-us.netdash.io:2000"),
        )

    def process_request(self, request, spider=None):
        # spider=None: Scrapy 2.13 passes the spider, 2.14 and later don't.
        if "proxy" in request.meta:
            return None
        spider = self.crawler.spider
        country = str(request.meta.get("netdash_country", getattr(spider, "country", "US"))).upper()
        country = "GB" if country == "UK" else country
        user = f"{self.username}-cc-{country}"
        session = request.meta.get("netdash_session", getattr(spider, "session", None))
        if session:
            ttl = request.meta.get("netdash_ttl", getattr(spider, "ttl", 30))
            user += f"-session-{session}-ttl-{ttl}"
        request.meta["proxy"] = f"http://{user}:{self.password}@{self.gateway}"
        return None

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.NetdashProxyMiddleware": 350,
}
NETDASH_GATEWAY = "scale-us.netdash.io:2000"

Replace myproject with your project's module name. Scrapy 2.14 deprecated requiring a spider argument in custom downloader middleware methods in favor of the crawler's spider attribute; the spider=None default keeps the middleware working on Scrapy 2.13 and on newer releases without a deprecation warning.

Sticky sessions per crawl or per item

Without a session ID, every request uses a rotating session and the gateway assigns exit IPs for you. A sticky session keeps the same exit IP for the TTL you set while that IP stays available, and reusing the session ID reuses the session. Use letters and digits only; the dashboard generates 8 lowercase characters.

• One session for the whole crawl: pass it as a spider argument, scrapy crawl prices -a session=a1b2c3d4 -a ttl=30. The -a arguments become spider attributes, which the middleware reads.
• One session per item: set netdash_session in the meta of the requests that belong together. Meta isn't inherited by follow-up requests, so pass it on explicitly:

import secrets

import scrapy


class PricesSpider(scrapy.Spider):
    name = "prices"
    start_urls = ["https://example.com/catalog"]

    def parse(self, response):
        for href in response.css("a.product::attr(href)").getall():
            # one sticky session per product: its pages share an exit IP
            yield response.follow(href, self.parse_product,
                                  meta={"netdash_session": secrets.token_hex(4)})

    def parse_product(self, response):
        specs = response.css("a.specs::attr(href)").get()
        if specs:
            yield response.follow(specs, self.parse_specs,
                                  meta={"netdash_session": response.meta["netdash_session"]})

    def parse_specs(self, response):
        yield {"url": response.url, "title": response.css("h1::text").get()}

TTL values match the dashboard presets: 15, 30, 720 (12 hours) or 10080 (7 days) minutes. Rotating and sticky sessions explains how to choose.

Country per spider run

The exit country is a spider argument too, so the same spider can run once per market. European exits use the Europe gateway, which -s sets for that run only:

scrapy crawl prices -a country=DE -s NETDASH_GATEWAY=scale-eu.netdash.io:2000

-a country=DE becomes the spider attribute country that the middleware reads, and -s overrides the one custom setting. Use ISO 3166-1 alpha-2 codes, and GB for the United Kingdom, not UK. Targeting is country-level; see the country targeting syntax.

Crawl politely

netdash sets no per-account concurrency limit, so the limits in settings.py are the ones that protect the websites you crawl. Start conservative:

ROBOTSTXT_OBEY = True
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0
DOWNLOAD_DELAY = 1.0
CONCURRENT_REQUESTS_PER_DOMAIN = 4
DOWNLOAD_TIMEOUT = 30
RETRY_TIMES = 2
USER_AGENT = "examplebot/1.0 (+https://example.com/bot)"

AutoThrottle raises the delay when responses slow down and never goes below DOWNLOAD_DELAY; CONCURRENT_REQUESTS_PER_DOMAIN stays a hard cap on top of it. A low RETRY_TIMES keeps failures from multiplying traffic. Name your crawler in USER_AGENT and link to a page that explains it, so site owners can reach you.

Proxy credentials since Scrapy 2.6.2

Scrapy 2.6.2 changed how HttpProxyMiddleware treats the Proxy-Authorization header when the proxy meta key changes between requests. It matters if you switch countries or sessions within one crawl.

Errors and fixes

Next steps

For a complete Python pipeline without Scrapy, see the complete scraper tutorial. Collecting training data? Read responsible AI training data collection first. Price checks by market are covered in e-commerce price monitoring, and every other tool is in the proxy integration guides. New to netdash? Start with the quickstart. To size a pipeline, see pricing: residential GB are sold in one-time orders with graduated per-GB pricing, and purchased GB never expire.

• Python proxy setup

  requests, httpx and aiohttp with the same proxy_url() helper.

• Country targeting

  ISO country codes in the username, GB for the United Kingdom, and the matching gateway.

• Web data collection

  Scheduled collection of public pages by country, at a rate each site can handle.

Stuck? See troubleshooting or email support with your username (without the password) and the error message.