Web Scraping with Python and Proxies

This tutorial covers web scraping with Python and proxies from start to finish. You'll set up a project, check robots.txt, fetch pages through a netdash proxy with requests, parse them with Beautiful Soup, store the results, and pace your requests at a rate the site can handle. Proxy configuration gets one short section here; for every option in requests and httpx, see how to configure a proxy in Python requests.

You need Python 3.9 or later and a netdash account with a residential plan; we tested the code on Python 3.14. It runs as shown, and only the credentials come from you.

What you will build

A script of about 190 lines, scrape_books.py, that:

confirms the proxy works and logs the exit IP for the run;
reads robots.txt through the same proxy and stops if the rules don't allow the pages;
fetches catalogue pages of books.toscrape.com one at a time, with a pause between pages;
extracts each book's title, price as shown, availability and URL, and skips entries that don't parse;
writes everything to books.csv.

Books to Scrape is a sandbox built for scraping practice. Its pages describe it as a demo website for web scraping, with prices and ratings assigned at random, so you can run this tutorial without affecting a real business. When you adapt the script to another site, read that site's robots.txt and terms first.

Set up the project

Create a folder, a virtual environment and the two dependencies:

Terminal

mkdir books-scraper && cd books-scraper
python3 -m venv .venv
source .venv/bin/activate
pip install requests beautifulsoup4
echo ".env" >> .gitignore

Then create a .env file with your netdash credentials. You'll find them in the dashboard under Residential, then Generate:

.env

NETDASH_USERNAME=USERNAME
NETDASH_PASSWORD=PASSWORD

Load the file into your shell before running the script, for example with set -a; . ./.env; set +a.

Credentials

Keep NETDASH_USERNAME and NETDASH_PASSWORD in a .env file or your secret manager, never in source control or in a chat with an assistant.

Check robots.txt and the site's terms

Before the first page request, read the site's robots.txt. The Robots Exclusion Protocol (RFC 9309) defines how crawlers should treat it, and three of its rules shape the code below:

A crawler follows the group of rules whose user agent matches its product token, or the * group if none does. That's why the script sends one fixed user agent and checks the rules under the same name.
If /robots.txt returns a 4xx status, there are no rules. If it returns a 5xx status or can't be reached, the crawler must assume it may not crawl anything.
The file is UTF-8 text, and its rules aren't access authorization: they state what the site asks of crawlers, and the site's terms still apply.

Give your scraper an honest user agent with a product token and a URL where the site's owner can find out who you are and contact you; replace example.com/contact in the examples with a page of your own. Then fetch robots.txt through the same session and proxy as your pages, so the check uses the same client and exit IP as the crawl:

robots.py

from urllib.parse import urljoin
from urllib.robotparser import RobotFileParser

import requests

BASE_URL = "https://books.toscrape.com/"
USER_AGENT = "books-tutorial/1.0 (+https://example.com/contact)"


# PROXIES, TIMEOUT and log are defined in the next sections and in the complete script.
def load_robots(session: requests.Session) -> RobotFileParser:
    """Fetch robots.txt through the same proxy and session as the pages."""
    robots_url = urljoin(BASE_URL, "/robots.txt")
    parser = RobotFileParser(robots_url)
    try:
        response = session.get(robots_url, proxies=PROXIES, timeout=TIMEOUT)
    except requests.RequestException as exc:
        log.warning("robots.txt unreachable (%s): not crawling", type(exc).__name__)
        parser.parse(["User-agent: *", "Disallow: /"])
        return parser
    if response.status_code in (401, 403) or response.status_code >= 500:
        parser.parse(["User-agent: *", "Disallow: /"])  # forbidden or unreachable: don't crawl
    elif response.status_code >= 400:
        parser.parse([])  # no robots.txt (for example 404): no rules apply
    else:
        parser.parse(response.content.decode("utf-8", errors="replace").splitlines())
    return parser

Syntax-checked

The script decodes robots.txt as UTF-8. It treats 401 and 403 like an unreachable file, which is stricter than the RFC and matches what Python's own RobotFileParser.read() does. Before each page, it calls robots.can_fetch(USER_AGENT, url), and it waits between pages for the longer of its own delay and any Crawl-delay the site publishes for your user agent (robots.crawl_delay(USER_AGENT); Crawl-delay isn't part of RFC 9309, but some sites use it).

One caveat: RFC 9309 says the longest matching rule wins, and the result of urllib.robotparser depends on your Python version. In our test, Python 3.13.16 and 3.14.8 used the longest match and understood * and $ wildcards, while Python 3.12 and earlier applied rules in file order and ignored the wildcards. On an older version, or for sites with complex rules, check the result with an RFC 9309 parser such as Protego, which Scrapy uses by default.

At the time of writing, Books to Scrape returns 404 for /robots.txt, so no rules apply and the crawl goes ahead.

robots.txt isn't the whole story. Read the site's terms of service as well, and look for an official API or data export: where one exists, it's usually the better source.

Collect politely

Respect robots.txt and each site's terms, keep request rates reasonable and back off when a site slows you down, don't collect personal data without a lawful basis, and don't access content behind logins or paywalls without authorization. netdash doesn't solve CAPTCHAs or other challenges, and your code shouldn't try to get past them: if a site rate-limits or challenges your requests, slow down, check its terms, or use its official API. Read how we expect netdash to be used.

Web scraping with Python through a proxy

The fetching code uses one requests.Session for the whole run, so connections are reused and every request carries your user agent. Three details matter:

Both proxy keys. The proxies dictionary needs an http and an https entry. Without the https key, requests sends HTTPS requests directly instead of through the proxy.
proxies= on every call. requests documents that proxies from environment variables overwrite Session.proxies, so the script passes proxies=PROXIES to each request instead of setting it on the session.
Timeouts. requests has no default timeout. The script uses 10 seconds to connect and 30 seconds to read.

This is the first part of the script. It runs on its own and prints the exit IP:

scrape_books.py (top)

import os
import secrets
from typing import Optional

import requests

NETDASH_USERNAME = os.environ["NETDASH_USERNAME"]
NETDASH_PASSWORD = os.environ["NETDASH_PASSWORD"]

USER_AGENT = "books-tutorial/1.0 (+https://example.com/contact)"
TIMEOUT = (10, 30)  # connect and read timeouts, in seconds


def proxy_url(country: str = "US", session_id: Optional[str] = None,
              ttl_minutes: int = 30, gateway: str = "scale-us.netdash.io:2000") -> str:
    """netdash proxy URL: rotating without session_id, sticky with one."""
    cc = country.upper()
    cc = "GB" if cc == "UK" else cc
    username = f"{NETDASH_USERNAME}-cc-{cc}"
    if session_id is not None:
        username += f"-session-{session_id}-ttl-{ttl_minutes}"
    return f"http://{username}:{NETDASH_PASSWORD}@{gateway}"


def new_session_id() -> str:
    return secrets.token_hex(4)  # 8 characters, letters and digits


# One sticky session for the whole run, so the site sees one client at a steady pace.
PROXY = proxy_url("US", session_id=new_session_id(), ttl_minutes=30)
PROXIES = {"http": PROXY, "https": PROXY}  # both keys, or HTTPS requests skip the proxy

session = requests.Session()
session.headers["User-Agent"] = USER_AGENT

# proxies= on each request: proxies from environment variables would overwrite session.proxies.
check = session.get("https://api.ipify.org?format=json", proxies=PROXIES, timeout=TIMEOUT)
check.raise_for_status()
print("exit IP for this run:", check.json()["ip"])

Checked against a local test proxy

proxy_url() builds the netdash username: -cc-US asks for an exit in the United States, and -session- with a random ID plus -ttl-30 keeps one exit IP for 30 minutes while that IP stays available. A sticky session isn't required for a catalogue crawl, but it keeps the run on one exit IP, so the site sees one client at a steady pace. If a run takes longer than the TTL, pick a longer lifetime, such as ttl_minutes=720.

The proxy URL starts with http:// although the site is HTTPS: netdash is an HTTP proxy, and HTTPS websites go through a CONNECT tunnel. Hosts, ports and the username syntax are in the proxy format reference, and session lifetimes in sticky and rotating sessions. If this is your first request through netdash, the quickstart covers credentials and the exit-IP check step by step.

Parse the data

Each book on a catalogue page sits in an article.product_pod element. The full title is in the title attribute of the link inside h3 (the visible link text is shortened), the price is in p.price_color, availability in p.availability, and the link to the next page in li.next a.

parse.py

import logging
from typing import Dict, List, Optional, Tuple
from urllib.parse import urljoin

from bs4 import BeautifulSoup

log = logging.getLogger("books")


def parse_page(html: bytes, page_url: str) -> Tuple[List[Dict[str, str]], Optional[str]]:
    """Return the books on one catalogue page and the URL of the next page, if any."""
    soup = BeautifulSoup(html, "html.parser")
    books = []
    for card in soup.select("article.product_pod"):
        link = card.select_one("h3 a")
        price = card.select_one("p.price_color")
        stock = card.select_one("p.availability")
        if link is None or price is None or stock is None:
            log.warning("skipping an incomplete entry on %s", page_url)
            continue
        books.append({
            "title": link.get("title") or link.get_text(strip=True),
            "price": price.get_text(strip=True),
            "availability": stock.get_text(strip=True),
            "url": urljoin(page_url, link.get("href", "")),
        })
    next_link = soup.select_one("li.next a")
    next_url = urljoin(page_url, next_link["href"]) if next_link and next_link.get("href") else None
    return books, next_url

Syntax-checked

Two details keep this robust:

Defensive selectors. select_one() returns None when nothing matches, so a missing field skips that entry with a log line instead of crashing the run. If every entry is skipped, the site's HTML has changed and the selectors need an update.
Bytes, not text. The script passes response.content to Beautiful Soup. This site sends text/html without a charset, so requests decodes response.text as ISO-8859-1 and the pound sign comes out as Â£. From the raw bytes, Beautiful Soup reads the page's own charset declaration and gets £ right.

Store the results

Keep values as the site shows them (£51.77, In stock) and convert them in a separate step. If the site changes a format, your raw data stays intact and only the conversion needs fixing.

store.py

import csv
import sqlite3
from typing import Dict, List

FIELDS = ["title", "price", "availability", "url"]


def save_csv(rows: List[Dict[str, str]], path: str = "books.csv") -> None:
    with open(path, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=FIELDS)
        writer.writeheader()
        writer.writerows(rows)


def save_sqlite(rows: List[Dict[str, str]], path: str = "books.db") -> None:
    db = sqlite3.connect(path)
    with db:  # commits on success, rolls back on error
        db.execute("CREATE TABLE IF NOT EXISTS books (title TEXT, price TEXT, availability TEXT, url TEXT PRIMARY KEY)")
        db.executemany("INSERT OR REPLACE INTO books VALUES (:title, :price, :availability, :url)", rows)
    db.close()

Syntax-checked

The complete script uses save_csv(). save_sqlite() takes the same rows and keys them by URL, so running the script again updates books instead of duplicating them.

Pace your requests

A polite scraper fetches one page at a time from a site, pauses between pages, and listens when the site pushes back:

A fixed pause between pages: 2 seconds here, or the site's Crawl-delay if that's longer.
429 and 503 ask the client to wait: 429 means too many requests (RFC 6585), and both can carry a Retry-After header (RFC 9110). The script waits as long as Retry-After asks, whether it's given in seconds or as a date, and stops if the wait is longer than 5 minutes.
Other 5xx responses and network errors get a capped backoff: 2, 4 and 8 seconds. If the fourth attempt fails too, the run stops and saves what it has.
403 means the site has refused the request. The script stops instead of retrying.
407 comes from the proxy, not the site, and means the credentials are wrong.
Certificate errors are raised, never worked around by turning off verification.

polite.py

import logging
import time
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime

import requests

MAX_WAIT_SECONDS = 300   # stop if the site asks for a longer pause than this
log = logging.getLogger("books")
# PROXIES and TIMEOUT come from the top of scrape_books.py.


class StopCrawl(Exception):
    """The site asked us to stop, or a page kept failing."""


def retry_after_seconds(response: requests.Response, default: float) -> float:
    """Read Retry-After as a number of seconds or as an HTTP date."""
    value = response.headers.get("Retry-After", "").strip()
    if value.isdigit():
        return float(value)
    try:
        when = parsedate_to_datetime(value)
    except (TypeError, ValueError):
        return default
    if when.tzinfo is None:
        when = when.replace(tzinfo=timezone.utc)
    return max(0.0, (when - datetime.now(timezone.utc)).total_seconds())


def polite_get(session: requests.Session, url: str, attempts: int = 4) -> requests.Response:
    """GET through the proxy. Waits on 429 and 503, backs off on 5xx and network errors, stops on 403."""
    for attempt in range(1, attempts + 1):
        wait = float(min(2 ** attempt, 30))
        try:
            response = session.get(url, proxies=PROXIES, timeout=TIMEOUT)
        except requests.exceptions.ProxyError as exc:
            if "407" in str(exc):
                raise SystemExit("Proxy authentication failed (407): check your netdash username and password.") from exc
            log.warning("proxy error on %s, retrying in %.0fs", url, wait)
        except requests.exceptions.SSLError:
            raise  # a certificate problem: fix it, never turn verification off
        except (requests.ConnectionError, requests.Timeout):
            log.warning("network error on %s, retrying in %.0fs", url, wait)
        else:
            status = response.status_code
            if status == 407:
                raise SystemExit("Proxy authentication failed (407): check your netdash username and password.")
            if status == 403:
                raise StopCrawl(f"403 on {url}: stopping. Check the site's terms or use its official API.")
            if status in (429, 503):
                wait = retry_after_seconds(response, default=60.0)
                if wait > MAX_WAIT_SECONDS:
                    raise StopCrawl(f"{url} asked for a {wait:.0f}s pause: stopping, try again later.")
                log.warning("%s on %s, waiting %.0fs as asked", status, url, wait)
            elif status >= 500:
                log.warning("%s on %s, retrying in %.0fs", status, url, wait)
            else:
                response.raise_for_status()  # any other 4xx is a problem with the URL: don't retry
                return response
        if attempt < attempts:
            time.sleep(wait)
    raise StopCrawl(f"giving up on {url} after {attempts} attempts")

Syntax-checked

netdash forwards your traffic unchanged. It doesn't solve CAPTCHAs, render JavaScript or retry requests. Targeting is country-level.

Retries and pacing are your code's job: netdash passes each request through as your client sent it and returns the site's response as it came.

The complete script

Everything together, in order. Save it as scrape_books.py:

scrape_books.py

"""Collect book titles, prices and availability from books.toscrape.com through netdash.

books.toscrape.com is a sandbox built for scraping practice. Before you point this script
at another site, read that site's robots.txt and terms.
"""
import csv
import logging
import os
import secrets
import time
from datetime import datetime, timezone
from email.utils import parsedate_to_datetime
from typing import Dict, List, Optional, Tuple
from urllib.parse import urljoin
from urllib.robotparser import RobotFileParser

import requests
from bs4 import BeautifulSoup

NETDASH_USERNAME = os.environ["NETDASH_USERNAME"]
NETDASH_PASSWORD = os.environ["NETDASH_PASSWORD"]

BASE_URL = "https://books.toscrape.com/"
START_URL = urljoin(BASE_URL, "catalogue/page-1.html")
USER_AGENT = "books-tutorial/1.0 (+https://example.com/contact)"
DELAY_SECONDS = 2.0      # pause between pages
MAX_PAGES = 5            # stop after this many catalogue pages
MAX_WAIT_SECONDS = 300   # stop if the site asks for a longer pause than this
TIMEOUT = (10, 30)       # connect and read timeouts, in seconds
OUTPUT = "books.csv"
FIELDS = ["title", "price", "availability", "url"]

log = logging.getLogger("books")


def proxy_url(country: str = "US", session_id: Optional[str] = None,
              ttl_minutes: int = 30, gateway: str = "scale-us.netdash.io:2000") -> str:
    """netdash proxy URL: rotating without session_id, sticky with one."""
    cc = country.upper()
    cc = "GB" if cc == "UK" else cc
    username = f"{NETDASH_USERNAME}-cc-{cc}"
    if session_id is not None:
        username += f"-session-{session_id}-ttl-{ttl_minutes}"
    return f"http://{username}:{NETDASH_PASSWORD}@{gateway}"


def new_session_id() -> str:
    return secrets.token_hex(4)  # 8 characters, letters and digits


# One sticky session for the whole run, so the site sees one client at a steady pace.
PROXY = proxy_url("US", session_id=new_session_id(), ttl_minutes=30)
PROXIES = {"http": PROXY, "https": PROXY}


class StopCrawl(Exception):
    """The site asked us to stop, or a page kept failing."""


def retry_after_seconds(response: requests.Response, default: float) -> float:
    """Read Retry-After as a number of seconds or as an HTTP date."""
    value = response.headers.get("Retry-After", "").strip()
    if value.isdigit():
        return float(value)
    try:
        when = parsedate_to_datetime(value)
    except (TypeError, ValueError):
        return default
    if when.tzinfo is None:
        when = when.replace(tzinfo=timezone.utc)
    return max(0.0, (when - datetime.now(timezone.utc)).total_seconds())


def polite_get(session: requests.Session, url: str, attempts: int = 4) -> requests.Response:
    """GET through the proxy. Waits on 429 and 503, backs off on 5xx and network errors, stops on 403."""
    for attempt in range(1, attempts + 1):
        wait = float(min(2 ** attempt, 30))
        try:
            response = session.get(url, proxies=PROXIES, timeout=TIMEOUT)
        except requests.exceptions.ProxyError as exc:
            if "407" in str(exc):
                raise SystemExit("Proxy authentication failed (407): check your netdash username and password.") from exc
            log.warning("proxy error on %s, retrying in %.0fs", url, wait)
        except requests.exceptions.SSLError:
            raise  # a certificate problem: fix it, never turn verification off
        except (requests.ConnectionError, requests.Timeout):
            log.warning("network error on %s, retrying in %.0fs", url, wait)
        else:
            status = response.status_code
            if status == 407:
                raise SystemExit("Proxy authentication failed (407): check your netdash username and password.")
            if status == 403:
                raise StopCrawl(f"403 on {url}: stopping. Check the site's terms or use its official API.")
            if status in (429, 503):
                wait = retry_after_seconds(response, default=60.0)
                if wait > MAX_WAIT_SECONDS:
                    raise StopCrawl(f"{url} asked for a {wait:.0f}s pause: stopping, try again later.")
                log.warning("%s on %s, waiting %.0fs as asked", status, url, wait)
            elif status >= 500:
                log.warning("%s on %s, retrying in %.0fs", status, url, wait)
            else:
                response.raise_for_status()  # any other 4xx is a problem with the URL: don't retry
                return response
        if attempt < attempts:
            time.sleep(wait)
    raise StopCrawl(f"giving up on {url} after {attempts} attempts")


def load_robots(session: requests.Session) -> RobotFileParser:
    """Fetch robots.txt through the same proxy and session as the pages."""
    robots_url = urljoin(BASE_URL, "/robots.txt")
    parser = RobotFileParser(robots_url)
    try:
        response = session.get(robots_url, proxies=PROXIES, timeout=TIMEOUT)
    except requests.RequestException as exc:
        log.warning("robots.txt unreachable (%s): not crawling", type(exc).__name__)
        parser.parse(["User-agent: *", "Disallow: /"])
        return parser
    if response.status_code in (401, 403) or response.status_code >= 500:
        parser.parse(["User-agent: *", "Disallow: /"])  # forbidden or unreachable: don't crawl
    elif response.status_code >= 400:
        parser.parse([])  # no robots.txt (for example 404): no rules apply
    else:
        parser.parse(response.content.decode("utf-8", errors="replace").splitlines())
    return parser


def parse_page(html: bytes, page_url: str) -> Tuple[List[Dict[str, str]], Optional[str]]:
    """Return the books on one catalogue page and the URL of the next page, if any."""
    soup = BeautifulSoup(html, "html.parser")
    books = []
    for card in soup.select("article.product_pod"):
        link = card.select_one("h3 a")
        price = card.select_one("p.price_color")
        stock = card.select_one("p.availability")
        if link is None or price is None or stock is None:
            log.warning("skipping an incomplete entry on %s", page_url)
            continue
        books.append({
            "title": link.get("title") or link.get_text(strip=True),
            "price": price.get_text(strip=True),
            "availability": stock.get_text(strip=True),
            "url": urljoin(page_url, link.get("href", "")),
        })
    next_link = soup.select_one("li.next a")
    next_url = urljoin(page_url, next_link["href"]) if next_link and next_link.get("href") else None
    return books, next_url


def save_csv(rows: List[Dict[str, str]], path: str = OUTPUT) -> None:
    with open(path, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=FIELDS)
        writer.writeheader()
        writer.writerows(rows)


def main() -> None:
    logging.basicConfig(level=logging.INFO, format="%(levelname)s %(message)s")
    session = requests.Session()
    session.headers["User-Agent"] = USER_AGENT

    check = polite_get(session, "https://api.ipify.org?format=json")
    log.info("exit IP for this run: %s", check.json()["ip"])

    robots = load_robots(session)
    delay = max(DELAY_SECONDS, robots.crawl_delay(USER_AGENT) or 0)

    rows: List[Dict[str, str]] = []
    url: Optional[str] = START_URL
    pages = 0
    while url and pages < MAX_PAGES:
        if not robots.can_fetch(USER_AGENT, url):
            log.info("robots.txt disallows %s, stopping", url)
            break
        try:
            response = polite_get(session, url)
        except (StopCrawl, requests.HTTPError) as exc:
            log.warning("%s", exc)
            break
        books, url = parse_page(response.content, response.url)
        rows.extend(books)
        pages += 1
        log.info("page %d: %d books", pages, len(books))
        if url:
            time.sleep(delay)

    save_csv(rows)
    log.info("saved %d books to %s", len(rows), OUTPUT)


if __name__ == "__main__":
    main()

Checked against a local test proxy

Run it with your credentials loaded:

Terminal

set -a; . ./.env; set +a
python scrape_books.py

The output looks like this (your exit IP will differ):

Output

INFO exit IP for this run: 203.0.113.10
INFO page 1: 20 books
INFO page 2: 20 books
INFO page 3: 20 books
INFO page 4: 20 books
INFO page 5: 20 books
INFO saved 100 books to books.csv

MAX_PAGES keeps the first run short. The catalogue has more pages; raise the limit once you're happy with the output, and leave the delay where it is.

Scaling up

Scale across sites before you scale within one. Ten sites at one request every few seconds each is a very different load from one site at ten parallel requests.

httpx with asyncio. httpx.AsyncClient(proxy=...) takes the same proxy URL (httpx 0.28 and later use proxy=, not proxies=). Bound the work with an asyncio.Semaphore per site, so concurrency grows with the number of sites, not the load on any one of them. The Python requests proxy setup guide has sync and async httpx examples.
Scrapy. For larger crawls, Scrapy handles scheduling, retries and politeness settings for you. Projects created with scrapy startproject obey robots.txt by default (ROBOTSTXT_OBEY), and the AutoThrottle extension (AUTOTHROTTLE_ENABLED) adjusts delays to the site's response times. See Scrapy proxy setup for the netdash configuration.
One session per worker. If several workers each need a stable exit IP, give each its own session ID. Workers that share an ID share the exit IP.

netdash sets no per-account concurrency limit, so the limit is yours to set: decide it per website, from the site's terms, its Crawl-delay and how it responds. For other languages and tools, see all proxy integration guides.

Troubleshooting

Symptom	Cause	Fix
`Proxy authentication failed (407)`	Wrong password, or a typo in the username or its parameters	Copy the credentials from the dashboard again; check `-cc-` and `-session-…-ttl-…`
The exit-IP check prints your own IP	The `https` key is missing from `proxies`, so HTTPS requests skip the proxy	Set both `http` and `https`
Prices show `Â£` instead of `£`	Parsing `response.text`, which requests decoded as ISO-8859-1	Pass `response.content` to Beautiful Soup
Every entry is skipped as incomplete	The site's HTML changed	Inspect a page in your browser's developer tools and update the selectors
`robots.txt unreachable` and nothing is collected	The robots.txt request failed, so the script treats the site as off limits	Check the exit-IP output and your network, then try again later
`JSONDecodeError` on the exit-IP check	The test URL is missing `?format=json`	Use `https://api.ipify.org?format=json`

More causes and fixes are in troubleshooting. Stuck? Email support.

Next steps

Python requests proxy setup
requests and httpx in detail: authentication, sticky sessions, country targeting and errors.
Scrapy proxy setup
Request meta or a downloader middleware, with polite crawl settings.
Web data collection
Workflows, configuration and responsible-use guidance for collecting public web data.

Web scraping with Python and proxies: a complete tutorial

What you will build

Set up the project

Check robots.txt and the site's terms

Web scraping with Python through a proxy

Parse the data

Store the results

Pace your requests

The complete script

Scaling up

Troubleshooting

Next steps

Sources

Privacy choices