Collecting web data for AI training, responsibly

AI training data scraping means collecting web pages at scale to train, fine-tune or evaluate models. Technically it's an ordinary crawl. The difference is what happens afterwards: the data may end up in a model that's hard to change later, so decisions about permission, copyright and personal data carry more weight than they do for a price check.

Whether you can collect a given dataset depends on the site's terms, the data and your jurisdiction; get legal advice for your case. This guide is general information, not legal advice. It walks through the steps a careful team takes before and during collection, with Python examples that use requests and a netdash proxy.

Start with purpose and scope

Write a short data plan before the first request. It keeps the crawl narrow, and it's the first thing you'll need if a site owner, a rights holder or a regulator asks what you collected and why.

• Purpose: what the model is for, and why it needs web data.
• Sources: which sites or sections, and why those.
• Data types: text, images, code or metadata, and whether personal data could be included.
• Retention: how long raw pages are kept, and when they're deleted.
• Access: who can see the raw pages and the datasets built from them.
• Contact: who answers questions from site owners, and where they can reach you.

Read robots.txt for your crawler

The Robots Exclusion Protocol (RFC 9309) is the standard way for a site to tell crawlers what they may fetch. The basics:

• The file lives at /robots.txt on each host. Rules are grouped by user agent, and a crawler follows the group whose name matches its product token (case-insensitive), or the * group if none does.
• The most specific matching rule wins; on a tie, Allow wins over Disallow.
• A 4xx response means there are no rules. A 5xx response or an unreachable file means the crawler must assume it may not crawl anything.
• Crawlers shouldn't use a cached copy for more than 24 hours.
• The rules aren't access authorization, and an allowed path isn't a license to use the content for any purpose.

Name your crawler honestly. Give it a stable product token, put a URL in the user agent where site owners can read what it does and contact you, and don't borrow another crawler's name. A stable name is what lets a site owner write rules for you.

Rules aimed at AI training crawlers

Sites can also publish rules for the crawlers that AI companies document for training:

• GPTBot: OpenAI's crawler for content that may be used to train its generative AI foundation models. OpenAI documents separate agents for search (OAI-SearchBot) and for visits a user asks for (ChatGPT-User).
• ClaudeBot: Anthropic's crawler for content that may contribute to model training. Anthropic documents Claude-User and Claude-SearchBot separately.
• Google-Extended: a product token rather than a separate crawler. It controls whether content Google crawls may be used to train future Gemini models and for grounding, and Google states that it doesn't affect inclusion or ranking in Google Search.

These rules don't name your crawler, but they tell you what a site has said about AI training. If a site disallows GPTBot, ClaudeBot or Google-Extended while allowing other crawlers, we recommend treating that as an opt-out for your training crawl too, or at least flagging the site for review. Some sites also state limits on AI use in their terms, or in comments inside robots.txt, without naming any AI crawler in the rules, so the rules alone don't settle the question.

First, a session that sends your crawler's user agent through a netdash proxy with an exit in Germany. It runs on its own and prints the exit IP:

import os
import secrets
from typing import Optional

import requests

NETDASH_USERNAME = os.environ["NETDASH_USERNAME"]
NETDASH_PASSWORD = os.environ["NETDASH_PASSWORD"]

USER_AGENT = "ExampleDataBot/1.0 (+https://example.com/bot)"


def proxy_url(country: str = "US", session_id: Optional[str] = None,
              ttl_minutes: int = 30, gateway: str = "scale-us.netdash.io:2000") -> str:
    """netdash proxy URL: rotating without session_id, sticky with one."""
    cc = country.upper()
    cc = "GB" if cc == "UK" else cc
    username = f"{NETDASH_USERNAME}-cc-{cc}"
    if session_id is not None:
        username += f"-session-{session_id}-ttl-{ttl_minutes}"
    return f"http://{username}:{NETDASH_PASSWORD}@{gateway}"


def new_session_id() -> str:
    return secrets.token_hex(4)  # 8 characters, letters and digits


# German exit IP, one sticky session for this collection run.
proxy = proxy_url("DE", session_id=new_session_id(), gateway="scale-eu.netdash.io:2000")
PROXIES = {"http": proxy, "https": proxy}

session = requests.Session()
session.headers["User-Agent"] = USER_AGENT

check = session.get("https://api.ipify.org?format=json", proxies=PROXIES, timeout=30)
check.raise_for_status()
print("exit IP:", check.json()["ip"])

Replace ExampleDataBot and the URL with your crawler's name and a page that describes it. Then fetch robots.txt through the same session and check it for your crawler and for the documented AI training tokens:

from typing import Dict, List
from urllib.parse import urlsplit
from urllib.robotparser import RobotFileParser

import requests

# Product tokens that OpenAI, Anthropic and Google document for AI training.
AI_TRAINING_TOKENS = ["GPTBot", "ClaudeBot", "Google-Extended"]


def load_robots(session: requests.Session, proxies: Dict[str, str], url: str) -> RobotFileParser:
    """Fetch and parse robots.txt for the site that serves `url`."""
    parts = urlsplit(url)
    parser = RobotFileParser(f"{parts.scheme}://{parts.netloc}/robots.txt")
    try:
        response = session.get(parser.url, proxies=proxies, timeout=30)
    except requests.RequestException:
        response = None
    if response is None or response.status_code >= 500 or response.status_code in (401, 403):
        parser.parse(["User-agent: *", "Disallow: /"])  # unreachable or forbidden: don't crawl
    elif response.status_code >= 400:
        parser.parse([])  # no robots.txt: no rules apply
    else:
        parser.parse(response.content.decode("utf-8", errors="replace").splitlines())
    return parser


def robots_decision(parser: RobotFileParser, user_agent: str, url: str) -> Dict[str, object]:
    """What robots.txt says for your crawler, and which AI training tokens it shuts out."""
    opted_out: List[str] = [t for t in AI_TRAINING_TOKENS if not parser.can_fetch(t, url)]
    return {"allowed": parser.can_fetch(user_agent, url), "ai_training_opt_out": opted_out}


# With the session and PROXIES from session.py:
# decision = robots_decision(load_robots(session, PROXIES, url), USER_AGENT, url)
# Skip the URL if not decision["allowed"]; review the site if decision["ai_training_opt_out"].

Fetching robots.txt through the proxy means the check uses the same client and exit IP as the crawl. The function treats 401 and 403 like an unreachable file, which is stricter than RFC 9309. In Python 3.12 and earlier, urllib.robotparser applies rules in file order and ignores * and $ wildcards (in our test, Python 3.13.16 and 3.14.8 used the longest match instead), so on an older version, or for sites with complex rules, compare the result with an RFC 9309 parser such as Protego. The Python scraping tutorial walks through the same check line by line.

Keep NETDASH_USERNAME and NETDASH_PASSWORD in a .env file or your secret manager, never in source control or in a chat with an assistant. For more on configuring requests and httpx, see the Python requests proxy setup guide.

Check the site's terms and licenses

robots.txt says what a crawler may fetch. Terms and licenses say what you may do with what you fetched.

• Terms of service. Site terms can restrict automated access, commercial reuse or AI training. Our own Terms of Service prohibit using netdash for "Scraping or collecting data from websites that explicitly prohibit such actions in their Terms of Service, where such prohibition is legally enforceable." Record the date you reviewed each site's terms.
• Content licenses. Open licenses grant specific permissions with conditions, such as attribution or share-alike, and those conditions travel with the data. A page can declare its license with a rel="license" link, which the manifest code below records.
• Official APIs and data exports. Where a site offers an API, a bulk export or a dataset license, it usually comes with clearer terms and puts less load on the site than crawling HTML.

Copyright and opt-outs

Copyright rules for text and data mining differ by jurisdiction, and many questions about AI training are still open. Without drawing legal conclusions, two points are worth knowing:

• The EU has an opt-out model. Article 4 of Directive (EU) 2019/790 requires member states to provide an exception for reproductions and extractions of lawfully accessible works for text and data mining. The exception applies only if rights holders haven't expressly reserved that use in an appropriate manner, such as machine-readable means for content made publicly available online. Copies made under it may be kept for as long as necessary for the purposes of the mining.
• Elsewhere, the position varies and is unsettled in places. Don't assume an exception in one country covers a dataset collected from, or used in, another.

In practice, look for reservations in robots.txt rules for AI crawlers, in the site's terms and in page metadata. Record what you found for each source, exclude content where a reservation applies, and ask your lawyer about the rest.

Personal data

Web pages contain personal data: names, contact details, photos, posts and reviews. If your sources may include it, data protection law applies to what you collect. Using the EU's GDPR as an example framework:

• Lawful basis. Processing personal data needs one of the bases in Article 6(1). For collected web data, a common candidate is legitimate interests (Article 6(1)(f)), which requires weighing your interests against the rights and interests of the people concerned.
• Minimization and retention. Article 5 includes data minimization (only what's necessary for the purpose) and storage limitation (identifiable data kept no longer than necessary).
• Transparency. Article 14 sets out what people must be told when their data wasn't obtained from them, with limited exceptions.
• Erasure. Article 17 gives people a right to erasure in the circumstances it lists, so you need a way to find and delete their data, including in datasets you've built from it.

Whatever framework applies, these practical steps are a sensible baseline: leave out page sections such as comments and profiles unless your purpose needs them, filter contact details and other identifiers before storage, keep a deletion path keyed by URL and content hash, and don't collect from areas behind logins.

Collect at a pace the site can handle

Training crawls tend to be large, so politeness has to be built in rather than added later:

• Crawl one site at a time per worker, with a fixed pause between requests, and respect Crawl-delay where a site publishes one.
• Treat 429 (too many requests, RFC 6585) and 503 as signals to slow down: pause the whole site for as long as Retry-After asks (RFC 9110), then put it back in the schedule.
• Use conditional requests. Send If-None-Match with the page's ETag and If-Modified-Since with its Last-Modified date, and an unchanged page comes back as a short 304 response instead of the full page.
• Cache what you've fetched, and don't refetch pages you already have without a reason.

import time
from typing import Dict, Optional

import requests

DELAY_SECONDS = 5.0  # pause after every request to the same site


class SlowDown(Exception):
    """The site asked for a pause (429 or 503). Reschedule the site, don't retry in a loop."""

    def __init__(self, url: str, seconds: int):
        super().__init__(f"{url}: wait {seconds}s")
        self.seconds = seconds


def fetch_if_changed(session: requests.Session, proxies: Dict[str, str], url: str,
                     validators: Dict[str, Dict[str, str]]) -> Optional[requests.Response]:
    """Conditional GET. Returns None if the page is unchanged since the last fetch."""
    headers = {}
    known = validators.get(url, {})
    if known.get("etag"):
        headers["If-None-Match"] = known["etag"]
    if known.get("last_modified"):
        headers["If-Modified-Since"] = known["last_modified"]
    try:
        response = session.get(url, headers=headers, proxies=proxies, timeout=30)
    finally:
        time.sleep(DELAY_SECONDS)
    if response.status_code in (429, 503):
        retry_after = response.headers.get("Retry-After", "")
        raise SlowDown(url, int(retry_after) if retry_after.isdigit() else 300)
    if response.status_code == 304:
        return None
    response.raise_for_status()  # 403 and other errors stop here: review before trying again
    validators[url] = {
        "etag": response.headers.get("ETag", ""),
        "last_modified": response.headers.get("Last-Modified", ""),
    }
    return response

On SlowDown, pause the whole site, not just the URL, and reschedule it. On a 403 or a challenge page, stop collecting from that site and review: slow down, check its terms, or use its official API. netdash doesn't solve CAPTCHAs or other challenges, and your crawler shouldn't try.

netdash sets no per-account concurrency limit, so the limit is yours to set, per website, from the site's terms, its Crawl-delay and how it responds.

Keep provenance records

For every document you keep, record where and when it came from and what you checked. A JSON Lines manifest, one JSON object per line, is easy to append to, diff and filter:

import hashlib
import json
from datetime import datetime, timezone
from typing import Dict, Optional
from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup


def license_link(html: bytes, base_url: str) -> Optional[str]:
    """The page's rel="license" link, if it declares one."""
    tag = BeautifulSoup(html, "html.parser").select_one('[rel~="license"][href]')
    return urljoin(base_url, tag["href"]) if tag else None


def record(manifest_path: str, response: requests.Response, robots: Dict[str, object],
           exit_country: str) -> None:
    """Append one JSON line per stored document."""
    entry = {
        "url": response.url,
        "fetched_at": datetime.now(timezone.utc).isoformat(timespec="seconds"),
        "status": response.status_code,
        "exit_country": exit_country,
        "robots_allowed": robots["allowed"],
        "ai_training_opt_out": robots["ai_training_opt_out"],
        "license_link": license_link(response.content, response.url),
        "sha256": hashlib.sha256(response.content).hexdigest(),
    }
    with open(manifest_path, "a", encoding="utf-8") as f:
        f.write(json.dumps(entry, ensure_ascii=False) + "\n")

One line of the manifest looks like this:

{"url": "https://example.com/articles/42", "fetched_at": "2026-10-09T11:05:59+00:00", "status": 200, "exit_country": "DE", "robots_allowed": true, "ai_training_opt_out": [], "license_link": "https://creativecommons.org/licenses/by/4.0/", "sha256": "9f2c…"}

With a manifest you can answer a deletion request (find every document from a URL or with a given hash), honor a later opt-out (re-check robots.txt and remove affected documents), remove duplicates by hash, and rebuild any version of a dataset from its sources. Store the date you reviewed each site's terms alongside it.

Where proxies fit

A proxy changes where your requests come from, not what you're allowed to collect. It's useful in two cases:

• Country-specific public content. Some sites serve different language versions, prices or content to visitors in different countries. With netdash you choose the exit country with -cc- and a two-letter ISO code; targeting is country-level.
• Consistent sessions. A sticky session keeps one exit IP for a collection run, for the TTL you set and while that IP stays available, so the site sees one client at a steady pace rather than the same load spread across addresses.

It doesn't fit, and you shouldn't use it, for anything a site has decided against: content behind logins or paywalls you aren't authorized to access, getting around rate limits, or getting past challenges. Section 2 of our Terms of Service covers use "for legitimate business purposes, including data collection, market research, and ad verification", and the norms below apply to every crawl.

For agent workflows that browse rather than crawl, see proxies for AI agents. For scheduled collection of public pages, see web data collection. The proxy format reference lists every host, port and username option.

For a collection larger than 50,000 GB in one order, or for custom terms, talk to sales.

A pre-collection checklist

Related

• Proxies for AI agents

  An exit country per task, a sticky session per run, and where an agent should stop and hand off.

• Web data collection

  Workflows, configuration and responsible-use guidance for collecting public web data.

• Scrapy proxy setup

  Request meta or a downloader middleware, with polite crawl settings.