Rate limits, robots.txt, and content truncation

The pipeline works on your laptop. Running it against the open web at scale is a different beast. Sites push back with rate limits, some forbid scraping entirely, and the LLM context has a hard ceiling. Three small habits keep you a good web citizen and keep the bill predictable.

Three hygiene habits

The non-negotiables for any scraping pipeline.

mini-perplexity.ipynb
python
import time
import urllib.robotparser

def is_allowed(url: str, user_agent: str) -> bool:
    """Check robots.txt before fetching."""
    from urllib.parse import urlparse
    parsed = urlparse(url)
    robots_url = f"{parsed.scheme}://{parsed.netloc}/robots.txt"
    rp = urllib.robotparser.RobotFileParser()
    try:
        rp.set_url(robots_url)
        rp.read()
        return rp.can_fetch(user_agent, url)
    except Exception:
        return True  # be permissive on parse errors

# Inside the extraction loop
for i, result in enumerate(state["search_results"][:3]):
    url = result.get("href", "")
    if not is_allowed(url, "mini-perplexity-bot"):
        print(f"Skipping {url} due to robots.txt")
        continue

    content = extract_content_from_url(url, session)
    # Per-page cap keeps total tokens bounded
    content = content[:2500]

    time.sleep(0.5)  # small courtesy delay between fetches

Three cheap habits that pay off every time. Robots.txt checks cost one extra request per host. Sleep between fetches costs half a second. Per-page truncation costs zero and saves tokens on every call.

In most jurisdictions, robots.txt is a strong norm rather than a hard law. Ignoring it rarely ends in court, but it often ends with your IP blocked, your User-Agent flagged, or your relationship with a data provider damaged. Treat it as a polite contract. If a site says no, find another source. The research assistant is better off with three willing sources than four where one got you banned.

Quiz: Quiz

Loading practice…