Fetch pages politely

Search gave you URLs. Now you need the actual content. A naive requests.get works for some sites but fails on others because many servers reject clients that look like bots. A realistic User-Agent, a session for connection reuse, and a short timeout fix most of it.

Always pass a timeout. Without it, one slow server will stall your whole pipeline for minutes. Eight seconds is a reasonable ceiling: long enough for real pages to load, short enough that a dead server does not take you down with it.

Retrying blindly hurts more than it helps. If a site returns 429, retrying immediately just gets you blocked harder. If it returns 500, the issue is on their end and a retry probably fails the same way. For a research pipeline, the cleaner answer is to skip the bad URL, log it, and move to the next result. The LLM can still synthesize a good answer from the remaining sources.