Cleaning real web content

Real websites are noisy. Headers, navigation, footers, scripts, and ads all live in the HTML alongside the content you actually want. If you embed that noise, every query has to fight through it. Clean ingestion is the foundation of every good RAG system.

The ingestion pipeline

From raw URL to clean, searchable chunks.

rag_utils.py
python
class WebScraper:
    """Extracts clean text content from websites"""

    def __init__(self):
        self.client = httpx.AsyncClient(timeout=10.0, follow_redirects=True)
        self.text_splitter = RecursiveCharacterTextSplitter(
            chunk_size=500,
            chunk_overlap=50,
            separators=["\n\n", "\n", " ", ""]
        )

    async def scrape_url(self, url: str):
        response = await self.client.get(url)
        response.raise_for_status()

        soup = BeautifulSoup(response.content, "html.parser")

        # Strip the boilerplate that poisons retrieval
        for element in soup(["script", "style", "nav", "footer", "header", "aside"]):
            element.decompose()

        # Prefer semantic containers over the whole body
        main_content = soup.find("main") or soup.find("article") or soup.body
        text = main_content.get_text(separator=" ", strip=True)
        return self._clean_text(text)

The decompose() calls are the key line of defense. Every script, nav, and footer that makes it into your index will show up as noise in future queries.

500 characters is roughly a paragraph. Small enough to embed precisely, large enough to carry a complete thought. The 50-character overlap stops ideas from getting cut in half at the boundary between chunks. These defaults work well for most web content, but you should tune them for your own sources. Legal documents want bigger chunks. Chat logs want smaller ones.

Quiz: Quiz

Loading practice…