Extract main article text with BeautifulSoup

Raw HTML is full of noise: nav menus, cookie banners, sidebars, footers. If you feed all of that to the LLM, you waste tokens and pollute the context. BeautifulSoup lets us strip the noise and keep the article.

mini-perplexity.ipynb
python
from bs4 import BeautifulSoup

def extract_content_from_url(url, session):
    """Extract main text content from a given URL."""
    try:
        response = session.get(url, timeout=8)
        soup = BeautifulSoup(response.content, "html.parser")

        # Remove unwanted tags
        for element in soup(["script", "style", "nav", "header", "footer"]):
            element.decompose()

        # Try to find the main content area
        main_content = (
            soup.find("main")
            or soup.find("article")
            or soup.find("div", class_=lambda x: x and any(
                keyword in str(x).lower()
                for keyword in ["content", "main", "post", "article"]
            ))
        )

        text = (main_content or soup).get_text(separator=" ", strip=True)
        return text[:2500]

    except Exception as e:
        return f"Error extracting content: {str(e)}"

First we drop tags that never carry article content. Then we try to find a main, article, or content-looking div. If none match, we fall back to the whole page. The 2500 character cap keeps the LLM context bill predictable.

mini-perplexity.ipynb
python
def content_extraction_node(state):
    """Extract article content from search results."""
    state["current_step"] = "extracting_content"

    session = requests.Session()
    session.headers.update({
        "User-Agent": "Mozilla/5.0 ..."
    })

    extracted_content = []
    sources = []

    for i, result in enumerate(state["search_results"][:3]):  # top 3
        title = result.get("title", "No title")
        url = result.get("href", "")
        snippet = result.get("body", "")

        content = extract_content_from_url(url, session)

        extracted_content.append({
            "title": title,
            "url": url,
            "snippet": snippet,
            "content": content,
            "source_id": i + 1,
        })
        sources.append({"title": title, "url": url, "snippet": snippet})

    state["extracted_content"] = extracted_content
    state["sources"] = sources
    return state

We cap at the top 3 results. More than that and the synthesis context gets noisy without improving the answer. Each entry gets a source_id that the LLM will use for citations.

Different sites use different conventions. Modern sites use the semantic main tag. Blogs usually wrap posts in article. Older sites rely on divs with class names like post-content or main-article. Checking in that order gives you a clean hit on most pages and falls back gracefully when none match. Boilerplate extraction is inherently fuzzy. A small chain of heuristics outperforms any single rule.

Fill in the blanks: Fill in the tags we strip

Loading practice…

Checkpoint: Retrieval phase checkpoint

Loading practice…