Scrape and split

Custom pipelines start the same way: you need the raw content, and you need it in chunks the embedder and the model can handle. LangChain ships battle-tested primitives for both steps.

The custom pipeline end to end

Four LangChain primitives and two providers, composed into one route.

src/app/api/rag2/route.ts
typescript
import { CheerioWebBaseLoader } from '@langchain/community/document_loaders/web/cheerio';
import { RecursiveCharacterTextSplitter } from '@langchain/textsplitters';

const cheerioLoader = new CheerioWebBaseLoader(remoteUrl, { selector: 'p' });
const htmlText = await cheerioLoader.load();

const splitter = new RecursiveCharacterTextSplitter({
  chunkSize: 3000,
  chunkOverlap: 400,
});

const chunkDocuments = await splitter.splitDocuments(htmlText);

CheerioWebBaseLoader fetches the page and keeps only the p tags, which drops most of the navigation noise. RecursiveCharacterTextSplitter then breaks the text into overlapping chunks so retrieval can span paragraph boundaries.

Chunk size depends on the content and the model. Long-form articles benefit from larger chunks because a single idea often spans paragraphs. Overlap prevents important sentences from being chopped in half between chunks. Smaller chunks make sense for dense technical docs where each paragraph is self contained. When in doubt, start in the 1000 to 3000 range and measure retrieval quality.

Quiz: Quiz

Loading practice…