Tradeoffs of native-tool RAG
Native tools are fast to ship but they are opinionated. Understanding the tradeoffs is what keeps you from reaching for them every time and then being stuck when your product outgrows them.
Matching exercise: Match each behavior to the right flavor
Loading practice…
latency-notes.md
text
Approximate budget for one question:
/api/chat (no retrieval)
Latency: LLM first token, then stream.
/api/rag (urlContext)
Latency: provider fetches URL, processes content, starts generating.
Typically a noticeable bump over direct chat, measured in hundreds of milliseconds.
/api/rag2 (custom pipeline)
Latency: scrape + split + embed + search + LLM.
Easily measured in whole seconds, especially on cold starts.
These numbers move with network and cache state. Measure your own before arguing.Use these as mental defaults until you benchmark the real thing yourself. Latency always has to be measured on your own data.
No. urlContext fetches the URL like any anonymous visitor. If the page requires authentication, session cookies, or a browser-rendered SPA, the tool usually falls short. That is a concrete case where you drop to the custom pipeline and fetch the page yourself with real credentials.
Quiz: Quiz
Loading practice…
Checkpoint: Native-tool checkpoint
Loading practice…