Same questions, both retrievers

Intuition about when graphs beat vectors only develops when you watch them fail on the same question. The compare endpoint runs both retrievers independently and returns both answers with their raw context. Now you can point at specific questions where one approach clearly wins.

Side-by-side comparison flow

One question fans out to both retrievers, and each answer comes back with the raw evidence it used.

graph_rag.py
python
def compare(self, question, top_k=5):
    graph_answer = self.ask(question)
    vector_chunks = []
    vector_answer = ""
    if self.vector_store is not None:
        vector_chunks = self.vector_store.query(question, top_k=top_k)
        if vector_chunks:
            llm = _build_chat_model()
            joined = "\n---\n".join(vector_chunks)
            prompt = (
                "Answer the question using ONLY the context below. "
                "If the answer is not in the context, say you do not know.\n\n"
                f"Context:\n{joined}\n\nQuestion: {question}\nAnswer:"
            )
            vector_answer = llm.invoke(prompt).content
    return {
        "question": question,
        "graph_rag": graph_answer,
        "vector_rag": {"answer": vector_answer, "context": vector_chunks},
    }

The vector side runs its own mini RAG chain: retrieve chunks, stuff them into a prompt, ask the LLM. This is the simplest possible vector RAG so the comparison shows Graph RAG versus a clean baseline, not Graph RAG versus a fancy hybrid that already uses graph ideas.

router.py
python
@router.post("/compare", response_model=CompareResponse)
async def compare(request: CompareRequest):
    """Answer the same question with Graph RAG and Vector RAG side-by-side."""
    try:
        result = get_graph_rag().compare(request.question, top_k=request.top_k)
    except GraphStoreUnavailable as e:
        raise _neo4j_unreachable(e)
    except Exception as e:
        raise HTTPException(status_code=500, detail=f"Comparison failed: {e}")
    return CompareResponse(
        question=result["question"],
        graph_rag=AskResponse(**result["graph_rag"]),
        vector_rag=result["vector_rag"],
    )

The route returns both answers with their full context so you can inspect which retriever saw what. When you are deciding which mode to route a user question to, this endpoint is your evaluation harness.

Validation checklist: Run three comparisons

Loading practice…