Multi-hop RAG: Query decomposition with LangGraph

You built a RAG system and it handles simple questions well. Then a user asks: “What is the competitor to the product mentioned in our latest press release?” The retriever finds the press release, the model names the product, and the answer stops there. Nothing ever searched for the competitor.

Multi-hop RAG is a retrieval pattern that answers questions needing more than one lookup by splitting the question into sub-questions and running one retrieval per hop. Each hop’s answer feeds the next search. This guide builds it in LangGraph: a planner that decomposes the query, an executor that loops over the hops, and a final answer node.

Key takeaways

• A multi-hop question needs facts from more than one document, and the second search depends on what the first search found.
• Single-shot RAG fails on multi-hop queries because it runs one search with the original question and never searches for the missing link.
• Query decomposition uses an LLM to split the question into simple, searchable sub-questions that run as separate retrieval hops.
• Each later sub-question must be rewritten with the facts from earlier hops, or it searches for “the product mentioned” instead of the product’s name.
• In LangGraph, a planner node, an executor node with a conditional loop edge, and a final answer node give you multi-hop RAG with a hop cap.

What are multi-hop questions?

A multi-hop question is one whose answer needs facts from different places, where finding one fact tells you what to search for next. “Who is the competitor to the product in our latest press release?” needs the product name first, then a search about that product.

Single-hop questions are different. “What is our refund policy?” lives in one chunk, and one search finds it. The MultiHop-RAG benchmark built a dataset of multi-hop queries and found that existing RAG methods retrieve and answer them poorly.

Why does one-shot retrieval fail on multi-hop queries?

Because the retriever embeds the original question once, and the missing fact is not in that question. Trace the failure:

1. The question is embedded and matched against the corpus.
2. The press release scores highest, because the question mentions a press release.
3. The model reads it and learns the product is “Model-V”.
4. Nothing searches for “competitors of Model-V”, so the answer is a guess or “I don’t know”.

Raising top-k rarely fixes this. The competitor document shares few words with the original question, so it does not rank near the top. For how k behaves on single-hop questions, see tuning retriever top-k.

What is query decomposition in RAG?

Query decomposition is the step that turns one complex question into an ordered list of simple sub-questions, each answerable by one search. An LLM does the split, because it can read which part of the question depends on which.

For the press release question, a good plan is:

1. “What product is mentioned in the latest press release?”
2. “Who is the main competitor to that product?”

The second sub-question still says “that product”. Before it reaches the retriever, it must be rewritten with the answer from the first hop. That rewrite makes the hops a chain rather than unrelated searches. Self-Ask and IRCoT use the same idea: ask a follow-up question, answer it with retrieval, and use that answer to form the next step.

How do you build multi-hop RAG in LangGraph?

You build a graph with three nodes and one loop. Plan the sub-questions, run one hop per pass through the executor, and generate the final answer once the to-do list is empty.

Step 1: the graph state

The state holds a to-do list of sub-questions and a done list of facts. The answers field uses the operator.add reducer, so each hop appends a fact instead of replacing the list.

# filename: multihop/state.py
# description: Graph state for multi-hop RAG: questions to answer and facts found.
from operator import add
from typing import Annotated
from typing_extensions import TypedDict

MAX_HOPS = 4  # cap the loop so a bad plan cannot run forever


class GraphState(TypedDict):
    original_query: str
    questions: list[str]                # to-do: sub-questions not yet answered
    answers: Annotated[list[str], add]  # done: one fact per hop, appended
    final_answer: str

Step 2: the planner node

The planner asks the model for an ordered list of sub-questions. Structured Outputs with a JSON schema makes the reply parse into a list every time. If the plan comes back empty, the planner falls back to the original question as a single hop.

# filename: multihop/planner.py
# description: Planner node that decomposes the query into sub-questions.
import json
from openai import OpenAI

client = OpenAI()
MODEL = 'gpt-5-mini'  # any chat model that supports Structured Outputs

PLAN_SCHEMA = {
    'name': 'plan',
    'strict': True,
    'schema': {
        'type': 'object',
        'properties': {'questions': {'type': 'array', 'items': {'type': 'string'}}},
        'required': ['questions'],
        'additionalProperties': False,
    },
}


def plan_queries(state: GraphState) -> dict:
    prompt = (
        'Break the question into the shortest ordered list of simple, searchable '
        'sub-questions. A later sub-question may refer to the answer of an earlier one.\n\n'
        f"Question: {state['original_query']}"
    )
    response = client.chat.completions.create(
        model=MODEL,
        messages=[{'role': 'user', 'content': prompt}],
        response_format={'type': 'json_schema', 'json_schema': PLAN_SCHEMA},
    )
    questions = json.loads(response.choices[0].message.content)['questions']
    return {'questions': (questions or [state['original_query']])[:MAX_HOPS]}

Step 3: the executor node

The executor takes the first sub-question and rewrites it with the facts found so far. Then it runs your existing RAG pipeline on it and appends the result. Most tutorials skip the rewrite, and it is the step that makes the second hop search for “Model-V” instead of “that product”.

# filename: multihop/executor.py
# description: Executor node that runs one retrieval hop per pass.
def simple_rag_pipeline(query: str) -> str:
    # Retrieve from your vector store and answer from the top chunks.
    ...


def rewrite_with_facts(question: str, facts: list[str]) -> str:
    if not facts:
        return question
    prompt = (
        'Rewrite the question so it stands alone. Replace references such as '
        '"that product" with the names given in the facts. Return only the question.\n\n'
        'Facts:\n' + '\n'.join(facts) + f'\n\nQuestion: {question}'
    )
    response = client.chat.completions.create(
        model=MODEL, messages=[{'role': 'user', 'content': prompt}]
    )
    return response.choices[0].message.content.strip()


def execute_search(state: GraphState) -> dict:
    question = rewrite_with_facts(state['questions'][0], state['answers'])
    answer = simple_rag_pipeline(question)
    return {
        'questions': state['questions'][1:],  # pop the hop we just ran
        'answers': [f'Q: {question}\nA: {answer}'],  # appended by the reducer
    }

Step 4: the final answer and the loop

The final node answers the original question from the collected facts only. The conditional edge sends the executor back to itself while sub-questions remain. add_edge(START, ...) is the current way to set the entry point in LangGraph.

# filename: multihop/graph.py
# description: Wire the planner, the executor loop and the final answer.
from langgraph.graph import END, START, StateGraph


def generate_final_answer(state: GraphState) -> dict:
    facts = '\n\n'.join(state['answers'])
    prompt = (
        'Answer the question using only these facts. '
        'If they are not enough, say what is missing.\n\n'
        f"Facts:\n{facts}\n\nQuestion: {state['original_query']}"
    )
    response = client.chat.completions.create(
        model=MODEL, messages=[{'role': 'user', 'content': prompt}]
    )
    return {'final_answer': response.choices[0].message.content}


def check_for_more_questions(state: GraphState) -> str:
    return 'continue' if state['questions'] else 'end'


workflow = StateGraph(GraphState)
workflow.add_node('plan_queries', plan_queries)
workflow.add_node('execute_search', execute_search)
workflow.add_node('generate_final_answer', generate_final_answer)
workflow.add_edge(START, 'plan_queries')
workflow.add_edge('plan_queries', 'execute_search')
workflow.add_conditional_edges(
    'execute_search',
    check_for_more_questions,
    {'continue': 'execute_search', 'end': 'generate_final_answer'},
)
workflow.add_edge('generate_final_answer', END)
app = workflow.compile()

result = app.invoke({
    'original_query': 'What is the competitor to the product in our latest press release?',
    'questions': [],
    'answers': [],
    'final_answer': '',
})
print(result['final_answer'])

The run traces two hops. The first hop finds “Model-V” in the press release. The executor rewrites the second hop to “Who is the main competitor to Model-V?” and retrieves the competitor document. The final node answers from both facts.

When should sub-questions run in parallel instead of in sequence?

Run them in parallel when no sub-question depends on another. “Compare the refund policies of plan A and plan B” splits into independent lookups, so they can run at once.

LangGraph does this with Send, which starts one node run per item and passes each its own state. The sequential loop above stays the right shape for chained questions, where the second hop needs the first hop’s answer.

# filename: multihop/parallel.py
# description: Fan out independent sub-questions with Send.
from langgraph.types import Send


def fan_out(state: GraphState) -> list[Send]:
    return [Send('answer_one', {'question': q}) for q in state['questions']]

# workflow.add_conditional_edges('plan_queries', fan_out)

How do you do multi-hop web research programmatically?

Keep the same loop and route each hop to the right tool. Some sub-questions are about your own documents, such as “Who is our CEO?”. Others need the public web, such as “What did our competitor announce this week?”.

Add a router inside execute_search. A cheap model call, or a keyword rule, labels each rewritten sub-question as internal or web. Internal questions go to simple_rag_pipeline. Web questions go to a web search tool, and the top results become the fact for that hop.

The planner, the rewrite and the final answer do not change. To split the tools across separate agents, the multi-agent system with MCP lab builds that version.

How do you make multi-hop RAG reliable in production?

Bound it, check it and trace it. These controls cover most failures:

1. Cap the hops. MAX_HOPS stops a long plan from running a search per sub-question without limit. Log when the cap is hit, because it often means the planner misread the question.
2. Check each fact. A wrong answer at the first hop sends the second hop to the wrong place. A per-claim check like the one in fact-checking RAG answers catches unsupported facts before the next hop uses them.
3. Persist the run. Compile the graph with a checkpointer so a failed hop can resume. LangGraph persistence and thread models covers the setup.

What to do next

1. Collect 20 to 50 questions from your logs that single-shot RAG answered wrong, and label which ones need more than one fact.
2. Add the planner node and print its sub-questions for those questions. Fix the prompt until the plans look right.
3. Add the executor with the rewrite step and a hop cap.
4. Compare answers with and without multi-hop on the same questions before you ship it.
5. If some hops need public data, add a router that sends them to web search.
6. Trace each run so you can see which hop failed when an answer is wrong.

Frequently asked questions

What is multi-hop RAG?

Multi-hop RAG is retrieval-augmented generation that runs more than one search to answer a question. It splits the question into sub-questions, retrieves for each in turn, and uses earlier answers to shape later searches. It handles questions such as “who competes with the product in our press release”, where the second search depends on the first answer.

When should I use multi-hop RAG instead of simple retrieval?

Use it when your logs show wrong answers on compound questions that need facts from more than one document. Each hop adds a model call and a search, so latency and cost grow with the number of hops. Keep single-shot retrieval for lookup questions, and route only compound questions to the multi-hop graph.

How do I decompose a query into sub-questions?

Ask an LLM to split the question into the shortest ordered list of simple, searchable sub-questions, returned as JSON. Use Structured Outputs or a JSON schema so the reply always parses. Then rewrite each later sub-question with the facts found so far, so it names the entity instead of saying “that product”.

How do I handle multi-hop reasoning in a RAG pipeline without LangGraph?

Write the same loop in plain Python. Call the planner once, then loop over the sub-questions: rewrite each with the facts so far, retrieve, answer and append the fact. Stop at a hop cap, then generate the final answer from the facts. LangGraph adds checkpointing, tracing and parallel fan-out, but the pattern does not need it.

Does a bigger top-k fix multi-hop questions?

Rarely. The document that answers the second hop often shares few words with the original question, so it ranks low no matter how large k is. Raising k mostly adds noise and tokens. A second search with a rewritten question finds that document directly, which is why decomposition works where a larger k does not.

Sources

• MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries, Tang and Yang, January 2024: existing RAG methods retrieve and answer multi-hop queries poorly.
• Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions, Trivedi et al., ACL 2023: each reasoning step guides the next retrieval (IRCoT).
• Measuring and Narrowing the Compositionality Gap in Language Models, Press et al., Findings of EMNLP 2023: Self-Ask poses follow-up questions and answers them, optionally with a search engine.
• Graph API overview, LangChain LangGraph docs, October 2026: START edges, conditional edges, operator.add reducers and Send.