When an LLM rerank actually helps

An LLM call per item per request costs money and time. So you never call an LLM at the retrieval stage. You call it on the top twenty candidates, after the ranker has already done the heavy lifting, when the surface justifies a slightly slower response.

recsys/inference/llm_reranker.py
python
class LLMRerankerService:
    def rerank(self, customer_features, candidates, max_items=20):
        candidates = list(candidates)[:max_items]
        if not self._enabled or not candidates:
            return candidates
        prompt = PROMPT_TEMPLATE.format(
            age=customer_features.get('age', 30),
            month_sin=customer_features.get('month_sin', 0),
            month_cos=customer_features.get('month_cos', 0),
            articles_json=json.dumps(candidates, default=str),
            n=len(candidates),
        )
        response = litellm.completion(
            model=settings.LLM_MODEL_ID,
            messages=[{'role': 'user', 'content': prompt}],
            temperature=0.2,
            max_tokens=256,
        )
        scores = json.loads(self._extract_json(response.choices[0].message.content or '[]'))
        for cand, score in zip(candidates, scores):
            cand['llm_score'] = float(score)
        return sorted(candidates, key=lambda c: c.get('llm_score', 0.0), reverse=True)

LiteLLM keeps the provider neutral so you can swap OpenAI for Claude or Gemini without touching the orchestrator.

The LLM rerank pays off on surfaces where the ranker has thin signal. Cold-start customers. Long-tail items. Editorial guard-rails that change weekly. On a saturated home feed where the CatBoost ranker already saw millions of impressions, the LLM is usually noise.

Matching exercise: When to add an LLM rerank

Loading practice…

AI prompt: Try it: cost model the LLM rerank

Loading practice…