Evaluating ranked retrieval
Three offline metrics matter for ranked retrieval. Recall@K asks whether any ground-truth item showed up in the top K. NDCG@K rewards landing them near the top. MAP@K rewards finding several at once. Pick one as your primary metric and watch the others.
def recall_at_k(predicted, truth, k):
return len(set(predicted[:k]) & truth) / max(len(truth), 1)
def ndcg_at_k(predicted, truth, k):
dcg = sum(1.0 / np.log2(i + 2) for i, item in enumerate(predicted[:k]) if item in truth)
ideal = sum(1.0 / np.log2(i + 2) for i in range(min(k, len(truth))))
return dcg / ideal if ideal > 0 else 0.0
recall_scores, ndcg_scores = [], []
for c in customers:
truths = ... # held-out article ids for the customer
pred = [it["article_id"] for it in pipeline.recommend(c, top_k=K)["items"]]
recall_scores.append(recall_at_k(pred, set(truths), K))
ndcg_scores.append(ndcg_at_k(pred, set(truths), K))Hold out one purchase per customer. See if it shows up in the top K.
On the full H&M competition set the public leaderboard sits around 0.04 recall at twelve. On our sampled slice you will see far higher numbers because the catalogue is much smaller. Do not chase absolute numbers across datasets. Chase deltas on a fixed holdout.
Quiz: Quiz
Loading practice…
AI prompt: Try it: bridging offline and online
Loading practice…