metric

`mrr_10`

better: higher
kind: quality: measured against a reference or by a judge
decides: no goal
weighed by: no goal
recorded on: scifact_200

01what it measures, and how

Reciprocal rank of the first relevant document if it is in the first 10, else 0.

A project explains its own metric at length with METRIC_DESCRIPTIONS = {"mrr_10": {"summary": "…", "details": "…"}}.

02on scifact_200

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

arm	mean mrr_10	items	scored against
sf_glm_s_e5first_n200_v1 · pair	0.774	200	gold/scifact_200 gold
sf_qwen_s_e5first_n200_v1 · pair	0.747	200	gold/scifact_200 gold
sf_deepseek_m_n200_v1	0.737	200	gold/scifact_200 gold
sf_glm_s_n200_v1	0.736	200	gold/scifact_200 gold
sf_glm_s_bm25first_n200_v1 · pair	0.734	200	gold/scifact_200 gold
sf_glm_m_n200_v1	0.732	200	gold/scifact_200 gold
sf_deepseek_s_n200_v1	0.730	200	gold/scifact_200 gold
sf_gemma_m_n200_v1	0.729	200	gold/scifact_200 gold
sf_gemma_l_n200_v1	0.729	200	gold/scifact_200 gold
sf_llama_l_n200_v1	0.723	200	gold/scifact_200 gold
sf_llama_s_n200_v1	0.707	200	gold/scifact_200 gold
sf_qwen_s_n200_v1	0.706	200	gold/scifact_200 gold
sf_llama_l_n200_v1 · repeats	0.704	200	gold/scifact_200 gold
sf_llama_s_n200_v1 · repeats	0.704	200	gold/scifact_200 gold
sf_qwen_s_n200_v1 · repeats	0.702	200	gold/scifact_200 gold
sf_qwen_s_bm25first_n200_v1 · pair	0.701	200	gold/scifact_200 gold
sf_llama_m_n200_v1	0.700	200	gold/scifact_200 gold
sf_mistral_s_n200_v1	0.699	200	gold/scifact_200 gold
sf_gemma_s_n200_v1	0.694	200	gold/scifact_200 gold
sf_e5_base_n200_v1	0.689	200	gold/scifact_200 gold
sf_bge_small_n200_v1	0.681	200	gold/scifact_200 gold
sf_mpnet_n200_v1	0.635	200	gold/scifact_200 gold
sf_minilm_n200_v1	0.621	200	gold/scifact_200 gold
sf_bm25_n200_v1	0.615	200	gold/scifact_200 gold
sf_random_n200_v1	0.001	200	gold/scifact_200 gold
sf_first_k_n200_v1	0.000	200	gold/scifact_200 gold