eval-harness

metric

rank_of_first

better
neutral: a description
kind
descriptive: says what the output was, never whether it was good
decides
no goal
weighed by
no goal
recorded on
scifact_200

01what it measures, and how

Rank of the first relevant document anywhere in the list, 0 when none was returned. Descriptive: 0 means never found, so it must not be averaged as a rank.

A project explains its own metric at length with METRIC_DESCRIPTIONS = {"rank_of_first": {"summary": "…", "details": "…"}}.

02on scifact_200

gold/scifact_200sf_mpnet_n200_v1: 6.88sf_e5_base_n200_v1: 6.455sf_minilm_n200_v1: 6.21sf_qwen_s_e5first_n200_v1 · pair: 5.915sf_glm_s_e5first_n200_v1 · pair: 5.77sf_bge_small_n200_v1: 4.95sf_bm25_n200_v1: 3.34sf_mistral_s_n200_v1: 3.155sf_gemma_s_n200_v1: 3.025sf_llama_m_n200_v1: 2.96sf_llama_s_n200_v1 · repeats: 2.875sf_llama_s_n200_v1: 2.825sf_qwen_s_bm25first_n200_v1 · pair: 2.76sf_llama_l_n200_v1 · repeats: 2.745sf_qwen_s_n200_v1 · repeats: 2.7sf_llama_l_n200_v1: 2.64sf_deepseek_s_n200_v1: 2.625sf_qwen_s_n200_v1: 2.62sf_gemma_l_n200_v1: 2.605sf_gemma_m_n200_v1: 2.555sf_deepseek_m_n200_v1: 2.55sf_glm_s_n200_v1: 2.52sf_glm_s_bm25first_n200_v1 · pair: 2.51sf_glm_m_n200_v1: 2.495sf_first_k_n200_v1: 2.34sf_random_n200_v1: 0.0450.045better · 6.88

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

armmean rank_of_firstitemsscored against
sf_mpnet_n200_v16.880200gold/scifact_200 gold
sf_e5_base_n200_v16.455200gold/scifact_200 gold
sf_minilm_n200_v16.210200gold/scifact_200 gold
sf_qwen_s_e5first_n200_v1 · pair5.915200gold/scifact_200 gold
sf_glm_s_e5first_n200_v1 · pair5.770200gold/scifact_200 gold
sf_bge_small_n200_v14.950200gold/scifact_200 gold
sf_bm25_n200_v13.340200gold/scifact_200 gold
sf_mistral_s_n200_v13.155200gold/scifact_200 gold
sf_gemma_s_n200_v13.025200gold/scifact_200 gold
sf_llama_m_n200_v12.960200gold/scifact_200 gold
sf_llama_s_n200_v1 · repeats2.875200gold/scifact_200 gold
sf_llama_s_n200_v12.825200gold/scifact_200 gold
sf_qwen_s_bm25first_n200_v1 · pair2.760200gold/scifact_200 gold
sf_llama_l_n200_v1 · repeats2.745200gold/scifact_200 gold
sf_qwen_s_n200_v1 · repeats2.700200gold/scifact_200 gold
sf_llama_l_n200_v12.640200gold/scifact_200 gold
sf_deepseek_s_n200_v12.625200gold/scifact_200 gold
sf_qwen_s_n200_v12.620200gold/scifact_200 gold
sf_gemma_l_n200_v12.605200gold/scifact_200 gold
sf_gemma_m_n200_v12.555200gold/scifact_200 gold
sf_deepseek_m_n200_v12.550200gold/scifact_200 gold
sf_glm_s_n200_v12.520200gold/scifact_200 gold
sf_glm_s_bm25first_n200_v1 · pair2.510200gold/scifact_200 gold
sf_glm_m_n200_v12.495200gold/scifact_200 gold
sf_first_k_n200_v12.340200gold/scifact_200 gold
sf_random_n200_v10.045200gold/scifact_200 gold