eval-harness

metric

n_relevant

better
neutral: a description
kind
descriptive: says what the output was, never whether it was good
decides
no goal
weighed by
no goal
recorded on
scifact_200

01what it measures, and how

How many documents are judged relevant (grade above 0) for the query.

A project explains its own metric at length with METRIC_DESCRIPTIONS = {"n_relevant": {"summary": "…", "details": "…"}}.

02on scifact_200

gold/scifact_200sf_bge_small_n200_v1: 1.135sf_bm25_n200_v1: 1.135sf_deepseek_m_n200_v1: 1.135sf_deepseek_s_n200_v1: 1.135sf_e5_base_n200_v1: 1.135sf_first_k_n200_v1: 1.135sf_gemma_l_n200_v1: 1.135sf_gemma_m_n200_v1: 1.135sf_gemma_s_n200_v1: 1.135sf_glm_m_n200_v1: 1.135sf_glm_s_n200_v1: 1.135sf_llama_l_n200_v1: 1.135sf_llama_m_n200_v1: 1.135sf_llama_s_n200_v1: 1.135sf_minilm_n200_v1: 1.135sf_mistral_s_n200_v1: 1.135sf_mpnet_n200_v1: 1.135sf_qwen_s_n200_v1: 1.135sf_random_n200_v1: 1.135sf_glm_s_bm25first_n200_v1 · pair: 1.135sf_glm_s_e5first_n200_v1 · pair: 1.135sf_qwen_s_bm25first_n200_v1 · pair: 1.135sf_qwen_s_e5first_n200_v1 · pair: 1.135sf_llama_l_n200_v1 · repeats: 1.135sf_llama_s_n200_v1 · repeats: 1.135sf_qwen_s_n200_v1 · repeats: 1.1350.635better · 1.635

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

armmean n_relevantitemsscored against
sf_bge_small_n200_v11.135200gold/scifact_200 gold
sf_bm25_n200_v11.135200gold/scifact_200 gold
sf_deepseek_m_n200_v11.135200gold/scifact_200 gold
sf_deepseek_s_n200_v11.135200gold/scifact_200 gold
sf_e5_base_n200_v11.135200gold/scifact_200 gold
sf_first_k_n200_v11.135200gold/scifact_200 gold
sf_gemma_l_n200_v11.135200gold/scifact_200 gold
sf_gemma_m_n200_v11.135200gold/scifact_200 gold
sf_gemma_s_n200_v11.135200gold/scifact_200 gold
sf_glm_m_n200_v11.135200gold/scifact_200 gold
sf_glm_s_n200_v11.135200gold/scifact_200 gold
sf_llama_l_n200_v11.135200gold/scifact_200 gold
sf_llama_m_n200_v11.135200gold/scifact_200 gold
sf_llama_s_n200_v11.135200gold/scifact_200 gold
sf_minilm_n200_v11.135200gold/scifact_200 gold
sf_mistral_s_n200_v11.135200gold/scifact_200 gold
sf_mpnet_n200_v11.135200gold/scifact_200 gold
sf_qwen_s_n200_v11.135200gold/scifact_200 gold
sf_random_n200_v11.135200gold/scifact_200 gold
sf_glm_s_bm25first_n200_v1 · pair1.135200gold/scifact_200 gold
sf_glm_s_e5first_n200_v1 · pair1.135200gold/scifact_200 gold
sf_qwen_s_bm25first_n200_v1 · pair1.135200gold/scifact_200 gold
sf_qwen_s_e5first_n200_v1 · pair1.135200gold/scifact_200 gold
sf_llama_l_n200_v1 · repeats1.135200gold/scifact_200 gold
sf_llama_s_n200_v1 · repeats1.135200gold/scifact_200 gold
sf_qwen_s_n200_v1 · repeats1.135200gold/scifact_200 gold