eval-harness

metric

unknown_ids

better
neutral: a description
kind
descriptive: says what the output was, never whether it was good
decides
no goal
weighed by
no goal
recorded on
scifact_200

01what it measures, and how

Ids in the ranking that are not corpus documents: an LLM inventing documents rather than reordering them. They keep their slot.

A project explains its own metric at length with METRIC_DESCRIPTIONS = {"unknown_ids": {"summary": "…", "details": "…"}}.

02on scifact_200

gold/scifact_200sf_bge_small_n200_v1: 0sf_bm25_n200_v1: 0sf_deepseek_m_n200_v1: 0sf_deepseek_s_n200_v1: 0sf_e5_base_n200_v1: 0sf_first_k_n200_v1: 0sf_gemma_l_n200_v1: 0sf_gemma_m_n200_v1: 0sf_gemma_s_n200_v1: 0sf_glm_m_n200_v1: 0sf_glm_s_n200_v1: 0sf_llama_l_n200_v1: 0sf_llama_m_n200_v1: 0sf_llama_s_n200_v1: 0sf_minilm_n200_v1: 0sf_mistral_s_n200_v1: 0sf_mpnet_n200_v1: 0sf_qwen_s_n200_v1: 0sf_random_n200_v1: 0sf_glm_s_bm25first_n200_v1 · pair: 0sf_glm_s_e5first_n200_v1 · pair: 0sf_qwen_s_bm25first_n200_v1 · pair: 0sf_qwen_s_e5first_n200_v1 · pair: 0sf_llama_l_n200_v1 · repeats: 0sf_llama_s_n200_v1 · repeats: 0sf_qwen_s_n200_v1 · repeats: 0-0.5better · 0.5

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

armmean unknown_idsitemsscored against
sf_bge_small_n200_v10.000200gold/scifact_200 gold
sf_bm25_n200_v10.000200gold/scifact_200 gold
sf_deepseek_m_n200_v10.000200gold/scifact_200 gold
sf_deepseek_s_n200_v10.000200gold/scifact_200 gold
sf_e5_base_n200_v10.000200gold/scifact_200 gold
sf_first_k_n200_v10.000200gold/scifact_200 gold
sf_gemma_l_n200_v10.000200gold/scifact_200 gold
sf_gemma_m_n200_v10.000200gold/scifact_200 gold
sf_gemma_s_n200_v10.000200gold/scifact_200 gold
sf_glm_m_n200_v10.000200gold/scifact_200 gold
sf_glm_s_n200_v10.000200gold/scifact_200 gold
sf_llama_l_n200_v10.000200gold/scifact_200 gold
sf_llama_m_n200_v10.000200gold/scifact_200 gold
sf_llama_s_n200_v10.000200gold/scifact_200 gold
sf_minilm_n200_v10.000200gold/scifact_200 gold
sf_mistral_s_n200_v10.000200gold/scifact_200 gold
sf_mpnet_n200_v10.000200gold/scifact_200 gold
sf_qwen_s_n200_v10.000200gold/scifact_200 gold
sf_random_n200_v10.000200gold/scifact_200 gold
sf_glm_s_bm25first_n200_v1 · pair0.000200gold/scifact_200 gold
sf_glm_s_e5first_n200_v1 · pair0.000200gold/scifact_200 gold
sf_qwen_s_bm25first_n200_v1 · pair0.000200gold/scifact_200 gold
sf_qwen_s_e5first_n200_v1 · pair0.000200gold/scifact_200 gold
sf_llama_l_n200_v1 · repeats0.000200gold/scifact_200 gold
sf_llama_s_n200_v1 · repeats0.000200gold/scifact_200 gold
sf_qwen_s_n200_v1 · repeats0.000200gold/scifact_200 gold