metric

`llm_named_unknown`

better: neutral: a description
kind: descriptive: says what the output was, never whether it was good
decides: no goal
weighed by: no goal
recorded on: scifact_200

01what it measures, and how

Ids the reranking LLM named that were not among the candidates it was shown: invented or out-of-list documents, which are dropped from its ordering.

A project explains its own metric at length with METRIC_DESCRIPTIONS = {"llm_named_unknown": {"summary": "…", "details": "…"}}.

02on scifact_200

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

arm	mean llm_named_unknown	items	scored against
sf_llama_s_n200_v1 · repeats	0.245	200	gold/scifact_200 gold
sf_llama_s_n200_v1	0.200	200	gold/scifact_200 gold
sf_mistral_s_n200_v1	0.185	200	gold/scifact_200 gold
sf_gemma_m_n200_v1	0.175	200	gold/scifact_200 gold
sf_glm_m_n200_v1	0.100	200	gold/scifact_200 gold
sf_gemma_s_n200_v1	0.045	200	gold/scifact_200 gold
sf_llama_m_n200_v1	0.030	200	gold/scifact_200 gold
sf_llama_l_n200_v1 · repeats	0.025	200	gold/scifact_200 gold
sf_deepseek_s_n200_v1	0.020	200	gold/scifact_200 gold
sf_gemma_l_n200_v1	0.005	200	gold/scifact_200 gold
sf_llama_l_n200_v1	0.005	200	gold/scifact_200 gold
sf_deepseek_m_n200_v1	0.000	200	gold/scifact_200 gold
sf_glm_s_n200_v1	0.000	200	gold/scifact_200 gold
sf_qwen_s_n200_v1	0.000	200	gold/scifact_200 gold
sf_glm_s_bm25first_n200_v1 · pair	0.000	200	gold/scifact_200 gold
sf_glm_s_e5first_n200_v1 · pair	0.000	200	gold/scifact_200 gold
sf_qwen_s_bm25first_n200_v1 · pair	0.000	200	gold/scifact_200 gold
sf_qwen_s_e5first_n200_v1 · pair	0.000	200	gold/scifact_200 gold
sf_qwen_s_n200_v1 · repeats	0.000	200	gold/scifact_200 gold