goal

Evidence retrieval for scientific claims (SciFact)

Pick a retriever that ranks the abstracts supporting or refuting a scientific claim at the top, over the full corpus.

01what it is for

decision

Which retrievers and rerankers can be told apart on nDCG@10, and what each costs.

audience

someone checking a scientific claim against the literature

decides

ndcg_10, higher is better

weighs

quality (ndcg_10); cost per item; speed per item

strategy

frugal: contenders: anything not significantly different from the leader by the paired test; the cheaper wins

tie-break

cost, quality

output contract

ranked_list

status

active

guardrails

one run per arm: Between-run variance is unmeasured: second runs moved nDCG@10 by up to 0.0126, each mixed with an adapter difference. (research/REPORT_RETRIEVAL.md, between-run variance)

links

the findings on the site

02the answer so far

scifact_200 · baseline promoted 2026-10-25 · scored against gold

sf_e5_base_n200_v1

Build e5_base retrieval first: 0.7191 at $0 per query, not significantly different from any of the twelve BM25-then-rerank pipelines; add a reranker on its top 20 only if about +0.07 nDCG@10 is worth about $732 per 1M queries. REPORT_RETRIEVAL, executive summary.

deploy sheet: what production must match

measured by: sf_e5_base_n200_v1_20260928T174656Z on scifact_200
against: gold/scifact_200
model sent: intfloat/e5-base-v2
model record: intfloat-e5-base-v2
server: not recorded
build: f52bf8ec8c7124536f0efb74aca902b2995e5bcd
settings sent: corpus_id: scifact
device: cpu
doc_prefix: passage:
encode_batch: 32
model: intfloat/e5-base-v2
precision: fp32
provider: dense
query_prefix: query:
top_k: 100
not sent: bookkeeping: family, price_tier
declared: deployment.corpus_id: scifact
deployment.corpus_n_docs: 5183
deployment.corpus_sha256: 297643e2d005109d9fc25d460c080faa45b0cc3ad881ea002941c67b6e41984e
deployment.doc_prefix: passage:
deployment.index_built_this_run: False
deployment.provider: dense
deployment.query_prefix: query:
deployment.scorer_sha256: 51d9e8bd57bc2e75ceccce80b7d60157c42f9959cadf3d7e9cf738f73fd10d04
deployment.tokenizer_sha256: 67a185c8c30efefa133188f15ba429328086f837a642a17e347a5a2ee437b79d
deployment.top_k: 100
deployment.weight_files: ['model.safetensors:437955512']
adapter: examples/retrieval-scifact/adapter.py 46cd2bd89061
harness: 0e20aaeb3dca

No capture here says what production ran (the adapter's production_ran).

Command: evh deploy-sheet --config sf_e5_base_n200_v1 [--capture C]

inspect it

scifact_40

No decision yet. When the tests separate an arm worth keeping, promote its run with the reason they give: evh run-promote --run <run_id> --reason "…"

03how we got here: rounds

Rounds are optional: each one sets out to learn something, changes a few arms, drops what lost and says why, and ends in a conclusion. Decisions are made with evh round-open, round-retire and round-close; evh rounds says the same as this page. Open a round for what changed, what was retired and why, and what to do next.

r3 closed
from r2

Second runs moved nDCG@10 by −0.0126 (llama_l), −0.0047 (llama_s) and +0.0004 (qwen_s), each mixed with an adapter difference: between-run variance is still unmeasured. (REPORT_RETRIEVAL, between-run variance.)

How much does a reranker's score move when it is run again?

0 arms (3 retired)

3 retired

arms

retired 3: a second run of the record's config; data/runs is the record
sf_llama_l_n200_v1, sf_llama_s_n200_v1, sf_qwen_s_n200_v1

r2 closed
from r1

glm_s reranking e5_base's candidates scores 0.7891, inside the registered range and +0.0700 over e5_base alone (p 0.0001): one run, under a different adapter from the main table. (REPORT_RETRIEVAL §3.4.)

Does swapping only the first stage to e5_base move a reranking pipeline into the pre-registered range 0.786–0.797?

0 arms (4 retired)

4 retired

arms

retired 4: a paired side run, not on the leaderboard; data/runs is the record
sf_glm_s_bm25first_n200_v1, sf_glm_s_e5first_n200_v1, sf_qwen_s_bm25first_n200_v1, sf_qwen_s_e5first_n200_v1

r1 closed

e5_base (e5-base-v2, MIT, 438 MB, CPU, $0 per query) scores 0.7191, +0.0740 over BM25, and is not significantly different from any of the twelve BM25-then-rerank pipelines. Build e5_base retrieval first; add an LLM reranker on its top 20 only if about +0.07 nDCG@10 is worth about $732 per 1M queries. (REPORT_RETRIEVAL, executive summary.)

Which first stage (BM25, four bi-encoders) and which of the 12 cheapest hosted LLMs reranking BM25's top 20 retrieves the evidence for a scientific claim best, and at what cost?

19 arms inspect
best sf_glm_s_n200_v1 0.744

04scifact_200 · test

scifact_200: 200 items · test

200 seeded queries

references: gold/scifact_200

Mean ndcg_10 over each arm's items, in groups by what each arm is scored against: numbers from different references, or from a judge, are different measurements and are never ranked together. A mean is not a verdict: the inspector tests each arm against its group's leader (Holm across them). Flagged arms are never the leader. filed: the arm's config names this goal (goal:); an arm without it is listed because it records the measure.

scored against gold/scifact_200 gold

Test every arm in this group against its leader

Recommended: sf_e5_base_n200_v1

Strategy frugal. Quality leader: sf_glm_s_n200_v1 (ndcg_10 0.7437). Contenders beside it (not significantly different by the paired test): 12 arms
which
sf_glm_m_n200_v1, sf_deepseek_m_n200_v1, sf_deepseek_s_n200_v1, sf_gemma_m_n200_v1, sf_gemma_l_n200_v1, sf_llama_l_n200_v1, sf_e5_base_n200_v1, sf_qwen_s_n200_v1, sf_llama_s_n200_v1, sf_bge_small_n200_v1, sf_mistral_s_n200_v1, sf_gemma_s_n200_v1
By cost (cost_usd): sf_e5_base_n200_v1 $0.000000, sf_bge_small_n200_v1 $0.000000, sf_deepseek_s_n200_v1 $0.000300, sf_gemma_m_n200_v1 $0.000361, sf_mistral_s_n200_v1 $0.000426, sf_gemma_s_n200_v1 $0.000481, sf_glm_s_n200_v1 $0.000711, sf_llama_s_n200_v1 $0.000729, sf_qwen_s_n200_v1 $0.000779, sf_gemma_l_n200_v1 $0.000850, sf_llama_l_n200_v1 $0.000865, sf_deepseek_m_n200_v1 $0.001190, sf_glm_m_n200_v1 $0.002288.
By quality (ndcg_10): sf_bge_small_n200_v1 0.7097, sf_e5_base_n200_v1 0.7191.
Recommended: sf_e5_base_n200_v1.

Strategy frugal (contenders: anything not significantly different from the leader by the paired test; the cheaper wins). Weighs quality (ndcg_10); cost per item; speed per item. Among the contenders on quality, the tie-break is cost, quality. The rule recommends; promoting is evh run-promote; the goal chooses its strategy (strategy:).

arm	mean ndcg_10	items	cost per item	speed per item	flags
sf_glm_s_e5first_n200_v1 · pair filed retired in r2	0.789	200	$0.000732 · 4,178 tok	p50 3,007 · p95 4,028 ms	—	test vs leader
sf_qwen_s_e5first_n200_v1 · pair filed retired in r2	0.771	200	$0.000754 · 4,485 tok	p50 4,102 · p95 13,396 ms	—	test vs leader
sf_glm_s_n200_v1 filed	0.744	200	$0.000711 · 4,182 tok	p50 3,004 · p95 3,980 ms	—	leader
sf_glm_s_bm25first_n200_v1 · pair filed retired in r2	0.742	200	$0.000741 · 4,182 tok	p50 3,019 · p95 3,653 ms	—	test vs leader
sf_glm_m_n200_v1 filed	0.742	200	$0.002288 · 4,171 tok	p50 7,430 · p95 10,170 ms	—	test vs leader
sf_deepseek_m_n200_v1 filed	0.740	200	$0.001190 · 3,869 tok	p50 773 · p95 2,389 ms	—	test vs leader
sf_deepseek_s_n200_v1 filed	0.738	200	$0.000300 · 3,872 tok	p50 2,392 · p95 7,323 ms	—	test vs leader
sf_gemma_m_n200_v1 filed	0.736	200	$0.000361 · 4,165 tok	p50 4,730 · p95 13,747 ms	—	test vs leader
sf_gemma_l_n200_v1 filed	0.736	200	$0.000850 · 4,176 tok	p50 8,641 · p95 17,014 ms	—	test vs leader
sf_llama_l_n200_v1 filed	0.728	200	$0.000865 · 3,977 tok	p50 3,701 · p95 9,656 ms	—	test vs leader
sf_e5_base_n200_v1 filed baseline recommended	0.719	200	$0.000000	p50 56 · p95 75 ms	—	test vs leader
sf_qwen_s_n200_v1 · repeats filed retired in r3	0.719	200	$0.000779 · 4,514 tok	p50 4,298 · p95 12,577 ms	—	test vs leader
sf_qwen_s_n200_v1 filed	0.718	200	$0.000779 · 4,514 tok	p50 4,050 · p95 12,207 ms	—	test vs leader
sf_llama_s_n200_v1 filed	0.717	200	$0.000729 · 4,013 tok	p50 13,661 · p95 42,002 ms	—	test vs leader
sf_llama_l_n200_v1 · repeats filed retired in r3	0.715	200	$0.000846 · 3,981 tok	p50 10,149 · p95 28,200 ms	—	test vs leader
sf_qwen_s_bm25first_n200_v1 · pair filed retired in r2	0.714	200	$0.000775 · 4,505 tok	p50 4,032 · p95 12,558 ms	—	test vs leader
sf_llama_s_n200_v1 · repeats filed retired in r3	0.713	200	$0.000524 · 4,014 tok	p50 3,903 · p95 10,564 ms	—	test vs leader
sf_bge_small_n200_v1 filed	0.710	200	$0.000000	p50 26 · p95 45 ms	—	test vs leader
sf_llama_m_n200_v1 filed	0.709	200	$0.000717 · 4,109 tok	p50 3,734 · p95 7,923 ms	—	test vs leader
sf_mistral_s_n200_v1 filed	0.707	200	$0.000426 · 4,191 tok	p50 6,068 · p95 13,800 ms	—	test vs leader
sf_gemma_s_n200_v1 filed	0.702	200	$0.000481 · 4,119 tok	p50 6,611 · p95 11,109 ms	—	test vs leader
sf_mpnet_n200_v1 filed	0.664	200	$0.000000	p50 61 · p95 92 ms	—	test vs leader
sf_minilm_n200_v1 filed	0.653	200	$0.000000	p50 16 · p95 23 ms	—	test vs leader
sf_bm25_n200_v1 filed	0.645	200	$0.000000	p50 20 · p95 38 ms	—	test vs leader
sf_random_n200_v1 filed	0.001	200	$0.000000	p50 1.6 · p95 1.9 ms	—	test vs leader
sf_first_k_n200_v1 filed	0.000	200	$0.000000	p50 0.23 · p95 0.38 ms	—	test vs leader

ndcg_10 against cost (USD per item): better is left top. Named and joined: the frontier, the arms no other arm beats on both.

ndcg_10 against speed p95 (ms): better is left top. Named and joined: the frontier, the arms no other arm beats on both.

05scifact_40 · pilot

scifact_40: 40 items · pilot

the pilot slice; all 40 of its items are in scifact_200

references: gold/scifact_40

No arm on this dataset records ndcg_10.