Free retrieval first; the best reranker's +0.0246 is not significant.
Retrieve the abstracts that support a scientific claim. The free e5_base retriever is not significantly different from the twelve LLM rerankers; the best of them, glm_s (0.7437), is 0.0246 ahead of e5_base (0.7191), p 0.27. One further run, glm_s reranking e5_base's top 20, scored 0.7891 (+0.0700 over e5_base, p 0.0001 uncorrected; one run, under a different adapter). The pick: build e5_base first.
The report's pick, as run here. Build e5_base retrieval first (0.7191, $0 per query); among the compared field the best reranker, glm_s (0.7437), is 0.0246 ahead of it, p 0.27, not significant. One further run, glm_s on e5_base's top 20, scored 0.7891 (+0.0700 over e5_base, p 0.0001 uncorrected; one run, under a different adapter from the main table) at about $732 per 1M queries. Only the twelve cheapest hosted LLMs were tried.
The goal's rule, in the catalog. Evidence retrieval for scientific claims (SciFact) (frugal, on ndcg_10) recommends e5_base, the promoted baseline. The goal page, with its reasons → An agent gets the same answer from the MCP server: the tool goals with goal: scientific_retrieval_scifact at https://evals.humancopilot.eu/mcp (how to connect).
Every system, one axis.
What the money bought.
What this eval found.
LLM rerankers lead, and the leader is not significantly different from twelve others.
glm_s reranking BM25's top 20 scores 0.7437 nDCG@10; 12 arms are not significantly different from it, including the free retrievers e5_base and bge_small.
The dearest model finished 2nd of 19.
glm_m billed $0.4576 for 200 items. Only the twelve cheapest hosted models ran here, so this is the dearest of those; no Anthropic or OpenAI arm was run. The cheapest model not significantly different from the leader, deepseek_s, cost 7.6× less.
The pilot slice's winner finished 10th — among the leader's contenders.
Tuned on a 40-query pilot slice, the leaderboard's winner there, llama_s, finished 10th of 19 on the 200-query test slice, 0.0265 behind the leader and not significantly different from it under Holm: a rank inside a tie, not a wrong pick shown at this n.
Measure the noise before the difference.
The pilot slice's leader finished 10th of 19, a rank inside a tie.
This habit, in the method →Check it yourself.
First, with nothing installed: test every arm against glm_s, item by item, in the catalog's inspector, with the same paired test and Holm correction as this page; glm_m, the most expensive model, among them. The goal page has the rest: what the goal recommends, and why.
The 7 free arms rerun on a laptop CPU for $0, pinned to the weights that were measured. The 12 hosted arms billed $1.94 when measured and rerun with one OpenRouter key.
make compare writes a replication card: each rerun against its committed run, item by item, and every significant difference and tie on this page re-tested with your reruns in place.
$ make plan EXP=scifact $ make data EXP=scifact $ make reproduce EXP=scifact LEVEL=local # with OPENROUTER_API_KEY set $ make reproduce EXP=scifact LEVEL=hosted $ make compare EXP=scifact