Retrieval · SciFact

Rerankers lead a crowd.

Retrieve the abstracts that support a scientific claim. LLM rerankers lead a twelve-way tie; a free dense retriever sits inside it.

Question

Rank 5,183 scientific abstracts by how well they bear on a claim.

Data

200 items from the SciFact test split

Systems

19 compared: 12 hosted models billed per call; 7 local models and baselines, free

Metric

nDCG@10 — the field's standard; higher is better

Calibration

bm25 0.6523 vs 0.6650 published

Billed

$1.94 for the whole sweep

Read the full report Protocol All experiments

01The field

Every system, one axis.

hosted · proprietaryhosted · open weightslocal · freestatistically tied with the leaderhigher score: further right →

Each dot is one system, by nDCG@10. The ringed dot is the most expensive hosted model; the shaded band is the group the leader cannot be statistically separated from. A dot left of the ⫽ break sits far below the rest, off the scale; tap it for its score.

02Cost

What the money bought.

hosted · proprietaryhosted · open weightslocal · freebetter: up · cheaper: left

Billed cost per 1,000 items, read from the provider's response, against nDCG@10. Free models sit in the left lane. A dot below the ⫽ break sits far below the rest, off the scale; tap it for its score.

03Findings

What this experiment found.

LLM rerankers lead — inside a twelve-way tie.
glm_s reranking BM25's top 20 scores 0.7437 nDCG@10; 12 arms cannot be separated from it, including the free dense retriever e5_base.
The dearest model finished 2nd of 19.
glm_m billed $0.4576 for 200 items. The cheapest model statistically tied with the leader, deepseek_s, cost 7.6× less.
The dev slice picked the wrong winner.
Tuned on a 40-query dev slice, the leaderboard's winner there was not the winner on the 160 queries it never saw.

04The method, here

Measure the noise before the difference.

The dev slice picked the wrong winner — its leader finished 10th of 19.

This habit, in the method →

05Reproduce it

Check it yourself.

The 7 free arms rerun on a laptop CPU for $0, pinned to the weights that were measured. The 12 hosted arms billed $1.94 when measured and rerun with one OpenRouter key.

make compare writes a replication card: each rerun against its committed run, item by item, and every separation and tie on this page re-tested with your reruns in place.

The guide This example's code

$ make plan      EXP=scifact
$ make data      EXP=scifact
$ make reproduce EXP=scifact LEVEL=local
# with OPENROUTER_API_KEY set
$ make reproduce EXP=scifact LEVEL=hosted
$ make compare   EXP=scifact

Rerankers lead a crowd.

Every system, one axis.

What the money bought.

What this experiment found.

LLM rerankers lead — inside a twelve-way tie.

The dearest model finished 2nd of 19.

The dev slice picked the wrong winner.

Measure the noise before the difference.

Check it yourself.