dataset
scifact_200
200 items · 26 arms. Headline: each arm's own primary metric, mean over its items; arms with different primary metrics are not ranked against each other.
baseline, promoted 2026-10-25
Build e5_base retrieval first: 0.7191 at $0 per query, not significantly different from any of the twelve BM25-then-rerank pipelines; add a reranker on its top 20 only if about +0.07 nDCG@10 is worth about $732 per 1M queries. REPORT_RETRIEVAL, executive summary.
Inspect the 26 arms on ndcg_10, scored against gold/scifact_200Open its sheet
01arms
scored against silverscored by a judgescored against a label set, not a reference setunmarked rows are scored against gold
26 shown
| arm | metric | mean | items | cost per item | speed per item | runs | scored against | |
|---|---|---|---|---|---|---|---|---|
| sf_glm_s_e5first_n200_v1 · pair | ndcg_10 | 0.789 | 200 | $0.000732 · 4,178 tok | p50 3,007 · p95 4,028 ms | 1 | gold/scifact_200 gold | inspect |
| sf_qwen_s_e5first_n200_v1 · pair | ndcg_10 | 0.771 | 200 | $0.000754 · 4,485 tok | p50 4,102 · p95 13,396 ms | 1 | gold/scifact_200 gold | inspect |
| sf_glm_s_n200_v1 | ndcg_10 | 0.744 | 200 | $0.000711 · 4,182 tok | p50 3,004 · p95 3,980 ms | 1 | gold/scifact_200 gold | inspect |
| sf_glm_s_bm25first_n200_v1 · pair | ndcg_10 | 0.742 | 200 | $0.000741 · 4,182 tok | p50 3,019 · p95 3,653 ms | 1 | gold/scifact_200 gold | inspect |
| sf_glm_m_n200_v1 | ndcg_10 | 0.742 | 200 | $0.002288 · 4,171 tok | p50 7,430 · p95 10,170 ms | 1 | gold/scifact_200 gold | inspect |
| sf_deepseek_m_n200_v1 | ndcg_10 | 0.740 | 200 | $0.001190 · 3,869 tok | p50 773 · p95 2,389 ms | 1 | gold/scifact_200 gold | inspect |
| sf_deepseek_s_n200_v1 | ndcg_10 | 0.738 | 200 | $0.000300 · 3,872 tok | p50 2,392 · p95 7,323 ms | 1 | gold/scifact_200 gold | inspect |
| sf_gemma_m_n200_v1 | ndcg_10 | 0.736 | 200 | $0.000361 · 4,165 tok | p50 4,730 · p95 13,747 ms | 1 | gold/scifact_200 gold | inspect |
| sf_gemma_l_n200_v1 | ndcg_10 | 0.736 | 200 | $0.000850 · 4,176 tok | p50 8,641 · p95 17,014 ms | 1 | gold/scifact_200 gold | inspect |
| sf_llama_l_n200_v1 | ndcg_10 | 0.728 | 200 | $0.000865 · 3,977 tok | p50 3,701 · p95 9,656 ms | 1 | gold/scifact_200 gold | inspect |
| sf_e5_base_n200_v1 baseline | ndcg_10 | 0.719 | 200 | $0.000000 | p50 56 · p95 75 ms | 1 | gold/scifact_200 gold | inspect |
| sf_qwen_s_n200_v1 · repeats | ndcg_10 | 0.719 | 200 | $0.000779 · 4,514 tok | p50 4,298 · p95 12,577 ms | 1 | gold/scifact_200 gold | inspect |
| sf_qwen_s_n200_v1 | ndcg_10 | 0.718 | 200 | $0.000779 · 4,514 tok | p50 4,050 · p95 12,207 ms | 1 | gold/scifact_200 gold | inspect |
| sf_llama_s_n200_v1 | ndcg_10 | 0.717 | 200 | $0.000729 · 4,013 tok | p50 13,661 · p95 42,002 ms | 1 | gold/scifact_200 gold | inspect |
| sf_llama_l_n200_v1 · repeats | ndcg_10 | 0.715 | 200 | $0.000846 · 3,981 tok | p50 10,149 · p95 28,200 ms | 1 | gold/scifact_200 gold | inspect |
| sf_qwen_s_bm25first_n200_v1 · pair | ndcg_10 | 0.714 | 200 | $0.000775 · 4,505 tok | p50 4,032 · p95 12,558 ms | 1 | gold/scifact_200 gold | inspect |
| sf_llama_s_n200_v1 · repeats | ndcg_10 | 0.713 | 200 | $0.000524 · 4,014 tok | p50 3,903 · p95 10,564 ms | 1 | gold/scifact_200 gold | inspect |
| sf_bge_small_n200_v1 | ndcg_10 | 0.710 | 200 | $0.000000 | p50 26 · p95 45 ms | 1 | gold/scifact_200 gold | inspect |
| sf_llama_m_n200_v1 | ndcg_10 | 0.709 | 200 | $0.000717 · 4,109 tok | p50 3,734 · p95 7,923 ms | 1 | gold/scifact_200 gold | inspect |
| sf_mistral_s_n200_v1 | ndcg_10 | 0.707 | 200 | $0.000426 · 4,191 tok | p50 6,068 · p95 13,800 ms | 1 | gold/scifact_200 gold | inspect |
| sf_gemma_s_n200_v1 | ndcg_10 | 0.702 | 200 | $0.000481 · 4,119 tok | p50 6,611 · p95 11,109 ms | 1 | gold/scifact_200 gold | inspect |
| sf_mpnet_n200_v1 | ndcg_10 | 0.664 | 200 | $0.000000 | p50 61 · p95 92 ms | 1 | gold/scifact_200 gold | inspect |
| sf_minilm_n200_v1 | ndcg_10 | 0.653 | 200 | $0.000000 | p50 16 · p95 23 ms | 1 | gold/scifact_200 gold | inspect |
| sf_bm25_n200_v1 | ndcg_10 | 0.645 | 200 | $0.000000 | p50 20 · p95 38 ms | 1 | gold/scifact_200 gold | inspect |
| sf_random_n200_v1 | ndcg_10 | 0.001 | 200 | $0.000000 | p50 1.6 · p95 1.9 ms | 1 | gold/scifact_200 gold | inspect |
| sf_first_k_n200_v1 | ndcg_10 | 0.000 | 200 | $0.000000 | p50 0.23 · p95 0.38 ms | 1 | gold/scifact_200 gold | inspect |
02what's inside
200 items, text, 4–28 words (median 12, middle half 9–15).
Source: BeIR/scifact (queries split). The text stays there; each item below links to its row.
the gold
gold/scifact_200: a list per item, 1–5 entries, median 1
200 shown
| item | words | gold | open |
|---|---|---|---|
100 | 8 | 1 entries | source ↗ · inspect |
1012 | 9 | 1 entries | source ↗ · inspect |
1014 | 9 | 1 entries | source ↗ · inspect |
1020 | 22 | 1 entries | source ↗ · inspect |
1021 | 22 | 1 entries | source ↗ · inspect |
1024 | 11 | 1 entries | source ↗ · inspect |
1029 | 21 | 3 entries | source ↗ · inspect |
1041 | 15 | 2 entries | source ↗ · inspect |
1049 | 11 | 1 entries | source ↗ · inspect |
1088 | 12 | 1 entries | source ↗ · inspect |
1099 | 4 | 1 entries | source ↗ · inspect |
1104 | 24 | 1 entries | source ↗ · inspect |
1107 | 10 | 1 entries | source ↗ · inspect |
1110 | 8 | 1 entries | source ↗ · inspect |
1121 | 13 | 1 entries | source ↗ · inspect |
113 | 13 | 1 entries | source ↗ · inspect |
1130 | 17 | 1 entries | source ↗ · inspect |
1144 | 17 | 1 entries | source ↗ · inspect |
115 | 11 | 1 entries | source ↗ · inspect |
1163 | 10 | 1 entries | source ↗ · inspect |
1175 | 8 | 1 entries | source ↗ · inspect |
118 | 12 | 1 entries | source ↗ · inspect |
1185 | 28 | 1 entries | source ↗ · inspect |
1187 | 23 | 1 entries | source ↗ · inspect |
1191 | 11 | 1 entries | source ↗ · inspect |
1194 | 21 | 1 entries | source ↗ · inspect |
1196 | 12 | 1 entries | source ↗ · inspect |
1199 | 18 | 1 entries | source ↗ · inspect |
1200 | 21 | 1 entries | source ↗ · inspect |
1204 | 14 | 1 entries | source ↗ · inspect |
1207 | 20 | 1 entries | source ↗ · inspect |
1213 | 13 | 1 entries | source ↗ · inspect |
1216 | 11 | 1 entries | source ↗ · inspect |
1225 | 8 | 1 entries | source ↗ · inspect |
124 | 13 | 1 entries | source ↗ · inspect |
1245 | 10 | 1 entries | source ↗ · inspect |
1259 | 21 | 1 entries | source ↗ · inspect |
1262 | 12 | 1 entries | source ↗ · inspect |
1266 | 23 | 1 entries | source ↗ · inspect |
127 | 10 | 1 entries | source ↗ · inspect |
1270 | 14 | 1 entries | source ↗ · inspect |
1271 | 21 | 1 entries | source ↗ · inspect |
1272 | 12 | 1 entries | source ↗ · inspect |
1273 | 11 | 1 entries | source ↗ · inspect |
1274 | 25 | 3 entries | source ↗ · inspect |
1278 | 15 | 1 entries | source ↗ · inspect |
1279 | 12 | 1 entries | source ↗ · inspect |
128 | 8 | 1 entries | source ↗ · inspect |
1280 | 14 | 1 entries | source ↗ · inspect |
1281 | 10 | 1 entries | source ↗ · inspect |
1282 | 15 | 1 entries | source ↗ · inspect |
129 | 15 | 1 entries | source ↗ · inspect |
1292 | 10 | 1 entries | source ↗ · inspect |
1298 | 23 | 1 entries | source ↗ · inspect |
13 | 10 | 1 entries | source ↗ · inspect |
130 | 15 | 1 entries | source ↗ · inspect |
1303 | 7 | 1 entries | source ↗ · inspect |
1316 | 10 | 1 entries | source ↗ · inspect |
1319 | 10 | 1 entries | source ↗ · inspect |
132 | 6 | 1 entries | source ↗ · inspect |
1320 | 16 | 1 entries | source ↗ · inspect |
133 | 19 | 5 entries | source ↗ · inspect |
1335 | 9 | 1 entries | source ↗ · inspect |
1336 | 8 | 1 entries | source ↗ · inspect |
1339 | 13 | 1 entries | source ↗ · inspect |
1359 | 20 | 1 entries | source ↗ · inspect |
1362 | 8 | 1 entries | source ↗ · inspect |
137 | 13 | 1 entries | source ↗ · inspect |
1379 | 16 | 4 entries | source ↗ · inspect |
1385 | 6 | 1 entries | source ↗ · inspect |
1389 | 8 | 1 entries | source ↗ · inspect |
1395 | 21 | 1 entries | source ↗ · inspect |
142 | 20 | 1 entries | source ↗ · inspect |
143 | 17 | 1 entries | source ↗ · inspect |
146 | 18 | 1 entries | source ↗ · inspect |
148 | 5 | 1 entries | source ↗ · inspect |
163 | 9 | 1 entries | source ↗ · inspect |
179 | 7 | 4 entries | source ↗ · inspect |
180 | 19 | 1 entries | source ↗ · inspect |
185 | 9 | 1 entries | source ↗ · inspect |
198 | 5 | 1 entries | source ↗ · inspect |
212 | 7 | 1 entries | source ↗ · inspect |
213 | 14 | 1 entries | source ↗ · inspect |
216 | 9 | 1 entries | source ↗ · inspect |
218 | 8 | 1 entries | source ↗ · inspect |
219 | 8 | 1 entries | source ↗ · inspect |
232 | 12 | 1 entries | source ↗ · inspect |
236 | 10 | 1 entries | source ↗ · inspect |
239 | 8 | 1 entries | source ↗ · inspect |
248 | 7 | 1 entries | source ↗ · inspect |
249 | 7 | 1 entries | source ↗ · inspect |
269 | 5 | 1 entries | source ↗ · inspect |
275 | 14 | 3 entries | source ↗ · inspect |
294 | 12 | 1 entries | source ↗ · inspect |
295 | 18 | 1 entries | source ↗ · inspect |
303 | 13 | 1 entries | source ↗ · inspect |
312 | 14 | 1 entries | source ↗ · inspect |
324 | 5 | 1 entries | source ↗ · inspect |
343 | 15 | 2 entries | source ↗ · inspect |
350 | 14 | 1 entries | source ↗ · inspect |
362 | 24 | 1 entries | source ↗ · inspect |
380 | 12 | 1 entries | source ↗ · inspect |
385 | 13 | 2 entries | source ↗ · inspect |
386 | 16 | 1 entries | source ↗ · inspect |
388 | 9 | 1 entries | source ↗ · inspect |
399 | 11 | 1 entries | source ↗ · inspect |
410 | 9 | 1 entries | source ↗ · inspect |
411 | 9 | 1 entries | source ↗ · inspect |
415 | 13 | 1 entries | source ↗ · inspect |
421 | 13 | 1 entries | source ↗ · inspect |
431 | 12 | 1 entries | source ↗ · inspect |
436 | 13 | 1 entries | source ↗ · inspect |
439 | 13 | 1 entries | source ↗ · inspect |
440 | 13 | 1 entries | source ↗ · inspect |
443 | 9 | 1 entries | source ↗ · inspect |
475 | 10 | 1 entries | source ↗ · inspect |
478 | 24 | 1 entries | source ↗ · inspect |
48 | 14 | 1 entries | source ↗ · inspect |
491 | 14 | 1 entries | source ↗ · inspect |
50 | 7 | 1 entries | source ↗ · inspect |
501 | 7 | 1 entries | source ↗ · inspect |
502 | 16 | 1 entries | source ↗ · inspect |
507 | 15 | 1 entries | source ↗ · inspect |
51 | 9 | 1 entries | source ↗ · inspect |
514 | 19 | 1 entries | source ↗ · inspect |
517 | 8 | 1 entries | source ↗ · inspect |
521 | 25 | 1 entries | source ↗ · inspect |
527 | 17 | 1 entries | source ↗ · inspect |
528 | 24 | 1 entries | source ↗ · inspect |
53 | 10 | 1 entries | source ↗ · inspect |
532 | 7 | 1 entries | source ↗ · inspect |
533 | 7 | 1 entries | source ↗ · inspect |
535 | 9 | 1 entries | source ↗ · inspect |
539 | 6 | 1 entries | source ↗ · inspect |
540 | 8 | 2 entries | source ↗ · inspect |
544 | 9 | 1 entries | source ↗ · inspect |
549 | 7 | 1 entries | source ↗ · inspect |
552 | 20 | 1 entries | source ↗ · inspect |
56 | 15 | 1 entries | source ↗ · inspect |
560 | 13 | 1 entries | source ↗ · inspect |
57 | 15 | 1 entries | source ↗ · inspect |
577 | 24 | 1 entries | source ↗ · inspect |
589 | 21 | 1 entries | source ↗ · inspect |
597 | 7 | 3 entries | source ↗ · inspect |
598 | 21 | 1 entries | source ↗ · inspect |
619 | 15 | 2 entries | source ↗ · inspect |
623 | 13 | 1 entries | source ↗ · inspect |
637 | 13 | 1 entries | source ↗ · inspect |
641 | 9 | 2 entries | source ↗ · inspect |
649 | 13 | 1 entries | source ↗ · inspect |
659 | 7 | 1 entries | source ↗ · inspect |
660 | 6 | 1 entries | source ↗ · inspect |
674 | 11 | 1 entries | source ↗ · inspect |
690 | 19 | 1 entries | source ↗ · inspect |
692 | 10 | 1 entries | source ↗ · inspect |
693 | 10 | 1 entries | source ↗ · inspect |
70 | 6 | 2 entries | source ↗ · inspect |
700 | 11 | 1 entries | source ↗ · inspect |
702 | 12 | 1 entries | source ↗ · inspect |
715 | 15 | 1 entries | source ↗ · inspect |
721 | 13 | 1 entries | source ↗ · inspect |
723 | 15 | 1 entries | source ↗ · inspect |
728 | 12 | 2 entries | source ↗ · inspect |
729 | 11 | 1 entries | source ↗ · inspect |
75 | 15 | 1 entries | source ↗ · inspect |
759 | 19 | 1 entries | source ↗ · inspect |
768 | 11 | 1 entries | source ↗ · inspect |
775 | 16 | 1 entries | source ↗ · inspect |
781 | 14 | 1 entries | source ↗ · inspect |
783 | 13 | 1 entries | source ↗ · inspect |
785 | 15 | 1 entries | source ↗ · inspect |
800 | 19 | 1 entries | source ↗ · inspect |
805 | 7 | 1 entries | source ↗ · inspect |
808 | 9 | 1 entries | source ↗ · inspect |
821 | 8 | 1 entries | source ↗ · inspect |
831 | 12 | 1 entries | source ↗ · inspect |
832 | 6 | 1 entries | source ↗ · inspect |
834 | 10 | 1 entries | source ↗ · inspect |
837 | 8 | 1 entries | source ↗ · inspect |
839 | 14 | 1 entries | source ↗ · inspect |
847 | 18 | 1 entries | source ↗ · inspect |
859 | 7 | 1 entries | source ↗ · inspect |
870 | 4 | 1 entries | source ↗ · inspect |
879 | 10 | 1 entries | source ↗ · inspect |
880 | 8 | 1 entries | source ↗ · inspect |
882 | 10 | 1 entries | source ↗ · inspect |
887 | 12 | 1 entries | source ↗ · inspect |
903 | 9 | 1 entries | source ↗ · inspect |
907 | 17 | 1 entries | source ↗ · inspect |
911 | 14 | 1 entries | source ↗ · inspect |
913 | 6 | 1 entries | source ↗ · inspect |
914 | 7 | 1 entries | source ↗ · inspect |
921 | 10 | 1 entries | source ↗ · inspect |
922 | 12 | 1 entries | source ↗ · inspect |
936 | 7 | 1 entries | source ↗ · inspect |
956 | 13 | 1 entries | source ↗ · inspect |
960 | 5 | 1 entries | source ↗ · inspect |
967 | 9 | 2 entries | source ↗ · inspect |
975 | 9 | 1 entries | source ↗ · inspect |
993 | 10 | 1 entries | source ↗ · inspect |
Each row is the profile evh dataset-items --dataset-id scifact_200 prints; inspect opens the item in the inspector with the dataset's largest group of comparable arms, and its text where the inspector runs on a machine that has it.
03reference sets
- gold/scifact_200 gold · promoted from person the dataset's authors
04labels
No labels yet. A sheet to fill: evh label-sheet --dataset-id scifact_200 --out sheet.csv; a run's outputs as a model set: evh label-from-run --run R --set S.