Skip to content

Reference — every metric, model and dataset in this repo, and where each one comes from

A single index for the five worked examples. Nothing here is a summary of results; it is the glossary, so that reading an example does not require already knowing what rougeLsum recall or nDCG@10 is, or which weights gemma_m actually loads.

Every row links to the primary source: the HuggingFace repo for a model or dataset, the paper or specification for a metric.

  • Metrics — what each number means, how it is computed here, and what it cannot see
  • Success criteria per experiment — what "good" meant, and why
  • Models — every arm, with the exact weights or upstream model id
  • Datasets — every corpus, its licence, and its download recipe
  • Statistical tests — how separation is decided
  • Terminology — the canonical name for every experiment, arm and term
  • papers/ — every cited work with the redistribution licence read from the publisher's page, and a fetcher for the twelve that permit it

Metrics

Each example declares PRIMARY_METRIC (the one thing arms are ranked by) and METRIC_KINDS, which splits everything else into quality (a claim about how good the output is) and descriptive (a diagnostic that must never be ranked). The split is enforced in code, not convention: harness/scripts/leaderboard.py will not rank a descriptive metric.

Summarisation — continuous overlap

metric kind what it is
coverage ← primary quality ROUGE-Lsum recall against the human reference, after clipping the output to a word budget. "How much of what a human thought mattered did it get?"
concision quality ROUGE-Lsum precision over the whole output. "What share of what it wrote earned its place." Padding is punished here exactly as much as raw recall rewards it.
rouge1 quality Unigram overlap — ROUGE in its simplest form. Reported because it is what most papers quote.
rougeLsum quality Longest-common-subsequence overlap, computed per sentence and unioned. Rewards preserving order, unlike rouge1.
grounding quality Ours, ~20 lines, not comparable to anyone else's number. Share of the summary's bigrams that also appear in the article. High = extractive and safe; low = abstractive, which is either a good paraphrase or a fabrication. It cannot tell those apart — but it can tell you the model left the source behind, which ROUGE cannot.
compression, length_vs_reference descriptive Output length relative to article and to the human reference. ROUGE rises with length, so this is how a verbose winner gets caught.
fmt_bullets, fmt_markdown, fmt_narration, fmt_label, fmt_paragraphs, fmt_overlong descriptive Format compliance. A model that answers with a markdown heading is not wrong about the content, and this keeps that out of the quality number.

What ROUGE cannot see: it is n-gram overlap. A factually wrong summary that reuses the article's wording scores well; a correct paraphrase scores badly. Read grounding beside it, never instead of it. ROUGE is the standard because it is cheap and reproducible, not because it is right — see Kryscinski et al., Neural Text Summarisation: A Critical Evaluation.

Implementation: google-research/rouge via the rouge-score package.

Classification — binary per item

metric kind what it is
correct ← primary quality 1 if the predicted label equals the gold label, else 0. Its mean is accuracy.
parsed quality Whether the output could be read as a label at all. Quality, not descriptive: an arm that cannot be parsed cannot be deployed, whatever it knows.
correct_after_repair descriptive Whether a lenient re-parse would have rescued it. The gap between this and correct is what the parser is worth — 8.6 accuracy points to one arm.
confidence descriptive The model's own probability for its answer. Only the local ML arms supply it; no hosted arm does, because logprobs were never requested.
answer_words, fmt_verbose, fmt_markdown descriptive Did it answer with one word, or a paragraph containing one.

Macro-F1 is not here, and that is deliberate. Macro-F1 needs a confusion matrix over the whole run — there is no per-item number whose mean is macro-F1. It lives in harness/scripts/classification_report.py instead. See scikit-learn's explanation of micro vs macro averaging.

Extraction (NER) — a set vs a set

metric kind what it is
f1 ← primary quality Harmonic mean of precision and recall over the entity set for one item, matched one-to-one after normalisation. Type must match — one of Few-NERD's 8 coarse types (person, location, organization, …).
precision, recall quality Of what it found, how much was right; of what was there, how much it found.
untyped_f1 quality The same, ignoring the type label. The pair is the point: f1 asks "did it label it right", untyped_f1 asks "did it find it at all".
type_penalty descriptive untyped_f1 − f1. Credit lost purely to mislabelling something it did find. A type-confusion problem is fixed differently from a detection problem, and this says which you have.
tp/fp/fn (+ untyped) descriptive Raw counts, so a report script can pool them into a corpus micro-F1 — a different number from the mean of per-item F1s, answering a different question.
parsed quality Could the output be read as a set. An unreadable answer scores 0, not "found nothing" — those are different, and conflating them once paid a non-answer 1.0.

Background: CoNLL-2003 shared task defines the standard exact-match entity F1 this follows.

Retrieval — a ranked list vs a set

metric kind what it is
ndcg_10 ← primary quality Normalised Discounted Cumulative Gain at rank 10. Position-weighted: a relevant document at rank 1 is worth more than at rank 10, discounted by 1/log2(rank+1). Normalised by the best possible ordering, so 1.0 is perfect.
recall_10, recall_100 quality Share of relevant documents appearing in the top 10 / top 100. recall_100 is what a two-stage pipeline depends on — a reranker cannot retrieve what the first stage never returned.
mrr_10 quality Mean Reciprocal Rank — 1/rank of the first relevant document, 0 if outside the top 10.
unknown_ids descriptive Document ids the model named that are not in the corpus. Hallucinated ids keep their slot in the ranking — dropping them would slide real documents up and make invention a scoring strategy.
duplicate_ids descriptive Ids repeated in one answer. Collapsed, not counted, or one document could fill the whole top-10.
rank_of_first, found_any, n_returned, n_relevant descriptive rank_of_first is never ranked: 0 means "never found", which would sort as the best possible position.

Gain function is 2^rel − 1, matching pytrec_eval and BEIR. On SciFact every judgment is binary, so this is indistinguishable from linear gain — it is declared anyway, because a metric that silently changes meaning between corpora is worse than one that is merely wrong.

Shared across all examples

metric kind what it is
cost_usd descriptive What the provider billed (usage.cost), not a price table. See REPORT_RETRIEVAL §7 for why that distinction cost four reports a correction.
latency_ms descriptive Per item, excluding warm-up. Index builds and model loads are reported separately as warmup_ms. A local arm's first one to three items still pay a one-off initialisation cost the load does not cover, so prefer the median for local arms (KNOWN_ISSUES.md). Figures from different machines are not comparable: see harness/data/runs-linux/.
truncated descriptive The provider stopped at max_tokens. Distinguishes "the model is bad" from "the budget was too small".
reasoning_tokens descriptive Tokens spent on hidden reasoning. All examples run reasoning: {enabled: false}; this verifies it took.

Success criteria per experiment

"Which arm is best" is decided by exactly one metric per example, fixed before any arm ran. Everything else is read after the ranking, to understand it rather than to change it.

Experiment Ranked by Floors (non-model arms, checked first) Read after ranking
Summarisation · CNN/DM coverage lead3 — the first three sentences. A model below it has not earned its inference cost. grounding (did it leave the source?), length_vs_reference (did it win by padding?), concision
Classification · AG News correct constant (always one class, ≈0.25) and keyword (~20 regex rules) parsed and correct_after_repair — the parser is part of the system; macro-F1 for concentrated blind spots; confidence calibration
Classification · DBpedia correct same two same, plus the label-noise pass — on a saturated task the field's consensus errors are the ceiling
Extraction · Few-NERD f1 nothing (predicts the empty set; must score exactly the empty-gold rate) and capitalized (every capitalised run) untyped_f1 and type_penalty — detection vs labelling; per-type breakdown; consensus type disagreements
Retrieval · SciFact ndcg_10 random (must score ≈ k/N) and first_k (corpus order) recall_100 — the pipeline ceiling; the oracle reordering; unknown_ids for invention

Why each example has floors that are not models. A floor is any non-model arm — a rule, a constant or a random draw. Most also have scores predictable in advance (nothing, random, constant, first_k), and those are calibration checks on the scorer before they are baselines on the task: if nothing does not score exactly the empty-gold rate, no other row in the table can be trusted. On Few-NERD this caught a gold entity that the normaliser deleted (the currency symbol £), which moved the floor by 1/280.

Models

Hosted — 24 arms, 8 vendors × 3 price tiers

Every example runs the same 24, at temperature 0 with reasoning disabled, so a difference between examples is the task and not the configuration. Measured through a LiteLLM proxy routing to OpenRouter; a rerun can go to OpenRouter directly with one key, through the same alias table (examples/models.json, Reproduce it).

Arm names are <vendor>_<tier>: _l / _m / _s = the vendor's large / medium / small price tier.

arm alias (examples/models.json) upstream model (OpenRouter) weights · licence
anthropic_l eval-opus-5 anthropic/claude-opus-5 proprietary
anthropic_m eval-claude-sonnet anthropic/claude-sonnet-5 proprietary
anthropic_s eval-claude-haiku anthropic/claude-haiku-4.5 proprietary
openai_l eval-gpt-6sol openai/gpt-6-sol proprietary
openai_m eval-gpt-55 openai/gpt-5.5 proprietary
openai_s eval-gpt-mini openai/gpt-5.4-mini proprietary
deepseek_l eval-deepseek-pro deepseek/deepseek-v4-pro open · MIT
deepseek_m eval-deepseek-41flash deepseek/deepseek-v4.1-flash open · MIT
deepseek_s eval-deepseek-flash deepseek/deepseek-v4-flash open · MIT
gemma_l eval-gemma-31b google/gemma-4-31b-it open · Apache-2.0
gemma_m eval-gemma-26b google/gemma-4-26b-a4b-it open · Apache-2.0
gemma_s eval-gemma-3-27b google/gemma-3-27b-it open · Gemma, gated
llama_l eval-llama-maverick meta-llama/llama-4-maverick open · Llama, gated
llama_m eval-llama-70b meta-llama/llama-3.3-70b-instruct open · Llama 3.3, gated
llama_s eval-llama-scout meta-llama/llama-4-scout open · Llama, gated
mistral_l eval-mistral-large mistralai/mistral-large-2512 open · Apache-2.0
mistral_m eval-mistral-medium mistralai/mistral-medium-3.1 proprietary
mistral_s eval-mistral-small mistralai/mistral-small-3.2-24b-instruct open · Apache-2.0
qwen_l eval-qwen-3max qwen/qwen3-max proprietary
qwen_m eval-qwen-37plus qwen/qwen3.7-plus proprietary
qwen_s eval-qwen-flash qwen/qwen3.8-flash proprietary
glm_l eval-glm-5 z-ai/glm-5 open · MIT
glm_m eval-glm-46 z-ai/glm-4.6 open · MIT
glm_s eval-glm-45-air z-ai/glm-4.5-air open · MIT

14 of these 24 have downloadable weights and could be self-hosted. Verified on 2026-09-29 against the HuggingFace API. Ten are unrestricted (MIT / Apache-2.0); four are gated behind licence acceptance and carry use conditions. An earlier version said 13 and inferred openness from OpenRouter's endpoint count — one provider implies closed — which is not a valid test: mistral_l is served only by Mistral and its weights are on HuggingFace under Apache-2.0. What that is worth is measured in REPORT_SYNTHESIS.md §0 — no proprietary model separated from the best open-weight model in any of the five experiments.

A hosted model has no verifiable identity. The fingerprint records identity_declared: true and revision_source: unavailable — we record the alias we asked for, not proof of what answered. A provider can change weights behind a name without notice. This is the single largest unverifiable assumption in the repo, and it is why the local arms record a revision and a weight-file size and the hosted ones cannot.

Local — downloaded weights, pinned by revision

Every one is loaded from HuggingFace. For the first ten, the commit SHA and model.safetensors byte size are recorded in the run fingerprint, so a silent upstream change is detectable. For five of the last six they are not (all but bert_base_ta): their runs record revision: null and no weight files (two snapshots in the cache — see KNOWN_ISSUES.md), so the revisions below were read from the cache by hand and are a record, not a fingerprint.

arm HuggingFace model weights size revision (HF commit) role
bart_l facebook/bart-large-cnn 1.6 GB 37f520fa929c BART fine-tuned on CNN/DailyMail — the in-distribution summariser
bart_mnli facebook/bart-large-mnli 1.6 GB d7645e127eaf Zero-shot classification by NLI entailment — no training on the task
bert_mini mrm8488/bert-mini-finetuned-age_news-classification 44 MB e94b474cce6f Fine-tuned on AG News. The 44 MB model that beat 24 LLMs
span_marker guishe/span-marker-generic-ner-v1-fewnerd-fine-super 499 MB f836dfa9b6b0 SpanMarker fine-tuned on Few-NERD, predicting the 66 fine types mapped down to 8
gliner urchade/gliner_medium-v2.1 781 MB 40ec419335d0 GLiNER — zero-shot NER where the caller supplies the label set. The matched control for span_marker
minilm sentence-transformers/all-MiniLM-L6-v2 91 MB 1110a243fdf4 General sentence-similarity encoder, symmetric, no prefix
mpnet sentence-transformers/all-mpnet-base-v2 438 MB e8c3b32edf54 The larger general encoder. Same recipe as MiniLM, so the pair isolates size
bge_small BAAI/bge-small-en-v1.5 133 MB 5c38ec7c405e Trained for retrieval, with an asymmetric query instruction. Pairs with MiniLM to isolate training objective
e5_base intfloat/e5-base-v2 438 MB f52bf8ec8c71 Retrieval-trained, asymmetric on both sides (query: / passage:)
bart_m sshleifer/distilbart-cnn-12-6 1.2 GB a4f8f3ea906e BART-large-CNN distilled to 12 encoder / 6 decoder layers
bart_s sshleifer/distilbart-cnn-6-6 460 MB d2fde4ca965b Distilled to 6 / 6 layers
bart_l_xsum facebook/bart-large-xsum 1.6 GB 2179ab81d3f1 BART-large fine-tuned on XSum — same task, another corpus. The out-of-distribution control for bart_l
bert_base_ta textattack/bert-base-uncased-ag-news 438 MB fe417ad660b1 BERT-base fine-tuned on AG News (TextAttack; card reports 0.9514)
bert_base_fy (AG) fabriceyhc/bert-base-uncased-ag_news 438 MB c14e3b32fe1f BERT-base fine-tuned on AG News by a second author — a check on the first
bert_base_fy (DBpedia) fabriceyhc/bert-base-uncased-dbpedia_14 438 MB 1fbfc3deaa28 BERT-base fine-tuned on DBpedia-14

The last six need torch ≥ 2.6, and ran on 2026-09-30 on Linux. Their main branches ship only pytorch_model.bin, which transformers refuses to torch.load below torch 2.6 (CVE-2025-32434); x86_64 macOS has no torch above 2.2.2. The revisions above are the main commits the runs loaded — read from the local cache, because these runs' fingerprints record revision: null (see KNOWN_ISSUES.md). Each also has a safetensors conversion PR on the Hub — refs/pr/29 (12-6), refs/pr/2 (6-6), refs/pr/9 (xsum) and refs/pr/1 for the three BERTs. On torch 2.2.2, through each arm's own adapter (evidence/conversion_pr_check.py, results in conversion_pr_check.json), five of the six reproduce the recorded outputs byte for byte: the three BERTs on every item (200, 200, 280) and bart_m and bart_l_xsum on 20 of 20. bart_s does not run: its PR is fp16, as its pickle is, and torch 2.2.2 has no fp16 LayerNorm on CPU. Cast to fp32 it matches 19 of 20, but that is a different computation from the recorded run, which ran at fp16.

Non-model arms

The floors. Each is first a check on the scorer — its score can be stated before it runs, so a mismatch indicts the scorer, not a model — and only then a bar for models to clear. bm25 is the exception: a real retrieval baseline, listed here because it uses no model weights.

arm what it does expected score
lead3 first three sentences of the article the extractive floor for summarisation
constant always the same class ≈ 1/num_classes
keyword ~20 hand-written regex rules whatever rules alone are worth
nothing predicts the empty set exactly the empty-gold rate
capitalized every capitalised run, minus a stoplist high untyped recall, near-random types
random a seeded permutation of the corpus ≈ k/N for recall@k
first_k the corpus in id order ≈ 0
bm25 Okapi BM25 via rank_bm25 no fixed prediction — a strong baseline that beats weak dense models on SciFact (calibrated: 0.6523 vs BEIR's 0.665)

Datasets

What of each is in this repository, what it asks of you, and what the committed run outputs contain: Data and licences.

None is committed. Each example ships a fetch.py download recipe; the slice lands on your machine and is gitignored. Dataset identity travels as items_sha256 (and corpus_sha256 for retrieval), not as bytes in the repo.

Dataset Source Licence Used for
CNN/DailyMail abisee/cnn_dailymail Apache-2.0 cnn_dailymail_200: the first 200 test articles — all from CNN (the split puts CNN's 9% first; 91% of it is DailyMail). cnn_dailymail_dm200: 200 DailyMail test articles, seeded random. Human highlights as gold. Corpus from Hermann et al. 2015; adapted for summarisation by Nallapati et al. 2016
AG News fancyzhx/ag_news unknown on the card; the corpus description says non-commercial research use 200 snippets, 4 topics (paper)
DBpedia-14 fancyzhx/dbpedia_14 CC-BY-SA 3.0 + GFDL, inherited from Wikipedia 280 abstracts, 14 ontology classes
Few-NERD DFKI-SLT/few-nerd CC BY-SA 4.0 — the cleanest of the five 280 sentences, 8 coarse entity types (paper)
BEIR SciFact BeIR/scifact + BeIR/scifact-qrels CC BY-SA 4.0 200 claims over a 5,183-abstract corpus (BEIR, SciFact)

Licence is a selection criterion here, not a footnote. Few-NERD was chosen over WNUT-2017 — which is closer to noisy podcast speech, the actual application — purely because WNUT is licensed other. That is relevance given up for a clean grant, and the NER report says so.

Statistical tests

Ranking arms by a mean is not a result until you know which gaps survive noise.

Tool What it answers Method
family_test.py Is arm A better than each of a pre-declared family? Sign-flip permutation on paired per-item scores, then Holm step-down over the whole family. The family is declared before p-values are read — otherwise you are choosing the comparison after seeing the answer.
holdout_significance.py Does the ordering hold on items a smaller slice never contained? Friedman-style permutation on within-item ranks, plus a Nemenyi critical difference.
rank_stability.py How many items before the leaderboard stops moving? Two disjoint halves, ranked independently, correlated by Spearman's ρ with midranks for ties. Neither half is treated as truth.
bootstrap_test.py Confidence interval for macro-F1, which has no per-item value Paired percentile bootstrap. Documented as anti-conservative — 11 separations against the exact test's 8.
rescore.py What would this look like under a different scorer? Recomputes from stored outputs. No API calls, no money. This is why a scorer bug costs nothing to fix.

Why Holm and not Bonferroni: Holm is uniformly more powerful and controls the same family-wise error rate. Why permutation and not a t-test: these per-item scores are not normal — correct is Bernoulli, ndcg_10 on a single-relevant query takes one of eleven values — and permutation assumes only exchangeability under the null.

Further reading

Terminology

Fixed names, used the same way in every document here. Where a table's first column is "experiment", every cell in it reads Task · Dataset — mixing a task name in one row with a dataset name in the next is what this convention exists to stop.

canonical label task dataset report
Summarisation · CNN/DM summarisation CNN/DailyMail REPORT_SUMMARIZATION
Classification · AG News classification AG News REPORT_CLASSIFICATION
Classification · DBpedia classification DBpedia-14 REPORT_CLASSIFICATION
Extraction · Few-NERD extraction (NER) Few-NERD REPORT_NER
Retrieval · SciFact retrieval BEIR SciFact REPORT_RETRIEVAL

Two experiments share the classification task and differ only in dataset, which is why the dataset is never optional in these labels.

Other fixed terms

term means instead of
arm one configuration under test — a model plus its prompt, parameters and parsing "model", which is only part of an arm
experiment one task on one dataset, with its own report section "dataset" or "task" alone
dev slice / measurement slice the small tuning set / the frozen set results are reported on "train/test", which implies we trained something
floor a non-model arm (rule, constant or random: lead3, keyword, constant, nothing, capitalized, random, first_k); those with a score predictable in advance double as scorer calibration checks "baseline" — the predictable ones are scorer checks first
separated the difference survives Holm step-down at α = 0.05 "better", which only needs a point estimate
billed what the provider charged (usage.cost) "cost", which used to mean a price-table estimate
local ML / open-weight / proprietary the three tiers in REPORT_SYNTHESIS §0 "free vs paid", which conflates tiers 1 and 2

Spelling

Prose uses British -isation (summarisation, normalisation, normaliser) — the majority form across this repo. Identifiers keep whatever they were named, because renaming them breaks paths and hashes: REPORT_SUMMARIZATION.md, examples/summarization-cnn-dailymail/, normalizer_sha256, normalize(). The two spellings coexisting in one sentence is expected when one of them is code.

Credits

Everything below is somebody else's work. Versions are the ones actually recorded in the run fingerprints, not the ones a lockfile would prefer.

Datasets

Dataset Authors Licence
CNN/DailyMail Hermann et al. (2015); summarisation framing by Nallapati et al. (2016); split by See et al. (2017) Apache-2.0
AG News Zhang, Zhao & LeCun (2015); corpus by ComeToMyHead card says unknown; corpus states non-commercial research use
DBpedia-14 Lehmann et al., DBpedia; split by Zhang et al. (2015) CC-BY-SA 3.0 + GFDL
Few-NERD Ding et al. (2021) CC BY-SA 4.0
BEIR SciFact Wadden et al. (2020); BEIR by Thakur et al. (2021) CC BY-SA 4.0

Models

Licences below were read from the HuggingFace API on 2026-09-29, not assumed. Two state no licence at all, which is a reuse blocker rather than a permission — follow each link and check before reusing any of them commercially.

model licence
facebook/bart-large-cnn MIT
facebook/bart-large-mnli MIT
facebook/bart-large-xsum MIT
BAAI/bge-small-en-v1.5 MIT
intfloat/e5-base-v2 MIT
sentence-transformers/all-MiniLM-L6-v2 Apache-2.0
sentence-transformers/all-mpnet-base-v2 Apache-2.0
urchade/gliner_medium-v2.1 Apache-2.0
sshleifer/distilbart-cnn-12-6 Apache-2.0
sshleifer/distilbart-cnn-6-6 Apache-2.0
fabriceyhc/bert-base-uncased-ag_news Apache-2.0
fabriceyhc/bert-base-uncased-dbpedia_14 Apache-2.0
guishe/span-marker-generic-ner-v1-fewnerd-fine-super CC-BY-SA-4.0
mrm8488/bert-mini-finetuned-age_news-classification none stated
textattack/bert-base-uncased-ag-news none stated

An earlier version of this line claimed the set was "variously MIT, Apache-2.0 and CC-BY-NC-4.0". No model here is CC-BY-NC-4.0; one is CC-BY-SA-4.0 and two state nothing. That was asserted rather than checked, and it is the same error the paper citations had.

facebook/bart-large-cnn · facebook/bart-large-mnli · mrm8488/bert-mini-finetuned-age_news-classification · guishe/span-marker-generic-ner-v1-fewnerd-fine-super · urchade/gliner_medium-v2.1 · sentence-transformers/all-MiniLM-L6-v2 · sentence-transformers/all-mpnet-base-v2 · BAAI/bge-small-en-v1.5 · intfloat/e5-base-v2

Underlying architectures: BART, BERT, MPNet, DeBERTa-v3 (GLiNER's encoder), Sentence-BERT, E5, BGE, in C-Pack, GLiNER, SpanMarker.

Software

Package Version recorded Licence
transformers 4.55.4 Apache-2.0
torch 2.2.2 (macOS runs) · 2.14.0 (Linux runs) BSD-3-Clause
sentence-transformers 3.4.1 Apache-2.0
datasets 5.0.1 Apache-2.0
rouge-score 0.1.2 Apache-2.0
rank_bm25 — (not recorded; lockfile has 0.2.2) Apache-2.0
gliner — (not recorded; requires ≥ 0.2.13) Apache-2.0
span-marker — (not recorded; requires ≥ 1.5) Apache-2.0
scikit-learn 1.9.1 BSD-3-Clause
numpy 1.26.4 or 2.4.6, varies by run BSD-3-Clause
openai (client only) 2.48.0 – 3.20.0, varies by run Apache-2.0
LiteLLM (proxy, not a library) — (not recorded) MIT
uv (environment manager) — (not recorded) Apache-2.0 / MIT

Infrastructure

Hosted inference is routed through OpenRouter, which resells capacity from Anthropic, OpenAI, Google, Meta, Mistral, DeepSeek, Alibaba (Qwen), Z.ai and their serving partners (Novita, DeepInfra, Nebius, Parasail and others). Dataset slices are fetched through the HuggingFace datasets-server.

Method

The statistical approach follows standard practice rather than inventing any: Holm step-down (Holm 1979 (A Simple Sequentially Rejective Multiple Test Procedure, Scandinavian Journal of Statistics 6(2):65–70)), Friedman and Nemenyi for multiple classifiers over multiple datasets (Demšar 2006), and permutation tests as described in Dror et al., The Hitchhiker's Guide to Testing Statistical Significance in NLP.

No corpus, no model weight, no paper and no provider output is redistributed by this repository. Every example ships a download recipe instead, and papers/README.md records the licence of each cited work — twelve of the twenty-four may be mirrored, twelve may not, and the split follows publisher policy rather than importance.

Every citation above was verified on 2026-09-29 by fetching it and comparing the real title to the claim made about it. That check found four errors in the first draft: a fabricated paper title, a GitHub advisory ID pointing at an unrelated NuGet CVE, a dataset credited to the wrong authors, and a moved repository. One citation — Holm (1979) — could not be verified programmatically and is therefore given as a full text reference rather than a link.