goals
Goals
What each evaluation is for: who uses the output, the decision it informs, and the measure that decides. Runs name their goal in their config (goal:).
all time, 2026-09-25 to 2026-11-02
01goals
| goal | statement | decides | recommended | datasets | configs filed | runs in period | spend in period | changes in period |
|---|---|---|---|---|---|---|---|---|
| Named-entity extraction (Few-NERD) | Pick an extractor that finds the named entities in a sentence and types them into eight coarse types. | f1 (higher) | few_nerd_280: fn_ | few_ few_ | 28 | 28 | — | 3 |
| Encyclopedia topic classification (DBpedia) | Pick a classifier that labels an encyclopedia abstract with one of fourteen classes, at a known cost per item. | correct (higher) | dbpedia_280: db_ | dbpedia_ dbpedia_ | 28 | 29 | — | 5 |
| Short news summaries (CNN) | Pick a model that writes a short summary of a CNN-style news article, at a known cost, for a reader who wants the story's main points. | coverage (higher) | cnn_dailymail_200: cnn_ | cnn_ cnn_ cnn_ | 36 | 42 | — | 7 |
| News topic classification (AG News) | Pick a classifier that labels a news headline and snippet with one of four topics, at a known cost per item. | correct (higher) | none yet | ag_ ag_ | 30 | 33 | — | 5 |
| Evidence retrieval for scientific claims (SciFact) | Pick a retriever that ranks the abstracts supporting or refuting a scientific claim at the top, over the full corpus. | ndcg_10 (higher) | scifact_200: sf_ | scifact_ scifact_ | 23 | 26 | — | 7 |
| A scorer that agrees with experts on summaries (SummEval) | Find a summary scorer whose ranking of summaries agrees with expert judges, so it can stand in for them. | tau (higher) | se_ | summeval_ | 35 | 41 | — | 5 |
| Triage decisions on Orrery's error signals | Decide for each error signal whether to file an issue, dismiss it, escalate it to a person, or clean it up, the way the person who later closed the issue would have wanted. | triage_cost (lower) | none yet | orrery_ | 78 | 91 | — | 7 |
02what happened in the period
Nothing happened to a goal in this period.