metric

`fmt_verbose`

better: neutral: a description
kind: descriptive: says what the output was, never whether it was good
decides: no goal
weighed by: no goal
recorded on: ag_news_200, dbpedia_280

01what it measures, and how

1 when the reply ran past two words, where the prompt asks for the label alone.

A project explains its own metric at length with METRIC_DESCRIPTIONS = {"fmt_verbose": {"summary": "…", "details": "…"}}.

02on ag_news_200

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

arm	mean fmt_verbose	items	scored against
ag_glm_l_n200_v1	0.090	200	gold/ag_news_200 gold
ag_anthropic_l_n200_v1	0.000	200	gold/ag_news_200 gold
ag_anthropic_m_n200_v1	0.000	200	gold/ag_news_200 gold
ag_anthropic_s_n200_v1	0.000	200	gold/ag_news_200 gold
ag_bart_mnli_n200_v1	0.000	200	gold/ag_news_200 gold
ag_bert_base_fy_n200_v1	0.000	200	gold/ag_news_200 gold
ag_bert_base_ta_n200_v1	0.000	200	gold/ag_news_200 gold
ag_bert_mini_n200_v1	0.000	200	gold/ag_news_200 gold
ag_constant_n200_v1	0.000	200	gold/ag_news_200 gold
ag_deepseek_l_n200_v1	0.000	200	gold/ag_news_200 gold
ag_deepseek_m_n200_v1	0.000	200	gold/ag_news_200 gold
ag_deepseek_s_n200_v1	0.000	200	gold/ag_news_200 gold
ag_gemma_l_n200_v1	0.000	200	gold/ag_news_200 gold
ag_gemma_m_n200_v1	0.000	200	gold/ag_news_200 gold
ag_gemma_s_n200_v1	0.000	200	gold/ag_news_200 gold
ag_glm_m_n200_v1	0.000	200	gold/ag_news_200 gold
ag_glm_s_n200_v1	0.000	200	gold/ag_news_200 gold
ag_keyword_n200_v1	0.000	200	gold/ag_news_200 gold
ag_llama_l_n200_v1	0.000	200	gold/ag_news_200 gold
ag_llama_m_n200_v1	0.000	200	gold/ag_news_200 gold
ag_llama_s_n200_v1	0.000	200	gold/ag_news_200 gold
ag_mistral_l_n200_v1	0.000	200	gold/ag_news_200 gold
ag_mistral_m_n200_v1	0.000	200	gold/ag_news_200 gold
ag_mistral_s_n200_v1	0.000	200	gold/ag_news_200 gold
ag_openai_l_n200_v1	0.000	200	gold/ag_news_200 gold
ag_openai_m_n200_v1	0.000	200	gold/ag_news_200 gold
ag_openai_s_n200_v1	0.000	200	gold/ag_news_200 gold
ag_qwen_l_n200_v1	0.000	200	gold/ag_news_200 gold
ag_qwen_m_n200_v1	0.000	200	gold/ag_news_200 gold
ag_qwen_s_n200_v1	0.000	200	gold/ag_news_200 gold
ag_bert_base_fy_n200_v1 · linux	0.000	200	gold/ag_news_200 gold
ag_bert_base_ta_n200_v1 · linux	0.000	200	gold/ag_news_200 gold
ag_bert_mini_n200_v1 · linux	0.000	200	gold/ag_news_200 gold
ag_glm_l_n200_v1 · adjudicated	0.090	200	gold/ag_news_200_adj1 gold
ag_anthropic_l_n200_v1 · adjudicated	0.000	200	gold/ag_news_200_adj1 gold
ag_anthropic_m_n200_v1 · adjudicated	0.000	200	gold/ag_news_200_adj1 gold
ag_anthropic_s_n200_v1 · adjudicated	0.000	200	gold/ag_news_200_adj1 gold
ag_bart_mnli_n200_v1 · adjudicated	0.000	200	gold/ag_news_200_adj1 gold
ag_bert_base_fy_n200_v1 · adjudicated	0.000	200	gold/ag_news_200_adj1 gold
ag_bert_base_ta_n200_v1 · adjudicated	0.000	200	gold/ag_news_200_adj1 gold
ag_bert_mini_n200_v1 · adjudicated	0.000	200	gold/ag_news_200_adj1 gold
ag_constant_n200_v1 · adjudicated	0.000	200	gold/ag_news_200_adj1 gold
ag_deepseek_l_n200_v1 · adjudicated	0.000	200	gold/ag_news_200_adj1 gold
ag_deepseek_m_n200_v1 · adjudicated	0.000	200	gold/ag_news_200_adj1 gold
ag_deepseek_s_n200_v1 · adjudicated	0.000	200	gold/ag_news_200_adj1 gold
ag_gemma_l_n200_v1 · adjudicated	0.000	200	gold/ag_news_200_adj1 gold
ag_gemma_m_n200_v1 · adjudicated	0.000	200	gold/ag_news_200_adj1 gold
ag_gemma_s_n200_v1 · adjudicated	0.000	200	gold/ag_news_200_adj1 gold
ag_glm_m_n200_v1 · adjudicated	0.000	200	gold/ag_news_200_adj1 gold
ag_glm_s_n200_v1 · adjudicated	0.000	200	gold/ag_news_200_adj1 gold
ag_keyword_n200_v1 · adjudicated	0.000	200	gold/ag_news_200_adj1 gold
ag_llama_l_n200_v1 · adjudicated	0.000	200	gold/ag_news_200_adj1 gold
ag_llama_m_n200_v1 · adjudicated	0.000	200	gold/ag_news_200_adj1 gold
ag_llama_s_n200_v1 · adjudicated	0.000	200	gold/ag_news_200_adj1 gold
ag_mistral_l_n200_v1 · adjudicated	0.000	200	gold/ag_news_200_adj1 gold
ag_mistral_m_n200_v1 · adjudicated	0.000	200	gold/ag_news_200_adj1 gold
ag_mistral_s_n200_v1 · adjudicated	0.000	200	gold/ag_news_200_adj1 gold
ag_openai_l_n200_v1 · adjudicated	0.000	200	gold/ag_news_200_adj1 gold
ag_openai_m_n200_v1 · adjudicated	0.000	200	gold/ag_news_200_adj1 gold
ag_openai_s_n200_v1 · adjudicated	0.000	200	gold/ag_news_200_adj1 gold
ag_qwen_l_n200_v1 · adjudicated	0.000	200	gold/ag_news_200_adj1 gold
ag_qwen_m_n200_v1 · adjudicated	0.000	200	gold/ag_news_200_adj1 gold
ag_qwen_s_n200_v1 · adjudicated	0.000	200	gold/ag_news_200_adj1 gold

03on dbpedia_280

scored against silverscored by a judgewrote the reference it is scored againstscored against a label set, not a reference setunmarked rows are scored against gold

arm	mean fmt_verbose	items	scored against
db_glm_l_n200_v1	0.100	280	gold/dbpedia_280 gold
db_anthropic_s_n200_v1	0.007	280	gold/dbpedia_280 gold
db_deepseek_m_n200_v1	0.004	280	gold/dbpedia_280 gold
db_llama_s_n200_v1	0.004	280	gold/dbpedia_280 gold
db_anthropic_l_n200_v1	0.000	280	gold/dbpedia_280 gold
db_anthropic_m_n200_v1	0.000	280	gold/dbpedia_280 gold
db_bart_mnli_n200_v1	0.000	280	gold/dbpedia_280 gold
db_bert_base_fy_n200_v1	0.000	280	gold/dbpedia_280 gold
db_constant_n200_v1	0.000	280	gold/dbpedia_280 gold
db_deepseek_l_n200_v1	0.000	280	gold/dbpedia_280 gold
db_deepseek_s_n200_v1	0.000	280	gold/dbpedia_280 gold
db_gemma_l_n200_v1	0.000	280	gold/dbpedia_280 gold
db_gemma_m_n200_v1	0.000	280	gold/dbpedia_280 gold
db_gemma_s_n200_v1	0.000	280	gold/dbpedia_280 gold
db_glm_m_n200_v1	0.000	280	gold/dbpedia_280 gold
db_glm_s_n200_v1	0.000	280	gold/dbpedia_280 gold
db_keyword_n200_v1	0.000	280	gold/dbpedia_280 gold
db_llama_l_n200_v1	0.000	280	gold/dbpedia_280 gold
db_llama_m_n200_v1	0.000	280	gold/dbpedia_280 gold
db_mistral_l_n200_v1	0.000	280	gold/dbpedia_280 gold
db_mistral_m_n200_v1	0.000	280	gold/dbpedia_280 gold
db_mistral_s_n200_v1	0.000	280	gold/dbpedia_280 gold
db_openai_l_n200_v1	0.000	280	gold/dbpedia_280 gold
db_openai_m_n200_v1	0.000	280	gold/dbpedia_280 gold
db_openai_s_n200_v1	0.000	280	gold/dbpedia_280 gold
db_qwen_l_n200_v1	0.000	280	gold/dbpedia_280 gold
db_qwen_m_n200_v1	0.000	280	gold/dbpedia_280 gold
db_qwen_s_n200_v1	0.000	280	gold/dbpedia_280 gold
db_bert_base_fy_n200_v1 · linux	0.000	280	gold/dbpedia_280 gold