Leaderboard

Overview

Each task's main metric on its test rows, in percent: accuracy for choice questions, balanced accuracy for yes or no questions, accuracy within one level for scores, and field F1 for extraction. In parentheses: the share of rows the provider can answer on its own at 95 percent accuracy, with the threshold fitted on the calibration rows (only for providers that report a confidence). n/a: the provider does not answer this kind of question. Status: verified runs were done by the maintainers; submitted runs were sent in by others and checked by CI.

Provider	Status	banking77 (accuracy)	clinc150 (accuracy)	massive (accuracy)	multiwoz (accuracy)	jailbreak (balanced)	civil_comments (balanced)	helpsteer2 (within one)	wnut17 (field F1)
GLiNER 2.5 base	verified	75.8 (35)	58.8 (3)	34.6 (0)	40.6 (3)	n/a	n/a	n/a	22.7 (0)
Laya	verified	40.2 (0)	48.4 (0)	19.4 (0)	32.8 (0)	99.8 (100)	77.0 (51)	64.8 (1)	n/a
Qwen 3.7 Flash (reasoning off)	verified	75.2	87.4	73.4	72.2	96.4	77.2	61.2	55.3
Qwen3-1.7B (local, reasoning off)	verified	53.0	57.8	52.2	51.8	82.2	66.8	53.4	37.4

banking77

Banking77: fine-grained banking support intents. 77 intents. Test rows from the test split, calibration rows from the train split.

Provider	Accuracy (95% CI)	Task metric	ECE	Coverage at 95%	Accuracy there	AURC	p50	Cost per 1k
GLiNER 2.5 base	75.8 (71.9 to 79.3)	macro-F1 73.1	0.099	35.0	94.9	0.090	41 ms	$0
Qwen 3.7 Flash (reasoning off)	75.2 (71.2 to 78.8)	macro-F1 72.4	n/a	n/a	n/a	n/a	569 ms	$0.020
Qwen3-1.7B (local, reasoning off)	53.0 (48.6 to 57.3)	macro-F1 50.0	n/a	n/a	n/a	n/a	548 ms	$0
Laya	40.2 (36.0 to 44.6)	macro-F1 36.8	0.197	0.0	n/a	0.466	88 ms	$0

clinc150

CLINC150 plus: 150 intents across 10 domains, and out of scope. The 'out of scope' option covers requests outside every intent; about one row in five.

Provider	Accuracy (95% CI)	Task metric	ECE	Coverage at 95%	Accuracy there	AURC	p50	Cost per 1k
Qwen 3.7 Flash (reasoning off)	87.4 (84.2 to 90.0)	macro-F1 86.9	n/a	n/a	n/a	n/a	556 ms	$0.027
GLiNER 2.5 base	58.8 (54.4 to 63.0)	macro-F1 64.0	0.078	3.2	87.5	0.211	57 ms	$0
Qwen3-1.7B (local, reasoning off)	57.8 (53.4 to 62.1)	macro-F1 62.4	n/a	n/a	n/a	n/a	610 ms	$0
Laya	48.4 (44.0 to 52.8)	macro-F1 39.2	0.265	0.0	n/a	0.339	92 ms	$0

massive

MASSIVE: 60 assistant intents in five languages. 100 test rows each from en-US, de-DE, es-ES, hi-IN and ja-JP; options are in English.

Provider	Accuracy (95% CI)	Task metric	ECE	Coverage at 95%	Accuracy there	AURC	p50	Cost per 1k
Qwen 3.7 Flash (reasoning off)	73.4 (69.4 to 77.1)	macro-F1 73.4	n/a	n/a	n/a	n/a	534 ms	$0.014
Qwen3-1.7B (local, reasoning off)	52.2 (47.8 to 56.5)	macro-F1 50.6	n/a	n/a	n/a	n/a	389 ms	$0
GLiNER 2.5 base	34.6 (30.6 to 38.9)	macro-F1 32.4	0.275	0.0	n/a	0.463	33 ms	$0
Laya	19.4 (16.2 to 23.1)	macro-F1 15.4	0.238	0.0	n/a	0.642	88 ms	$0

Accuracy by language:

Provider	de-DE	en-US	es-ES	hi-IN	ja-JP
GLiNER 2.5 base	30.0	50.0	45.0	5.0	43.0
Laya	13.0	37.0	20.0	7.0	20.0
Qwen 3.7 Flash (reasoning off)	73.0	74.0	77.0	70.0	73.0
Qwen3-1.7B (local, reasoning off)	54.0	55.0	56.0	46.0	50.0

multiwoz

MultiWOZ 2.2: the user's intent at a turn of a conversation. The input is the conversation so far as a list of messages, ending with a user turn. Only user turns with at least two earlier turns and at most one active intent are used; 'none' means no active intent, such as a thank-you.

Provider	Accuracy (95% CI)	Task metric	ECE	Coverage at 95%	Accuracy there	AURC	p50	Cost per 1k
Qwen 3.7 Flash (reasoning off)	72.2 (68.1 to 75.9)	macro-F1 64.1	n/a	n/a	n/a	n/a	585 ms	$0.012
Qwen3-1.7B (local, reasoning off)	51.8 (47.4 to 56.1)	macro-F1 44.4	n/a	n/a	n/a	n/a	391 ms	$0
GLiNER 2.5 base	40.6 (36.4 to 45.0)	macro-F1 34.7	0.116	2.6	100.0	0.433	29 ms	$0
Laya	32.8 (28.8 to 37.0)	macro-F1 29.1	0.379	0.0	n/a	0.546	82 ms	$0

jailbreak

Jailbreak classification: prompts that try to override an assistant's rules. The whole test split less one repeated prompt (399 rows, 140 jailbreaks); calibration rows from the train split.

Provider	Accuracy (95% CI)	Task metric	ECE	Coverage at 95%	Accuracy there	AURC	p50	Cost per 1k
Laya	99.7 (98.6 to 100.0)	balanced 99.8	0.003	100.0	99.7	0.000	72 ms	$0
Qwen 3.7 Flash (reasoning off)	97.5 (95.4 to 98.6)	balanced 96.4	n/a	n/a	n/a	n/a	589 ms	$0.012
Qwen3-1.7B (local, reasoning off)	86.7 (83.0 to 89.7)	balanced 82.2	n/a	n/a	n/a	n/a	314 ms	$0

Not answered (question kind unsupported): GLiNER 2.5 base.

civil_comments

Civil Comments: toxic comments. Toxic means a toxicity score of 0.5 or more. Sampled half toxic and half not. Contains offensive language.

Provider	Accuracy (95% CI)	Task metric	ECE	Coverage at 95%	Accuracy there	AURC	p50	Cost per 1k
Qwen 3.7 Flash (reasoning off)	77.2 (73.3 to 80.7)	balanced 77.2	n/a	n/a	n/a	n/a	572 ms	$0.006
Laya	77.0 (73.1 to 80.5)	balanced 77.0	0.394	51.2	94.5	0.080	48 ms	$0
Qwen3-1.7B (local, reasoning off)	66.8 (62.6 to 70.8)	balanced 66.8	n/a	n/a	n/a	n/a	310 ms	$0

Not answered (question kind unsupported): GLiNER 2.5 base.

helpsteer2

HelpSteer2: how helpful an assistant's reply is. Human helpfulness ratings 0 to 4 mapped to five levels, sampled evenly per level. Test rows from the validation split, calibration rows from the train split. Human ratings are noisy, so mean absolute error and within-one accuracy are reported next to exact accuracy.

Provider	Accuracy (95% CI)	Task metric	ECE	Coverage at 95%	Accuracy there	AURC	p50	Cost per 1k
Qwen 3.7 Flash (reasoning off)	30.6 (26.7 to 34.8)	MAE 1.28, within one 61.2	n/a	n/a	n/a	n/a	633 ms	$0.020
Laya	26.0 (22.3 to 30.0)	MAE 1.34, within one 64.8	0.099	0.8	75.0	0.665	80 ms	$0
Qwen3-1.7B (local, reasoning off)	25.4 (21.8 to 29.4)	MAE 1.56, within one 53.4	n/a	n/a	n/a	n/a	491 ms	$0

Not answered (question kind unsupported): GLiNER 2.5 base.

wnut17

WNUT 2017: emerging entities in social media text. Rows with at most one entity of each type; four in five rows mention at least one entity. Tokens are joined with spaces, as in the source.

Provider	Accuracy (95% CI)	Task metric	ECE	Coverage at 95%	Accuracy there	AURC	p50	Cost per 1k
Qwen 3.7 Flash (reasoning off)	48.8 (44.4 to 53.2)	field F1 55.3	n/a	n/a	n/a	n/a	1010 ms	$0.015
Qwen3-1.7B (local, reasoning off)	35.0 (30.9 to 39.3)	field F1 37.4	n/a	n/a	n/a	n/a	1502 ms	$0
GLiNER 2.5 base	16.2 (13.2 to 19.7)	field F1 22.7	0.442	0.0	n/a	0.770	43 ms	$0

Not answered (question kind unsupported): Laya.

Runs

Provider	Spec	Model	Hardware	ThinkLess	Date	Notes
GLiNER 2.5 base	gliner	fastino/gliner2.5-base-v1	NVIDIA GeForce RTX 5060 Laptop GPU	0.3.0	2026-09-25	fastino/gliner2.5-base-v1 at default settings on an RTX 5060 laptop GPU. Training data not published; overlap with these sources unknown.
Laya	laya	laya-rl-agent	NVIDIA GeForce RTX 5060 Laptop GPU	0.3.0	2026-09-25	convaiinnovations/laya at default settings (label budget of 256 tokens) on an RTX 5060 laptop GPU. Training data not published; overlap with these sources unknown.
Qwen 3.7 Flash (reasoning off)	llm:openrouter:qwen/qwen3.7-flash --reasoning off	qwen/qwen3.7-flash	NVIDIA GeForce RTX 5060 Laptop GPU	0.3.0	2026-09-25	Through OpenRouter with reasoning off, 4 requests in parallel, rate-limited rows retried; cost is what OpenRouter billed.
Qwen3-1.7B (local, reasoning off)	llm:local:Qwen/Qwen3-1.7B --reasoning off	Qwen/Qwen3-1.7B	NVIDIA GeForce RTX 5060 Laptop GPU	0.3.0	2026-09-25	Qwen/Qwen3-1.7B through transformers with thinking off, on an RTX 5060 laptop GPU. Training data not published; overlap with these sources unknown.