Benchmarks¶
Two benchmarks ship with ThinkLess. The support benchmark runs one realistic agent end to end with three different decision planes. The intent benchmarks measure providers on public datasets and simulate the cascade at every threshold. Both are one command, and both write their full results next to the summary.
Hosted models were run through OpenRouter, and every dollar figure for them is the cost OpenRouter billed for each call, not an estimate. Local models ran on an NVIDIA RTX 5060 laptop GPU (8 GB), Windows 11, Python 3.12, PyTorch 2.14 with CUDA 13.0.
At a glance¶
Same agent, same 53 labeled support tickets. llm sends every decision to the
LLM; hybrid asks rules, GLiNER and Laya first and the LLM only when they are
unsure.
| Fallback LLM | Task success, llm |
Task success, hybrid |
Billed per 1k tickets, llm |
Billed per 1k tickets, hybrid |
Decision time per ticket, llm to hybrid |
|---|---|---|---|---|---|
| Qwen 3.7 Flash (reasoning off) | 92.5% | 98.1% | $0.034 | $0.019 | 1.27 s to 0.58 s |
| Gemini 2.5 Flash Lite | 98.1% | 98.1% | $0.104 | $0.059 | 1.17 s to 0.76 s |
| GPT-5.6 Luna (reasoning low) | 98.1% | 100% | $0.235 | $0.141 | 2.50 s to 1.18 s |
| Claude Haiku 4.5 | 98.1% | 98.1% | $1.338 | $0.788 | 2.21 s to 1.34 s |
| Qwen3-1.7B, local | 86.8% | 94.3% | free | free | 1.84 s to 0.43 s |
On every model, hybrid matched or beat the LLM-only design at 40 to 44 percent lower billed cost, and cut the time spent on decisions by 35 to 55 percent. On Banking77, a GLiNER-then-LLM cascade was more accurate than either of its parts while sending a quarter of the traffic to the LLM (below).
Support benchmark¶
thinkless bench support --llm openrouter:qwen/qwen3.7-flash --reasoning off
thinkless bench support --llm openrouter:anthropic/claude-haiku-4.5
thinkless bench support # fully local
Setup¶
- The agent is the bundled support agent: the same code in every mode.
- The tickets are 53 hand-written support messages with labeled expected outcomes, covering every path of the agent plus hard cases: order numbers without the word "order", a customer asking about someone else's order, prompt injections, requests for a person, signed-out users, and Spanish and German messages. They are synthetic and small: read the numbers as evidence of the mechanism and its failure modes, not as a general accuracy ranking.
- The modes differ only in the decision cascade:
| Mode | Decision cascade |
|---|---|
llm |
The LLM answers every question: the common agent design |
hybrid |
Rules, then GLiNER, then Laya, then the LLM for anything they are unsure about |
models |
Rules, GLiNER and Laya only; unsure decisions go to a person |
- The baseline is the strongest reasonable version of the LLM design. All six triage questions (intent, urgency, churn risk, request for a person, prompt injection, order number) go to the LLM in one structured prompt per ticket, not one call per question. Deterministic checks (authentication, order ownership, duplicate detection, refund limits, reply grounding) are code in every mode.
- Thresholds for the routing questions were calibrated on a separate 48-row labeled set with accuracy targets fixed in advance; see calibration. Nothing was tuned on the 53 tickets.
- Replies are generated by the same LLM in every mode, including
models. - Task success is the share of tickets where the agent took the expected action.
Hosted results¶
Full reports, results files and trace viewers are in
benchmarks/results,
one directory per model (support-openrouter-*).
| Model | Mode | Success | Order id | LLM calls for decisions | Decision time | p50 / p95 | Billed per 1k |
|---|---|---|---|---|---|---|---|
| Qwen 3.7 Flash | llm |
92.5% | 94.1% | 1.06 | 1.27 s | 2.42 / 3.15 s | $0.034 |
hybrid |
98.1% | 100% | 0.70 | 0.58 s | 1.77 / 2.36 s | $0.019 | |
models |
73.6% | 100% | 0 | 0.12 s | 1.33 / 1.88 s | $0.012 | |
| Gemini 2.5 Flash Lite | llm |
98.1% | 100% | 1.06 | 1.17 s | 1.98 / 3.59 s | $0.104 |
hybrid |
98.1% | 100% | 0.70 | 0.76 s | 1.80 / 3.51 s | $0.059 | |
models |
73.6% | 100% | 0 | 0.12 s | 0.98 / 1.74 s | $0.035 | |
| GPT-5.6 Luna | llm |
98.1% | 100% | 1.06 | 2.50 s | 3.37 / 5.06 s | $0.235 |
hybrid |
100% | 100% | 0.72 | 1.18 s | 3.07 / 4.59 s | $0.141 | |
models |
73.6% | 100% | 0 | 0.16 s | 1.56 / 2.33 s | $0.082 | |
| Claude Haiku 4.5 | llm |
98.1% | 100% | 1.06 | 2.21 s | 4.10 / 8.98 s | $1.338 |
hybrid |
98.1% | 100% | 0.70 | 1.34 s | 3.58 / 7.25 s | $0.788 | |
models |
73.6% | 100% | 0 | 0.16 s | 2.25 / 5.28 s | $0.515 |
Across 373 LLM decision calls and 581 generated replies, no answer was invalid, none was cut off at the token limit, and every reply passed the grounding check. Structured outputs were not needed; the prompt-and-parse decider held up on every model.
Local results¶
| Mode | Success | Order id | LLM calls for decisions | Decision time | p50 / p95 |
|---|---|---|---|---|---|
llm |
86.8% | 94.1% | 1.04 | 1.84 s | 3.43 / 4.49 s |
hybrid |
94.3% | 100% | 0.68 | 0.43 s | 1.85 / 2.73 s |
models |
73.6% | 100% | 0 | 0.11 s | 1.55 / 2.32 s |
Qwen3-1.7B running in-process through Transformers on the laptop GPU, so every
number reproduces offline with no key
(report).
With a weak fallback, hybrid beats the LLM-only design outright: the small
models and rules are more accurate than the 1.7B model on the questions they
take, and all three hybrid misses are intents the 1.7B model answered as
other.
Reading the results¶
Hybrid matches the LLM at about 60 percent of the cost. The saving is the
decisions that never reach the LLM: 87 percent of hybrid decisions were made
by rules or small models. What remains of the bill is mostly reply
generation, which every mode pays (models mode, which never calls the LLM
for decisions, shows that floor).
End-to-end latency moves less than decision time. With hosted models, the reply and the network dominate each ticket, so p50 improves by 9 to 27 percent while the decision step itself gets 35 to 55 percent faster.
Extraction is where small models are most reliable. Rules and GLiNER
found every order number on every run. Gemini, GPT-5.6 Luna and Claude Haiku
did too in llm mode; Qwen 3.7 Flash missed three written without the word
"order", and the local 1.7B model missed two.
One ticket is contested. T-031, "Can I still change the shipping address on an order I placed this morning?", failed in 11 of the 12 hosted model and mode runs, split between reading it as an order question (asking for the order number) and as a policy question. It is labeled as a policy question and left as is; changing the label after seeing results would be tuning on the test.
Small models alone hand a quarter of tickets to people. models mode
never calls an LLM for decisions and sends 12 tickets to human triage because
intent confidence is below threshold. That is the intended behavior of that
design, and it shows the next lever: intent accuracy on the small models.
Intent benchmarks¶
thinkless bench intents --dataset banking77 --llm openrouter:qwen/qwen3.7-flash --reasoning off
thinkless bench intents --dataset emotion
Each run samples 500 test examples with a fixed seed (the same 500 in every
run), asks every provider the same Choice question, and reports accuracy,
expected calibration error, latency, billed cost and the threshold each
provider needs for 95 percent accuracy on the answers it accepts. It then
simulates the cascade (GLiNER, then Laya, then the LLM) at every threshold.
Banking77: the cascade matches or beats its fallback at a quarter of the cost¶
77 fine-grained banking intents (PolyAI).
| Provider | Accuracy | p50 latency | Billed per 1k |
|---|---|---|---|
| GLiNER 2.5 base (local) | 71.8% | 33 ms | $0 |
| Laya (local) | 35.0% | 51 ms | $0 |
| Qwen3-1.7B (local) | 51.4% | 405 ms | $0 |
| Qwen 3.7 Flash | 73.8% | 576 ms | $0.019 |
| Claude Haiku 4.5 | 76.2% | 1412 ms | $0.681 |
With Qwen 3.7 Flash as the fallback (report):
| Cascade threshold | Accuracy | Calls reaching the LLM | Mean latency | Billed per 1k |
|---|---|---|---|---|
| 0.50 | 73.8% | 7.0% | 80 ms | $0.0013 |
| 0.80 | 74.8% | 24.2% | 196 ms | $0.0046 |
| 0.90 | 74.8% | 34.2% | 264 ms | $0.0065 |
| LLM only | 73.8% | 100% | 687 ms | $0.0190 |
With Claude Haiku 4.5 as the fallback (report):
| Cascade threshold | Accuracy | Calls reaching the LLM | Mean latency | Billed per 1k |
|---|---|---|---|---|
| 0.50 | 74.4% | 7.0% | 150 ms | $0.048 |
| 0.80 | 76.2% | 24.2% | 432 ms | $0.165 |
| 0.90 | 76.2% | 34.2% | 599 ms | $0.234 |
| LLM only | 76.2% | 100% | 1803 ms | $0.681 |
At 0.80, with a quarter of the traffic reaching the LLM, the cascade beat both GLiNER and Qwen 3.7 Flash on their own, and matched Claude Haiku 4.5 on its own exactly (76.2 percent) at a quarter of its cost and latency. GLiNER's confident answers are more accurate than the LLM's on the same inputs, and the LLM takes the ones GLiNER is unsure about. With the local 1.7B model as the fallback the same cascade could not beat GLiNER alone, because that model was 20 points weaker: a cascade helps exactly when the fallback is stronger than the model it replaces on the inputs it escalates, and the benchmark shows where that line is.
Laya's weak result on many-option questions matches its own project's report;
questions with dozens of options belong with GLiNER (providers=(...) on the
question).
Emotion¶
Six emotions on short social media text (dair-ai),
local models only: GLiNER 51.6 percent, Laya 54.6 percent, Qwen3-1.7B 47.0
percent. The labels are noisy and overlap (joy and love), and no zero-shot
setup does well. Tasks like this call for a fine-tuned classifier, which slots
in through HFClassifier.
Strong areas¶
- Cost and decision latency, on every model tried. 40 to 44 percent lower billed cost and 35 to 55 percent less decision time than the LLM-only design, at equal or better task success.
- Extraction. Rules plus GLiNER found every order number on every run.
- Reliability of the LLM path. No invalid or truncated decision on 373 hosted LLM calls once reasoning was configured, and every reply grounded.
- Safety by construction. Uncertain decisions never trigger actions, money only moves after deterministic checks, and the subtlest injection in the set (T-051) is caught by rules even when the local LLM misses it.
- Cascades with a strong fallback. On Banking77 the cascade beat GLiNER and Qwen 3.7 Flash alone and matched Claude Haiku 4.5 alone, each time with a quarter of the calls reaching the LLM.
Weak areas, and what was done about them¶
| Weakness | Evidence | Status |
|---|---|---|
| Reasoning models spent the whole token budget reasoning and returned empty replies or no decisions | Qwen 3.7 Flash in the first smoke test: 6 of 6 answers missing, empty reply | Fixed. --reasoning off/low, and LLMDecider detects a truncated reply, logs why and retries once with a larger budget |
| An escalated question reached the LLM without its siblings, so requests for a human were flagged as injections | Hybrid-only misses on T-046 and T-048 with Qwen, Gemini, Haiku and the local model; on the calibration set, one model flagged 5 of 10 human requests when asked in isolation | Fixed. Escalation context passes settled sibling decisions to the LLM. Validated on the calibration set first (false positives on human requests went from 3 to 0 in the real cascade, with missed injections and LLM calls unchanged), then confirmed on T-046 and T-048 across all four hosted models. Adds 6 to 13 percent to hybrid bills |
| Cost was an estimate | Price table times tokens from another tokenizer | Fixed. OpenRouter's billed cost is recorded per call (cost_source="reported") |
| An endpoint could reject JSON mode and silently turn every LLM decision into an abstention | Found in review | Fixed. One retry without response_format, then JSON mode stays off for that client |
| Prompt injection is Laya's weakest question | 35 percent of injection questions escalate; its answers sit near a coin flip on ordinary messages | Mitigated by rules and a calibrated threshold. A dedicated injection classifier was tested and rejected: it made confident mistakes on ordinary tickets (see the production guide) |
| Intent is the largest remaining cost driver | Intent escalates on 59 percent of tickets under calibrated thresholds; models mode hands 12 tickets to triage |
Next. Base accuracy is the limit (64.6 percent for GLiNER on the calibration intents), not the threshold. A fine-tuned intent classifier through HFClassifier, better option descriptions, or a larger calibration set |
| Replies dominate what is left of the bill and the latency | models mode, with no LLM decisions, still pays 55 to 65 percent of the hybrid bill |
Next, application side. Templates for deterministic outcomes, or a smaller reply model |
| One contested ticket | T-031 fails in 11 of 12 hosted runs | Reported, not relabeled |
Not measured yet¶
- TypeSafe Jev. Supported (
--jevwithTYPESAFE_API_KEY) and its wire format is tested against TypeSafe's own SDK models, but it has not been run live. - The direct Anthropic SDK path. Claude was benchmarked through
OpenRouter;
AnthropicLLMis unit tested against a mocked client. - A larger support set, CPU-only and Apple Silicon numbers, and a faster local serving stack (Ollama or vLLM instead of the Transformers loop).
Runs on other hardware and models are very welcome; include the
results.json.
Spend¶
Everything on this page that used a hosted model, including smoke tests and the calibration experiments, cost $0.59 in OpenRouter credits, measured on the key.
Reproducing¶
conda env create -f environment.yml && conda activate thinkless
echo "OPENROUTER_API_KEY=..." > .env
thinkless bench support --llm openrouter:qwen/qwen3.7-flash --reasoning off
thinkless bench support --llm openrouter:anthropic/claude-haiku-4.5
thinkless bench support
thinkless bench intents --dataset banking77 --llm openrouter:qwen/qwen3.7-flash --reasoning off
python benchmarks/calibrate_support.py
Local decoding is greedy, so local reruns reproduce accuracy exactly; hosted models are close but not bit-for-bit. Latency does not reproduce tightly: on a laptop, absolute times move by a factor of two or more between sessions, so compare modes within one run.