Decision models next to LLMs and classifiers
A decision model answers a typed question and returns a distribution. TypeSafe’s Jev uses Choice, Score, and Noul. Artificial Analysis’s Decision Index and Arize’s bake-off put Jev next to other decision models and next to language models used as judges. Encoder classifiers stay in their own tables.
Compiled by jevmodel.ai. Numbers stay with the cited authors. Decision Index figures are the 13 October 2026 launch snapshot. The three Banking77 setups stay in separate tables. We have not rerun the experiments.
- Jev 1.13
- 58 on the index. 0.13 seconds per task. $42 per million tasks.
- Haiku 5.5 (Low)
- 61. 1.4 seconds. $184 per million tasks.
- GPT-6 Luna (Low)
- 55. 1.5 seconds. $131 per million tasks.
- Other decision models
- GLiDE and Microsoft-Decision-1 at 54. Decider v1.1 27B at $17. Celeris-1 at 0.06 seconds.
What a decision model returns
TypeSafe’s API takes state and a typed question. Choice selects among options the application supplies. Score evaluates a rubric. Noul returns a probability for a statement. Choice and Score include a confidence field. Noul does not.
Artificial Analysis’s Decision Index methodology says a decision model answers every question in one call. For a choice, the index scores the option with the highest probability. For a yes/no question it takes yes at probability 0.5 or more. Language models on that index receive a fixed system prompt and must return JSON through structured outputs. They supply no probability for the other options.
Sources:[1] TypeSafe AI[2] TypeSafe AI[140] Artificial Analysis
Decision models next to language models
Artificial Analysis launched Decision Index v1.0 on 13 October 2026. The launch article scores Jev 1.13 at 58. Claude Haiku 5.5 (Low) scores 61 at 1.4 seconds per task and $184 per million tasks. GPT-6 Luna (Low) scores 55 at 1.5 seconds and $131. Jev’s median is 0.13 seconds and $42.
Among decision models, GLiDE and Microsoft-Decision-1 sit at 54. Decider v1.1 27B has the lowest listed cost at $17 per million tasks. Celeris-1 Decision is the fastest at 0.06 seconds. The article says decision models are poorly calibrated on knowledge and reasoning, and relatively well calibrated on tool calling and classification. These figures are the launch snapshot. The live leaderboard can change.
Sources:[139] Artificial Analysis[138] Artificial Analysis[140] Artificial Analysis
Jev next to encoder classifiers
Lucas Biagetti’s v1.2 brief, at commit efbfb42, reports zero-shot accuracy on Banking77 subsets of 5, 10, 15, and 20 classes. jev-1.13.0 scores 94.00%, 92.25%, 89.83%, and 86.12%. gpt-5-nano-2025-08-07, with reasoning effort set to minimal, scores 92.00%, 88.25%, 87.33%, and 85.00%. MiniLM plus logistic regression, trained on 100 examples per class, scores 98.00%, 96.50%, 95.83%, and 92.38%, above both zero-shot rows. The brief calls the comparison exploratory and uses one seed. The same commit’s manifest names google-bert/bert-base-uncased trained for two epochs. The brief table we cite does not print that BERT row’s accuracies.
OpenRouter’s full Banking77 test split is a different setup: 3,080 utterances and all 77 intents. Jev 1.13 scores 81.0% there against 84.4% for Claude Opus 5 with reasoning off. The write-up says fine-tuned encoders trained on the 10,003 training examples sit in the low 90s. Those three Banking77 setups stay separate.
Sources:[117] Lucas Biagetti[118] Lucas Biagetti[116] Lucas Biagetti[87] Kenny Rogers
Jev next to OpenAI Decisions
Jim Bennett’s Arize bake-off, posted 7 October 2026, runs eight decision models on 2,675 RAGTruth rows, 655 Phoenix evaluator cases, and 52 routing decisions. OpenAI Decisions uses gpt-6-luna. On RAGTruth, Jev 1.13 scores 85.5% balanced accuracy at a cutoff fit on half the data, against 81.5% for OpenAI. The 95% intervals touch, so Arize records a draw on accuracy and sends Jev through on speed and cost: 0.08 seconds against 0.16, and $0.045 per 1,000 against $0.090.
On 655 Phoenix cases Jev scores 91.1% and OpenAI 90.0%. On 52 routing cases Jev is 49 of 52 and OpenAI 48 of 52. OpenAI is the only model in that round that does not accept Jev’s /v1/systemone layout. Arize wrote each case as labelled text sections and says a different layout could score differently.
Sources:[145] Jim Bennett[144] Arize AI
Splitting work between a decision model and an LLM
Several sourced workflows keep the language model and ask Jev a typed question first. OpenRouter’s Banking77 write-up keeps Jev 1.13 when confidence is at least 0.90 and sends the rest to Claude Opus 5, leaving 75.9% of the 3,080 utterances on Jev. The workflow page collects that cascade next to compaction, browser action selection, and agent handoffs.
TypeSafe’s coding-agents page says Jev is not a drop-in replacement for the LLM behind Claude Code, Cursor, Muse Spark, or similar tools. Keep the coding-agent model and use Jev separately for typed decisions.
Questions about decision models
What is a decision model?
In TypeSafe’s API, a decision model takes state and a typed question and returns Choice, Score, or Noul. Artificial Analysis’s Decision Index scores a decision model as one call per question, taking the highest-probability option, or yes at probability 0.5 or more.
How did Jev compare with language models on Decision Index v1.0?
Artificial Analysis scores Jev 1.13 at 58. Claude Haiku 5.5 (Low) scores 61 at 1.4 seconds per task and $184 per million tasks. GPT-6 Luna (Low) scores 55 at 1.5 seconds and $131. Jev’s median is 0.13 seconds and $42. These are launch figures from 13 October 2026.
How did Jev compare with MiniLM on Banking77 subsets?
On Lucas Biagetti’s zero-shot rows, jev-1.13.0 scores 94.00%, 92.25%, 89.83%, and 86.12% at 5, 10, 15, and 20 classes. MiniLM plus logistic regression, trained on 100 examples per class, scores 98.00%, 96.50%, 95.83%, and 92.38%. The brief calls the comparison exploratory and uses one seed.
Did OpenAI Decisions beat Jev on Arize’s RAGTruth run?
Jev 1.13 scores 85.5% balanced accuracy and OpenAI Decisions on gpt-6-luna scores 81.5%. The 95% intervals touch, so Arize records a draw on accuracy. Jev is faster and cheaper on that round: 0.08 seconds against 0.16, and $0.045 per 1,000 against $0.090.
Sources:[145] Jim Bennett
Are the Banking77 tables the same experiment?
No. Lucas’s table uses subsets of 5 to 20 classes with 40 examples per class. OpenRouter’s run uses 3,080 utterances and all 77 intents, 81.0% for Jev 1.13 against 84.4% for Claude Opus 5. The 37-dataset paper scores 3,076 Banking77 examples. The three setups stay separate.
Sources and reading notes
The summaries below are paraphrases. Each link opens the original source, including its surrounding context.
TypeSafe AI
Source checked:
[1] Jev and the Choice, Score and Noul primitives
The documentation describes Jev as a System One model that evaluates state and typed questions. Choice selects an option, Score evaluates a rubric, and Noul returns a probability for a statement.
API behavior documented by the provider. The page describes the three question types and includes no latency, accuracy, or cost comparison.
TypeSafe AI
Source checked:
[2] Confidence, probabilities and fallback decisions
Choice and Score include confidence derived from their probability distributions. Noul has no separate confidence field. TypeSafe recommends choosing thresholds for the domain and consequences of an action.
Implementation reference. A high confidence value can still sit on a wrong decision.
Artificial Analysis
Published:
Source checked:
[140] Artificial Analysis Decision Index methodology
Version 1.0 weights four evaluations at 25% each. A decision model answers every question in one call. For a choice, Artificial Analysis scores the option with the highest probability. For a yes/no question it takes yes at probability 0.5 or more. Language models receive a fixed system prompt and must return JSON through structured outputs.
AA-Omniscience-Decision non-hallucination is 100 × (1 − hallucination rate), where hallucination is a wrong answer instead of “I don’t know”. HLE-MultiChoice uses the 513 text-only multiple-choice questions from cais/hle, scored as an exact letter match. When2Call uses NVIDIA’s multiple-choice test set after filtering 229 tasks. WildGuardTest keeps rows where all three annotators agreed on prompt harm, response harm and refusal, and drops questions that consistently trigger model refusals. Cost per million tasks is one pass through the index, divided by the combined task count, times one million.
Artificial Analysis
Published:
Source checked:
[139] Artificial Analysis Decision Index v1.0 launch article
The 13 October 2026 launch article scores Jev 1.13 at 58 on Decision Index v1.0, ahead of GLiDE and Microsoft-Decision-1 at 54. Claude Haiku 5.5 (Low) scores 61 at 1.4 seconds per task and $184 per million tasks. GPT-6 Luna (Low) scores 55 at 1.5 seconds and $131. Jev’s median is 0.13 seconds and $42 per million tasks.
The index is an equal mix of AA-Omniscience-Decision (500 private five-option questions, accuracy and non-hallucination at 12.5% each), HLE-MultiChoice (513 text-only multiple-choice items), When2Call (3,423 NVIDIA tool-calling tasks after dropping 229 ambiguous ask-for-information items), and WildGuardTest (835 agreed rows, average of three F1 scores). Language models answer with structured JSON that names one option and supplies no probability. Jev scores 21 on HLE-MultiChoice, 35 accuracy and 35 non-hallucination on AA-Omniscience-Decision, 81 on When2Call, and 94 on WildGuardTest. Haiku 5.5 (Low) scores 24, 52, 73 and 93 on those four. Luna (Low) scores 22, 66, 77 and 89. The article says these are launch figures; the live leaderboard can change.
Artificial Analysis (@ArtificialAnlys)
Published:
Source checked:
[138] Artificial Analysis Decision Index launch thread
The opening post says Jev leads among decision models on Decision Index v1.0, and that language models such as Claude Haiku 5.5 still score higher at much higher cost and latency. The key-results post gives Jev 1.13 a score of 58, ahead of GLiDE and Microsoft-Decision-1 at 54, with Haiku 5.5 (Low) at 61, 1.4 seconds per task and $184 per million tasks against 0.13 seconds and $42 for Jev.
Thread on 13 October 2026 linking the launch article, methodology, and live leaderboard. Follow-up posts repeat the Pareto cost figures: Decider v1.1 27B at $17 per million tasks, Microsoft-Decision-1 at $31, and Jev 1.13 at $42. Celeris-1 Decision is 0.06 seconds per task. The thread says decision models are poorly calibrated on knowledge and reasoning, and relatively well calibrated on WildGuardTest and When2Call. Live scores can move after this launch snapshot.
Lucas Biagetti
Published:
Source checked:
[117] Banking77 v1.2 brief, 56 quality conditions
The v1.2 brief reports accuracy for 14 method and budget variants at 5, 10, 15, and 20 Banking77 classes. Zero-shot Jev scores 94.00%, 92.25%, 89.83%, and 86.12%. The OpenAI zero-shot row scores 92.00%, 88.25%, 87.33%, and 85.00%.
Checked at commit efbfb42. The brief says each width uses 40 held-out examples per class, one seed, and that the cohorts overlap, so the widths are not independent samples. It reports 1, 2, and 3 missing Jev probability vectors at 10, 15, and 20 classes. It makes no cross-provider cost or latency claim. MiniLM plus logistic regression at 100 examples per class scores 98.00%, 96.50%, 95.83%, and 92.38%. The model identifiers are in the manifest, not in this brief.
Lucas Biagetti
Published:
Source checked:
[118] Banking77 v1.2 manifest: jev-1.13.0 and gpt-5-nano-2025-08-07
The manifest pins Jev to jev-1.13.0 and the OpenAI zero-shot row to gpt-5-nano-2025-08-07 with reasoning effort minimal. Both use zero labeled examples. The campaign seed is 42, the class counts are 5, 10, 15, and 20, and the test set has 40 examples per class.
Same commit as the v1.2 brief. The MiniLM row uses sentence-transformers/all-MiniLM-L6-v2 at revision 1110a243, frozen, with logistic regression. BERT is google-bert/bert-base-uncased for two epochs. The manifest does not store latency or a billed cost for the zero-shot rows.
Lucas Biagetti (@LucasBiagettiA)
Published:
Source checked:
[116] Banking77 text-classification benchmark updated with Jev
Lucas Biagetti says he updated his text-classification benchmark, which he now frames as a decision-model comparison, to include Jev and to fill some gaps. He writes that Jev wins zero-shot classification in his exploratory analysis.
Post on 1 October 2026. It does not give the table, the class counts, or the OpenAI model identifier. The reply in the thread attaches a chart. The numbers and the model pins are in the repository brief and manifest at commit efbfb42.
Kenny Rogers
Published:
Source checked:
[87] Banking77 intents: Jev 1.13 and Claude Opus 5
OpenRouter ran the 3,080-utterance Banking77 test split through Jev 1.13 and Claude Opus 5. Accuracy is 81.0% and 84.4%, macro-F1 is 80.5% and 83.6%, median round trip is 175 ms and 2,266 ms, and the billed totals are $0.34 and $7.44.
Kenny Rogers, OpenRouter, 22 September 2026. Both models received the same one-line criteria, written from label names without looking at the test data. Opus ran at temperature 0 with reasoning off and a strict JSON enum. Eight requests were in flight. Latency is the client round trip through OpenRouter. The Opus system prompt was cached. The write-up says the uncached list rate would have been about $19 per 1,000 requests. Confidence at or above 0.99 covered 58% of utterances at 96.3% accuracy. Below 0.5 covered 3.5% at 29.6% accuracy. Sending confidence under 0.90 to Opus left 75.9% on Jev, at 84.0% accuracy and $0.69 per 1,000. That cascade table reuses the same 3,080 examples, and the write-up calls it an upper bound. Both models scored 0 of 40 on get_physical_card. One dataset, one afternoon. Opus was not run with reasoning on.
Jim Bennett
Published:
Source checked:
[145] Arize decision-model benchmark: Jev, OpenAI Decisions and six others
On 2,675 RAGTruth rows, Jev 1.13 scores 85.5% balanced accuracy at a tuned cutoff, against 81.5% for OpenAI Decisions on gpt-6-luna. Median latency is 0.08 seconds against 0.16, and cost per 1,000 is $0.045 against $0.090. The 95% intervals on accuracy touch, so Arize records a draw there and sends Jev through on speed and cost. On 655 Phoenix cases Jev scores 91.1% and OpenAI 90.0%. On 52 routing cases Jev is 49 of 52 and OpenAI 48 of 52.
Blog dated October 2026, posted on X on 7 October. OpenAI is the only model that does not accept Jev’s /v1/systemone layout; Arize wrote each case as labelled text sections and says a different layout could score differently. Accuracy wins require non-overlapping 95% intervals. Each model gets its own yes/no cutoff, fit on half the data and scored on the other half, same seed as Arize’s earlier Jev-as-a-judge work. An earlier Arize Opus 5 judge scored 86.6% balanced accuracy at $14.30 per 1,000. Reversing the hallucination question drops Liquid d1 from 0.92 ROC AUC to 0.17; Jev, Clef 27B and OpenAI stay near 0.93, 0.92 and 0.90 on both wordings.
Arize AI (@arizeai)
Published:
Source checked:
[144] Arize thread on eight decision models
The opening post says Arize benchmarked eight decision models on hallucination detection, 13 Phoenix evaluator suites, and agent routing. It writes that Jev, Clef 27B and Liquid sit within a point of each other on hallucination, and within the noise of an LLM judge that costs 60 to 400 times more.
Thread on 7 October 2026 linking the blog. A follow-up names Jev for a hosted API, Kev for self-hosting, and Clef 27B for open weights with a hosted option, and says to test the question as hard as the model. The thread itself does not print the RAGTruth percentages.
TypeSafe AI
Source checked:
[162] Jev is not a drop-in for Muse Spark or other coding-agent LLMs
TypeSafe says Jev is not a drop-in replacement for the LLM behind Claude Code, Cursor, opencode, Copilot, Muse Spark, Grok Bot, or similar tools. There is no model: "jev-latest" setting that turns a coding agent into a Jev-powered agent. The page tells readers to keep the coding-agent LLM and use Jev separately for typed decisions.
Official coding-agents page as accessed 15 October 2026. It names Muse Spark, not Muse Wonder. Muse Wonder is Meta’s later coding and agent model.