Jev benchmark reports, with their limits
These numbers come from the cited authors and repositories. We read the sources and did not rerun the API calls. Each dataset and pipeline has its own table.
Business-email classification
Nikhil Mudholkar: 1,565 emails across 10 categories, including 1,201 real emails and 364 AI-written examples.
| Model | Overall accuracy | Equal-weight category accuracy | Cost per 1,000 emails |
|---|---|---|---|
| Jev | 96.4% | 92.0% | $0.08 |
| Gemini 3.5 Flash-Lite | 97.5% | 94.6% | $0.80 |
| Gemini 3.8 Flash | 98.5% | 96.9% | $1.79 |
- These are author-reported results against the reference labels. Each model runs on its own. The thread has no combined pipeline.
- The emails are mostly German, with English examples. Attachments are excluded and the AI-written examples target rare categories.
- Cost is normalized to 1,000 emails, not the total cost of the dataset. The author notes that the savings are small at normal inbox volumes.
Sources:[9] nikhil mudholkar[10] nikhil mudholkar[11] nikhil mudholkar[13] nikhil mudholkar[14] nikhil mudholkar
Read the model report and user feedbackLabeling 1,000 app reviews
Rahul Kumar’s Column Race: sentiment, topic, bug and churn labels for 1,000 Android reviews. Both lanes use batches of 20 at concurrency 8.
| Model | Whole-run duration | Estimated run cost | Sentiment correlation with stars |
|---|---|---|---|
| Jev 1.13.0 | 4.6 seconds | $0.023 | 0.803 |
| Gemini 3.8 Flash | 18.8 seconds | $0.158 | 0.822 |
- The public repository contains one recorded run pair with these results. Durations cover all 1,000 reviews. The repo publishes no per-request median.
- Sentiment uses Spearman correlation with star ratings, a weak proxy. Topic agreement of 84.4% and bug agreement of 92.6% have no human ground truth, so those rates say the models matched each other.
- The rounded costs use reported token counts and list prices, not invoices. No JSON retries occurred in this recorded pair.
Sources:[15] Rahul Kumar[16] goodrahstar
Read the model report and user feedbackEmail screening with a Kimi K3 fallback
Hassan: 100 emails, evenly split between legitimate and fraudulent examples. Jev sends 31 cases below 95% confidence to Kimi K3.
| Workflow | Whole-run duration | Reported run cost | Correct decisions |
|---|---|---|---|
| Jev + Kimi K3 | 16 seconds | About $0.07 | 96 of 100 |
- Jev’s first pass takes 1.42 seconds. The full pipeline duration includes Kimi’s work.
- The author attributes $0.003 to Jev and $0.068 to Kimi. The roughly $0.07 total is rounded.
- This is an author-reported demonstration on the demo’s 95% cutoff. It stands apart from a production fraud evaluation and from the two Gemini datasets.
Sources:[19] Hassan
Read the model report and user feedbackDoes the cited passage support the sentence?
Jamie Watters, jev-bench: 42 labeled claims, of which 24 are constructed controls and 18 are real published sentences. One schema-only run per model on 20 September 2026, and a scratchpad rerun of the three frontier models on 21 September.
| Model | All 42 | Hard 18, schema only | Hard 18, with scratchpad | Cost per claim, schema run |
|---|---|---|---|---|
| Jev (table label jev-1.13.0; calls used jev-latest) | 85.7% | 77.8% | not re-run | $0.000025 |
| GPT-5.4 | 85.7% | 66.7% | 61.1% | $0.001289 |
| Claude Sonnet 5 | 85.7% | 66.7% | 44.4% | $0.002228 |
| Gemini 3.1 Pro (preview) | 81.0% | 55.6% | 55.6% | $0.004824 |
- Schema-run times are 0.23, 1.01, 2.76 and 3.78 seconds. Scratchpad times for the three frontier models are 3.77, 6.23 and 8.71 seconds, at $0.005450, $0.006439 and $0.011545 per claim.
- The author says the 18 real sentences are the headline set. Two of Jev’s six errors on the full 42 are in the easy controls. Every frontier error in the schema-only run is in the hard 18.
- Unsupported-claim recall for Jev is 78.9%, short of the 80% bar written down before the run, with 88.2% precision against 81.0% for GPT-5.4. The write-up calls Jev a pre-filter on that rule.
- McNemar p-values for the hard-18 scratchpad change are 1.000, 0.219 and 1.000. The README says a claim that reasoning did not help is supportable, and a claim that reasoning made the models worse is not. For GPT-5.4 and Sonnet 5 the verdict call transcribes the scratchpad.
- One run per model. Labels come from the author’s pipeline, and one hard row is contested. Frontier calls go through OpenRouter. Costs use the repo’s price file, not invoices.
Sources:[39] Jamie Watters
Read the model report and user feedbackSemEval-2026 aspect sentiment
Eastwood: all ten SemEval-2026 DimABSA subtask 1 test sets. The aspect is given. Scores are official RMSE_VA, equal-weighted across the ten sets. Lower is a smaller error.
| Method | Mean RMSE_VA | Test sets with lower error than Jev |
|---|---|---|
| Kimi-K2 Thinking, official one-shot | 1.8731 | 10 of 10 |
| Qwen3-14B, QLoRA fine-tune | 2.1889 | 7 of 10 |
| Jev 1.13.0, descriptive scale, not fine-tuned | 2.4750 | reference row |
- Jev’s mean error is 32.1% higher than the Kimi-K2 Thinking figure. The record against Kimi is 0 wins and 10 losses, and against the Qwen fine-tune 3 wins and 7 losses.
- The chart caption says the metric skips normalization and that training and example conditions differ across methods. The per-dataset plot shows Jev above Kimi on every set.
- Kimi-K2 Thinking and the Kimi K3 in Hassan’s email handoff are different models. This table stands apart from that demo.
Sources:[45] Eastwood
Read the model report and user feedbackStagehand act trials against Gemini 3.8 Flash
miguelg719, open pull request 2953, read 22 September 2026. Experimental Jev act path plus Gemini 3.8 Flash fallback, compared with the existing Gemini 3.8 Flash act path. Off unless experimentalJevAct is set.
| Suite | Gemini 3.8 Flash baseline | Jev arm with fallback | Median latency | Acts that called the language model |
|---|---|---|---|---|
| Act, 40 tasks (Jev arm, 3 trials) | 39/40 | 118/120 | 1.97 s to 0.46 s | 4 of 147 |
| Breadth, 40 real sites (not in the pull request) | 35/40 | 39/40 | 3.19 s to 0.77 s | 11 of 49 |
- Thresholds were tuned on these suites. With the language model disabled, act scores 27/40 and the breadth set scores 26/40. Three trials of breadth on two models move from 204/240 to 229/240.
- Jev token use is logged and is still absent from result metadata. The pull request was open when read, so these figures describe the branch under review.
- A companion observe pull request reports 9/12 to 10/12, with 11 of 16 observes answered by Jev in 0.18 seconds against a 1.99 second baseline, and 5 of 16 sent to the language model. Its cache-check threshold of 0.35 has no end-to-end eval.
- Kyle Jeong’s 17 September post is a different measurement: one remote-browser task at a reported $0.001. That dollar figure is separate from this median.
Sixty-email classifier against DeepSeek V4.1 Flash
matyas: a 60-email table attached to the post. Both columns use the author’s labels. The corpus and the Jev version are unnamed.
| Model | Correct on 60 | Median latency | 95th percentile | Cost for the 60 |
|---|---|---|---|---|
| DeepSeek V4.1 Flash | 60/60 | 3.361 s | 19.105 s | $0.01912 |
| Jev | 60/60 | 0.313 s | 0.465 s | $0.00278 |
- Average latency in the same table is 6.600 seconds versus 0.348 seconds. Both sides record zero API or schema errors.
- The post’s 10,000-email line, about $3.19 versus $0.46 and about 9 hours 20 minutes versus 52 minutes, multiplies the per-1,000 cost by 10 and the median latency by 10,000. The post shows no log of that batch.
- The 10.7 times figure is the ratio of the medians. The 85.5% figure matches the cost ratio. V4.1 Flash is the name on this table, not the V4 Flash in the intent-routing note.
Sources:[46] matyas[22] LimboAI
Read the model report and user feedbackProduct-configurator fan-out, Jev and Gemini
Nikhil Mudholkar, 25 September 2026: 300 synthetic customer requests. Accuracy is on the 270 requests he calls solvable. Neither model version is named.
| Measure | Jev | Gemini |
|---|---|---|
| Accuracy before fan-out | 47.0% | 52.2% |
| Accuracy after fan-out | 55.6% | 62.6% |
| Median time | 4.60 s to 1.89 s | not reported |
| Wrong answers the later check stopped | 83 | 72 |
| Correct answers the same check stopped | 6 | 20 |
- The catalogue is fictional: eight linked dropdowns, about 120 million combinations on paper, 578,143 valid. Model calls fall from about 7 to 3. The requirement check was developed on these same cases.
- The thread gives no price table for the reported one-eleventh cost per correct configuration. Gemini’s median time is absent.
- At about 300 questions in one call, Jev answered and Gemini rejected all 300 requests before generation, on about 55,000 tokens. Configurator questions worked at 126 and failed at 127. Yes/no questions reached 329. An 801-question email figure in the same thread is a different test.
- hev’s reply describes a batch-size limit on their reranker. That repository caps a Jev call at about 30 passages and 32,000 tokens. The cap is separate from the 126/127 split.
Sources:[68] Nikhil Mudholkar[69] Nikhil Mudholkar[70] Nikhil Mudholkar[71] Nikhil Mudholkar[72] Nikhil Mudholkar[73] Nikhil Mudholkar[74] hev mind[76] hev
Read the model report and user feedbackReranking one BM25 shortlist with Jev, Claude and GPT-5.6 Luna
hev, RESULTS.md generated 17 September 2026, commit 1eb4726. jev-1.13.0 permutes the same BM25 top-30 shortlist as the other systems. SciFact and FiQA have 300 test queries. NFCorpus has 323. The score is nDCG@10.
| System | SciFact | NFCorpus | FiQA | SciFact run cost |
|---|---|---|---|---|
| BM25 order, no rerank | 0.667 | 0.310 | 0.234 | — |
| Jev 1.13.0, 30 Nouls in one call | 0.768 | 0.358 | 0.376 | $0.18 |
| Jev 1.13.0, one Noul per document | 0.772 | 0.358 | 0.376 | $0.28 |
| Voyage rerank-3 | 0.755 | 0.357 | 0.402 | $0.21 |
| Cohere rerank-v3.5 | 0.745 | 0.339 | 0.374 | $0.60 |
| GPT-5.6 Luna, listwise, reasoning off | 0.747 | 0.355 | 0.363 | $0.71 |
| Claude Haiku 4.5, listwise | 0.723 | not run | not run | $4.18 |
| Claude Opus 5, low effort, listwise | 0.756 | not run | not run | $28.40 |
- Costs are that run’s token usage at list price. The file has no later invoice. The SciFact column is 300 queries. The Jev batch costs $0.19 on NFCorpus and $0.13 on FiQA. Voyage rerank-3 costs $0.15 on FiQA.
- Opus 5 refused 12 of 300 SciFact queries. Those queries are scored as zero in the 0.756. The file says substituting BM25 order on the 12 gives 0.778.
- A repeat of 30 batch requests moved Noul probabilities by a mean absolute 0.004. Pair-Noul calibration on SciFact: a probability at or above 0.9 was relevant 76 percent of the time, and one under 0.1 was relevant 0.5 percent of the time.
- nDCG@10 is a ranking score on a shared shortlist. An accuracy rate would be a different measurement. We have not rerun the queries.
Rubric judging against Luna, Gemini 3.8 Flash and DeepSeek V4.1 Flash
Delip Rao and Chris Callison-Burch, arXiv:2609.29769, submitted 24 September 2026. Nine panels, 5,003 pairs. Jev Choice, GPT-5.6 Luna, Gemini 3.8 Flash and DeepSeek V4.1 Flash read the same criterion texts. Language-model judges are called once per criterion.
| Judge | Cost against Jev Choice | Wall time for the nine panels |
|---|---|---|
| Jev Choice | $0.063 | 29 seconds |
| GPT-5.6 Luna | 29 times | 859 seconds |
| DeepSeek V4.1 Flash | 66 times | 6,298 seconds |
| Gemini 3.8 Flash | 325 times | 950 seconds |
- Accuracy separates in 8 of 27 paired comparisons. Jev’s leads fall mostly on binary criteria. Its deficits fall on graded criteria, chiefly against Gemini, which the paper calls usually the best language-model judge.
- Requests used the jev-latest alias. A same-day stability check logged jev-1.13.0. The main runs did not log the version. DeepSeek’s three failed calls on the binary panels were recorded as UNMET.
- Jev sent 8 requests at a time on the graded panels, with Choice and Score sharing one API. Each language-model judge sent 16 to 32. The wall-time ratios understate a comparison at the same concurrency.
- On Jev’s most confident errors, 96.0% of the language-model verdicts repeat the wrong answer. A cascade replayed on the recorded verdicts gains at most 1.5 points over the best single judge with cross-fitted thresholds, and at most 2.0 with oracle thresholds.
Banking77 intents against Claude Opus 5
Kenny Rogers, OpenRouter, 22 September 2026. The Banking77 test split: 3,080 utterances, 40 for each of 77 intents. Same one-line criteria, written from the label names. Opus ran with reasoning off.
| Model | Accuracy | Macro-F1 | Median round trip | Billed cost |
|---|---|---|---|---|
| Jev 1.13 | 81.0% (79.6 to 82.3) | 80.5% | 175 ms | $0.34 |
| Claude Opus 5, reasoning off | 84.4% (83.1 to 85.6) | 83.6% | 2,266 ms | $7.44 |
- The paired accuracy gap is 3.3 points, with a 95% bootstrap interval of 2.3 to 4.4. The models agree on 89.3% of utterances. Eight requests were in flight. Latency is the client round trip through OpenRouter.
- Per 1,000 requests the bills are $0.11 and $2.42. The Opus system prompt was cached. The write-up says the uncached list rate would have been about $19 per 1,000.
- Confidence at or above 0.99 covers 58% of utterances at 96.3% accuracy. Below 0.5 covers 3.5% at 29.6%. Sending confidence under 0.90 to Opus leaves 75.9% on Jev, at 84.0% accuracy and $0.69 per 1,000. That cascade reuses these 3,080 examples, and the write-up calls it an upper bound.
- Both models score 0 of 40 on get_physical_card. One dataset, one afternoon. Opus was not run with reasoning on.
Sources:[87] Kenny Rogers
Read the model report and user feedbackOpenRouter Jev Router on DeepSWE
Theo, 26 September 2026. One author-reported DeepSWE run of typesafe/jev-router against GPT-6 Astra on low. He spent $1,000 on the bench.
| Setup | DeepSWE result | Cost against Astra low | Runtime against Astra low |
|---|---|---|---|
| GPT-6 Astra, low effort | comparison baseline | baseline | baseline |
| OpenRouter Jev Router | roughly the same | slightly more | almost 5 times as long |
- The posts do not publish a DeepSWE score, a task count, or a dollar gap. The $1,000 is the reported cost of the bench, not a per-request price.
- Hours earlier, Theo described the in-progress router as basically a DeepSeek 4.1 Flash router. That note does not give a mix across models. The follow-up says to use Astra on low.
- His separate objection is that a prompt does not show the codebase, the tools, or the size of the change, so Jev is categorizing without the context a coding router would need.
Reasoning effort for DeepSeek on 30 tasks
pcparts001, pi-jev-reasoning-router-lite at commit 2215f7c. A 30-task pre-port benchmark on a direct DeepSeek route. The extension asks Jev for none, low or high, then sets only the thinking level.
| Policy | Per-task cost | Answer quality on that run |
|---|---|---|
| Always high, corrected to the same max tokens | $0.3352 | baseline for the 15.2% comparison |
| Jev routing, corrected | $0.2844 | Spearman rho +0.504 against difficulty |
| Always low | $0.2112 | equal to routing |
- The uncorrected pair is $0.2685 for routing against $0.3193 for always-high, a 15.9% difference. Routing against always-low is $0.2845 against $0.2112.
- Spearman p = 0.0040, n = 30. The README says none was never returned, so the observed router was low versus high. The file limits those quality and cost figures to the direct DeepSeek route.
- The measured-results table lists decision latency of 409 to 856 ms, typically about 500 ms, added before the model request. A missing key, a timeout, or a bad answer falls back to high.
Sources:[88] pcparts001
Read the model report and user feedbackPublic JevBench tasks against Luna and Astra
Zefan Cai, Open-Jev docs/jevbench-public.md at commit 3308a15. All 231 available public tasks: 72 original, 48 easy, 111 hard. Jev 1.13.0, GPT-5.6 Luna with reasoning effort none, and GPT-6 Astra with reasoning effort low. One pass, no retries.
| Model | Correct / 231 | Hard / 111 | P50 full response |
|---|---|---|---|
| Open-Jev 2B | 150 | 46 | 138.0 ms, local H100 |
| Open-Jev 9B | 179 | 66 | 189.2 ms, local H100 |
| Open-Jev 27B v1.1 | 197 | 80 | omitted from the HTTP table |
| Jev 1.13.0 | 200 | 81 | 291.3 ms, hosted HTTPS |
| GPT-5.6 Luna, reasoning effort none | 206 | 89 | 953.8 ms, hosted HTTPS |
| GPT-6 Astra, reasoning effort low | 231 | 111 | 2,206.4 ms, hosted HTTPS |
- The easy tier is 48 of 48 for every row. On the original 72, Jev scores 71, Luna 69 and Astra 72. The note limits these scores to the public 231. The other 303 of 534 tasks were unavailable, and no 534-task score is reported.
- Open-Jev and Jev return native probabilities. GPT returns verbalized probabilities in JSON. Candidate order differs on 119 of 139 Choice tasks. One Jev vector was renormalized under the note’s rounding rule. Brier means are 0.1811, 0.2074 and 0.0085 for Jev, Luna and Astra. Top-label ECE is 0.0318, 0.0932 and 0.0149. The 27B report does not export ECE.
- Each P50 is one full-response observation per task, concurrency one, with no warmup and no retry. The note says this is not a repeated latency test. 27B ran by direct execution, and its timings are left out of that table. Tariff estimates for this mixture are $0.03842, $0.17914 and $8.81978 per 1,000 Jev, Luna and Astra decisions. The note says they are not billing receipts.
- A cookbook snapshot dated 26 September also uses 231 planned public tasks and stops early. Its Jev row, 116 correct with 110 tasks unattempted, is a different run from this 200 of 231.
Fifty 10-K questions: Jev RAG and an agent loop
David Arias, 27 and 28 September 2026. Fifty questions on the Apple, Microsoft, Nvidia and Amazon 10-Ks. Jev ranks a shortlist. The language model reads two passages. The agent searches EDGAR and can read whole filings.
| Writer | Agentic RAG | Jev RAG | Agent tokens read | Jev RAG tokens read |
|---|---|---|---|---|
| DeepSeek V4 Flash, thinking on | 46 of 50 | 50 of 50 | 82,908 | 840 |
| Qwen 3.7 Flash | 41 of 50 | 49 of 50 | 82,428 | 988 |
| GPT-5 nano | 34 of 50 | 48 of 50 | 64,634 | 815 |
- Code looks up the latest filing before Jev runs. All four DeepSeek agent misses opened the previous year’s filing. Across the three agent runs, 21 of 29 misses were that wrong-year mistake. GPT-5 nano also asked a clarifying question four times instead of searching.
- A hand reading changed 13 automatic grades, in both directions. Keyword plus vector search put a correct passage first on 20 of 50. Jev Choice did so on 45 of 50. Two shuffles of the same 30 candidates scored 46 and 44.
- Jev is jev-latest through OpenRouter. The X post says that alias resolved to Jev 1.13. DeepSeek Flash runs on DeepSeek’s API. The other two writers use qwen/qwen3.7-flash and openai/gpt-5-nano through OpenRouter. For DeepSeek, the agent averages 4.7 calls, 11.1 seconds and $0.0031 cached or $0.0133 uncached per question. Jev RAG averages 1 call, 2.0 seconds and $0.0007. One ranking call, about 12,000 tokens, had a median of 0.5 seconds and cost about $0.0005.
- The article treats four large-cap filings and 50 questions as too small to settle the comparison. Jev chooses passages. The language model reads the figures. Questions about earlier years would need a date parser the Jev pipeline does not have.
Sources:[89] David Arias[90] David Arias
Read the model report and user feedbackUnfinished five-alias run on 231 planned tasks
Datawhale Jev Cookbook, chapter 6 at commit 546050a, snapshot dated 26 September 2026. One run. Five aliases. 231 planned public tasks. Conditional accuracy uses scorable answers only.
| Alias | Valid / 231 | Correct / 231 | p50 | Recorded spend |
|---|---|---|---|---|
| DeepSeek Flash | 110 | 110 | 1.40 s | $0.020312 |
| Doubao 2.1 Pro | 105 | 105 | 4.49 s | $0.040792 |
| GLM 5.3 Codeplan | 65 | 65 | 6.51 s | $0.01376 |
| Jev | 120 | 116 | 0.77 s | $0.001986 |
| Kimi K3 | 139 | 139 | 9.63 s | $0.011435 |
- The names are aliases from the run files. They do not include a full version string. Jev’s conditional accuracy is 116 of 120. The other four aliases are correct on every scorable answer. DeepSeek’s 110 of 110 leaves 67 tasks unattempted. Kimi K3 has a row for every task and 92 errors, 89 of them empty probability parses.
- Jev stops with 110 tasks unattempted after the run exceeds its reservation, plus one HTTP 520. Doubao stops the same way, with 110 unattempted. DeepSeek and GLM stop after consecutive infrastructure errors, with 67 and 91 tasks unattempted. Hard-tier valid answers are 2, 1 and 29 of 111 for DeepSeek, Jev and Kimi.
- The chapter rounds the stored composites to 85.9 for Jev, 75.0 for DeepSeek, 73.7 for GLM, 73.2 for Kimi and 68.2 for Doubao. It says the highest score describes this incomplete run. p95 on the timed requests is 2.09, 1.38 and 15.60 seconds for DeepSeek, Jev and Kimi. Raw prompts and provider responses are not in the repository.
- This is a different run from the completed public-231 audit, where Jev 1.13.0 scores 200 of 231. Conditional accuracy in this snapshot uses the parsed answers as the denominator.
EosHQ decisions against Gemini 3 Flash Preview
EosHQ, 28 September 2026. Agent selection and automation detection are 18 Jev-enabled decisions each. Label selection is one observed request. The Jev path includes Gemini fallbacks. The article does not name a Jev version.
| Decision | Gemini 3 Flash Preview | Jev path, including fallback | Estimated cost per 1,000 |
|---|---|---|---|
| Choose a supplied label (one request) | 3,070 ms | 455 ms | $1.0905 to $0.0150 |
| Select an agent (18 decisions) | 3,380 ms | 1,616 ms | $1.293 to $0.526 |
| Detect an automation (18 decisions) | 3,432 ms | 925 ms | $1.366 to $0.234 |
- Fallbacks were 6 of 18 agent decisions and 3 of 18 automation decisions. Jev-only averages are 365 ms and 350 ms. The invoice-trigger example needed Gemini on all three repetitions and was slightly slower with Jev in front.
- The label row is one observed request per side. The article says the Gemini side also dropped the supplied labels and wrote a rationale. A separate timing of the Jev network request was 0.251 seconds. Costs are estimates from token counts and list prices, not invoices.
- Gemini 3 Flash Preview is priced in the article at $0.50 per million input tokens and $3 per million output tokens, including thinking. Jev is $0.042 per million input tokens, with output free. An earlier skill-selection row, 2,636 ms to 477 ms, is a step those requests now skip. One complete classification response then moved from 3.29 seconds to 2.63 seconds. The article says these samples are not a production latency guarantee.
Sources:[100] Kishore Mandyam[101] EosHQ
Read the model report and user feedbackClaude Code hook log for the OpenRouter jev-router alias
Christoph Görn, 28 September 2026. Eight prompts from earlier sessions, two of each size. The cloud call is OpenRouter’s typesafe/jev-router, asked for a JSON size label. Kev-4B is a local model on an RTX 4090.
| Sizer | Score on that set | Latency | Mean confidence |
|---|---|---|---|
| Kev-4B, local | 1 of 7 (14%) | 100 to 180 ms | 13% |
| typesafe/jev-router via OpenRouter | 1 of 5 (20%) | 700 to 1400 ms | 87% |
- The denominators are the author’s. The set has 8 prompts. He reports API errors on the OpenRouter side. Neither column is a score for TypeSafe’s Jev systemone API.
- A separate log of the first 16 prompts counts 3 tiny, 1 everyday, 1 large, 4 left on the main session, and 7 errors. The 9 successful calls average 2.99 seconds, from 1.26 to 4.41, and cost about $0.0032. answered_by names gpt-6-luna five times, deepseek-v4.1-flash three times, and kimi-k3 once.
- The X post’s 10 to 15 times figure is the latency pair in this table. Görn says the router log has no successful Kev decision yet, and that the router code is in a private infra repository. The main Claude session still reads every prompt.
Jev 1.13.0 and hosted judges on one contract
Yubo Li, Yidi Miao, Ramayya Krishnan and Rema Padman, arXiv:2609.26550, 22 September 2026. Base-order accuracy. Invalid outputs count as errors. RewardBench is 400 pairs, JudgeBench 350, HaluEval 240.
| Judge | RewardBench | JudgeBench | HaluEval | Valid responses |
|---|---|---|---|---|
| Jev 1.13.0 | 92.2% | 78.6% | 87.5% | 1312/1312 |
| Claude Sonnet 5 | 91.2% | 88.9% | 85.8% | 1308/1312 |
| Gemini 3 Flash | 92.8% | 76.0% | 83.3% | 1312/1312 |
| Gemini 3.1 Pro | 94.5% | 87.4% | 87.1% | 1312/1312 |
| GPT-6 Astra | 93.5% | 93.1% | 86.7% | 1312/1312 |
- Identifiers are jev-1.13.0, claude-sonnet-5, gemini-3-flash-preview, gemini-3.1-pro-preview and gpt-6-astra. Reasoning models run at low effort under a JSON schema, and the prompt asks for a label and label probabilities with no written rationale. JudgeBench is the GPT-4o split. RewardBench here is 100 pairs from each of four categories.
- Sonnet 5’s four failures are HTTP 200 contract breaks. Valid-only accuracy, which drops them, is 91.7%, 89.4% and 85.8%. Primary accuracy keeps them in the denominator. Flash and Pro are valid on every base item, so their two rates match.
- On 150 saved replies the scores are 94.0%, 94.7%, 95.3%, 94.7% and 96.7%, in the same row order. Macro-F1 on those labels is 0.923, 0.934, 0.939, 0.923 and 0.947. A frozen cascade on 510 later pairs accepts Jev at 0.9 and otherwise takes Astra: 53.7% accepted, 92.5% against 93.1%, fee ratio 0.568. Sonnet 5 at 0.6 accepts 90.8% and scores 88.0% against 89.2%. Flash at 0.6 accepts 90.8% and scores 87.1% against 82.9%. Pro at 0.9 accepts 53.7% and scores 89.6% against 90.4%.
- The printed latency medians on the 120-judgment panel are 0.152 seconds and $0.044 per 1,000 for Jev, and 1.885 seconds and $12.182 for GPT-6. The same sentence does not give Claude or Gemini medians. Fees use reported usage and are not invoices. The cascade fee ratios are offline simulations.
Sources:[102] Yubo Li, Yidi Miao, Ramayya Krishnan and Rema Padman
Read the model report and user feedbackGPT Researcher chunk filter before gpt-5.4
Assaf Elovic, eval note at commit 2906e632, 26 September 2026. 28 saved tasks, 20 from SimpleQA and 8 open-ended. gpt-5.4 writes every report. The client calls jev-latest. The note does not record the resolved version.
| Filter | Context precision | SimpleQA | Cost per report | Filter median |
|---|---|---|---|---|
| Jev, score at least 1.5, top 10 | 0.73 | 19 of 20 | $0.115 | 1.7 s at 64 in flight |
| Embeddings, cosine above 0.42, top 10 | 0.46 | 18 of 20 | $0.117 | 1.0 s |
| BM25 keyword fallback | 0.51 | 19 of 20 | $0.116 | 0.02 s |
| No filter | not reported | 20 of 20 | $0.192 | 0.0 s |
- Blind comparisons count only when gpt-5.4 prefers the same side in both orders. Jev against embeddings is 15 wins, 10 ties and 3 losses. The sign test on the 18 decided pairs is about p = 0.008. The keyword fallback against embeddings is 14 wins, 8 ties and 6 losses, and that sign test is about p = 0.12.
- On the eight open-ended questions, the unfiltered report wins 6 of the decided comparisons with Jev. The note says a one-question SimpleQA gap is noise, and that the writer and the pairwise judge are the same model family. Forcing Jev to fill all 10 slots, with no score floor, drops precision from 0.73 to 0.50.
- Jev’s filter median is 2.4 seconds with 32 calls in flight, and 1.7 seconds at the default of 64. Embeddings filter p90 is 1.3 seconds. select.py uses Jev when a TypeSafe key is set and the keyword fallback otherwise. Context p50 is 4.5k tokens for Jev, 7.6k for embeddings, 6.4k for the keyword row and 33.4k with no filter. The precision column is undefined for the unfiltered row.
A 60-tick Bay Area conflict window
Daniel Liao, jev-tower at commit 20dcbbb. One recorded ADS-B window, 60 ticks. A hazard is a closest approach inside 3 nautical miles and 1,000 feet within 120 seconds. A flag is a probability above 0.5. Jev is called as jev-latest.
| Model | Average latency | True positives | False positives | Misses |
|---|---|---|---|---|
| Jev beside gpt-4o | 99 ms | 385 | 335 | 0 |
| gpt-4o | 2,356 ms | 329 | 32 | 56 |
| Jev beside gpt-4o-mini | 103 ms | 385 | 335 | 0 |
| gpt-4o-mini | 2,242 ms | 225 | 22 | 160 |
| Jev beside claude-opus-5-5 | 119 ms | 385 | 335 | 0 |
| claude-opus-5-5 | 12,318 ms | 378 | 174 | 0 |
- The three Jev rows are the same window scored beside each chat model. True positives plus misses are 385 on the OpenAI logs. On the Opus log they are 378, and the file prints one fetch failure, so that miss count covers the pairs the tick returned.
- The README lists OpenAI false positives as 1 and 0, and one Jev latency of 103 ms. The JSON files show 32 and 22 false positives, and Jev averages of 99, 103 and 119 ms. The post’s 900 flights describe a live session. They are a different count from this window.
- The resolved Jev version is not in the logs. The README says the geometry rule is not a certified separation standard. OpenAI calls use temperature 0. Opus is claude-opus-5-5.
Citation-coverage changes on one Glean answer
Tony Gentilcore, 28 September 2026, with Aviral Singh and Manav Agrawal. One 1,448-word answer, checked three times. Jev’s time is sequential client wall time. The Luna times add up model-call durations.
| Judge | Paragraphs that changed | Run pairs that disagree | Time | Cost |
|---|---|---|---|---|
| Jev | 1 of 28 | 2 of 84 | 6.6 s | $0.014 |
| GPT-5.6 Luna, no reasoning | 7 of 28 | 15 of 84 | 81.3 to 85.5 s | $0.030 to $0.047 |
| GPT-5.6 Luna, xhigh | 5 of 28 | 10 of 84 | 227.4 to 259.7 s | $0.047 to $0.060 |
- Jev’s one paragraph change is whether that paragraph needed a citation. The article says the systems used different support criteria, and that the team did not collect independent human labels. This table is a repeatability check.
- The same article reports three other checks that are not in this table. Query classification agrees with Glean’s production LLM on 66.8% of requests for zero-shot Jev. The request count is not in the text. A routing replay of 751 entries has its accuracy percentages in a chart. On 40 entries that transferred, the median speedup is 8.1 times. Choice reranking on 4,855 queries raises Recall@6 from 40.2% to 44.3% when production order breaks identical scores.
- Costs include the caching the article observed. Gentilcore calls the rerank a cheap baseline, and says it does not replace Glean’s production ranker.
Sources:[105] Tony Gentilcore
Read the model report and user feedbackTwo early Decisions API checks against Jev
Dan Shipper, Every, 29 September 2026, reporting checks by Jack Cheng and Kieran Klaassen. OpenAI says the API uses GPT-6 Luna. The computer replay is text only. The article does not name a Jev version.
| Check | Decisions API | Jev | Decisions time | Jev time |
|---|---|---|---|---|
| Correct control on a text replay | 76 of 78 | 73 of 78 | 230 ms, typical | 500 ms, typical |
| Conversation classification | Accuracy not published | Accuracy not published | 309 ms median | 161 ms median |
- Shipper writes that the classification test was effectively tied on accuracy. The article’s opening says Decisions trailed Jev on Every’s broader evals. Those scores, the thread count, and the list of computer tasks are not in the article.
- OpenAI’s follow-up says the preview accepts text or images and that a broader release is planned in the coming days. Every expects a launch in a few weeks and says pricing has not been announced. The computer replay did not use images.
Sources:[110] Dan Shipper[108] OpenAI Developers[109] OpenAI Developers
Read the model report and user feedbackLedger-account shortlists on Crypto Accounting Bench
Entendre, 28 September 2026. Jev chooses among 2, 3, or 5 real accounts on the 118-task CAB set. Every shortlist contains the correct account. The incorrect options are other plausible accounts from the same chart.
| Condition | Jev accuracy | Coverage |
|---|---|---|
| 2 real accounts | 51.7% | Reported as the overall rate |
| 3 real accounts | 47.5% | Reported as the overall rate |
| 5 real accounts | 48.3% | Reported as the overall rate |
| 5 accounts, confidence at least 0.75 | 61.5% | 39 of 118 tasks |
- Successful calls averaged 635 ms. The post does not name a Jev version or count failed calls. It says the highest percentages in the confidence charts come from the smallest samples.
- The same post sets 48.3% beside 46.3% for GPT-5.6 Luna and 46.9% for DeepSeek V4 Flash, and 51.7% equal to a recorded Claude Opus 5 rate, with GPT-5.6 Sol at 51.4% and Claude Sonnet 5 at 49.4%. Those language-model figures are account correctness on full journal entries written from the complete chart. They are separate from these shortlist rows.
- The CAB paper’s Pass@3 on full entries is 47.46% for GPT-5.6 Luna, 50.00% for GPT-5.6 Sol, 45.76% for Claude Opus 5, 47.46% for Claude Sonnet 5, and 46.61% for deepseek-v4-flash-0731. Pass@3 requires a perfect rubric and every gate on one of three attempts. The median chart is 569 accounts. The public dataset does not include this Jev run.
Sources:[111] Entendre[112] Entendre[113] Entendre[114] Kareem Khattab, Omar Khattab and Mohamed Ibrahem[115] Entendre Finance
Read the model report and user feedbackBanking77 subsets at four class counts
Lucas Biagetti, v1.2 brief at commit efbfb42. Zero-shot rows on held-out Banking77 subsets. Each width uses 40 examples per class and seed 42. The cohorts overlap. The OpenAI row is gpt-5-nano-2025-08-07 with reasoning effort minimal. Jev is jev-1.13.0.
| Method | 5 classes | 10 classes | 15 classes | 20 classes |
|---|---|---|---|---|
| jev-1.13.0, zero-shot | 94.00% | 92.25% | 89.83% | 86.12% |
| gpt-5-nano-2025-08-07, zero-shot | 92.00% | 88.25% | 87.33% | 85.00% |
| MiniLM + logistic regression, 100 per class | 98.00% | 96.50% | 95.83% | 92.38% |
- The brief calls the comparison exploratory. One seed. Bootstrap intervals use 2,000 gold-stratified resamples, seed 20260925, and 95% percentile intervals with no multiplicity adjustment. The intervals condition on the fitted models and do not estimate training-seed variability.
- The columns cover 200, 400, 600, and 800 examples. Cohorts overlap and must not be pooled. Changing the class count adds classes and examples, so the trend does not isolate a causal effect of cardinality. Jev is missing 1, 2, and 3 probability vectors at 10, 15, and 20 classes. OpenAI returned no probabilities. The brief makes no cross-provider cost or latency claim. Class definitions were assistant-reviewed, without a separate human validation.
- These percentages are subset accuracies. OpenRouter’s Banking77 run uses all 77 intents and 3,080 utterances. The 37-dataset paper scores 3,076 Banking77 examples. The three setups stay separate.
Sources:[116] Lucas Biagetti[117] Lucas Biagetti[118] Lucas Biagetti[87] Kenny Rogers[123] Tobias Deußer, Lorenz Sparrenberg and Rafet Sifa
Read the model report and user feedbackjev-1.13.0 against Qwen3.8-27B and Gemma-4-E4B
Tobias Deußer, Lorenz Sparrenberg and Rafet Sifa, arXiv:2609.37647v1, submitted 29 September 2026. Zero-shot jev-1.13.0 on 37 datasets, 346,009 requests. Qwen3.8-27B and Gemma-4-E4B are scored on the same requests from next-token probabilities, with thinking off. They are the reference models on this card.
| Model | Banking77, n=3,076 | MMLU, n=14,042 | Datasets ahead of Jev |
|---|---|---|---|
| jev-1.13.0 | 79.7% | 91.8% | Reference on 37 datasets |
| Qwen3.8-27B | 77.4% | 81.7% | 9 of 37 |
| Gemma-4-E4B | 67.0% | 67.4% | 0 of 37 |
- Jev is ahead of Qwen on 27 datasets, tied on one, and behind on 9. Eleven of Jev’s leads have non-overlapping 95% bootstrap intervals. None of Qwen’s nine leads do. Jev is ahead of Gemma on all 37, and 31 of those gaps have non-overlapping intervals.
- The evaluation split cost US$9.15. With the pilot and the extra analyses, the study cost US$9.40. Mean client latency is 0.36 seconds at 32 concurrent requests. The full run took 5 hours 15 minutes, at up to 1,100 requests per minute.
- One template per dataset was frozen after 20 development examples, with no in-context examples. Rotating the options leaves accuracy unchanged. Withholding the question drops accuracy near chance. The authors say that rules out shallow memorization, and that it does not rule out memorized question-answer pairs.
- Jev’s lower absolute scores include Emotion at 58.5%, SST-5 accuracy at 57.9%, GoEmotions macro-F1 of 0.243, AGB-DE F1 of 0.204, and UNFAIR-ToS micro-F1 of 0.499 at a 0.5 threshold. Tuning thresholds on training data raises the UNFAIR-ToS figure to 0.748. Spearman correlation on SST-5 is 0.851. On rubric dimensions, Qwen’s correlation is higher on 9 of 12, mostly by small margins. Jev leads on ToxiGen and on SummEval consistency and fluency. Gemma trails on every rubric dimension.
- Scoring the open models took 6.4 A40 hours for Gemma and 17.4 A100 hours for Qwen. The paper says those hours depend on the hardware and are not comparable to API prices. Banking77 here is 3,076 examples. Lucas’s subset table and OpenRouter’s 3,080-utterance run are different setups. Commit 6bbdeb3 links the responses at doi.org/10.5281/zenodo.23039006. The Jev response license allows research and evaluation and bars distillation and training a model to imitate Jev.
Sources:[123] Tobias Deußer, Lorenz Sparrenberg and Rafet Sifa[124] Tobias Deußer[117] Lucas Biagetti[87] Kenny Rogers
Read the model report and user feedbackGemini 4 Argon announcement figures
Koray Kavukcuoglu, 30 September 2026. Google’s announcement of Gemini 4 Argon. The post includes no Jev arm, no API model id, and no date past the Fairwind rollout to trusted cyber defenders.
| Figure | Google’s report |
|---|---|
| DeepSWE v1.1 | 77.9% |
| AutomationBench | 51.3%, ranked first |
| LVBench | 91.7% |
| CWE-bench v1 remediation | 68%, tied for first |
| Output token limit | 1,000,000, up from 64,000 |
| Introductory price | $2 / $10 per million input / output tokens |
| Price after the introductory period | $4 / $20 per million input / output tokens |
| Cached input | 95% off the input token price |
- These rows are Google’s announcement figures. The post does not name Jev. Theo’s DeepSWE note on typesafe/jev-router publishes no score, so that run stays in its own report.
- Google says Argon leads the Vals Index and does not print the score. The 68% is the remediation score on CWE-bench v1, and the post does not name the model it ties.
- Access at announcement is the Fairwind Program. Paid API customers and Google AI Ultra subscribers are named as the later group. The post gives no calendar date for that step.
Sources:[125] Koray Kavukcuoglu[82] Theo
Read the model report and user feedbackAgent Arena thread on Jev Router
Arena.ai, 7 October 2026. A product the thread calls Jev Router by @typesafeai, measured on more than 4,700 Agent Arena sessions. The comparison row is DeepSeek V4.1 Flash (Max). The thread prints no API model id.
| Measure | Jev Router | DeepSeek V4.1 Flash (Max) |
|---|---|---|
| End-to-end median | 6.18 seconds | 3.64 seconds |
| End-to-end P90 | 30.59 seconds | 14.26 seconds |
| Steerability | +10.0% | −0.2% |
| Praise vs complaint | +6.3% | +2.1% |
| Confirmed success | +2.2% | +8.3% |
| Bash recovery | −0.7% | +7.8% |
| Tool hallucination | +0.2% | +0.4% |
| Net improvement, if listed | +3.6% | Not printed |
| Median cost per task | $0.155 | Not printed |
- The opening post says similar performance costs 38% more, and that median model-request latency is 1.7 times higher. The later post is the source of the 6.18 and 3.64 second medians and the P90 pair. The thread does not print DeepSeek’s dollar cost or its net-improvement percentage.
- Call shares in the follow-up are 28.1% DeepSeek V4.1 Flash, 23.5% GPT-6 Astra, 22.8% Sol, and 11.1% Luna. The opening post names GPT-6.1 Sol and GPT-6 Luna as frequent choices and does not use the name Astra. Steerability against Claude Opus 5.5 (High) is +10.0% against +10.48. The opening post rounds the router figure to +10%.
- Four of the five most-used models sit on the Pareto frontier. The fifth model is not named. The thread links no code and no session export. It does not say the calls used OpenRouter’s typesafe/jev-router. Theo’s DeepSWE note on that alias publishes no score. Görn’s 16-prompt hook log is a different sample.
Sources:[127] Arena.ai[79] OpenRouter[82] Theo[99] Christoph Görn
Read the model report and user feedbackTool or reply, with Sonnet 4.6 behind Jev
Josh Longenecker, Grounded AI, 8 October 2026. 200 items from GeneralFunctionCall-Test, 100 tool-call labels and 100 reply labels. The label is what Kimi K2-thinking did next. Jev is jev/jev-latest. The judge is bedrock/us.anthropic.claude-sonnet-4-6.
| Setup | Sent to Sonnet | Mean time | Median time | Cost per 1M decisions | Agreement |
|---|---|---|---|---|---|
| Sonnet 4.6 alone | 200 of 200 | 2.59 seconds | 2.35 seconds | $7,751 | 0.675 |
| Cascade at 0.7 | 73 of 200 | 1.28 seconds | 0.20 seconds | $2,997 | 0.695 |
| Cascade at 0.9 | 112 of 200 | 1.84 seconds | 2.11 seconds | $4,572 | 0.685 |
- The write-up rounds bench.json at commit 523194a. That file stores means of 2.591, 1.276 and 1.838 seconds, medians of 2.355, 0.199 and 2.110 seconds, and costs of $7,750.875, $2,996.74 and $4,571.86. sent_to_sonnet is 0.365 and 0.56.
- The write-up calls the 0.7 row 51% faster and 61% cheaper, and calls 0.9 the default threshold. The X post says the accuracy was the same. The table is agreement with K2-thinking, and the write-up says some disagreements with that label are ones it accepts.
- List prices in the caveats are Jev at $0.042 per million input tokens, output free, and Sonnet 4.6 at $3 and $15. The JSON does not record a resolved Jev version.
Sources:[129] Josh Longenecker[130] Josh Longenecker[128] Josh Longenecker
Read the model report and user feedback100-turn routing against Gemma 4, Cygnet and Clef
David Chiu, ywchiu/jev_benchmark at commit ed4a0ab, 4 October 2026. 20 conversations, 5 turns each. A turn counts only when route, scope change, mode and restriction are all correct. Nothing is retrieved and no tool runs. Gemma and Clef here are self-hosted. Jev is hosted Jev 1.13.0.
| System | All four decisions | Route | Scope change | Restriction | p50 |
|---|---|---|---|---|---|
| Gemma 4 31B, one JSON object | 77.0% | 90.0% | 87.0% | 95.0% | 2,293 ms |
| Cygnet readout, same 31B | 69.0% | 86.0% | 80.0% | 95.0% | 2,196 ms |
| Jev 1.13.0, hosted | 61.4% | 90.6% | 85.8% | 84.4% | 749 ms |
| Cygnet, gemma-4-12B-it | 58.0% | 79.0% | 76.0% | 94.0% | 265 ms |
| Clef 27B, open weights | 54.0% | 85.0% | 86.0% | 84.0% | 1,544 ms |
| Clef-flash 9B, open weights | 28.0% | 82.0% | 72.0% | 51.0% | 1,994 ms |
- Mode is 92.0% for the Gemma JSON row and 91.6% for Jev. Jev’s p95 is 818 ms. The Gemma JSON p95 is 4,723 ms. The README says the 31B server also served live traffic, so that latency is only indicative. Each Cygnet row was run once. Clef was run twice, Clef-flash five times, and both Clef sizes repeated the same answers. The original five systems were run five times.
- Cloudflare’s hosted API quota ran out, so the Clef rows use the open-weight systemone() path. Ask-first accuracy is 43% for Gemma and Clef, 31% for Jev, and the README says Cygnet on 12B does not ask first. Laya scores 0.0% and SemIf 24.0% on the all-four metric.
- The README says Cygnet was No. 1 on JevBench v1.5.4. Those items are single-turn and about 700 tokens. This bench carries about 2,500 tokens of history. Gemma-4-E4B in arXiv:2609.37647 is a different model and a different task. None of these rows is a Gemini API call.
Sources:[131] David Chiu[123] Tobias Deußer, Lorenz Sparrenberg and Rafet Sifa
Read the model report and user feedbackVals 400-item claim verification
Connor Frank, Vals AI, 6 October 2026. 400 claim-verification items from 10-Ks of 48 companies, five construction rules of 80 items each. Twelve systems. Jev is jev-1.13.0. Gemini 3.8 Flash wrote 160 paraphrases and contradictions; labels come from the construction rule.
| Measure | Jev 1.13.0 | Comparator |
|---|---|---|
| Correct of 400 | 390 (0.975) | GPT-6 Astra, GPT-6.1 Sol, GPT-6 Luna also 390 |
| Top of twelve | 390 | Claude Sonnet 5.5, 398 |
| Range of twelve | 0.953 to 0.995 | No system missed more than 32 of 400 |
| Cost per 1,000, with probabilities | $0.02 | Luna $0.10, DeepSeek V4.1 Flash $0.55, Argon $6.46, Opus 5.5 $9.64, Astra $18.00 |
| Cost per 1,000, label only | $0.02 | Luna $0.08; X thread Astra $12.29 |
| 32 judgments about one document | 194× lower median than Astra | 241× lower than Gemini 4 Argon |
| 1 to 64 judgments | 0.10s to 0.12s | Astra 38s at 64; Argon 65s |
| 1% error budget, held-out 70% | 95% coverage at 1.6% error | Sonnet 5.5 100% at 0.4%; Argon 100% at 0.8%; DeepSeek 96% at 0.8% |
| ECE on claims | 0.011, lowest of twelve | 0.108 on the LegalBench slice, highest of twelve |
- The write-up’s 1/498th of Astra uses the $0.02 and $18.00 probability-priced pair. The X thread’s $0.025 against $12.29 is the label-only Astra cost. Asking an LLM for probabilities raises Luna 29% and Astra 46%. Argon’s probability calls cost 3% less than its label calls.
- The 1% budget was fit on the first 30% of items and frozen. Six of twelve systems exceeded it. Sweeping the threshold over all 400 items would have let Jev automate 91% within the budget, against 100% for Opus 5.5, Sonnet 5.5, and Argon.
- Latency under load used 8 Jev requests in flight against 4 per LLM, from one residential machine. Isolated single-call median is 0.10s against 0.11s under load. Native structured output. Intervals bootstrap whole filings. This set is a different 10-K task from David Arias’s 50-question reading test.
Vals 396-question LegalBench slice
Connor Frank, Vals AI, 6 October 2026. Twelve subtasks, 33 questions each, 396 questions sampled from 1,808 available items, fixed before any model was called. The page says the slice is not representative of the full LegalBench.
| Measure | Jev 1.13.0 | Comparator |
|---|---|---|
| Class-balanced accuracy | 0.730, last of twelve | Haiku 4.5 0.732; GPT-5.6 Terra 0.932 |
| Raw accuracy vs the field | Every other lead is significant after eleven comparisons | LegalBench is public and likely in LLM training data |
| contract_nli_permissible_copy | Yes on all 33; 0.485 against a 16/17 split | Every LLM 0.91 to 1.00 on the two contract-NLI subtasks |
| contract_nli_confidentiality_of_agreement | Yes on 30 of 33 | 33 items per subtask |
| ECE | 0.108, highest of twelve | 0.011 on the claim set, lowest of twelve |
| Answers still missing after 180s retries | 0 | Terra 10, Opus 5.5 6, Sonnet 5.5 2, GPT-5.4 mini 1 |
| Google block | None reported | Argon blocked on one learned_hands_crime item |
- Sixteen of the 19 missing answers after retries were on international_citizenship_questions. Systems are scored on the items they answered. Jev gave the same answer on all five repeats of 25 items.
- The class-balanced metric is the protocol’s focus because some subtasks are unbalanced. A constant Yes on permissible_copy scores 0.485.
- This slice is a different legal task from OpenRouter’s Banking77 run and from Grounded AI’s 200 tool-or-reply items.
Seven political-science replications against GPT-6 Luna
Steven Denney and Matthew DiGiuseppe, arXiv:2610.06625v2, 6 October 2026. Seven published annotation and scaling tasks re-run with Jev 1.13, GPT-6 Luna, and Qwen3.8-27B. Human-coded scores are the reference.
| Task | Jev 1.13 | GPT-6 Luna |
|---|---|---|
| Gilardi 2021 relevance accuracy | 89.0% | 91.1% |
| Gilardi 2023 relevance accuracy | 76.2% | 84.2% |
| Ornstein SCOTUS sentiment r | 0.740 | 0.816 |
| GTD attack-type accuracy | 0.715 | 0.709 |
| V-Dem 2023 mean correlation | 0.68 | 0.63 |
| Party-name CHES mean r, 8 years | 0.877 | 0.844 |
| UK manifesto economic r | 0.97 | 0.977 |
| EP coal-speech r, 35 speeches | 0.81 | 0.947 |
| One-at-a-time median, 7 tasks | 0.09 to 0.27 s | 0.75 to 1.04 s |
- Luna at OpenAI batch prices costs less on six of the seven primary tasks and about 3% more on the seventh. Against Luna’s standard prices, Jev is cheaper on six of seven. Speed rows are 200 timed items per application, one request at a time, from the Netherlands on 29 and 30 September 2026.
- Jev ran through OpenRouter except on V-Dem, which used TypeSafe’s endpoint. Qwen ran through OpenRouter on Parasail at FP8. On the party comparisons, 74% of Luna’s probabilities sit below 0.01 or above 0.99, against 18% for Jev. When each question is asked once, Jev’s ECE is lower than Luna’s token probabilities on all eight reading tasks with a categorical human label.
- The V-Dem and party-name tasks give the model a name rather than a text. Those expert codes were public before these models were trained. The comments on the abstract list no public code archive. Qwen3.8-27B is an open-weight comparator in this paper, not a pairing on this site.
203 sales threads, Jev against Gemini 3.8 Flash
Premsanth Rajamani, Salesgear, 30 September 2026. 203 hand-written synthetic threads, labels fixed before either model saw them. Jev is jev-1.13.0. Gemini is gemini-3.8-flash in JSON classification mode with low thinking. Commit 926e72e.
| Measure | Jev 1.13.0 | Gemini 3.8 Flash |
|---|---|---|
| Correct category | 193 of 203 | 199 of 203 |
| Tricky threads (68) | 60 of 68 | 64 of 68 |
| Real deferrals found | 60 of 61 | 60 of 61 |
| Top 10 worth reopening | 9 of 10 | 9 of 10 |
| Come-back window on deferrals | 91.8% | 77.0% |
| ECE | 0.0199 | 0.0230 |
| Cost per 10,000 threads | $0.45 | $12.34 |
| Median call | 0.346 s | 2.188 s |
| Workers in results.json | 8 | 2 |
- Both top-ten lists include thread m059, a travel out-of-office reply labelled not_sales. Deferral recall is 0.9836 for both. Jev’s run cost $0.00906; Gemini’s $0.25055. The README’s 8-minute and 48-minute 10,000-thread estimates assume eight workers on both sides; Gemini’s wall time in the JSON uses two workers.
- Claude helped draft the threads, so Gemini is the baseline. Warmth and date arithmetic run in code. One run per model. One team wrote the labels. The public repository is developer943/sales-pipeline-revival-jev; the blog also links pipeline-revival-jev.
- Mudholkar’s 1,565-email table and EosHQ’s Gemini 3 Flash Preview timings are different tasks.
Jack & Jill initial screen, TypeSafe case study
TypeSafe AI, 12 October 2026. Offline test on 150 live roles. Gemini 3.1 Flash Lite against Jev on the initial scoring stage that cuts 6,000 retrieved candidates to a 2,000-person shortlist. No Jev version and no public log.
| Measure | Jev | Gemini 3.1 Flash Lite |
|---|---|---|
| Winners kept on the shortlist | 94.6% | 93.9% |
| 90% CI on the 0.7 point gap | −0.4 to +1.8, not significant | Same test |
| AUC | 0.933 | 0.924 |
| Median time to screen a role | 10.3 s | 20.3 s |
| Cost per 1,000 candidates | $0.092 | $0.755 |
| List price, input / output per million tokens | $0.042 / free | $0.25 / $1.50 |
| Current-volume annual saving | $265,000 | The Gemini baseline |
| Projected saving with growth | $500,000 | Excludes extra workflows |
- The quality test used every past hiring-manager winner plus a sample of remaining candidates weighted to represent the original 6,000. A winner is a candidate a hiring manager later asked to meet.
- More than 15 extra workflows, including a profile check TypeSafe says dropped from about 3.6 seconds to about 0.2 seconds, sit outside the dollar figures. Matched roles at sign-up are billed at under $0.10 per candidate.
- This is TypeSafe’s write-up of a customer test. Salesgear’s 203-thread table and Mudholkar’s email table stay separate.
Sources:[137] TypeSafe AI
Read the model report and user feedbackArtificial Analysis Decision Index v1.0
Artificial Analysis, 13 October 2026 launch article. Equal mix of AA-Omniscience-Decision (500 private questions), HLE-MultiChoice (513 text-only items), When2Call (3,423 tasks) and WildGuardTest (835 rows, average of three F1 scores). Language models answer with structured JSON. These are launch figures; the live leaderboard can change.
| Measure | Jev 1.13 | Comparator |
|---|---|---|
| Decision Index | 58 | Haiku 5.5 (Low) 61; Luna (Low) 55; GLiDE 54; Microsoft-Decision-1 54 |
| Time per task | 0.13 s | Celeris-1 0.06 s; Haiku 5.5 (Low) 1.4 s; Luna (Low) 1.5 s |
| Cost per 1M tasks | $42 | Decider v1.1 27B $17; Microsoft-Decision-1 $31; Haiku 5.5 (Low) $184; Luna (Low) $131 |
| HLE-MultiChoice | 21 | Haiku 5.5 (Low) 24; Luna (Low) 22; other decision models 15 or less |
| AA-Omniscience accuracy | 35 | Luna (Low) 66; Haiku 5.5 (Low) 52; Liquid d1 41 |
| AA-Omniscience non-hallucination | 35 | Microsoft-Decision-1 40; Mercury Decide 32; d1 2 |
| When2Call | 81 | Celeris-1 and GLiDE 84; Luna (Low) 77; Haiku 5.5 (Low) 73 |
| WildGuardTest | 94 | GLiDE 95; Decider v1.1 27B 94; Haiku 5.5 (Low) 93; Luna (Low) 89 |
- A decision model answers every question in one call. Artificial Analysis scores the highest-probability option, and takes yes at probability 0.5 or more. Language models receive a fixed system prompt and must return JSON. They supply no probability for the other options.
- Non-hallucination is the share of questions not answered correctly where the model chose “I don’t know”. WildGuardTest keeps rows where all three annotators agreed, and drops questions that consistently trigger refusals. When2Call drops 229 ambiguous ask-for-information tasks from 3,652.
- Cost per million tasks is one pass through the index, divided by the combined task count, times one million. GLiDE’s index score of 54 costs $119, about three times Jev. Hugging Face’s earlier Decision Index is a different board.
Sources:[139] Artificial Analysis[138] Artificial Analysis[140] Artificial Analysis
Read the model report and user feedbackThree Japanese classification sets against Claude 5.5
driller, Zenn, 10 October 2026, edited 11 October. jev-latest against Claude Haiku 5.5, Sonnet 5.5 and Opus 5.5 through system-one-adapter. MASSIVE 1.1 ja-JP scenario labels, 18 classes, 216 items. chABSA sentiment, 3 classes, 210 items. JGLUE JNLI v1.3, 3 classes, 210 items. One run per setup, seed 42, equal counts per class.
| Set | jev-latest | Opus 5.5 probabilities |
|---|---|---|
| MASSIVE accuracy | 0.884 | 0.926; Haiku discrete 0.870; Haiku prob 0.894; Sonnet 0.912 |
| chABSA accuracy, after rewrite | 0.910 | 0.881; Haiku discrete 0.900; Haiku prob 0.862; Sonnet 0.886 |
| JNLI accuracy, after rewrite | 0.838 | 0.914; Haiku discrete 0.867; Haiku prob 0.843; Sonnet 0.876 |
| ECE, three sets | 0.049, 0.059, 0.056 | 0.075, 0.050, 0.113 |
| Median latency | 0.16 to 0.17 s | Claude rows 1.3 to 3.5 s |
| Cost per 1,000 | $0.02 to $0.04 | About 1/8 to 1/16 of Haiku, 1/145 to 1/210 of Sonnet, 1/310 to 1/430 of Opus |
- chABSA and JNLI figures are after the option text was rewritten to match the annotation guides. The first chABSA pass moved Jev from 0.824 to 0.910. JNLI moved by 2 points or less. McNemar finds Opus ahead on JNLI (4 Jev-only correct, 20 Opus-only, p=0.002). The MASSIVE Opus gap is p=0.035 and is not treated as significant after ten pairs on one set.
- Each set has about 210 items and one run, so the article puts accuracy error near ±5 points. ECE uses 10 bins, about 20 items each. A cascade that sends low-confidence Jev items to Sonnet or Opus replays these labels; the author calls those accuracies optimistic.
- Claude calls used structured outputs and each model’s default effort. Haiku also ran in discrete mode, where the chosen label has probability 1. The Claude API bill for the check is $14.41; Jev is about $0.03 on a separate bill. chABSA’s licence text disagrees between the GitHub README and Kaggle.
Sources:[142] driller[141] driller
Read the model report and user feedbackALFWorld and AndroidWorld agents, Jev against gpt-5.6-luna
Yuchen Ling, Shengcheng Yu, Zhenyu Chen and Chunrong Fang, arXiv:2610.13092v1, 9 October 2026. Five agent families, text-only observations, jev-1.13.0 for action selection, gpt-5.6-luna for every LLM decision and auxiliary call. ALFWorld 134 valid_unseen tasks. AndroidWorld 116 task types. Three runs, 4,500 executions.
| Measure | Jev variant | gpt-5.6-luna baseline |
|---|---|---|
| ALFWorld ReAct success | 96.02% | 1.49 points higher; LLM-Choice gap 1.00 points |
| ALFWorld Reflexion success | 98.01% | 1.74 points higher; LLM-Choice gap 0.00 points |
| AndroidWorld T3A success | 13.22 points lower | Interval −17.82 to −8.62 |
| AndroidWorld AutoDroid success | 2.30 points lower | Both variants below 20%; interval includes zero |
| AndroidWorld FastAgent success | 18.10 points lower | Interval −26.44 to −9.77 |
| Decision cost / time | 66.1% to 78.4% lower / 73.9% to 81.5% lower | Decision time 4.13 to 8.13 s down to 1.04 to 1.75 s |
| T3A cost per success | 73.97 mUSD | 68.70 mUSD; +7.7%, interval includes zero |
| FastAgent time per success | 377.8 s | 239.4 s |
- ALFWorld’s LLM-Choice control uses Luna with Jev’s candidate list and selection instructions. AndroidWorld agents already pick from predefined actions. FastAgent-J executes one tool per step; FastAgent-B can emit several. AutoDroid’s app-specific memory is off in both variants.
- Mean task cost falls in every family. ReAct task cost falls 50.9% and time 47.6%. Reflexion cost falls 32.6% and time 37.8%. T3A cost falls 17.1% and time 10.6%. AutoDroid cost falls 68.8% and time 5.7%. FastAgent cost falls 43.0% and time rises 0.7%. Auxiliary reasoning and extra interactions absorb part of the decision saving.
- Reflexion-J succeeds on 383 of 402 first trials against 376 of 402 for the baseline, then recovers 11 of 19 initial failures against 25 of 26. Paired intervals use 20,000 task-cluster bootstrap replicates. Experimental code is at anonymous.4open.science/r/jev-emp-5BF4/.
Sources:[143] Yuchen Ling, Shengcheng Yu, Zhenyu Chen and Chunrong Fang
Read the model report and user feedbackArize eight-model decision bake-off
Jim Bennett, Arize, 7 October 2026. Eight decision models. RAGTruth 2,675 rows. 13 Phoenix evaluator suites, 655 labelled cases. 52 routing decisions. Hosted APIs over the network; open models on a MacBook Pro M4 Max, 36 GB. Each model gets its own yes/no cutoff, fit on half the data and scored on the other half.
| Measure | Jev 1.13 | OpenAI Decisions (gpt-6-luna) |
|---|---|---|
| RAGTruth balanced accuracy | 85.5% | 81.5%; intervals touch, recorded as a draw |
| RAGTruth p50 | 0.08 s | 0.16 s |
| RAGTruth cost per 1,000 | $0.045 | $0.090 |
| Phoenix suites | 91.1% | 90.0%; Liquid d1 87.6% |
| Phoenix calibration error | 0.023 | 0.042; Liquid 0.051 |
| Routing, 52 cases | 49 of 52 | 48 of 52; Clef 27B 50 of 52 |
| Hallucination ROC, both wordings | 0.93 and 0.93 | 0.90 and 0.89; Liquid 0.92 and 0.17 |
- OpenAI is the only model that does not accept Jev’s /v1/systemone layout. Arize wrote each case as labelled text sections and says a different layout could score differently. Accuracy wins require non-overlapping 95% intervals. Cloud and local latencies are not comparable.
- An earlier Arize Opus 5 judge scored 86.6% balanced accuracy at $14.30 per 1,000. Jev, Liquid and Clef 27B land within the noise of that figure on RAGTruth. Clef 27B scores 85.2% there, 91.5% on Phoenix, and 50 of 52 on routing, and went out in round one on speed and cost against Liquid.
- The polarity swap uses the same 1,338 test rows. Jev, Clef 27B and OpenAI keep their ROC. Kev 9B drops to 0.54. Liquid, Jeeves and Strands invert. This table is a different hallucination set from Vals’s 10-K claims and from the Carnegie Mellon judge panel.
Sources:[145] Jim Bennett[144] Arize AI
Read the model report and user feedback100 fictional return emails, jev-1.14.0 against Haiku 5.5
Takuma Yoshida, UnlockJapan, 26 October 2026. 100 made-up Japanese return emails for a fictional kitchenware shop. Six answers, nine policy rules applied from the top. Jev is jev-1.14.0. Haiku is claude-haiku-5-5, medium effort, one JSON call through the Claude Code CLI. Clairic is Cloudflare’s preview extractor @cf/cloudflare/clairic as of 24 October 2026.
| Setup | Correct of 100 | Time per case | Cost per 100 |
|---|---|---|---|
| Person (the author) | 73 | 18.0 s | ¥751 |
| Claude Haiku 5.5, one call | 91 | 3.0 s | ¥8.1 |
| Exp. 1: Clairic then Jev, blog layout | 68 | 0.5 s | ¥4.9 |
| Exp. 2: Jev reads, blog layout | 48 | 0.3 s | ¥1.6 |
| Exp. 3: six steps, Clairic reads | 77 | 0.9 s | ¥8.8 |
| Exp. 4: six steps, no Clairic | 78 | 0.3 s | ¥1.5 |
- In experiment 4, Jev’s final Choice matched a Python encoding of the nine rules on all 100 cases. All 22 misses came from reading the email. Nineteen of those 22 read rule 1 (injury, a strong complaint, or uncovered circumstances) into an email that had none. Haiku got 15 of those 19 right.
- The emails and order records are fictional, written by Claude after the correct answer was fixed. Haiku wrote none of them. Text sent to Jev and Clairic is Japanese. Yen figures use 150 yen to the dollar. Haiku’s cost is a list-price estimate inside a subscription. Confidence thresholds such as 0.8 were drawn after seeing the 100 results. Layouts were chosen after seeing other layouts on the same 100 cases.
- This table is a different Japanese task from driller’s MASSIVE, chABSA and JNLI check, and a different six-way sort from OpenRouter’s Banking77 run.
Sanity docs-agent decision times against Haiku
John Siciliano and Knut Melvær, Sanity engineering, 24 September 2026. Four decisions on a documentation agent that answers from Sanity Knowledge Bases. Jev is called as jev-latest in the sample request; the sample response logs jev-1.13.0. The answering model in the 0.5-second demo is openai/gpt-oss-120b on Cerebras.
| Decision | LLM (Haiku) | Jev | Saved |
|---|---|---|---|
| Entry choice | 1,328 ms | 282 ms | 1,046 ms |
| Speculation while typing | n/a at enter | 0 ms at enter | 1,828 ms |
| Model routing | 818 ms | ~300 ms | 511 ms |
| MCP choice | 829 ms | 170 ms | 659 ms |
- The median full answer is 0.5 seconds, and three of four finish under 0.9 seconds. Combined entry, model, and MCP questions in one Jev request took about 340 ms. The authors score 20 of 22 answers as fully grounded in the docs, with the first token in about 350 ms.
- Routing to a smaller model cut the cost of those answers by about 4.5 times and added about half a second at the median, so they turned routing off for the speed demo. Speculation is about $0.00008 per call. The post calls the numbers indicative: one week of testing on one corpus.
- These Haiku times are a different Claude comparison from Takuma Yoshida’s 100-email sort and from driller’s MASSIVE, chABSA and JNLI check.
One-pass limits on state tracking and reasoning items
Wang Yang, Xinpeng Li, Yanyan Zhang, Chuang Ma, Shouren Wang, Jianzhe Wei, Liqin Ye, Xiaoran Cai, Mohsen Hariri, Vipin Chaudhary and Xiaotian Han, arXiv:2610.24929v1, 18 October 2026. jev-1.13.0 through the hosted API, plus 23 open Jev-style configurations. Controlled logs from 1k to 20k tokens. 738 real items after dropping HelpSteer2 and 93 disputed labels, graded by Qwen3.8-27B thinking tokens.
| Measure | Jev 1.13.0 | Note |
|---|---|---|
| Retrieval, selection, event lookup | All 105 cells correct | Up to 20k tokens and 64 options |
| Final state at 16 updates | 53 to 80% | Similar records of other users interfere |
| One-pass-solvable real items | 97.5% | 646 items Qwen3.8-27B solves in one pass |
| Items that need reasoning | 74% | 92 items; 50% above 4k thinking tokens |
| Own-error AUROC / stale-input AUROC | 0.83 / 0.53 | Final-state items |
| Hand-off option taken | 1.6% of items, 3 of 40 errors | Real-task set |
- A stale one-line summary is followed even when the log contradicts it. Prompting for step-by-step thinking does not recover the sixteen-update items. Removing other users’ records restores 100% on those items.
- Qwen3.8-27B is the reasoning reference. The paper does not score a GPT API model and does not score jev-1.14.0. The 738-item mix includes JevBench, PubMedQA, and Banking77 slices packaged by Jevals; it is not OpenRouter’s 3,080-utterance Banking77 run, Lucas’s 5-to-20-class subsets, or arXiv:2609.37647’s 3,076-example row.
- Open replicas share the same pattern: retrieval stays high, combined state tracking falls, and a stale summary is trusted.
ClickHouse Agentic Analytics: table ranking and failure labels
Eduardo Vellasques, Lionel Palacin and Al Brown, ClickHouse, 27 October 2026. Jev ranks warehouse tables and labels agent failures on the Agentic Analytics Benchmark. Development set: 136 questions over an 18-table warehouse. Board: 201 held-out questions. Failure labels: four-model development run, 544 attempts, 277 failures. In-loop hint: 20 models. No Jev version named.
| Measure | Jev result | Note |
|---|---|---|
| Table ranking nDCG@10 | 0.951 | 136-question development set, 18 tables |
| recall@10 / all-required in top 5 | 0.991 / 84% | Against tables the reference SQL reads |
| Ranking latency | About 0.4 s per question | One yes/no Noul per table |
| Wide-catalog pilot, recall@10 | 87% | 250 cases, 75 to 710 tables; 73% all-required; p90 3.4 s; about $0.50 |
| Labeled failures | 277 of 544 attempts | Four-model development run |
| Wrong data selection | 87 of 277 | Largest bucket; mostly entity resolution |
| Grader audit at 0.65 | 18 of 277 | “Correct despite fail” |
| In-loop hint, pooled net | +12 (−18 to +42) | Fired cells across 20 models; accuracy flat |
- The labeler sends two Choice questions and two Noul questions in one call per failed trajectory. Sub-modes nest under five coarse modes. The post revises an earlier headline: a share of wrong plan is wrong table or wrong entity.
- The ranking keeps the full schema in the prompt and never prunes lower-ranked tables. The in-loop gate fires when a query reads none of the tables scored at 0.5 or above, and adds a note at most once per run.
- Code is ClickHouse/data-agent-mnist at commit 403f3e7. hev’s BEIR nDCG@10 table and Stagehand’s act timings stay separate.
Sources:[225] Eduardo Vellasques, Lionel Palacin and Al Brown[224] ClickHouse[226] ClickHouse
Read the model report and user feedbackSources and methodology
nikhil mudholkar (@nikhilmudholkar)
Published:
Source checked:
[9] Business-email classification against Gemini
Mudholkar compares Jev with Gemini on 1,565 German and English business emails in 10 categories. Gemini performs better on overall accuracy; the author is interested in Jev for its uncertainty signal.
An author-reported benchmark. Each model runs on its own. The thread has no combined Jev and Gemini pipeline.
nikhil mudholkar (@nikhilmudholkar)
Published:
Source checked:
[10] Email dataset composition and shared instructions
The dataset contains 1,201 real emails and 364 AI-written examples for rare categories, mostly in German. The three models receive the same email text and category instructions.
Methodology from the benchmark thread. 364 of the emails were written to fill rare categories.
nikhil mudholkar (@nikhilmudholkar)
Published:
Source checked:
[11] Email accuracy results for Jev and two Gemini models
Reported overall accuracy is 96.4% for Jev, 97.5% for Gemini 3.5 Flash-Lite and 98.5% for Gemini 3.8 Flash. Equal weighting of categories yields 92.0%, 94.6% and 96.9%, respectively.
Scores against the author’s reference labels on this dataset.
nikhil mudholkar (@nikhilmudholkar)
Published:
Source checked:
[13] Reported cost per 1,000 business emails
The author reports $0.08 for Jev, $0.80 for Gemini 3.5 Flash-Lite and $1.79 for Gemini 3.8 Flash per 1,000 emails in these runs.
Costs for this workload, counted per 1,000 emails. They are separate from the provider’s list price and from the bill for all 1,565 emails. The author notes that savings are small at ordinary inbox volumes.
nikhil mudholkar (@nikhilmudholkar)
Published:
Source checked:
[14] Attachments and synthetic examples in the email test
Attachments were excluded, and 364 emails were generated to match chosen labels. The author describes the lack of non-text input as a barrier to production use.
Limitations supplied by the experiment’s author. Attachments were left out of the run.
Rahul Kumar (@rahulbuildsmore)
Published:
Source checked:
[15] Jev and Gemini labeling 1,000 app reviews
Kumar compares sentiment, topic, bug and churn labels on 1,000 app reviews. He reports Jev at 4.6 seconds and $0.023, versus Gemini 3.8 Flash at 18.8 seconds and $0.158.
The post links to a public project with recorded runs and measurement notes. The two models are compared side by side, and Kumar does not chain them into one workflow.
goodrahstar
Source checked:
[16] Column Race recorded runs and evaluation methodology
Both lanes process 20 reviews per request at concurrency 8. The recorded pair uses Jev 1.13.0 and Gemini 3.8 Flash. Sentiment correlation with star ratings is 0.803 and 0.822; topic agreement is 84.4% and bug agreement is 92.6%.
One recorded pair over 1,000 Android reviews. Stars are a weak sentiment proxy, and topic and bug labels have no human ground truth, so the agreement rates only say the models matched each other. Costs use token counts and list prices. The repository has no invoices.
Hassan (@nutlope)
Published:
Source checked:
[19] Jev first-pass email screening with Kimi K3 fallback
Hassan tests 50 legitimate and 50 fraudulent emails. Jev classifies them in 1.42 seconds; 31 cases below a 95% confidence threshold go to Kimi K3. The full pipeline takes 16 seconds, gets 96 of 100 correct and costs about $0.07.
Author-reported demonstration, not independently rerun here. The stated cost split is $0.003 for Jev and $0.068 for Kimi. The 1.42-second figure excludes the Kimi stage.
Jamie Watters
Source checked:
[39] 42-claim test of whether a cited passage supports a sentence
Watters scores 42 claim-and-passage pairs: 24 constructed controls and 18 real published sentences. The schema-only run reports 85.7% for Jev, GPT-5.4 and Claude Sonnet 5, and 81.0% for Gemini 3.1 Pro. On the hard 18, the README shows 77.8%, 66.7%, 66.7% and 55.6%. A scratchpad rerun of the frontier models leaves those hard-set scores no higher. Unsupported-claim recall for Jev is 78.9%, short of the 80% bar set before the run.
One run per model, labels from the author’s own pipeline, frontier calls through OpenRouter. The table names jev-1.13.0, and the known-weaknesses section says the calls used the jev-latest alias. Costs come from the repo’s price file. McNemar p-values on the hard-18 scratchpad change are 1.000, 0.219 and 1.000. At those p-values the README stops short of calling the drops significant. For GPT-5.4 and Sonnet 5, the verdict call transcribes a short scratchpad.
Eastwood (@Tsj_estwld)
Published:
Source checked:
[45] SemEval-2026 DimABSA scores for Jev, Kimi-K2 Thinking and Qwen
Eastwood reports Jev 1.13.0 on all ten SemEval-2026 DimABSA subtask 1 test sets. Equal-weight mean RMSE_VA is 2.4750 for untuned Jev, 2.1889 for a Qwen3-14B QLoRA fine-tune, and 1.8731 for the official Kimi-K2 Thinking one-shot result. That is 0 wins and 10 losses against Kimi, and 3 wins and 7 losses against Qwen. Mean error is 32.1% higher than Kimi’s.
Lower RMSE_VA is better. The aspect is given, and the model scores polarity and activation. The chart caption says results use the official test metric without normalization, and that training and example conditions differ across methods. Kimi-K2 Thinking and the Kimi K3 in Hassan’s email demo are different models. The per-dataset chart shows Jev higher than Kimi on every set.
miguelg719
Source checked:
[43] Experimental Jev decision tree for Stagehand act()
The pull request compares an experimental Jev act path plus Gemini 3.8 Flash fallback with the existing Gemini 3.8 Flash act path. On 40 tasks, the baseline is 39/40 and the Jev arm is 118/120 across three trials, with median latency 1.97 seconds to 0.46 seconds and 4 of 147 acts calling the language model. A 40-task breadth set outside the pull request moves from 35/40 to 39/40 and from 3.19 seconds to 0.77 seconds. With the language model disabled, act scores 27/40.
Open pull request on 22 September 2026. The path is off unless experimentalJevAct or STAGEHAND_EXPERIMENTAL_JEV_ACT is set. Thresholds were tuned on these suites. Jev token use is logged but not yet included in result metadata. Breadth at three trials on two models is reported as 204/240 to 229/240.
miguelg719
Source checked:
[44] Experimental Jev observe() and cache check
The observe pull request reports a suite move from 9/12 to 10/12 against Gemini 3.8 Flash. Eleven of 16 observes were answered by Jev in 0.18 seconds, against a 1.99 second baseline, and 5 of 16 went to the language model. With the language model disabled the suite scores 7/12.
Open pull request on 22 September 2026, off by default. The cache-check threshold of 0.35 comes from API probes, and no eval runs that path end to end.
Kyle Jeong (@kylejeong)
Published:
Source checked:
[41] One remote-browser task with Stagehand and Jev
Jeong describes a Stagehand loop that sends an accessibility tree as state and candidate actions as questions, lets Jev pick the next action, and reports about $0.001 for one remote-browser task. A follow-up shows a jev-latest request with operation, click, input and completion questions.
One demonstrated task. The thread leaves the site unnamed, shows a single run, and leaves the typing model unnamed. The later act-suite timings are a separate measurement.
matyas (@Matyas_Jirsa)
Published:
Source checked:
[46] Sixty-email comparison with DeepSeek V4.1 Flash
The attached table compares DeepSeek V4.1 Flash and Jev on 60 emails. Both score 60/60. Median latency is 3.361 seconds versus 0.313 seconds, average latency 6.600 versus 0.348, and the 95th percentile 19.105 versus 0.465. Cost for the 60 is $0.01912 versus $0.00278, with zero API or schema errors on both sides.
The post text scales the same rates to 10,000 emails, about $3.19 versus $0.46 and about 9 hours 20 minutes versus 52 minutes. Those durations equal the median latency multiplied by 10,000. The post shows no log of that batch. The corpus, label rules, and Jev version are unnamed. DeepSeek V4.1 Flash is the image title. LimboAI’s note names V4 Flash, a different label.
LimboAI (@limbopeng)
Published:
Source checked:
[22] Refactoring intent recognition with Jev and DeepSeek
LimboAI reports replacing intent-recognition work in a project with Jev alongside DeepSeek V4 Flash and describes a positive experience with speed and cost.
Qualitative personal feedback. No dataset, baseline, elapsed time or billing totals are given.
Nikhil Mudholkar (@nikhilmudholkar)
Published:
Source checked:
[68] Three hundred configurator requests for Jev and Gemini
Mudholkar says he tested Jev and Gemini on 300 customer requests. Asking more questions in one call cut the agent from about 7 model calls to 3 and raised accuracy for both. He says Jev had a better trade-off when a later check filtered wrong answers, and that Gemini then hit a structured-output limit.
Opening post of a seven-post thread. The model versions, the catalogue, and the counts are in the following posts.
Nikhil Mudholkar (@nikhilmudholkar)
Published:
Source checked:
[69] A fictional catalogue with 578,143 valid products
The agent chooses among eight linked dropdowns in a fictional industrial catalogue. Mudholkar counts about 120 million combinations on paper and 578,143 valid configurations, 0.48 percent.
Synthetic catalogue built for this test.
Nikhil Mudholkar (@nikhilmudholkar)
Published:
Source checked:
[70] Fan-out accuracy and Jev’s median time
On 270 solvable requests, fan-out moves accuracy from 47.0 percent to 55.6 percent for Jev and from 52.2 percent to 62.6 percent for Gemini. Jev’s median time moves from 4.60 seconds to 1.89 seconds. Model calls fall from about 7 to 3.
Gemini’s median time and both model versions are absent from the post. Accuracy here means agreement with the author’s solvable-request labels.
Nikhil Mudholkar (@nikhilmudholkar)
Published:
Source checked:
[71] A requirement check that also drops some correct answers
After fan-out, some answers satisfied the catalogue while dropping a requirement the customer had stated. The added check stops 83 wrong Jev answers and 6 correct ones, and 72 wrong Gemini answers and 20 correct ones. Mudholkar calls that about 14 mistakes stopped per correct answer lost for Jev, and 3.6 for Gemini.
Counts are on the solvable requests. The check compares explicit requirements with the final product in code.
Nikhil Mudholkar (@nikhilmudholkar)
Published:
Source checked:
[72] Gemini rejects a wide structured batch before generation
At about 300 questions per call, Jev answered and Gemini rejected all 300 test requests before generation. The input was about 55,000 tokens. Configurator questions worked at 126 and failed at 127. Simpler yes/no questions reached 329. He says Jev also answered 801 short questions in a separate email test.
He attributes the rejection to Gemini’s structured-output constraints. The post treats the million-token window as a different limit. The 801-question figure is a different email test.
Nikhil Mudholkar (@nikhilmudholkar)
Published:
Source checked:
[73] The configurator test is synthetic, and Gemini still leads on accuracy
Across the 300 requests, Gemini still leads on accuracy. With the requirement check, Mudholkar reports about one eleventh the cost per correct configuration for Jev, and says Jev wrongly rejected 70 percent fewer correct answers. He says the test is synthetic and that the check was developed on these same cases.
The one-eleventh figure has no price table in the post. A postscript says the thread leaves out further detail.
hev mind (@hevmind)
Published:
Source checked:
[74] A reranker author reports a separate batch-size limit
hev says their reranker sizing hit a batch-size limit, and that the thing to debug is structured-output enforcement. He sets the million-token window aside. The reply links their repository.
A reply to Mudholkar’s thread. The 126/127 configurator split is Mudholkar’s. The repository’s own limit is about 30 passages per Jev call.
hev
Source checked:
[76] How the Jev reranker batches documents
The reranker sends one Noul per document, up to about 30 documents, inside a request budget of about 32,000 tokens. Lists longer than that are split. The README says a probability at or above 0.9 was relevant 76 percent of the time on the SciFact run, and a probability under 0.1 was relevant 0.5 percent of the time.
Checked at commit 1eb4726. The calibration sentence matches the pair-Noul check in RESULTS.md. Hosted calls send the documents to TypeSafe.
hev
Source checked:
[75] jev-1.13.0 reranking BEIR shortlists
The file records jev-1.13.0 on BM25 top-30 shortlists. On SciFact’s 300 test queries, 30 Nouls in one call score nDCG@10 of 0.768 at $0.18 list price. Claude Opus 5 at low effort scores 0.756, with 12 of 300 queries refused and scored as zero; substituting BM25 order on those 12 gives 0.778. GPT-5.6 Luna scores 0.747, 0.355 and 0.363 on SciFact, NFCorpus and FiQA.
Generated 2026-09-17. Costs are token usage at list price. Haiku 4.5 and Opus 5 appear on SciFact only. NFCorpus has 323 queries. We have not rerun the queries.
Delip Rao and Chris Callison-Burch
Published:
Source checked:
[86] Jev Choice against three flash rubric judges
The paper compares Jev Choice with GPT-5.6 Luna, Gemini 3.8 Flash and DeepSeek V4.1 Flash on nine panels and 5,003 pairs, using the same criterion texts. Accuracy separates in 8 of 27 paired comparisons. Across the panels, Jev Choice cost $0.063 and finished in 29 seconds. Luna, DeepSeek and Gemini cost 29, 66 and 325 times as much, and took 859, 6,298 and 950 seconds.
arXiv:2609.29769, submitted 24 September 2026. Requests used the jev-latest alias. A same-day stability check logged jev-1.13.0. The main runs did not log the version. LLM judges are called once per criterion. On Jev’s most confident errors, 96.0% of the LLM verdicts repeat the wrong answer. A cascade replayed on the recorded verdicts gains at most 1.5 points over the best single judge with cross-fitted thresholds, and at most 2.0 with oracle thresholds. Jev’s concurrency was lower than the LLM judges’, so the wall-time ratios understate a same-concurrency comparison. DeepSeek’s three failed calls on the binary panels were recorded as UNMET.
Kenny Rogers
Published:
Source checked:
[87] Banking77 intents: Jev 1.13 and Claude Opus 5
OpenRouter ran the 3,080-utterance Banking77 test split through Jev 1.13 and Claude Opus 5. Accuracy is 81.0% and 84.4%, macro-F1 is 80.5% and 83.6%, median round trip is 175 ms and 2,266 ms, and the billed totals are $0.34 and $7.44.
Kenny Rogers, OpenRouter, 22 September 2026. Both models received the same one-line criteria, written from label names without looking at the test data. Opus ran at temperature 0 with reasoning off and a strict JSON enum. Eight requests were in flight. Latency is the client round trip through OpenRouter. The Opus system prompt was cached. The write-up says the uncached list rate would have been about $19 per 1,000 requests. Confidence at or above 0.99 covered 58% of utterances at 96.3% accuracy. Below 0.5 covered 3.5% at 29.6% accuracy. Sending confidence under 0.90 to Opus left 75.9% on Jev, at 84.0% accuracy and $0.69 per 1,000. That cascade table reuses the same 3,080 examples, and the write-up calls it an upper bound. Both models scored 0 of 40 on get_physical_card. One dataset, one afternoon. Opus was not run with reasoning on.
OpenRouter (@OpenRouter)
Published:
Source checked:
[79] OpenRouter announces typesafe/jev-router
OpenRouter introduced typesafe/jev-router as a cache-aware router that uses Jev to pick a model and a reasoning effort for each request, with the stated aim of balancing quality, speed and cost.
Product announcement. The post describes the router. It does not include a score, a task count, or a price for the routed models.
Theo (@theo)
Published:
Source checked:
[80] Why a prompt is a weak input for routing a coding agent
Theo writes that Jev categorizes and does not reason, and that a prompt alone does not show the codebase, the tools, or how large the change is. His example is a request to port a project to Rust, which could be a 100-line file or a million-line application.
Reply in the thread under OpenRouter’s Jev Router announcement. The argument is about what the router can see. It is not a benchmark score.
Theo (@theo)
Published:
Source checked:
[81] An in-progress look at Jev Router’s model mix
While the DeepSWE bench was still running, Theo wrote that Jev Router was so far basically a DeepSeek 4.1 Flash router.
Interim note, posted hours before the final DeepSWE comparison. The post does not give a percentage split across models.
Theo (@theo)
Published:
Source checked:
[82] DeepSWE run of Jev Router against GPT-6 Astra on low
Theo says he spent $1,000 and stayed up until 2am benchmarking Jev Router. On DeepSWE, performance was roughly the same as GPT-6 Astra on low, the router cost slightly more, and the run took almost 5 times as long.
Author-reported comparison with the OpenRouter router. The post gives no DeepSWE score, no task count, and no dollar gap beyond “slightly more” and the $1,000 spent on the bench.
Theo (@theo)
Published:
Source checked:
[83] Follow-up after the Jev Router bench
Immediately after the DeepSWE post, Theo’s update is to use Astra on low.
One-line follow-up in the same thread. It names the setup he would use instead of the router. It does not add a second measurement.
pcparts001
Source checked:
[88] Pi extension that sets DeepSeek reasoning effort with Jev
The extension asks Jev whether the next Pi prompt needs none, low or high reasoning, then changes only that thinking level. The README’s example model is deepseek-v4.1-flash. On 30 tasks against always-high, the corrected per-task cost is $0.2844 versus $0.3352. Against always-low, routing costs $0.2845 versus $0.2112, with equal answer quality on that run.
Read at commit 2215f7c. The default route calls jev-latest on TypeSafe’s API, which the README equates with jev-1.13.0. The 30-task figures are from a pre-port benchmark on a direct DeepSeek route, and the README limits those quality and cost figures to the direct route. Spearman rho against difficulty is +0.504 (p = 0.0040, n = 30). The measured-results table lists decision latency of 409 to 856 ms. A missing key, a timeout, or a bad answer falls back to high. The README says none was never returned in that set, so the observed router was low versus high.
Zefan Cai (@Zefan_Cai)
Published:
Source checked:
[91] Open-Jev 27B v1.1 on the public JevBench tasks
Zefan Cai reports Open-Jev 27B v1.1 at 197 of 231 public JevBench tasks, and 80 of 111 on the hard tier. Jev is 200 of 231 and 81 of 111. The post says the 231 tasks are a public subset of 534.
Announcement post. The GPT-5.6 Luna and GPT-6 Astra rows, timings and scoring rules are in the repository note at commit 3308a15, not in this post.
Zefan Cai
Published:
Source checked:
[92] Public JevBench note at Open-Jev commit 3308a15
The note scores six streams on all 231 available public tasks. Jev 1.13.0 is correct on 200. GPT-5.6 Luna, reasoning effort none, is correct on 206. GPT-6 Astra, reasoning effort low, is correct on 231. Easy tasks are 48 of 48 for each of those three. On the original 72 the counts are 71, 69 and 72. On the hard 111 they are 81, 89 and 111.
Read at commit 3308a15. The other 303 of 534 tasks were unavailable, and the note reports no 534-task score. Open-Jev and Jev return native probabilities. GPT returns verbalized probabilities in JSON. Candidate order differs on 119 of 139 Choice tasks. One Jev vector was renormalized. P50 full-response times, one observation per task, are 291.3 ms for Jev, 953.8 ms for Luna and 2,206.4 ms for Astra. The note says that sample is not a repeated latency test. Tariff estimates are $0.03842, $0.17914 and $8.81978 per 1,000 decisions, and the note says they are not billing receipts. Brier means are 0.1811, 0.2074 and 0.0085. Top-label ECE is 0.0318, 0.0932 and 0.0149. Open-Jev 27B v1.1 is 197 of 231, and its direct-execution timings are left out of the latency table. Upstream tasks are pinned at fstandhartinger/jevbench commit f8ce713.
Datawhale
Published:
Source checked:
[94] Jev Cookbook chapter 6 snapshot for 26 September 2026
The chapter records one run of five aliases on 231 planned public tasks. Kimi K3 has 139 valid answers and 139 correct. Jev has 120 valid and 116 correct, then leaves 110 tasks unattempted after a reservation stop. DeepSeek Flash has 110 valid and 110 correct, then stops after consecutive errors. Conditional accuracy uses only scorable answers.
Read at commit 546050a. snapshot.json in the same commit matches the counts. Recorded spend is $0.001986 for Jev, $0.011435 for Kimi K3 and $0.020312 for DeepSeek Flash. p50 latency is 0.77, 9.63 and 1.40 seconds. The aliases do not include a full version string. Hard-tier valid answers are 1, 29 and 2 of 111. The stored composite is highest for Jev, and the chapter says that score describes this incomplete run. Raw prompts and provider responses are not in the repository. This run is separate from the completed 231-task audit in the Open-Jev note.
David Arias (@Davidariasfin)
Published:
Source checked:
[89] Jev ranks 10-K passages before a language model answers
David Arias describes 50 questions about four annual reports. Jev ranks retrieved passages and a language model reads the top two. With DeepSeek Flash, that pipeline scores 50 of 50 against 46 of 50 for an agent that reads whole filings. The post says jev-latest resolved to Jev 1.13, and that search put the right passage first 40% of the time against 90% after Jev ranked the same candidates.
The X post is the write-up. The linked article has the per-model tables. The 40% figure matches the article’s keyword-plus-vector row, not every search method on that page.
David Arias
Published:
Source checked:
[90] 50 questions on four 10-Ks: Jev RAG and an agent loop
The article scores 50 analyst-style questions on the Apple, Microsoft, Nvidia and Amazon 10-Ks. Jev RAG with DeepSeek V4 Flash, thinking on, scores 50 of 50. The agent scores 46 of 50. The same passages sent to Qwen 3.7 Flash and GPT-5 nano score 49 and 48 of 50, against agent scores of 41 and 34. Code looks up the latest filing before Jev runs. A hand reading changed 13 automatic grades.
Read 28 September 2026. The page dates the tests to 27 and 28 September 2026. Jev is called as jev-latest through OpenRouter. The X post says that alias resolved to Jev 1.13. DeepSeek Flash runs on DeepSeek’s API. Qwen 3.7 Flash is qwen/qwen3.7-flash and GPT-5 nano is openai/gpt-5-nano, both through OpenRouter at provider defaults. For DeepSeek, the agent uses 4.7 calls, 82,908 tokens, 11.1 seconds and $0.0031 cached or $0.0133 uncached per question. Jev RAG uses 1 call, 840 tokens, 2.0 seconds and $0.0007. Keyword plus vector search is first on 20 of 50. Jev Choice is first on 45 of 50, and two shuffles score 46 and 44. One Jev call over about 12,000 tokens had a median of 0.5 seconds and cost about $0.0005.
wang yiming (@wangyiming34110)
Published:
Source checked:
[93] A five-model snapshot of 231 planning tasks
The post points to chapter 6 of the Jev Cookbook and says that, on one snapshot of 231 planning tasks, Jev produced 120 valid answers at a p50 of 0.77 seconds, while Kimi produced 139 valid answers, the most of the five models.
The post gives valid-answer counts and one latency figure. Stop reasons, correct-answer counts and version strings are in the cookbook chapter at commit 546050a.
Kishore Mandyam (@nmandyam)
Published:
Source checked:
[100] EosHQ timings for agent selection and automation detection
Kishore Mandyam says EosHQ uses Jev for fast decisions. In synthetic tests, agent selection took 52% less time and automation detection 73% less, including Gemini fallbacks. The post links the article for the numbers and the exceptions.
The percentages match the article’s agent-selection and automation rows. The article is the source for sample size, the Gemini version, and the cases that were slower.
EosHQ
Published:
Source checked:
[101] EosHQ decision timings against Gemini 3 Flash Preview
The article reports a Jev path, including Gemini 3 Flash Preview fallbacks, at 1,616 ms for agent selection and 925 ms for automation detection, against 3,380 ms and 3,432 ms. Label selection is one observed request, 3.07 seconds to 0.455 seconds. The article does not name a Jev version.
Measurements dated 28 September 2026. Agent selection and automation detection are 18 Jev-enabled decisions each, from six scenarios repeated three times. Fallbacks were 6 of 18 and 3 of 18. Jev-only averages are 365 ms and 350 ms. One invoice-trigger example needed Gemini on all three repetitions and was slightly slower. Costs per 1,000 are estimates from token counts and list prices: Gemini 3 Flash Preview at $0.50 per million input tokens and $3 per million output tokens, including thinking, and Jev at $0.042 per million input tokens with output free. The article says the samples are not a production latency guarantee.
Christoph Görn (@goern)
Published:
Source checked:
[98] A Claude Code router timed against local Kev
Christoph Görn says a local Kev-4B router on an RTX 4090 was 10 to 15 times faster than the cloud sizer he called Jev, 100 to 180 ms against 700 to 1400 ms, and that Kev’s accuracy was 14%. The post links his write-up.
The 14% figure is Kev’s score in the write-up, on 8 prompts. The cloud side of that table is OpenRouter’s typesafe/jev-router alias, and the write-up says the log named other models. The 16-prompt latency average is a different sample.
Christoph Görn
Published:
Source checked:
[99] What answered the Claude Code sizing hook
The hook calls OpenRouter’s typesafe/jev-router and asks for a JSON size label. On the first 16 prompts the log counts 3 tiny, 1 everyday, 1 large, 4 left on the main session, and 7 errors. The answered_by field names gpt-6-luna five times, deepseek-v4.1-flash three times, and kimi-k3 once.
Read on 28 September 2026. The 9 successful calls average 2.99 seconds, from 1.26 to 4.41, and sizing cost about $0.0032. A separate 8-prompt table scores Kev-4B at 1 of 7 and the alias at 1 of 5, with latencies of 100 to 180 ms and 700 to 1400 ms. The author says the router code is in a private infra repository and that the router log has no successful Kev decision yet.
Yubo Li, Yidi Miao, Ramayya Krishnan and Rema Padman
Published:
Source checked:
[102] Jev 1.13.0 as a judge next to GPT-6, Claude and Gemini
The paper scores jev-1.13.0 against hosted GPT, Claude and Gemini judges on the same Choice contract. On a 400-pair RewardBench sample Jev scores 92.2% and GPT-6 Astra 93.5%. Claude Sonnet 5’s primary scores are 91.2%, 88.9% and 85.8%, and four contract failures stay in that denominator. Gemini 3 Flash scores 92.8%, 76.0% and 83.3%. Gemini 3.1 Pro scores 94.5%, 87.4% and 87.1%.
arXiv:2609.26550v1, dated 22 September 2026. Jev is jev-1.13.0. GPT-6 Astra is gpt-6-astra at low effort with a JSON schema. Claude Sonnet 5 is claude-sonnet-5 and Gemini 3 Flash and 3.1 Pro are gemini-3-flash-preview and gemini-3.1-pro-preview, also at low effort. RewardBench here is 100 pairs from each of four categories, not the official subset weights. JudgeBench is the GPT-4o split. HaluEval is 240 evidence-grounded judgments. Existing-label accuracy is a fourth column on 150 replies: 94.0% for Jev, 94.7% for Sonnet 5, 95.3% for Flash, 94.7% for Pro and 96.7% for Astra. Invalid outputs count as errors in the primary accuracy. Sonnet 5 is valid on 1,308 of 1,312 items, all four failures HTTP 200 contract breaks. Table 6’s valid-only Sonnet 5 rates are 91.7%, 89.4% and 85.8%. Flash and Pro are valid on all 1,312. The isolated latency panel is 120 judgments, and the printed medians name Jev, GPT-4.1 mini and GPT-6. Fees use reported usage at collection-time prices and are not invoices. The single-order cascades read the threshold off the same items. The two-order policies were frozen earlier, on 510 extension pairs: Astra at 0.9 scores 92.5% against 93.1% at 56.8% of the fee; Sonnet 5 at 0.6 scores 88.0% against 89.2%; Flash at 0.6 scores 87.1% against 82.9%; Pro at 0.9 scores 89.6% against 90.4%. A human adjudicated disagreements with GPT-6. The second pass was an LLM, and an author resolved the 39 items where the two passes differed.
Assaf Elovic (@assaf_elovic)
Published:
Source checked:
[103] GPT Researcher replaces its embedding filter with Jev
Assaf Elovic says GPT Researcher replaced its embedding filter with Jev on 28 research tasks, that 73% of kept chunks were relevant against 46%, and that blind comparisons preferred the Jev reports 15 to 3 at the same cost. He says the project now runs on Jev by default.
Announcement post. The 15-to-3 figure omits the ties in the repository table. The task split, the writer model, the keyword fallback and the Jev alias are in the eval note at commit 2906e632.
Assaf Elovic
Published:
Source checked:
[104] GPT Researcher context-filter note at commit 2906e632
The note replays 28 saved research tasks, 20 from SimpleQA and 8 open-ended, and writes every report with gpt-5.4. Jev keeps 1,000-character chunks scoring at least 1.5 on a four-level usefulness rubric. Context precision is 0.73 against 0.46 for embeddings, and the blind comparison is 15 wins, 10 ties and 3 losses. Cost per report is $0.115 and $0.117.
Read at commit 2906e632. The precision judge is gpt-5.4-mini. A pairwise win counts only when gpt-5.4 prefers the same side in both orders. Jev’s filter time is 2.4 seconds at the median with 32 calls in flight, and 1.7 seconds at the default of 64. The client calls jev-latest unless JEV_MODEL is set, and the note does not record which version answered. select.py at the same commit uses Jev when a TypeSafe key is present and a BM25 keyword fallback otherwise. That keyword row is 19 of 20 on SimpleQA, precision 0.51, and 14 wins, 8 ties and 6 losses against embeddings, at $0.116. Its sign test is about p = 0.12. SimpleQA is 19, 18 and 20 of 20 for Jev, embeddings and no filter. The note says a one-question difference is noise. On the 8 open-ended questions, no filter wins 6 of the decided comparisons with Jev. Skipping the filter costs $0.192 per report.
Daniel Liao (@Daniel_Bernath_)
Published:
Source checked:
[106] Jev judges aircraft pairs over the San Francisco Bay
Daniel Liao says a Jev judge watched about 900 live flights from an E2B sandbox, flagged risky pairs in 103 ms, and missed none. He says gpt-4o missed 56 in the same sky and Claude Opus 5.5 took 12 seconds to miss none.
The post links 7datumba/jev-tower. The 103 ms, 56 misses and 12 second figures match files in that repository. The false-positive counts and the fetch error are in those files, not in this post. The post’s 900 flights are not the 60-tick window used for the ladder.
Daniel Liao
Published:
Source checked:
[107] jev-tower logs at commit 20dcbbb
tower.mjs replays one 60-tick ADS-B window. A pair is a hazard when the closest approach is inside 3 nautical miles and 1,000 feet within 120 seconds. Jev, called as jev-latest, and the chat model both flag a pair when their probability is above 0.5. Beside gpt-4o, Jev averages 99 ms with 0 misses and 335 false positives. gpt-4o averages 2,356 ms with 56 misses and 32 false positives.
Checked at commit 20dcbbb. data/tower-gpt4o.json, data/tower-gpt4o-mini.json and data/tower-claude-opus-5-5.json are the three logs. Jev’s counts stay at 385 true positives, 335 false positives and 0 misses. Average latency on the three runs is 99, 103 and 119 ms. gpt-4o-mini averages 2,242 ms, with 160 misses and 22 false positives. claude-opus-5-5 averages 12,318 ms, with 0 misses and 174 false positives on the pairs it scored, and 378 true positives against Jev’s 385. The Opus log prints one fetch failure. The README table lists Jev at 103 ms and the OpenAI false positives as 1 and 0, which these files do not show. The resolved Jev version is not in the logs. The ground truth is the script’s geometry rule.
Tony Gentilcore (@tonygentilcore)
Published:
Source checked:
[105] Four Glean checks, including a citation judge against GPT-5.6 Luna
Tony Gentilcore’s article names Aviral Singh, Manav Agrawal, and the accounts @eddiedzhou, @mr_cheu and @MatZhao. On one 1,448-word answer, Jev’s citation-coverage verdict changed on 1 of 28 paragraphs across three runs. GPT-5.6 Luna with no reasoning changed 7, and Luna at xhigh changed 5.
X article posted 28 September 2026. The citation timing is 6.6 seconds and $0.014 for Jev, against 81.3 to 85.5 seconds and $0.030 to $0.047 for Luna with no reasoning, and 227.4 to 259.7 seconds and $0.047 to $0.060 at xhigh. Jev’s time is sequential client wall time. The Luna times sum model-call durations. The article says the two systems used different support criteria and that there were no independent human labels. Query classification reports agreement with a production LLM, not a human label: 66.8% for Jev, 35.9% for base Laya and 74.5% for a fine-tuned Laya. The request count for that table is not in the text. A 751-entry routing replay uses three experts. The article says Jev was more accurate than the existing LLM router. The chart percentages are not transcribed here. The median speedup on 40 transfer calls is 8.1 times. Choice reranking on 4,855 queries gave identical scores to about 37 of 41 candidates. Using production order to break ties raised Recall@6 from 40.2% to 44.3%.
Dan Shipper
Published:
Source checked:
[110] Every’s early Decisions API checks against Jev
Dan Shipper reports two early checks of OpenAI’s Decisions API against Jev. On a text-only replay of computer tasks, Jack Cheng found 76 of 78 correct controls for Decisions and 73 of 78 for Jev, at typical times of 230 and 500 milliseconds. On conversation classification, Kieran Klaassen’s test was tied on accuracy, with medians of 161 and 309 milliseconds.
Every, 29 September 2026. The article says Decisions accepts images as well as text, and that this computer replay was text only. It says Decisions trailed Jev on Every’s broader evals and does not publish those scores, the Jev version, the thread count, or the task list. Pricing is unannounced. Shipper expects a launch in a few weeks. The post that names GPT-6 Luna is OpenAI’s, not this article.
OpenAI Developers (@OpenAIDevs)
Published:
Source checked:
[108] Decisions API preview, powered by GPT-6 Luna
OpenAI’s post says the Decisions API, powered by GPT-6 Luna, takes a question and a fixed set of answers. The caller uses the returned selection to classify content, route a request, or choose an agent’s next action. The post says the API is in limited preview.
Announcement post on 29 September 2026. It does not state a price, a latency, or a comparison with Jev. Image input and the preview’s release plan are in the follow-up post.
OpenAI Developers (@OpenAIDevs)
Published:
Source checked:
[109] Decisions API accepts text or images in the preview
The follow-up says a Decisions API call can send text or images as context, for example a support request and the teams that could take it. The API returns a selection. Preview access is limited to selected API customers, and the post says a broader release is planned in the coming days.
Second post in the announcement thread. It links to OpenAI’s DevDay 2026 recap and does not give a price or a measured latency. The model name GPT-6 Luna is in the first post.
Entendre (@EntendreAI)
Published:
Source checked:
[111] Jev picks a ledger account from two, three, or five options
Entendre tested Jev on Crypto Accounting Bench by asking it to choose among 2, 3, or 5 real accounts. Reported accuracy is 51.7%, 47.5%, and 48.3%. Successful calls averaged 635 ms. On the five-account list, confidence of at least 0.75 raised accuracy to 61.5% and covered 39 of 118 tasks.
First post in the 28 September 2026 thread. The post says the highest percentages in the confidence charts come from the smallest samples. It does not name a Jev version or say how many calls failed. The shortlist rules and the language-model comparison are in the next two posts.
Entendre (@EntendreAI)
Published:
Source checked:
[112] Every Crypto Accounting Bench shortlist contained the right account
Entendre says each option was a real ledger account, and the incorrect choices were plausible alternatives for that transaction. Every shortlist contained the correct account. The score is agreement with the journal entry the company recorded.
Second post in the thread. It points to the Crypto Accounting Bench write-up, the research paper, and a public dataset. The paper and dataset describe full journal entries. They do not contain this Jev shortlist run.
Entendre (@EntendreAI)
Published:
Source checked:
[113] Shortlist accuracy set beside full journal-entry rates
With five candidates, Entendre reports 48.3% for Jev, 46.3% for GPT-5.6 Luna, and 46.9% for DeepSeek V4 Flash. With two candidates, Jev’s 51.7% matches the recorded Claude Opus 5 rate in the post, above GPT-5.6 Sol at 51.4% and Claude Sonnet 5 at 49.4%.
Third post in the thread. Entendre says the language-model figures are account correctness when those models wrote full journal entries from the complete chart, while Jev received a shortlist that already contained the correct account. The paper’s Pass@3 table is a different metric.
Kareem Khattab, Omar Khattab and Mohamed Ibrahem
Published:
Source checked:
[114] Crypto Accounting Bench paper, arXiv:2609.14811
The paper evaluates full journal-entry reconstruction on 118 tasks from 7 organizations. Pass@3, a perfect rubric and every required gate on one of three attempts, is 47.46% for GPT-5.6 Luna, 50.00% for GPT-5.6 Sol, 45.76% for Claude Opus 5, 47.46% for Claude Sonnet 5, and 46.61% for deepseek-v4-flash-0731.
arXiv:2609.14811v1, submitted 13 September 2026. Mean Score is the paper’s primary metric. Pass@3 is the third column of Table 2 and matches Figure 3. The median chart is 569 accounts. The paper does not evaluate Jev. Entendre’s later shortlist percentages are not these Pass@3 figures.
Entendre Finance
Source checked:
[115] Public Crypto Accounting Bench dataset, 118 full-entry tasks
The dataset card describes 118 tasks whose target is the complete journal entry an organization posted, chosen from that organization’s chart of accounts. It repeats the paper’s best Mean Score of 77.43% and best Pass@3 of 56.78%. It does not include Entendre’s later Jev shortlist run.
Read 30 September 2026. The card says the public files are a transformed derivative and that the judge pipeline behind the paper’s headline numbers is not released. The deterministic scorer in the repository is described as a lower bound. No dataset revision hash is cited here.
Lucas Biagetti (@LucasBiagettiA)
Published:
Source checked:
[116] Banking77 text-classification benchmark updated with Jev
Lucas Biagetti says he updated his text-classification benchmark, which he now frames as a decision-model comparison, to include Jev and to fill some gaps. He writes that Jev wins zero-shot classification in his exploratory analysis.
Post on 1 October 2026. It does not give the table, the class counts, or the OpenAI model identifier. The reply in the thread attaches a chart. The numbers and the model pins are in the repository brief and manifest at commit efbfb42.
Lucas Biagetti
Published:
Source checked:
[117] Banking77 v1.2 brief, 56 quality conditions
The v1.2 brief reports accuracy for 14 method and budget variants at 5, 10, 15, and 20 Banking77 classes. Zero-shot Jev scores 94.00%, 92.25%, 89.83%, and 86.12%. The OpenAI zero-shot row scores 92.00%, 88.25%, 87.33%, and 85.00%.
Checked at commit efbfb42. The brief says each width uses 40 held-out examples per class, one seed, and that the cohorts overlap, so the widths are not independent samples. It reports 1, 2, and 3 missing Jev probability vectors at 10, 15, and 20 classes. It makes no cross-provider cost or latency claim. MiniLM plus logistic regression at 100 examples per class scores 98.00%, 96.50%, 95.83%, and 92.38%. The model identifiers are in the manifest, not in this brief.
Lucas Biagetti
Published:
Source checked:
[118] Banking77 v1.2 manifest: jev-1.13.0 and gpt-5-nano-2025-08-07
The manifest pins Jev to jev-1.13.0 and the OpenAI zero-shot row to gpt-5-nano-2025-08-07 with reasoning effort minimal. Both use zero labeled examples. The campaign seed is 42, the class counts are 5, 10, 15, and 20, and the test set has 40 examples per class.
Same commit as the v1.2 brief. The MiniLM row uses sentence-transformers/all-MiniLM-L6-v2 at revision 1110a243, frozen, with logistic regression. BERT is google-bert/bert-base-uncased for two epochs. The manifest does not store latency or a billed cost for the zero-shot rows.
Tobias Deußer, Lorenz Sparrenberg and Rafet Sifa
Published:
Source checked:
[123] 37-dataset Jev evaluation, arXiv:2609.37647
The paper evaluates jev-1.13.0 zero-shot on 37 datasets, 346,009 requests. Against Qwen3.8-27B and Gemma-4-E4B scored on the same requests, Jev is ahead on 27 datasets, tied on one, and behind on 9, and ahead of Gemma on all 37. The evaluation split cost US$9.15.
arXiv:2609.37647v1, submitted 29 September 2026. Banking77 accuracy is 79.7%, 77.4%, and 67.0% on 3,076 examples. MMLU is 91.8%, 81.7%, and 67.4% on 14,042. Eleven of Jev’s leads over Qwen have non-overlapping bootstrap intervals, and none of Qwen’s nine leads do. Mean latency is 0.36 seconds at 32 concurrent requests. The whole study, including the pilot and extra analyses, is US$9.40. The reference models are not GPT, Claude, Gemini, Grok, Kimi, or DeepSeek. The harness and response archive are in the repository.
Tobias Deußer
Published:
Source checked:
[124] jev-benchmarking harness at commit 6bbdeb3
The README pins the evaluation to jev-1.13.0 and compares it with Qwen3.8-27B and Gemma-4-E4B on identical requests. It links the paper and the raw responses at doi.org/10.5281/zenodo.23039006. Offline scoring rebuilds each request and looks up the cached response.
Commit 6bbdeb3, 30 September 2026. The README says a dataset that changes on Hugging Face would no longer match the cache. LLM-AggreFact is gated. The open-model path needs a GPU. The response license allows research and evaluation use of the Jev responses and bars distillation and training a model to imitate Jev.
Koray Kavukcuoglu
Published:
Source checked:
[125] Gemini 4 Argon announcement
Google announces Gemini 4 Argon on 30 September 2026 and says it is rolling out to trusted cyber defenders through the Fairwind Program. The introductory price is $2 per million input tokens and $10 per million output tokens, cached input is 95% off the input price, and the later price is $4 and $20. The output limit is 1 million tokens, up from 64,000. Reported scores are 77.9% on DeepSWE v1.1, 51.3% on AutomationBench, 91.7% on LVBench, and 68% tied for first on CWE-bench v1 remediation.
The announcement does not mention Jev, an API model id, or a date for paid API or Google AI Ultra access. It says Argon leads the Vals Index and does not print that score. The DeepSWE, AutomationBench, LVBench, and CWE-bench figures are Google’s. The libgav1 note says agents replaced 32,000 lines of SIMD and the decoder ran 2.7 times faster than the earlier Rust port, with the same video output. A healthcare vulnerability found with Wiz is described without a package name or CVE. Jev is listed here because the note is cited from the Jev and Gemini report.
Arena.ai (@arena)
Published:
Source checked:
[127] Agent Arena thread on Jev Router
Arena evaluates a product it calls Jev Router by @typesafeai on more than 4,700 Agent Arena sessions. It says the router does not improve on the Pareto frontier: similar performance to DeepSeek V4.1 Flash (Max), 38% more cost, and 1.7 times the median model-request latency. A later post in the thread gives end-to-end medians of 6.18 seconds and 3.64 seconds.
Thread on 7 October 2026. The opening post names DeepSeek V4.1 Flash as the most picked model, with GPT-6.1 Sol and GPT-6 Luna also frequent, and steerability of +10% against +10.48 for Claude Opus 5.5 (High). The next posts give call shares of 28.1% DeepSeek V4.1 Flash, 23.5% GPT-6 Astra, 22.8% Sol and 11.1% Luna, a hypothetical leaderboard row of +3.6% net improvement at $0.155 per task, and P90 latency of 30.59 seconds against 14.26. The thread prints no API model id, no code, and no session export. It does not say the calls used OpenRouter’s typesafe/jev-router. Theo’s DeepSWE note and Christoph Görn’s 16-prompt log stay separate.
Josh Longenecker
Published:
Source checked:
[129] CascadeEvaluator on 200 tool-or-reply decisions
The write-up measures CascadeEvaluator on 200 conversations from GeneralFunctionCall-Test. At a confidence of 0.7, 73 of 200 decisions go to Claude Sonnet 4.6. Mean time is 1.28 seconds against 2.59 for Sonnet alone, and the listed cost per million decisions is $2,997 against $7,751. Agreement with the reference is 0.695 against 0.675.
The page has no separate byline. The X post links it, and commit 523194a on grounded-ai/grounded_ai is by jlonge4 on 8 October 2026. The judge string is bedrock/us.anthropic.claude-sonnet-4-6. Jev is called as jev/jev-latest. Labels record what Kimi K2-thinking did next. List prices in the caveats are Jev at $0.042 per million input tokens, output free, and Sonnet 4.6 at $3 and $15. At 0.9, 112 of 200 decisions go to Sonnet, mean time is 1.84 seconds, cost is $4,572 per million, and agreement is 0.685. The write-up calls 0.9 the default threshold.
Josh Longenecker
Published:
Source checked:
[130] tool-call bench.json at commit 523194a
The JSON stores the 200-item run: Sonnet alone at 0.675 agreement, 2.591 seconds mean, 2.355 seconds median and $7,750.875 per million decisions. The 0.7 cascade stores sent_to_sonnet 0.365, agreement 0.695, mean 1.276 seconds and $2,996.74. The 0.9 cascade stores 0.56, 0.685, 1.838 seconds and $4,571.86.
Commit 523194a, 8 October 2026, message "Tool-call benchmark: CascadeEvaluator vs Sonnet 4.6 alone". The file names jev as jev/jev-latest and the judge as bedrock/us.anthropic.claude-sonnet-4-6. The write-up rounds these fields to 2.59 seconds, 2.35 seconds, $7,751, 1.28 seconds, $2,997, 1.84 seconds and $4,572. sent_to_sonnet of 0.365 and 0.56 is 73 and 112 of 200.
Josh Longenecker (@jldevtech)
Published:
Source checked:
[128] Jev in front of Claude Sonnet 4.6 on tool-call checks
The post says a cascade put TypeSafe’s Jev in front of Claude Sonnet 4.6, so Sonnet sees only the cases Jev is unsure about, and reports 51% faster, 61% cheaper, and the same accuracy.
Post on 8 October 2026 linking the Grounded AI write-up. The shorter wording says the accuracy was the same. The write-up and bench.json give 0.695, 0.685 and 0.675, and say the labels are Kimi K2-thinking’s next action.
David Chiu
Published:
Source checked:
[131] 100-turn routing bench at commit ed4a0ab
The README scores a 100-turn enterprise-agent routing bench. All four decisions are correct on 77.0% of turns for a self-hosted Gemma 4 31B JSON call and 61.4% for hosted Jev 1.13.0. Route is 90.0% and 90.6%. Restriction is 95.0% and 84.4%. Jev’s p50 is 749 ms and its p95 is 818 ms.
Commit ed4a0ab, 4 October 2026, on github.com/ywchiu/jev_benchmark. The account name on that commit is ywchiu and the author name is David Chiu. The bench is 20 conversations of 5 turns and does not retrieve documents or run tools. The original five systems were run 5 times. Of the October rows, Clef-flash was run 5 times, Clef twice, and each Cygnet row once. Clef and Clef-flash repeated the same answers. The 31B server also served live traffic, so that latency is only indicative. Clef was the open-weight systemone() path. Cloudflare’s hosted API quota ran out partway through. Cygnet reads gemma-4-12B-it one token per question. The same readout on the 31B checkpoint scores 69.0%. Ask-first accuracy is 43% for Gemma and Clef, 31% for Jev, and 0% for Cygnet on 12B. Laya scores 0.0% and SemIf 24.0% on the all-four metric. This Gemma 4 31B row is not Gemma-4-E4B in arXiv:2609.37647, and it is not a Gemini API model.
Connor Frank
Published:
Source checked:
[133] Independent evaluation of Jev on claim verification and LegalBench
Vals scores twelve systems on 400 claim-verification items built from 10-Ks and on a 396-question LegalBench slice. Jev 1.13.0 scores 0.975 on the claims, 390 of 400, tying GPT-6 Astra, GPT-6.1 Sol and GPT-6 Luna at $0.02 per 1,000 cases. On LegalBench it ranks last, with class-balanced accuracy 0.730 against 0.732 for Claude Haiku 4.5 and 0.932 for GPT-5.6 Terra.
Write-up dated 6 October 2026. Gemini 3.8 Flash wrote 160 paraphrases and contradictions; the labels come from the construction rule, and a person audited every item. LegalBench is 12 subtasks with 33 questions each, sampled from 1,808 available items, and the page says that slice is not representative of the full benchmark. At 32 judgments about one document, Jev’s median latency is 194 times lower than GPT-6 Astra’s and 241 times lower than Gemini 4 Argon’s. A 1% error budget, fit on the first 30% of claim items, automates 95% of the held-out 70% at 1.6% error. Native structured output, no JSON prompting. Intervals bootstrap whole filings and Bonferroni-correct eleven comparisons against Jev.
Vals AI (@ValsAI)
Published:
Source checked:
[132] Vals thread on a 400-item Jev claim check
The opening post says Vals tested TypeSafe’s Jev against 11 other models and matched GPT-6 Astra’s 97.5% accuracy on 400 claim-verification questions at about 1/500th the cost. A follow-up gives about $0.025 per 1,000 cases against $12.29 for Astra, and 194 times lower median latency at 32 judgments about one document.
Thread on 6 October 2026 linking the write-up. A later post in the thread says Jev ranked last on a 12-task LegalBench slice, and that a 1% error-budget threshold automated 95% of held-out claim-verification cases at 1.6% error. The write-up’s probability-priced table lists Jev at $0.02 per 1,000 cases and Astra at $18.00. The $12.29 figure is the thread’s Astra cost without that extra probability charge.
Steven Denney and Matthew DiGiuseppe
Published:
Source checked:
[134] Seven political-science replications, arXiv:2610.06625
The paper re-runs seven published political-science annotation and scaling tasks with Jev 1.13 against GPT-6 Luna and Qwen3.8-27B. Jev matches or comes close on the human-coded scores. Luna at OpenAI batch prices costs less on six of the seven primary tasks. One-at-a-time medians are 0.09 to 0.27 seconds for Jev and 0.75 to 1.04 seconds for Luna.
arXiv:2610.06625v2, 6 October 2026, 71 pages. v2 corrects author order in the metadata. Jev ran through OpenRouter except on the V-Dem recall task, which used TypeSafe’s endpoint. Qwen ran through OpenRouter on Parasail at FP8. Speed rows are 200 timed items per application, sent one at a time from the Netherlands on 29 and 30 September 2026. On Gilardi’s 2021 relevance tweets Jev scores 89.0% against Luna’s 91.1%; on the 2023 sample, 76.2% against 84.2%. Mean V-Dem correlation is 0.68 for Jev and 0.63 for Luna. On party-name comparisons, 74% of Luna’s probabilities sit below 0.01 or above 0.99, against 18% for Jev. The comments list no public code archive.
Premsanth Rajamani
Published:
Source checked:
[135] Jev against Gemini 3.8 Flash on 203 sales threads
Salesgear scores jev-1.13.0 and Gemini 3.8 Flash on 203 hand-written sales threads. Category accuracy is 193 of 203 for Jev and 199 of 203 for Gemini. Both find 60 of 61 real deferrals and put the same wrong thread in their top ten. Jev’s median is 0.35 seconds at $0.45 per 10,000 threads; Gemini’s is 2.2 seconds at $12.34.
Page dated 30 September 2026. The threads are synthetic, written one by one for a fictional seller, with labels fixed before either model saw them. Claude helped draft, so Gemini is the baseline. Gemini ran in JSON classification mode with low thinking. Warmth and date arithmetic run in code. The page links github.com/developer943/pipeline-revival-jev; the public repository is developer943/sales-pipeline-revival-jev.
Salesgear
Published:
Source checked:
[136] sales-pipeline-revival-jev at commit 926e72e
Commit 926e72e stores the 203-thread dataset, the six typed questions, the Gemini baseline, and results.json. That file records Jev accuracy 0.9507, Gemini 0.9803, medians 0.346 seconds and 2.188 seconds, and run costs $0.00906 and $0.25055. Jev ran with 8 workers; Gemini ran with 2.
Commit message dated 30 September 2026, “Jev vs Gemini on 203 dead sales deals: app, dataset, results”. results.json was generated 2026-09-30 16:36. Jev is jev-1.13.0. Gemini is gemini-3.8-flash. Deferral recall is 0.9836 for both. Come-back window accuracy on real deferrals is 0.9180 for Jev and 0.7705 for Gemini. ECE is 0.0199 and 0.0230. The README’s 8-minute and 48-minute 10,000-thread estimates assume eight workers on both sides; the Gemini wall time in the JSON uses two workers.
TypeSafe AI
Published:
Source checked:
[137] Jack & Jill candidate screen, TypeSafe case study
TypeSafe writes that Jack & Jill replaced Gemini 3.1 Flash Lite on the initial scoring stage of a candidate-matching pipeline. An offline test on 150 live roles kept 94.6% of hiring-manager winners on Jev’s shortlist against 93.9% for Gemini. Cost per 1,000 candidates is $0.092 against $0.755. Median time to screen all candidates for a role is 10.3 seconds against 20.3.
Case study dated 12 October 2026. The quality test used every past winner plus a sample of the remaining candidates, weighted to represent the original 6,000 retrieved for each role. The 0.7 point gap has a 90% confidence interval of −0.4 to +1.8, which TypeSafe calls not statistically significant. AUC is 0.933 against 0.924. List prices in the table are Gemini $0.25 input and $1.50 output per million tokens, and Jev $0.042 input with output free. $265,000 is current-volume savings; $500,000 is a projection that includes expected growth. Extra workflows, including a 3.6 second to 0.2 second profile check, sit outside those figures. The page names no Jev version and links no public log.
Artificial Analysis
Published:
Source checked:
[139] Artificial Analysis Decision Index v1.0 launch article
The 13 October 2026 launch article scores Jev 1.13 at 58 on Decision Index v1.0, ahead of GLiDE and Microsoft-Decision-1 at 54. Claude Haiku 5.5 (Low) scores 61 at 1.4 seconds per task and $184 per million tasks. GPT-6 Luna (Low) scores 55 at 1.5 seconds and $131. Jev’s median is 0.13 seconds and $42 per million tasks.
The index is an equal mix of AA-Omniscience-Decision (500 private five-option questions, accuracy and non-hallucination at 12.5% each), HLE-MultiChoice (513 text-only multiple-choice items), When2Call (3,423 NVIDIA tool-calling tasks after dropping 229 ambiguous ask-for-information items), and WildGuardTest (835 agreed rows, average of three F1 scores). Language models answer with structured JSON that names one option and supplies no probability. Jev scores 21 on HLE-MultiChoice, 35 accuracy and 35 non-hallucination on AA-Omniscience-Decision, 81 on When2Call, and 94 on WildGuardTest. Haiku 5.5 (Low) scores 24, 52, 73 and 93 on those four. Luna (Low) scores 22, 66, 77 and 89. The article says these are launch figures; the live leaderboard can change.
Artificial Analysis (@ArtificialAnlys)
Published:
Source checked:
[138] Artificial Analysis Decision Index launch thread
The opening post says Jev leads among decision models on Decision Index v1.0, and that language models such as Claude Haiku 5.5 still score higher at much higher cost and latency. The key-results post gives Jev 1.13 a score of 58, ahead of GLiDE and Microsoft-Decision-1 at 54, with Haiku 5.5 (Low) at 61, 1.4 seconds per task and $184 per million tasks against 0.13 seconds and $42 for Jev.
Thread on 13 October 2026 linking the launch article, methodology, and live leaderboard. Follow-up posts repeat the Pareto cost figures: Decider v1.1 27B at $17 per million tasks, Microsoft-Decision-1 at $31, and Jev 1.13 at $42. Celeris-1 Decision is 0.06 seconds per task. The thread says decision models are poorly calibrated on knowledge and reasoning, and relatively well calibrated on WildGuardTest and When2Call. Live scores can move after this launch snapshot.
Artificial Analysis
Published:
Source checked:
[140] Artificial Analysis Decision Index methodology
Version 1.0 weights four evaluations at 25% each. A decision model answers every question in one call. For a choice, Artificial Analysis scores the option with the highest probability. For a yes/no question it takes yes at probability 0.5 or more. Language models receive a fixed system prompt and must return JSON through structured outputs.
AA-Omniscience-Decision non-hallucination is 100 × (1 − hallucination rate), where hallucination is a wrong answer instead of “I don’t know”. HLE-MultiChoice uses the 513 text-only multiple-choice questions from cais/hle, scored as an exact letter match. When2Call uses NVIDIA’s multiple-choice test set after filtering 229 tasks. WildGuardTest keeps rows where all three annotators agreed on prompt harm, response harm and refusal, and drops questions that consistently trigger model refusals. Cost per million tasks is one pass through the index, divided by the combined task count, times one million.
driller
Published:
Source checked:
[142] Claude Max API credits used like Jev on three Japanese sets
The article scores jev-latest against Claude Haiku 5.5, Sonnet 5.5 and Opus 5.5 on MASSIVE ja-JP scenario labels (216 items), chABSA sentiment (210) and JGLUE JNLI v1.3 (210). After rewriting the chABSA and JNLI option text to match the annotation guides, Jev accuracy is 0.884, 0.910 and 0.838. Opus probabilities mode is 0.926, 0.881 and 0.914. Jev’s median latency is 0.16 to 0.17 seconds against 1.3 to 3.5 seconds for the Claude rows.
Published 10 October 2026, edited 11 October. Claude calls go through typesafe-ai/system-one-adapter-python with structured outputs. Haiku ran in discrete and probabilities modes; Sonnet and Opus ran in probabilities mode. Effort is each model’s default. Each class has the same count, seed 42, one run per setup. McNemar finds Opus ahead of Jev on JNLI (4 Jev-only correct, 20 Opus-only, p=0.002). The MASSIVE Opus gap is p=0.035 and the article says it is not significant after comparing ten pairs on one set. Cost per 1,000 items is $0.02 to $0.04 for Jev. The Claude API bill for the check is $14.41; Jev is about $0.03 on a separate bill. A cascade that sends low-confidence Jev items to Sonnet or Opus is a replay of these labels, and the author calls those accuracies optimistic.
driller (@patraqushe)
Published:
Source checked:
[141] Zenn post on Claude Max credits used like Jev
The post links a Zenn article that tries Claude Max plan API credits as a Jev-style classifier and compares jev-latest with Haiku 5.5, Sonnet 5.5 and Opus 5.5 on three Japanese tasks.
Posted 10 October 2026, the same day the article went up. The article was edited on 11 October. The post itself carries no scores.
Yuchen Ling, Shengcheng Yu, Zhenyu Chen and Chunrong Fang
Published:
Source checked:
[143] Jev action selection in ALFWorld and AndroidWorld, arXiv:2610.13092
The paper swaps gpt-5.6-luna action selection for jev-1.13.0 across five agent families, 4,500 executions, ALFWorld’s 134 valid_unseen tasks and AndroidWorld’s 116 task types, three runs each. ReAct-J succeeds on 96.02% of ALFWorld tasks, 1.49 points under the original agent. Reflexion-J succeeds on 98.01%, 1.74 points under. On AndroidWorld, Jev trails T3A by 13.22 points, AutoDroid by 2.30 and FastAgent by 18.10.
arXiv:2610.13092v1, 9 October 2026. All LLM decisions and auxiliary generation use gpt-5.6-luna. Jev accepts text-only observations. ALFWorld also has an LLM-Choice control that uses Luna with Jev’s candidate list; those gaps shrink to 1.00 and 0.00 points. Decision cost falls 66.1% to 78.4% and decision time 73.9% to 81.5%, from 4.13 to 8.13 seconds to 1.04 to 1.75 seconds. Mean task cost falls in every family. T3A’s cost per delivered success rises 7.7%, from 68.70 to 73.97 mUSD, with a 95% interval that includes zero. FastAgent’s time per success rises from 239.4 to 377.8 seconds. Experimental code is at anonymous.4open.science/r/jev-emp-5BF4/.
Jim Bennett
Published:
Source checked:
[145] Arize decision-model benchmark: Jev, OpenAI Decisions and six others
On 2,675 RAGTruth rows, Jev 1.13 scores 85.5% balanced accuracy at a tuned cutoff, against 81.5% for OpenAI Decisions on gpt-6-luna. Median latency is 0.08 seconds against 0.16, and cost per 1,000 is $0.045 against $0.090. The 95% intervals on accuracy touch, so Arize records a draw there and sends Jev through on speed and cost. On 655 Phoenix cases Jev scores 91.1% and OpenAI 90.0%. On 52 routing cases Jev is 49 of 52 and OpenAI 48 of 52.
Blog dated October 2026, posted on X on 7 October. OpenAI is the only model that does not accept Jev’s /v1/systemone layout; Arize wrote each case as labelled text sections and says a different layout could score differently. Accuracy wins require non-overlapping 95% intervals. Each model gets its own yes/no cutoff, fit on half the data and scored on the other half, same seed as Arize’s earlier Jev-as-a-judge work. An earlier Arize Opus 5 judge scored 86.6% balanced accuracy at $14.30 per 1,000. Reversing the hallucination question drops Liquid d1 from 0.92 ROC AUC to 0.17; Jev, Clef 27B and OpenAI stay near 0.93, 0.92 and 0.90 on both wordings.
Arize AI (@arizeai)
Published:
Source checked:
[144] Arize thread on eight decision models
The opening post says Arize benchmarked eight decision models on hallucination detection, 13 Phoenix evaluator suites, and agent routing. It writes that Jev, Clef 27B and Liquid sit within a point of each other on hallucination, and within the noise of an LLM judge that costs 60 to 400 times more.
Thread on 7 October 2026 linking the blog. A follow-up names Jev for a hosted API, Kev for self-hosting, and Clef 27B for open weights with a hosted option, and says to test the question as hard as the model. The thread itself does not print the RAGTruth percentages.
Takuma Yoshida
Published:
Source checked:
[208] Clairic and Jev on 100 return emails: four experiments
Four layouts sort 100 fictional Japanese return emails for a kitchenware shop. Claude Haiku 5.5 scores 91 of 100 in one call, at 3.0 seconds and ¥8.1 per 100. The six-step Jev layout without Clairic, experiment 4, scores 78 at 0.3 seconds and ¥1.5, using jev-1.14.0. In that layout Jev’s final choice matched a Python encoding of the nine rules on all 100 cases, so all 22 misses came from reading the email.
English write-up dated 26 October 2026. The emails and order records are fictional, written by Claude after the correct answer was fixed. The author is the one person in the person row. Text sent to Jev and Clairic is Japanese. Clairic is Cloudflare’s preview extraction model @cf/cloudflare/clairic as of 24 October 2026. Confidence thresholds such as 0.8 were drawn after seeing the 100 results. Layouts were chosen after seeing other layouts on the same 100 cases. Haiku’s cost is a list-price estimate inside a subscription. Yen figures use 150 yen to the dollar. UnlockJapan states it has no partnership with TypeSafe or Cloudflare.
Takuma Yoshida (@TakumaYoshidaUJ)
Published:
Source checked:
[207] 100 made-up refund emails: Clairic, Jev, Haiku
Takuma Yoshida reports 100 made-up refund emails scored with Clairic, Jev, and Claude Haiku. Jev plus Python is 78% right at ¥1.5 per 100. Clairic plus Jev is 68 to 77% at ¥4.9 to 8.8, which he says could be his setup. Haiku is 91% at ¥8.1. Keeping only the cases Jev is sure of, then doing the rest himself, moves 78% to 83%.
X post, 26 October 2026, linking the English write-up. The 83% line is the experiment-4 split that leaves high-confidence Jev answers in place and sends the rest to the author. Thresholds on that page were drawn after seeing the 100 results.
John Siciliano and Knut Melvær
Published:
Source checked:
[200] Sanity’s 0.5-second docs agent with Jev, Haiku and gpt-oss-120b
The write-up times four decisions that used to sit on an LLM. Entry choice is 282 ms for Jev against 1,328 ms for Haiku. Model routing is about 300 ms against 818 ms. MCP choice is 170 ms against 829 ms. Speculation while the user types removes 1,828 ms at enter. The median full answer is 0.5 seconds. Three of four finish under 0.9 seconds.
Engineering post dated 24 September 2026. The answering model in the demo is openai/gpt-oss-120b on Cerebras. The authors score 20 of 22 answers as fully grounded in the docs, with the first token in about 350 ms. Routing made answers cheaper, about 4.5 times, and slower at the median, so they turned routing off for the 0.5-second demo. Combined entry, model, and MCP questions took about 340 ms. The post calls the numbers indicative: one week of testing on one corpus. Jev is called as jev-latest in the sample request, and the sample response logs jev-1.13.0.
Wang Yang, Xinpeng Li, Yanyan Zhang, Chuang Ma, Shouren Wang, Jianzhe Wei, Liqin Ye, Xiaoran Cai, Mohsen Hariri, Vipin Chaudhary and Xiaotian Han
Published:
Source checked:
[210] What one-pass decision models cannot decide, arXiv:2610.24929
The paper evaluates jev-1.13.0 and 23 open Jev-style configurations. Retrieval and selection are perfect up to 20k tokens and 64 options. Final-state accuracy after sixteen updates of one user falls to 53 to 80%, because similar records of other users interfere. On 738 real items graded by how many thinking tokens Qwen3.8-27B needs, Jev answers 97.5% of the items that one forward pass of that model solves and 74% of the others.
arXiv:2610.24929v1, 18 October 2026. A stale one-line summary is followed even when the log contradicts it. Confidence flags Jev’s own errors (AUROC 0.83 on final-state items) and not errors inherited from a stale input (0.53). An offered hand-off option is taken on 1.6% of the real-task items and on 3 of Jev’s 40 errors. The 738-item set drops HelpSteer2 and 93 items whose thinking samples disagree with the gold label. Qwen3.8-27B is the reasoning reference, not a GPT API model. The paper does not score jev-1.14.0.
Eduardo Vellasques, Lionel Palacin and Al Brown
Published:
Source checked:
[225] Agentic Analytics Benchmark: ranking tables and labeling failures with Jev
On a 136-question development set over an 18-table warehouse, Jev ranks needed tables at mean nDCG@10 of 0.951, recall@10 of 0.991, and about 0.4 seconds per question, with every required table inside the top five for 84% of questions. A four-model development run of 544 attempts yields 277 labeled failures; wrong data selection is the largest bucket at 87 of 277. At a 0.65 threshold on “correct despite fail,” 18 answers look like grader errors. Feeding the ranking back as an in-loop hint leaves accuracy flat across 20 models: pooled net +12 on fired cells, 95% interval −18 to +42.
ClickHouse engineering blog dated 27 October 2026. A companion pilot on 250 public text-to-SQL cases with 75 to 710 tables reaches 87% recall@10 and finds every required table in 73% of cases, at a p90 of 3.4 seconds and about $0.50 for the run. The post does not name a Jev version. Code ships in ClickHouse/data-agent-mnist.
ClickHouse (@ClickHouseDB)
Published:
Source checked:
[224] ClickHouse thread on Jev table ranking
ClickHouse writes that it used TypeSafe’s Jev to label every failure on the Agentic Analytics Benchmark and to rank the tables each question needs, reporting 0.951 nDCG@10 in 0.4 seconds.
X post, 27 October 2026, linking the engineering blog. Detailed failure counts, the wide-catalog pilot, and the flat in-loop hint result sit in the article.
ClickHouse
Source checked:
[226] data-agent-mnist: optional Jev ranking and failure labeling
The harness README’s optional Jev section installs typesafe-sdk and points schema_retrieval.py, schema_retrieval_agent.py, and label_failure_modes.py at a TypeSafe key. Ranking scores each table with a yes/no judgment. The labeler reads a graded run and returns a failure sub-mode, a preventing rule, and a grader-error check. Apache 2.0.
Repository opened at commit 403f3e7 on 2 November 2026. The README says the tools are optional extras and name no fixed warehouse tables.