Anthropic · Community reports
Jev + Claude: 100 return emails, Decision Index and Japanese 5.5
Takuma Yoshida scores jev-1.14.0 at 78 of 100 fictional return emails against 91 for Claude Haiku 5.5. Haiku 5.5 (Low) scores 61 on Artificial Analysis’s Decision Index, 3 points above Jev 1.13.
Compiled by jevmodel.ai. We read the cited posts and documentation. The measurements stay with their authors. We have not rerun them.
Jev Claude Code compaction
tamara’s fast-jev-compaction project applies Jev to Claude Code’s conversation history. It asks whether a tool call and its result still matter, then keeps, truncates or removes that material. User and assistant text remains verbatim in the resulting conversation.
The implementation pairs calls with results and protects recent messages. The state sent to Jev omits full tool-result bodies and can shorten other material to fit its budget. The transcript Jev sees can be shorter than the conversation the user had.
Sources:[6] tamara[8] tamaratran
What a Claude Code user reported
Alex Volkov reports that the plugin reduced a session from nearly one million tokens to about 86,000 in roughly one second. His post links to the project and describes an enthusiastic first-hand experience.
Those numbers belong to his session. The post gives no typical reduction ratio, no billing saving, and no score for later coding tasks. The repository also describes its separate native demo app as scripted and making no API calls, so the animation has no live API timing.
Sources:[7] Alex Volkov[8] tamaratran
Claude Code hook settings and fallback
The checked repository documents an early-access Claude Code function-hook integration, requiring Claude Code 2.1.274 or later and an opt-in setting. Installation and environment configuration belong to that project’s README; check those instructions against the version you run.
The hook falls back to Claude Code’s built-in summary when Jev fails or the reduction is too small. Token counts in the library are estimates. The author warns that a probability can be high on a tool result that is unsafe to delete, even if the assistant can run the tool again.
Sources:[8] tamaratran
Jev picks the Claude model for each turn
Himanshu Kumar’s jev-model-router, at commit 8d754fa, is a local proxy in front of Claude Code. On each new message it asks Jev which of the account’s Haiku, Sonnet or Opus models should run that turn. Tool calls in the same turn stay on the chosen model. Choosing a model in /model, or naming one in the prompt, steps past the router. If the key is missing, Jev errors, or the call times out, the README says the turn continues within about three seconds: Opus on the first turn, otherwise the model already in use.
The same README says the path was checked on Claude Code 2.1.282, with 94 automated tests. The screenshots were captured with scripts/fake_jev.py, a local stand-in. Jev is described as seeing the new turn’s prompt, the current model, a rough context size, and the candidate list. The files, the tool output, and the earlier conversation stay out of that view. The project says it is not affiliated with Anthropic or TypeSafe.
Jev and Claude on one reranking shortlist
hev’s RESULTS.md records jev-1.13.0 and two Claude listwise runs on the same BM25 top-30 shortlist. On SciFact’s 300 test queries, 30 Jev probabilities in one call score nDCG@10 of 0.768. Claude Haiku 4.5 scores 0.723. Claude Opus 5 at low effort scores 0.756, after 12 of the 300 queries were refused and scored as zero. Substituting BM25 order on those 12, the file reports 0.778 for Opus. List-price cost for the SciFact runs is $0.18, $4.18 and $28.40.
Haiku and Opus are absent from the NFCorpus and FiQA tables. nDCG@10 ranks a shared shortlist. The benchmark page has the other systems, including Voyage rerank-3, which scores higher than this Jev call on FiQA.
Sources:[75] hev
Claude Sonnet 5 on the same 42-claim source check
Watters’s schema-only run gives Claude Sonnet 5 the same 85.7% as Jev and GPT-5.4 on all 42 claims, and 66.7% on the 18 real published sentences, where Jev scores 77.8%. Sonnet 5’s schema-run cost is $0.002228 per claim and the recorded time is 2.76 seconds, against Jev’s $0.000025 and 0.23 seconds.
Allowing a scratchpad first, Sonnet 5’s hard-set score falls to 44.4%, at $0.006439 and 6.23 seconds. The McNemar p-value on that 18-item change is 0.219. Watters says that p-value is too high to support a claim that reasoning made the model worse, and that the Sonnet 5 verdict call transcribes the scratchpad. The same limits apply here as on the GPT report: one run, labels from one publishing pipeline, OpenRouter latency, and Jev called through the jev-latest alias.
Lieflat’s metaphor check the same day includes Claude Opus 5 and Opus 4.8. On one 202-sentence article, Opus 5 in non-reasoning mode took 8.2 seconds, against 2.6 seconds for Jev. Of 10 sentences with explicit comparison markers, Opus 5 marked 5 and Jev marked none. Those marks are the author’s. The Gemini report has the timing for the other models in that test.
Jev and Claude Opus 5 on Banking77 intents
Kenny Rogers at OpenRouter ran the Banking77 test split, 3,080 customer-support utterances and 40 examples for each of 77 intents, through Jev 1.13 and Claude Opus 5 on 22 September 2026. Both models received the same one-line criteria, written from the label names without looking at the test set. Jev was a Choice over those 77 labels. Opus ran at temperature 0, with reasoning off and a strict JSON enum. Jev’s accuracy is 81.0% (79.6 to 82.3) and macro-F1 is 80.5%. Opus is 84.4% (83.1 to 85.6) and 83.6%. A paired bootstrap puts the accuracy gap at 3.3 points, with a 95% interval of 2.3 to 4.4. The models agree on 89.3% of utterances. Where they split, Opus is alone-right on 175 and Jev on 72.
From the same client, with 8 requests in flight, median round trip is 175 ms for Jev and 2,266 ms for Opus. The 95th percentiles are 270 ms and 3,004 ms, and the 99th are 353 ms and 3,835 ms. The runs billed $0.34 and $7.44, or $0.11 and $2.42 per 1,000. Opus’s system prompt was cached. The write-up says the uncached list rate would have been about $19 per 1,000. Mean input tokens are 2,605 against 3,750, of which 3,725 were cached, and mean output tokens are 826 against 17.
Rogers treats Jev’s confidence as a rank, and says it overestimated accuracy in the middle of the range. At or above 0.99, 58% of utterances, accuracy is 96.3%. Below 0.5, 3.5% of utterances, accuracy is 29.6%. Sending confidence under 0.90 to Opus leaves 75.9% of traffic on Jev, at 84.0% accuracy and $0.69 per 1,000. That table uses the same 3,080 examples as the accuracy measurement, so the write-up calls it an upper bound. Both models score 0 of 40 on get_physical_card. Excluding that class, the figures are 82.1% and 85.5%. Fine-tuned encoders trained on the 10,003 training examples sit in the low 90s. The run is one dataset, one afternoon, with Opus reasoning left off. Banking77 was published in 2020, and the write-up says this run cannot show whether Opus had seen it.
Sources:[87] Kenny Rogers
Jev Claude Code routing that only raises the tier
Engin Diri’s jev-router, at commit f909310, is a local proxy in front of Claude Code. It asks Jev when a person writes a new message. Tool steps, subagents, compaction and workflow requests stay on the session’s tier. Background calls go to Haiku 4.5. The default policy can raise that tier and does not lower it. The shipped config treats a session as fresh after 10 minutes without traffic, and again after the stored count of human turns shrinks. The code comment calls that shrink a compaction or a fork.
The Choice labels are mechanical, routine, complex and deep. In the Claude-only config they map to Haiku 4.5, Sonnet 5, and Opus 5.5 for both complex and deep. The Fable option adds Fable 5.1 for deep work and for the sensitive override. Accept bars in config/anthropic-only.json are 0.85, 0.6 and 0.3. Below the bar, the router takes the more capable of the top two answers, and the Fable config stops that step at Opus 5.5. With guards on, two Noul questions ride along. A probability of at least 0.7 that the request would change production systems, credentials, permissions, billing, shared infrastructure or unrestorable data sends the message to the top tier. A probability of at least 0.5 that the text tries to set the model, tier or effort keeps the result from falling below the reference tier.
The TypeSafe channel pins jev-1.13.0. The OpenRouter failover names typesafe/jev-1.13. The npm package is @ediri/jev-router. The README says the unscoped name jev-router is a different project. The 58 labeled prompts have not been used to tune the bars, and a live run of that set is priced at about $0.002. The offline suite has 176 tests against mocks. On 24 September 2026 a check with Jev mocked ran Claude Code 2.1.281. Haiku 4.5 and Sonnet 5 both returned 200, and the router made one Jev call per human message. Full Codex sessions were not in those live checks. The blog estimates one routing call at about $0.00003. This proxy is separate from Himanshu Kumar’s jev-model-router.
A Claude Code hook that called the OpenRouter alias
Christoph Görn’s UserPromptSubmit hook asks a sizer for tiny, everyday, large, hardest or contextual, and delegates when the chosen label’s probability is at least 0.6. The cloud backend calls OpenRouter’s typesafe/jev-router and asks for JSON. On the first 16 prompts the log counts 3 tiny, 1 everyday, 1 large, 4 left on the main session, and 7 errors. The 9 successful calls averaged 2.99 seconds, from 1.26 to 4.41, and sizing cost about $0.0032. The answered_by field names openai/gpt-6-luna five times, deepseek/deepseek-v4.1-flash three times, and moonshotai/kimi-k3 once. Görn writes that those calls were not TypeSafe’s Jev.
On 8 prompts from earlier sessions, two of each size, the same write-up scores local Kev-4B at 1 of 7 and the OpenRouter alias at 1 of 5. Latency in that table is 100 to 180 ms and 700 to 1400 ms. Mean confidence is 13% and 87%. The X post’s 10 to 15 times figure is that latency pair. The 2.99 second average belongs to the 16-prompt log. Görn says the router’s own log still has no successful Kev decision, and that the router code is in a private infra repository. The main Claude session still reads the prompt. A routed task is then handed to a helper agent.
Claude Sonnet 5 on the same judge contract
Yubo Li, Yidi Miao, Ramayya Krishnan and Rema Padman ask claude-sonnet-5 for a label and label probabilities, at low effort under a JSON schema, with no written rationale. Invalid outputs count as errors. Sonnet 5 scores 91.2% on the 400-pair RewardBench sample, 88.9% on JudgeBench’s 350 pairs, 85.8% on 240 HaluEval judgments, and 94.7% on 150 saved replies. Jev 1.13.0 scores 92.2%, 78.6%, 87.5% and 94.0% on the same four sets. Macro-F1 on the existing labels is 0.934 for Sonnet 5 and 0.923 for Jev.
Sonnet 5 returns a valid contract on 1,308 of 1,312 base items. The other four responses are HTTP 200 and break the contract. The provider-generation count and the transport count are both 0. Dropping those four, valid-only accuracy is 91.7% on RewardBench, 89.4% on JudgeBench and 85.8% on HaluEval. Primary accuracy keeps them in the denominator, so those two primary scores sit below the valid-only rates.
A frozen rule, with the threshold fit on 96 earlier pairs, accepts Jev when its maximum label probability is at least 0.6 and otherwise takes Sonnet 5. On 510 later preference pairs it accepts 90.8%, scores 88.0% against Sonnet 5 alone at 89.2%, and the fee ratio is 0.152 on reported usage and on the conservative reservation. The cascade minus Sonnet 5 is −1.2 points, interval −4.0 to 1.8. The paper prints latency medians for Jev, GPT-4.1 mini and GPT-6 on the 120-judgment panel. It does not print a Sonnet 5 median. The fee ratio is an offline simulation.
Sources:[102] Yubo Li, Yidi Miao, Ramayya Krishnan and Rema Padman
Claude Opus 5.5 on the 60-tick Bay Area window
Daniel Liao’s jev-tower log beside claude-opus-5-5, at commit 20dcbbb, replays the same 60 ticks of Bay Area ADS-B as the gpt-4o file. tower.mjs marks a pair as a hazard when the closest approach is inside 3 nautical miles and 1,000 feet within 120 seconds. A flag is a probability above 0.5. Opus 5.5 averages 12,318 ms, with 378 true positives, 174 false positives and 0 misses on the pairs that returned. Jev, called as jev-latest on that run, averages 119 ms, with 385 true positives, 335 false positives and 0 misses.
The log prints one fetch failure. Opus’s true positives plus misses are 378, against 385 for Jev, so the skipped tick includes hazards under the script’s rule. The post says Opus needed 12 seconds and missed none. Those words match this average and the pairs the log scored. The README lists one Jev latency of 103 ms, which matches the run beside gpt-4o-mini. This run’s Jev average is 119 ms. The resolved Jev version is not in the log. The README says the geometry rule is not a certified separation standard.
A two-account shortlist beside Claude Opus 5
Entendre’s 28 September 2026 thread scores Jev at 51.7% when it picks between two real ledger accounts on the 118-task Crypto Accounting Bench. The post sets that figure equal to a recorded Claude Opus 5 account rate, and lists GPT-5.6 Sol at 51.4% and Claude Sonnet 5 at 49.4% beside it. Sonnet 5’s number in that sentence is the post’s account rate. The judge-panel scores earlier on this page are a different contract.
Jev chose from a shortlist that already contained the correct account. The language-model rates are account correctness on full journal entries written from the complete chart. The CAB paper’s Pass@3 on those full entries is 45.76% for Claude Opus 5 and 47.46% for Claude Sonnet 5. Pass@3 counts a task when one of three attempts earns a perfect rubric and clears every required gate. The paper’s median chart size is 569 accounts. The public dataset does not include the Jev shortlist run. The Jev + GPT report has the two-account, three-account, and five-account Jev rates, including 48.3% on five accounts.
Sources:[111] Entendre[112] Entendre[113] Entendre[114] Kareem Khattab, Omar Khattab and Mohamed Ibrahem[115] Entendre Finance
maslul asks Jev for a difficulty tier
Ilia Tankelevich’s maslul 0.5.0, at commit de093d3, can ask Jev which of simple, medium, or hard a turn is. The classifier calls jev-latest with one Choice question. Confidence below 0.7, or a failed call, returns no tier, and the router continues with its own strategy. The example config sets simple and medium to anthropic:claude-haiku-4-5 and hard to anthropic:claude-sonnet-4-6. The default strategy is route_default and the default level is hard. That file configures only Anthropic, and it leaves the Jev extra off. A caller has to pass the hooks in. The README lists Gemini, Grok, and OpenAI as other providers. The wording check and the shipped example stay on the Claude config.
The post says about 0.3 seconds per decision. jev.py sets the HTTP timeout to 5 seconds and does not store that timing. The state sent to Jev is the last 6 text messages, cut at 20,000 characters, with media dropped. A follow-up says the wording of the level descriptions mattered more than the language. A comment in jev.py, dated 2026-10-01, records three requests: a moving plan, a savings split, and an insurance comparison. Abstract descriptions returned medium at 0.48 to 0.78, in English and in Hebrew, and hard never won. The concrete defaults returned hard at 0.97 to 0.99 on the same requests, and the easy requests stayed simple. Those concrete descriptions are now the defaults. The default hard description covers a multi-part plan, a comparison with trade-offs, financial, legal, or medical advice, and reasoning over many documents. decide() returns the raw pick. The 0.7 floor applies when the classifier is asked for a tier. The verifier asks a Noul and, on failure, accepts the reply by default.
Sources:[119] Ilia Tankelevich[120] Ilia Tankelevich[121] Ilia Tankelevich[122] Ilia Tankelevich
Jev keeps the tool-or-reply calls Sonnet 4.6 is unsure of
Josh Longenecker’s Grounded AI write-up measures CascadeEvaluator on 200 conversations from GeneralFunctionCall-Test: 100 where the reference next action was a tool call, and 100 where it was a reply. Jev is called as jev/jev-latest. The judge is bedrock/us.anthropic.claude-sonnet-4-6. At confidence 0.7, bench.json records sent_to_sonnet of 0.365, which is 73 of 200. Mean time is 1.276 seconds against 2.591 for Sonnet alone. Median time is 0.199 seconds against 2.355. Cost per million decisions is $2,996.74 against $7,750.875. Agreement with the reference is 0.695 against 0.675. The write-up rounds those fields to 1.28 seconds, 2.59 seconds, 0.20 seconds, 2.35 seconds, $2,997 and $7,751, and calls the result 51% faster and 61% cheaper.
At 0.9, which the write-up calls the default threshold, 112 of 200 decisions go to Sonnet. Mean time is 1.838 seconds, median 2.110 seconds, cost $4,571.86 per million, and agreement 0.685. The write-up rounds those to 1.84 seconds, 2.11 seconds and $4,572. The labels record what Kimi K2-thinking did next, not a human judgment. The write-up says both Jev and Sonnet sometimes disagree with that label for a reason it accepts, such as replying after a search came back empty. The X post says the accuracy was the same. The table shows 0.695 and 0.685 against 0.675. List prices in the caveats are Jev at $0.042 per million input tokens, with output free, and Sonnet 4.6 at $3 and $15. Commit 523194a stores the JSON. The resolved Jev version is not in the file.
Sources:[128] Josh Longenecker[129] Josh Longenecker[130] Josh Longenecker
Steerability next to Claude Opus 5.5 in Arena’s thread
Arena’s 7 October 2026 thread says steerability for the product it calls Jev Router is +10.0%, against +10.48 for Claude Opus 5.5 (High), a gap of 0.48 points. The opening post rounds the router figure to +10%. On the same comparison with DeepSeek V4.1 Flash (Max), the router is ahead on steerability, +10.0% against −0.2%, and on praise versus complaint, +6.3% against +2.1%. DeepSeek is ahead on confirmed success, +8.3% against +2.2%, bash recovery, +7.8% against −0.7%, and tool hallucination, +0.4% against +0.2%.
The thread says the router preferred OpenAI models over Anthropic models. It does not give an Anthropic call share. It prints no API model id. The more than 4,700 sessions, the $0.155 median cost, and the latency pair of 6.18 seconds and 3.64 seconds are in the DeepSeek report. This is a different measurement from Claude Sonnet 5 on the Carnegie Mellon judge contract.
Sources:[127] Arena.ai[102] Yubo Li, Yidi Miao, Ramayya Krishnan and Rema Padman
Vals: last of twelve on a LegalBench slice
The same Vals roster that ties Jev with GPT-6 Astra on claim verification ranks Jev last on a preregistered LegalBench slice. Connor Frank fixed 12 subtasks and 33 questions from each, 396 questions sampled from 1,808 available items, before calling any model. The page says that slice is not representative of the full benchmark, and that LegalBench is public and probably sits in the LLMs’ training data. On class-balanced accuracy, GPT-5.6 Terra leads at 0.932. Jev ranks last at 0.730, next to Claude Haiku 4.5 at 0.732. On raw accuracy, every other system’s lead remains statistically significant after correcting for eleven comparisons.
Jev struggled most on the two contract-entailment subtasks. On contract_nli_permissible_copy it answered Yes to all 33 items. The gold answers split 16 yes and 17 no, so a constant Yes scores 0.485. On contract_nli_confidentiality_of_agreement it answered yes on 30 of 33. Every LLM scored 0.91 to 1.00 on those items. Each subtask has only 33 items. Expected calibration error on LegalBench is 0.108, the highest of the twelve, against 0.011 on the claim set. Calls that timed out after 180 seconds were retried. After that, GPT-5.6 Terra was still missing 10 answers, Claude Opus 5.5 6, Sonnet 5.5 2, and GPT-5.4 mini 1; 16 of those 19 were on international_citizenship_questions. Systems are scored on the items they answered.
On the claim-verification 1% error budget, Claude Sonnet 5.5 automates 100% of the held-out items at 0.4% error. Claude Opus 5.5 automates 99% at 0.4%. Haiku 4.5 stays inside the budget by automating 22% at 0.0% error. Jev automates 95% at 1.6%. Sweeping the threshold over all 400 items would have let Jev automate 91% within the budget, against 100% for Opus 5.5, Sonnet 5.5, and Gemini 4 Argon. Probability-priced costs per 1,000 claim cases are $0.95 for Haiku 4.5, $4.95 for Sonnet 5.5, and $9.64 for Opus 5.5, against $0.02 for Jev. This LegalBench slice is a different task from OpenRouter’s Banking77 run and from Grounded AI’s 200 tool-or-reply items.
Sources:[133] Connor Frank[132] Vals AI[87] Kenny Rogers[130] Josh Longenecker
Decision Index v1.0 next to Claude Haiku 5.5
Artificial Analysis’s Decision Index v1.0, launched 13 October 2026, scores Claude Haiku 5.5 (Low) at 61 and Jev 1.13 at 58. Haiku takes 1.4 seconds per task against 0.13, and costs $184 per million tasks against $42. GPT-6 Luna (Low), on the GPT report, scores 55 at 1.5 seconds and $131. Among decision models, GLiDE and Microsoft-Decision-1 sit at 54. Decider v1.1 27B has the lowest listed cost at $17. Celeris-1 Decision is the fastest at 0.06 seconds.
The four evaluations are weighted equally. AA-Omniscience-Decision is 500 private five-option questions. HLE-MultiChoice is 513 text-only items. When2Call is 3,423 NVIDIA tool-calling tasks. WildGuardTest is 835 safety rows, scored as the average of three F1 figures. Language models answer with structured JSON and supply no probability for the other options. A decision model answers every question in one call.
Haiku 5.5 (Low) scores 52 accuracy on AA-Omniscience-Decision, 24 on HLE-MultiChoice, 73 on When2Call, and 93 on WildGuardTest. Jev scores 35, 21, 81 and 94. Jev’s non-hallucination rate on the knowledge set is 35, behind Microsoft-Decision-1 at 40. The article says language models lead on knowledge and reasoning, and that leading decision models sit close to them on classification and tool calling. These figures are the launch snapshot. The live leaderboard can change. Vals’s LegalBench slice, earlier on this page, is a different legal task.
Sources:[139] Artificial Analysis[138] Artificial Analysis[140] Artificial Analysis
Three Japanese sets against Haiku, Sonnet and Opus 5.5
driller’s Zenn article, published 10 October 2026 and edited the next day, asks whether Claude Max plan API credits can run a Jev-shaped classifier. The check uses typesafe-ai/system-one-adapter-python, so Claude returns the same Choice object as Jev. The three sets are MASSIVE 1.1 ja-JP scenario labels, 18 classes and 216 items; chABSA sentiment, 3 classes and 210 items; and JGLUE JNLI v1.3, 3 classes and 210 items. jev-latest is compared with Haiku 5.5 in discrete and probabilities modes, and with Sonnet 5.5 and Opus 5.5 in probabilities mode. Effort is each model’s default. Each class has the same count, seed 42, one run per setup.
After the option text on chABSA and JNLI was rewritten to match the annotation guides, Jev accuracy is 0.884, 0.910 and 0.838. Opus probabilities mode is 0.926, 0.881 and 0.914. Sonnet is 0.912, 0.886 and 0.876. Haiku discrete is 0.870, 0.900 and 0.867. The first chABSA pass, with a neutral description that did not match the guide, scored 0.824 for Jev and 0.810 for Opus. Rewriting that one description moved every model up 6 to 10 points. McNemar finds Opus ahead of Jev on JNLI, 4 Jev-only correct and 20 Opus-only, p=0.002. The MASSIVE Opus gap is p=0.035, and the article says it is not significant after comparing ten pairs on one set.
Jev’s expected calibration error is 0.049, 0.059 and 0.056 on the three sets. Median latency is 0.16 to 0.17 seconds against 1.3 to 3.5 seconds for the Claude rows. Cost per 1,000 items is $0.02 to $0.04 for Jev, about one eighth to one sixteenth of Haiku and a few hundredths of Sonnet or Opus. The Claude API bill for the check is $14.41; Jev is about $0.03 on a separate bill. A cascade that sends low-confidence Jev items to Sonnet or Opus is a replay of these labels, and the author calls those accuracies optimistic. Each set has about 210 items, so the article puts accuracy error near ±5 points.
Sources:[142] driller[141] driller
100 fictional return emails against Claude Haiku 5.5
Takuma Yoshida’s 26 October 2026 write-up sorts 100 made-up Japanese return emails for a fictional kitchenware shop, Midori Kitchen. Each case has an email, an order record, and one of six answers: refund, exchange, repair, ask for photos or information, decline, or pass to staff. Nine policy rules apply from the top. Claude Haiku 5.5, called as claude-haiku-5-5 at medium effort through the Claude Code CLI, scores 91 of 100 in one call, at 3.0 seconds per case and ¥8.1 per 100 on a list-price estimate. The person row is the author, 73 of 100, at 18.0 seconds.
Jev is jev-1.14.0. The six-step layout without Cloudflare’s Clairic extractor, experiment 4, scores 78 of 100 at 0.3 seconds and ¥1.5 per 100. Jev’s final Choice matched a Python encoding of the nine rules on all 100 cases, so every miss in that layout came from reading the email. Nineteen of the 22 misses read injury, a strong complaint, or circumstances the policy does not cover into an email that had none. Haiku got 15 of those 19 right. The same reading with the whole policy handed to Jev as prose, experiment 2, scored 48. Showing only the rules that apply is what moved 48 to 78.
Layouts that keep Clairic score 68 and 77. Clairic is @cf/cloudflare/clairic, a preview as of 24 October 2026. Yen figures use 150 yen to the dollar. The emails are fictional, written by Claude after the correct answer was fixed. Haiku wrote none of them. Confidence lines such as 0.8 were drawn after seeing the 100 results. On experiment 4, leaving answers at Jev confidence 0.8 or higher in place and sending the other 21 cases to the author scores 83, with 7 unseen misses. The author states UnlockJapan has no partnership with TypeSafe or Cloudflare. This Japanese reading task is a different setup from driller’s MASSIVE, chABSA and JNLI check.
Sanity times Jev against Haiku on a docs agent
John Siciliano and Knut Melvær’s 24 September 2026 docs-agent post times four decisions that used to sit on an LLM. Entry choice is 282 ms for Jev against 1,328 ms for Haiku. Model routing is about 300 ms against 818 ms. MCP choice is 170 ms against 829 ms. Speculation while the user types removes 1,828 ms at enter. Combined entry, model, and MCP questions in one Jev request took about 340 ms. The median full answer is 0.5 seconds, and three of four finish under 0.9 seconds.
The sample request sends model jev-latest. The sample response logs jev-1.13.0. Routing made answers cheaper, about 4.5 times, and slower at the median on that host, so they turned routing off for the 0.5-second demo. The post calls the numbers indicative: one week of testing on one corpus. The answering model is openai/gpt-oss-120b, on the GPT report. The later @s harness uses the same Jev-as-classifier idea for 66 internal agents, with a 0.4 confidence floor, and prints no Haiku timing.
Sources:[200] John Siciliano and Knut Melvær[199] Even Westvang, Rostislav Melkumyan and Knut Melvær[197] knut
Questions about Jev + Claude
Does fast-jev-compaction summarize Claude messages?
Its documented approach selects and prunes tool calls and results while retaining user and assistant text verbatim in the output. The separate state sent to Jev can be shortened to fit its budget.
Sources:[8] tamaratran
Is the reported one-second compaction a guarantee?
No. It is Alex Volkov’s report about one session. The sources give no typical latency and no downstream task score.
Sources:[7] Alex Volkov[8] tamaratran
Does the per-turn router replace Claude?
The README says Claude still does the work. Jev chooses among the account’s Haiku, Sonnet and Opus models for the turn. The screenshots were taken with scripts/fake_jev.py, a local stand-in, and the project is marked beta.
Sources:[78] Himanshu Kumar
Did a scratchpad close Sonnet 5’s gap on the hard claims?
On the 18 real sentences, Sonnet 5 moves from 66.7% in the schema-only run to 44.4% with a scratchpad. Watters reports McNemar p = 0.219 and treats that drop as short of significant.
Sources:[39] Jamie Watters
How did Jev compare with Claude Opus 5 on Banking77?
On 3,080 test utterances, OpenRouter reports 81.0% for Jev 1.13 and 84.4% for Claude Opus 5 with reasoning off. The paired gap is 3.3 points. Median round trip was 175 ms and the run cost $0.34, against 2,266 ms and $7.44 for Opus with a cached system prompt.
Sources:[87] Kenny Rogers
Does Engin Diri’s proxy move a Claude session down to Haiku?
At commit f909310 the default policy re-decides on each new human message and only moves the tier up. The shipped accept bars are 0.85, 0.6 and 0.3. The README says the 58-prompt set has not been used to tune them. The 24 September live check mocked Jev. The TypeSafe channel pins jev-1.13.0.
Sources:[97] Engin Diri
Did Christoph Görn’s Claude Code hook call TypeSafe Jev?
The hook called OpenRouter’s typesafe/jev-router. On the first 16 prompts the log names gpt-6-luna five times, deepseek-v4.1-flash three times, and kimi-k3 once. Seven calls returned no JSON. On 8 other prompts the alias scores 1 of 5, at 700 to 1400 ms. The router code is in a private repository.
How did Claude Sonnet 5 score against Jev on the judge panel?
On the base order, where invalid outputs count as errors, Claude Sonnet 5 scores 91.2% on RewardBench, 88.9% on JudgeBench and 85.8% on HaluEval. Jev 1.13.0 scores 92.2%, 78.6% and 87.5%. Sonnet 5 returned a valid contract on 1,308 of 1,312 items. The valid-only rates, which drop four HTTP 200 contract failures, are 91.7%, 89.4% and 85.8%. The identifier is claude-sonnet-5, at low effort.
Sources:[102] Yubo Li, Yidi Miao, Ramayya Krishnan and Rema Padman
Did Opus 5.5 miss conflicts in the Bay Area window?
On the pairs the Opus log scored, Opus 5.5 misses 0 of the script’s hazards and flags 174 false positives, at an average of 12,318 ms. Jev on that run misses 0 and flags 335, at 119 ms, with 385 true positives against Opus’s 378. The log prints one fetch failure. A hazard is the script’s closest-approach rule, and the README says it is not a certified separation standard.
Did Jev match Claude Opus 5 on crypto account selection?
On Entendre’s two-account shortlist, Jev scored 51.7%, which the post sets equal to Claude Opus 5’s recorded account rate, above GPT-5.6 Sol at 51.4% and Claude Sonnet 5 at 49.4%. Jev picked from a shortlist that included the correct account. The language models wrote full journal entries. The CAB paper’s Pass@3 is 45.76% for Opus 5 and 47.46% for Sonnet 5.
Sources:[113] Entendre[112] Entendre[114] Kareem Khattab, Omar Khattab and Mohamed Ibrahem
Does maslul send the turn to Jev instead of Claude?
The hook asks Jev for a simple, medium, or hard tier. The example config still sends the turn to Claude Haiku 4.5 for simple and medium, and to Claude Sonnet 4.6 for hard. The hook runs only when the application passes it in. A missed call leaves the router on its own strategy, and the example’s default level is hard.
Did maslul record a 0.3 second Jev decision?
The 0.3 second figure is in the post. jev.py sets a 5 second client timeout and does not record that measurement. decide() returns the raw pick. Confidence below 0.7, or a failed call, returns no tier when the classifier is asked for one. The router strategy then continues.
Did the Sonnet 4.6 cascade keep the same accuracy?
The X post says the accuracy was the same. bench.json records agreement with Kimi K2-thinking of 0.695 at confidence 0.7, 0.685 at 0.9, and 0.675 for Sonnet 4.6 alone, on 200 items. Those labels are the reference model’s next action. At 0.7, 73 of 200 decisions go to Sonnet.
Sources:[128] Josh Longenecker[130] Josh Longenecker[129] Josh Longenecker
Did Arena’s router match Claude Opus 5.5 on steerability?
The thread reports +10.0% for the router and +10.48 for Claude Opus 5.5 (High), a gap of 0.48 points. The opening post rounds the router figure to +10%. The thread prints no API model id and no Anthropic call share.
Sources:[127] Arena.ai
Did Jev beat Claude on Vals’s LegalBench slice?
Jev 1.13.0 ranks last of twelve. Class-balanced accuracy is 0.730, next to Claude Haiku 4.5 at 0.732. GPT-5.6 Terra leads at 0.932. On raw accuracy, every other system’s lead remains statistically significant after eleven comparisons. The slice is 396 questions, 33 from each of 12 subtasks.
Sources:[133] Connor Frank
Did Jev stay inside a 1% error budget on Vals’s claim items?
A threshold fit on the first 30% of items automates 95% of the held-out 70% at 1.6% error. Claude Sonnet 5.5 automates 100% at 0.4%. Claude Opus 5.5 automates 99% at 0.4%. Haiku 4.5 automates 22% at 0.0%. Six of twelve systems exceeded the budget, including Jev.
Sources:[133] Connor Frank[132] Vals AI
Did Jev beat Claude Haiku 5.5 on Artificial Analysis’s Decision Index?
On the 13 October 2026 launch snapshot, Claude Haiku 5.5 (Low) scores 61 and Jev 1.13 scores 58. Haiku takes 1.4 seconds per task against 0.13, and costs $184 per million tasks against $42. Haiku leads on AA-Omniscience-Decision accuracy, 52 against 35. Jev leads on When2Call, 81 against 73. The live leaderboard can change.
Did Opus 5.5 beat jev-latest on the three Japanese classification sets?
On JNLI, yes: 0.914 against 0.838, McNemar p=0.002. On chABSA, Jev scores 0.910 and Opus 0.881, a gap the article treats as chance after the option text was rewritten. On MASSIVE, Opus scores 0.926 against 0.884, p=0.035, which the article does not treat as significant after ten pairs on one set. Each set has about 210 items and one run.
Sources:[142] driller
Did jev-1.14.0 match Claude Haiku 5.5 on 100 return emails?
No. Haiku 5.5 scored 91 of 100. The best Jev layout, jev-1.14.0 with Python and no Clairic, scored 78. That layout matched a Python encoding of the nine rules on all 100 final choices, so the 22 misses came from reading the email. The emails are fictional and written in Japanese.
Did Sanity time Jev against Haiku on a docs agent?
Entry choice is 282 ms for Jev against 1,328 ms for Haiku. Model routing is about 300 ms against 818 ms. MCP choice is 170 ms against 829 ms. The median full answer is 0.5 seconds. The post calls those figures one week of testing on one corpus.
Sources and reading notes
The summaries below are paraphrases. Each link opens the original source, including its surrounding context.
tamara (@tamarajtran)
Published:
Source checked:
[6] Scoring tool calls for context compaction
The project author proposes using Jev to score tool calls and discard irrelevant material during compaction.
Project announcement. The repository says its native demo app is scripted and makes no API calls, so the animation has no timing to cite.
tamaratran
Source checked:
[8] fast-jev-compaction implementation and limitations
The plugin scores tool calls and results, preserves user and assistant text in its output, and keeps calls paired with their results. Its hook falls back to built-in compaction on errors or insufficient reduction.
The README documents early-access Claude Code function hooks, estimated token sizes, and a scripted native demo. The author warns that a probability can be high on a tool result that is unsafe to delete.
Alex Volkov (@altryne)
Published:
Source checked:
[7] A user report of Claude context reduction
Volkov reports reducing a Claude session from nearly one million tokens to about 86,000 in roughly one second using fast-jev-compaction.
One session. The post includes no downstream task score. The timestamp is the edited-post time shown by X.
Himanshu Kumar (@himanshu231204)
Published:
Source checked:
[77] A per-turn Claude Code router announced on X
Himanshu says Claude Code otherwise keeps one model for a session, while his router asks Jev on each new message and then keeps the tool loop on that choice. He says a failure leaves the session usable within 3 seconds, and links the repository.
Announcement post. The README at the linked commit is the source for which models are offered and for the screenshot setup.
Himanshu Kumar
Source checked:
[78] jev-model-router README at commit 8d754fa
The proxy asks Jev which of the account’s Haiku, Sonnet or Opus models should handle a new Claude Code turn. Tool calls in that turn reuse the choice. A missing key, a timeout or an error continues on a safe model within about three seconds. The README marks jev-codex experimental and says the project is not affiliated with Anthropic or TypeSafe.
Checked at commit 8d754fa on 25 September 2026. The README says the path was exercised on Claude Code 2.1.282 with 94 tests. The screenshots were captured with scripts/fake_jev.py, a local stand-in. The README presents them as captures from that stand-in.
hev
Source checked:
[75] jev-1.13.0 reranking BEIR shortlists
The file records jev-1.13.0 on BM25 top-30 shortlists. On SciFact’s 300 test queries, 30 Nouls in one call score nDCG@10 of 0.768 at $0.18 list price. Claude Opus 5 at low effort scores 0.756, with 12 of 300 queries refused and scored as zero; substituting BM25 order on those 12 gives 0.778. GPT-5.6 Luna scores 0.747, 0.355 and 0.363 on SciFact, NFCorpus and FiQA.
Generated 2026-09-17. Costs are token usage at list price. Haiku 4.5 and Opus 5 appear on SciFact only. NFCorpus has 323 queries. We have not rerun the queries.
Jamie Watters
Source checked:
[39] 42-claim test of whether a cited passage supports a sentence
Watters scores 42 claim-and-passage pairs: 24 constructed controls and 18 real published sentences. The schema-only run reports 85.7% for Jev, GPT-5.4 and Claude Sonnet 5, and 81.0% for Gemini 3.1 Pro. On the hard 18, the README shows 77.8%, 66.7%, 66.7% and 55.6%. A scratchpad rerun of the frontier models leaves those hard-set scores no higher. Unsupported-claim recall for Jev is 78.9%, short of the 80% bar set before the run.
One run per model, labels from the author’s own pipeline, frontier calls through OpenRouter. The table names jev-1.13.0, and the known-weaknesses section says the calls used the jev-latest alias. Costs come from the repo’s price file. McNemar p-values on the hard-18 scratchpad change are 1.000, 0.219 and 1.000. At those p-values the README stops short of calling the drops significant. For GPT-5.4 and Sonnet 5, the verdict call transcribes a short scratchpad.
躺在废墟里 Lieflat (@lieflat_3)
Published:
Source checked:
[35] Labeling awkward metaphors in three AI-written articles
The author asked Jev, Claude Opus 5, Claude Opus 4.8, GPT-5.6 Terra, GPT-5.6 Luna and Gemini to find awkward metaphors in three AI-written articles. On one 202-sentence article, Jev labeled every sentence in 2.6 seconds. Opus 5 in non-reasoning mode took 8.2 seconds, Gemini 88.3 seconds, and Terra and Luna each took more than 160 seconds. Of 10 sentences with explicit comparison markers, Jev marked none and Opus 5 marked 5. The sentence Jev scored as the worst metaphor was a definition of the g factor. The author counts that sentence as a definition.
The labels are the author’s judgment. The post leaves Gemini’s version unnamed. Jev scored every sentence. The other models returned only the sentences they flagged. A repeated ruler metaphor received Jev scores of 0.32, 0.31 and 0.30.
Kenny Rogers
Published:
Source checked:
[87] Banking77 intents: Jev 1.13 and Claude Opus 5
OpenRouter ran the 3,080-utterance Banking77 test split through Jev 1.13 and Claude Opus 5. Accuracy is 81.0% and 84.4%, macro-F1 is 80.5% and 83.6%, median round trip is 175 ms and 2,266 ms, and the billed totals are $0.34 and $7.44.
Kenny Rogers, OpenRouter, 22 September 2026. Both models received the same one-line criteria, written from label names without looking at the test data. Opus ran at temperature 0 with reasoning off and a strict JSON enum. Eight requests were in flight. Latency is the client round trip through OpenRouter. The Opus system prompt was cached. The write-up says the uncached list rate would have been about $19 per 1,000 requests. Confidence at or above 0.99 covered 58% of utterances at 96.3% accuracy. Below 0.5 covered 3.5% at 29.6% accuracy. Sending confidence under 0.90 to Opus left 75.9% on Jev, at 84.0% accuracy and $0.69 per 1,000. That cascade table reuses the same 3,080 examples, and the write-up calls it an upper bound. Both models scored 0 of 40 on get_physical_card. One dataset, one afternoon. Opus was not run with reasoning on.
Engin Diri (@_ediri)
Published:
Source checked:
[95] A Claude Code proxy that asks Jev per message
Engin Diri says he built a Jev router for Claude, with thresholds on the prompt and a dashboard for the decisions. The thread links the repository and a Pulumi blog post.
Announcement post. Model names, the ratchet, and the accept bars are in the repository at commit f909310 and in the blog, not in this post.
Engin Diri
Published:
Source checked:
[96] Route every Claude Code message with Jev
The post describes a local proxy that asks Jev on each new Claude Code message. Mechanical work is aimed at Haiku 4.5, routine work at Sonnet 5, and complex or deep work at Opus 5.5, with an optional Fable 5.1 tier. A session’s tier can rise and does not fall. The author estimates a routing call at about $0.00003.
Read on 28 September 2026. The live-view caption in the post is a picture of the UI. The repository at commit f909310 is the source for the shipped bars, the model pin jev-1.13.0, and which checks were still open.
Engin Diri
Source checked:
[97] jev-router README and config at commit f909310
The proxy asks Jev once per new human message and, by default, only moves a session’s tier up. The Claude-only config maps the four Choice labels onto Haiku 4.5, Sonnet 5 and Opus 5.5. Codex’s frontier tier is gpt-6-astra and its trusted tier is gpt-6-sol. The package name is @ediri/jev-router.
Checked at commit f909310, release 1.6.0. config/anthropic-only.json at that commit sets policy.accept to 0.85, 0.6 and 0.3, sensitiveOverride to 0.7, claimGuard to 0.5, and idleResetMinutes to 10. The TypeSafe channel pins jev-1.13.0. src/jev.mjs sends one Choice plus two Noul guards. The README says the 58-prompt set has not been used to tune the bars, full Codex sessions were not in the live checks, and the offline suite has 176 tests against mocks. A 24 September 2026 check with Jev mocked used Claude Code 2.1.281. The unscoped npm name jev-router is a different project.
Christoph Görn (@goern)
Published:
Source checked:
[98] A Claude Code router timed against local Kev
Christoph Görn says a local Kev-4B router on an RTX 4090 was 10 to 15 times faster than the cloud sizer he called Jev, 100 to 180 ms against 700 to 1400 ms, and that Kev’s accuracy was 14%. The post links his write-up.
The 14% figure is Kev’s score in the write-up, on 8 prompts. The cloud side of that table is OpenRouter’s typesafe/jev-router alias, and the write-up says the log named other models. The 16-prompt latency average is a different sample.
Christoph Görn
Published:
Source checked:
[99] What answered the Claude Code sizing hook
The hook calls OpenRouter’s typesafe/jev-router and asks for a JSON size label. On the first 16 prompts the log counts 3 tiny, 1 everyday, 1 large, 4 left on the main session, and 7 errors. The answered_by field names gpt-6-luna five times, deepseek-v4.1-flash three times, and kimi-k3 once.
Read on 28 September 2026. The 9 successful calls average 2.99 seconds, from 1.26 to 4.41, and sizing cost about $0.0032. A separate 8-prompt table scores Kev-4B at 1 of 7 and the alias at 1 of 5, with latencies of 100 to 180 ms and 700 to 1400 ms. The author says the router code is in a private infra repository and that the router log has no successful Kev decision yet.
Yubo Li, Yidi Miao, Ramayya Krishnan and Rema Padman
Published:
Source checked:
[102] Jev 1.13.0 as a judge next to GPT-6, Claude and Gemini
The paper scores jev-1.13.0 against hosted GPT, Claude and Gemini judges on the same Choice contract. On a 400-pair RewardBench sample Jev scores 92.2% and GPT-6 Astra 93.5%. Claude Sonnet 5’s primary scores are 91.2%, 88.9% and 85.8%, and four contract failures stay in that denominator. Gemini 3 Flash scores 92.8%, 76.0% and 83.3%. Gemini 3.1 Pro scores 94.5%, 87.4% and 87.1%.
arXiv:2609.26550v1, dated 22 September 2026. Jev is jev-1.13.0. GPT-6 Astra is gpt-6-astra at low effort with a JSON schema. Claude Sonnet 5 is claude-sonnet-5 and Gemini 3 Flash and 3.1 Pro are gemini-3-flash-preview and gemini-3.1-pro-preview, also at low effort. RewardBench here is 100 pairs from each of four categories, not the official subset weights. JudgeBench is the GPT-4o split. HaluEval is 240 evidence-grounded judgments. Existing-label accuracy is a fourth column on 150 replies: 94.0% for Jev, 94.7% for Sonnet 5, 95.3% for Flash, 94.7% for Pro and 96.7% for Astra. Invalid outputs count as errors in the primary accuracy. Sonnet 5 is valid on 1,308 of 1,312 items, all four failures HTTP 200 contract breaks. Table 6’s valid-only Sonnet 5 rates are 91.7%, 89.4% and 85.8%. Flash and Pro are valid on all 1,312. The isolated latency panel is 120 judgments, and the printed medians name Jev, GPT-4.1 mini and GPT-6. Fees use reported usage at collection-time prices and are not invoices. The single-order cascades read the threshold off the same items. The two-order policies were frozen earlier, on 510 extension pairs: Astra at 0.9 scores 92.5% against 93.1% at 56.8% of the fee; Sonnet 5 at 0.6 scores 88.0% against 89.2%; Flash at 0.6 scores 87.1% against 82.9%; Pro at 0.9 scores 89.6% against 90.4%. A human adjudicated disagreements with GPT-6. The second pass was an LLM, and an author resolved the 39 items where the two passes differed.
Daniel Liao (@Daniel_Bernath_)
Published:
Source checked:
[106] Jev judges aircraft pairs over the San Francisco Bay
Daniel Liao says a Jev judge watched about 900 live flights from an E2B sandbox, flagged risky pairs in 103 ms, and missed none. He says gpt-4o missed 56 in the same sky and Claude Opus 5.5 took 12 seconds to miss none.
The post links 7datumba/jev-tower. The 103 ms, 56 misses and 12 second figures match files in that repository. The false-positive counts and the fetch error are in those files, not in this post. The post’s 900 flights are not the 60-tick window used for the ladder.
Daniel Liao
Published:
Source checked:
[107] jev-tower logs at commit 20dcbbb
tower.mjs replays one 60-tick ADS-B window. A pair is a hazard when the closest approach is inside 3 nautical miles and 1,000 feet within 120 seconds. Jev, called as jev-latest, and the chat model both flag a pair when their probability is above 0.5. Beside gpt-4o, Jev averages 99 ms with 0 misses and 335 false positives. gpt-4o averages 2,356 ms with 56 misses and 32 false positives.
Checked at commit 20dcbbb. data/tower-gpt4o.json, data/tower-gpt4o-mini.json and data/tower-claude-opus-5-5.json are the three logs. Jev’s counts stay at 385 true positives, 335 false positives and 0 misses. Average latency on the three runs is 99, 103 and 119 ms. gpt-4o-mini averages 2,242 ms, with 160 misses and 22 false positives. claude-opus-5-5 averages 12,318 ms, with 0 misses and 174 false positives on the pairs it scored, and 378 true positives against Jev’s 385. The Opus log prints one fetch failure. The README table lists Jev at 103 ms and the OpenAI false positives as 1 and 0, which these files do not show. The resolved Jev version is not in the logs. The ground truth is the script’s geometry rule.
Entendre (@EntendreAI)
Published:
Source checked:
[111] Jev picks a ledger account from two, three, or five options
Entendre tested Jev on Crypto Accounting Bench by asking it to choose among 2, 3, or 5 real accounts. Reported accuracy is 51.7%, 47.5%, and 48.3%. Successful calls averaged 635 ms. On the five-account list, confidence of at least 0.75 raised accuracy to 61.5% and covered 39 of 118 tasks.
First post in the 28 September 2026 thread. The post says the highest percentages in the confidence charts come from the smallest samples. It does not name a Jev version or say how many calls failed. The shortlist rules and the language-model comparison are in the next two posts.
Entendre (@EntendreAI)
Published:
Source checked:
[112] Every Crypto Accounting Bench shortlist contained the right account
Entendre says each option was a real ledger account, and the incorrect choices were plausible alternatives for that transaction. Every shortlist contained the correct account. The score is agreement with the journal entry the company recorded.
Second post in the thread. It points to the Crypto Accounting Bench write-up, the research paper, and a public dataset. The paper and dataset describe full journal entries. They do not contain this Jev shortlist run.
Entendre (@EntendreAI)
Published:
Source checked:
[113] Shortlist accuracy set beside full journal-entry rates
With five candidates, Entendre reports 48.3% for Jev, 46.3% for GPT-5.6 Luna, and 46.9% for DeepSeek V4 Flash. With two candidates, Jev’s 51.7% matches the recorded Claude Opus 5 rate in the post, above GPT-5.6 Sol at 51.4% and Claude Sonnet 5 at 49.4%.
Third post in the thread. Entendre says the language-model figures are account correctness when those models wrote full journal entries from the complete chart, while Jev received a shortlist that already contained the correct account. The paper’s Pass@3 table is a different metric.
Kareem Khattab, Omar Khattab and Mohamed Ibrahem
Published:
Source checked:
[114] Crypto Accounting Bench paper, arXiv:2609.14811
The paper evaluates full journal-entry reconstruction on 118 tasks from 7 organizations. Pass@3, a perfect rubric and every required gate on one of three attempts, is 47.46% for GPT-5.6 Luna, 50.00% for GPT-5.6 Sol, 45.76% for Claude Opus 5, 47.46% for Claude Sonnet 5, and 46.61% for deepseek-v4-flash-0731.
arXiv:2609.14811v1, submitted 13 September 2026. Mean Score is the paper’s primary metric. Pass@3 is the third column of Table 2 and matches Figure 3. The median chart is 569 accounts. The paper does not evaluate Jev. Entendre’s later shortlist percentages are not these Pass@3 figures.
Entendre Finance
Source checked:
[115] Public Crypto Accounting Bench dataset, 118 full-entry tasks
The dataset card describes 118 tasks whose target is the complete journal entry an organization posted, chosen from that organization’s chart of accounts. It repeats the paper’s best Mean Score of 77.43% and best Pass@3 of 56.78%. It does not include Entendre’s later Jev shortlist run.
Read 30 September 2026. The card says the public files are a transformed derivative and that the judge pipeline behind the paper’s headline numbers is not released. The deterministic scorer in the repository is described as a lower bound. No dataset revision hash is cited here.
Ilia Tankelevich (@ilia_tankelevic)
Published:
Source checked:
[119] maslul 0.5.0 lets Jev pick the tier
Ilia Tankelevich says maslul 0.5.0 uses TypeSafe’s Jev model to pick the tier in his open-source Python LLM router. He reports about 0.3 seconds per decision and points to pip install "maslul[jev]".
Post on 1 October 2026. It does not name the Claude, Gemini, Grok, or GPT model that receives the tier, and it does not include a log of the 0.3 second figure. The hook’s defaults and the example Claude models are in the repository at commit de093d3. The follow-up post is about the level descriptions.
Ilia Tankelevich (@ilia_tankelevic)
Published:
Source checked:
[120] Level descriptions changed the Hebrew tier check
The follow-up says a check on real Hebrew conversations found that the wording of the level descriptions mattered more than the language. Abstract descriptions never let hard win, in English or Hebrew. Concrete examples put hard requests at 0.97 to 0.99 confidence, and those descriptions are now the defaults.
Reply on 1 October 2026 in the maslul thread. It does not list the requests or the confidence of the abstract run. Those details are in a comment in jev.py at commit de093d3. The post does not name a Jev version.
Ilia Tankelevich
Published:
Source checked:
[121] maslul Jev classifier and verifier at commit de093d3
jev.py adds an optional Jev classifier and verifier. The classifier asks one Choice question for simple, medium, or hard, calls jev-latest, and returns no tier below confidence 0.7 or when the call fails. The verifier asks a Noul and, on failure, accepts the reply by default.
Commit de093d3, 1 October 2026, the 0.5.0 hook. The HTTP timeout in this file is 5 seconds. The state is the last 6 text messages, cut to 20,000 characters, and media is dropped. A comment dated 2026-10-01 says abstract descriptions returned medium at 0.48-0.78 for a moving plan, a savings split, and an insurance comparison, in English and Hebrew, and the concrete defaults returned hard at 0.97-0.99 for the same requests. The file does not record a 0.3 second measurement. The example model names are in examples/maslul.toml.
Ilia Tankelevich
Published:
Source checked:
[122] maslul example tiers: Claude Haiku 4.5 and Sonnet 4.6
The example config at commit de093d3 sets simple and medium to anthropic:claude-haiku-4-5 and hard to anthropic:claude-sonnet-4-6. The default strategy is route_default and the default level is hard. The file configures only the Anthropic provider.
Same commit as the Jev hook. The comment says the file is a minimal example so the quickstart runs with an Anthropic key. It does not enable the Jev extra, and it does not name a Gemini, Grok, or OpenAI model. The README of the package documents those providers separately.
Josh Longenecker (@jldevtech)
Published:
Source checked:
[128] Jev in front of Claude Sonnet 4.6 on tool-call checks
The post says a cascade put TypeSafe’s Jev in front of Claude Sonnet 4.6, so Sonnet sees only the cases Jev is unsure about, and reports 51% faster, 61% cheaper, and the same accuracy.
Post on 8 October 2026 linking the Grounded AI write-up. The shorter wording says the accuracy was the same. The write-up and bench.json give 0.695, 0.685 and 0.675, and say the labels are Kimi K2-thinking’s next action.
Josh Longenecker
Published:
Source checked:
[129] CascadeEvaluator on 200 tool-or-reply decisions
The write-up measures CascadeEvaluator on 200 conversations from GeneralFunctionCall-Test. At a confidence of 0.7, 73 of 200 decisions go to Claude Sonnet 4.6. Mean time is 1.28 seconds against 2.59 for Sonnet alone, and the listed cost per million decisions is $2,997 against $7,751. Agreement with the reference is 0.695 against 0.675.
The page has no separate byline. The X post links it, and commit 523194a on grounded-ai/grounded_ai is by jlonge4 on 8 October 2026. The judge string is bedrock/us.anthropic.claude-sonnet-4-6. Jev is called as jev/jev-latest. Labels record what Kimi K2-thinking did next. List prices in the caveats are Jev at $0.042 per million input tokens, output free, and Sonnet 4.6 at $3 and $15. At 0.9, 112 of 200 decisions go to Sonnet, mean time is 1.84 seconds, cost is $4,572 per million, and agreement is 0.685. The write-up calls 0.9 the default threshold.
Josh Longenecker
Published:
Source checked:
[130] tool-call bench.json at commit 523194a
The JSON stores the 200-item run: Sonnet alone at 0.675 agreement, 2.591 seconds mean, 2.355 seconds median and $7,750.875 per million decisions. The 0.7 cascade stores sent_to_sonnet 0.365, agreement 0.695, mean 1.276 seconds and $2,996.74. The 0.9 cascade stores 0.56, 0.685, 1.838 seconds and $4,571.86.
Commit 523194a, 8 October 2026, message "Tool-call benchmark: CascadeEvaluator vs Sonnet 4.6 alone". The file names jev as jev/jev-latest and the judge as bedrock/us.anthropic.claude-sonnet-4-6. The write-up rounds these fields to 2.59 seconds, 2.35 seconds, $7,751, 1.28 seconds, $2,997, 1.84 seconds and $4,572. sent_to_sonnet of 0.365 and 0.56 is 73 and 112 of 200.
Arena.ai (@arena)
Published:
Source checked:
[127] Agent Arena thread on Jev Router
Arena evaluates a product it calls Jev Router by @typesafeai on more than 4,700 Agent Arena sessions. It says the router does not improve on the Pareto frontier: similar performance to DeepSeek V4.1 Flash (Max), 38% more cost, and 1.7 times the median model-request latency. A later post in the thread gives end-to-end medians of 6.18 seconds and 3.64 seconds.
Thread on 7 October 2026. The opening post names DeepSeek V4.1 Flash as the most picked model, with GPT-6.1 Sol and GPT-6 Luna also frequent, and steerability of +10% against +10.48 for Claude Opus 5.5 (High). The next posts give call shares of 28.1% DeepSeek V4.1 Flash, 23.5% GPT-6 Astra, 22.8% Sol and 11.1% Luna, a hypothetical leaderboard row of +3.6% net improvement at $0.155 per task, and P90 latency of 30.59 seconds against 14.26. The thread prints no API model id, no code, and no session export. It does not say the calls used OpenRouter’s typesafe/jev-router. Theo’s DeepSWE note and Christoph Görn’s 16-prompt log stay separate.
Connor Frank
Published:
Source checked:
[133] Independent evaluation of Jev on claim verification and LegalBench
Vals scores twelve systems on 400 claim-verification items built from 10-Ks and on a 396-question LegalBench slice. Jev 1.13.0 scores 0.975 on the claims, 390 of 400, tying GPT-6 Astra, GPT-6.1 Sol and GPT-6 Luna at $0.02 per 1,000 cases. On LegalBench it ranks last, with class-balanced accuracy 0.730 against 0.732 for Claude Haiku 4.5 and 0.932 for GPT-5.6 Terra.
Write-up dated 6 October 2026. Gemini 3.8 Flash wrote 160 paraphrases and contradictions; the labels come from the construction rule, and a person audited every item. LegalBench is 12 subtasks with 33 questions each, sampled from 1,808 available items, and the page says that slice is not representative of the full benchmark. At 32 judgments about one document, Jev’s median latency is 194 times lower than GPT-6 Astra’s and 241 times lower than Gemini 4 Argon’s. A 1% error budget, fit on the first 30% of claim items, automates 95% of the held-out 70% at 1.6% error. Native structured output, no JSON prompting. Intervals bootstrap whole filings and Bonferroni-correct eleven comparisons against Jev.
Vals AI (@ValsAI)
Published:
Source checked:
[132] Vals thread on a 400-item Jev claim check
The opening post says Vals tested TypeSafe’s Jev against 11 other models and matched GPT-6 Astra’s 97.5% accuracy on 400 claim-verification questions at about 1/500th the cost. A follow-up gives about $0.025 per 1,000 cases against $12.29 for Astra, and 194 times lower median latency at 32 judgments about one document.
Thread on 6 October 2026 linking the write-up. A later post in the thread says Jev ranked last on a 12-task LegalBench slice, and that a 1% error-budget threshold automated 95% of held-out claim-verification cases at 1.6% error. The write-up’s probability-priced table lists Jev at $0.02 per 1,000 cases and Astra at $18.00. The $12.29 figure is the thread’s Astra cost without that extra probability charge.
Artificial Analysis
Published:
Source checked:
[139] Artificial Analysis Decision Index v1.0 launch article
The 13 October 2026 launch article scores Jev 1.13 at 58 on Decision Index v1.0, ahead of GLiDE and Microsoft-Decision-1 at 54. Claude Haiku 5.5 (Low) scores 61 at 1.4 seconds per task and $184 per million tasks. GPT-6 Luna (Low) scores 55 at 1.5 seconds and $131. Jev’s median is 0.13 seconds and $42 per million tasks.
The index is an equal mix of AA-Omniscience-Decision (500 private five-option questions, accuracy and non-hallucination at 12.5% each), HLE-MultiChoice (513 text-only multiple-choice items), When2Call (3,423 NVIDIA tool-calling tasks after dropping 229 ambiguous ask-for-information items), and WildGuardTest (835 agreed rows, average of three F1 scores). Language models answer with structured JSON that names one option and supplies no probability. Jev scores 21 on HLE-MultiChoice, 35 accuracy and 35 non-hallucination on AA-Omniscience-Decision, 81 on When2Call, and 94 on WildGuardTest. Haiku 5.5 (Low) scores 24, 52, 73 and 93 on those four. Luna (Low) scores 22, 66, 77 and 89. The article says these are launch figures; the live leaderboard can change.
Artificial Analysis (@ArtificialAnlys)
Published:
Source checked:
[138] Artificial Analysis Decision Index launch thread
The opening post says Jev leads among decision models on Decision Index v1.0, and that language models such as Claude Haiku 5.5 still score higher at much higher cost and latency. The key-results post gives Jev 1.13 a score of 58, ahead of GLiDE and Microsoft-Decision-1 at 54, with Haiku 5.5 (Low) at 61, 1.4 seconds per task and $184 per million tasks against 0.13 seconds and $42 for Jev.
Thread on 13 October 2026 linking the launch article, methodology, and live leaderboard. Follow-up posts repeat the Pareto cost figures: Decider v1.1 27B at $17 per million tasks, Microsoft-Decision-1 at $31, and Jev 1.13 at $42. Celeris-1 Decision is 0.06 seconds per task. The thread says decision models are poorly calibrated on knowledge and reasoning, and relatively well calibrated on WildGuardTest and When2Call. Live scores can move after this launch snapshot.
Artificial Analysis
Published:
Source checked:
[140] Artificial Analysis Decision Index methodology
Version 1.0 weights four evaluations at 25% each. A decision model answers every question in one call. For a choice, Artificial Analysis scores the option with the highest probability. For a yes/no question it takes yes at probability 0.5 or more. Language models receive a fixed system prompt and must return JSON through structured outputs.
AA-Omniscience-Decision non-hallucination is 100 × (1 − hallucination rate), where hallucination is a wrong answer instead of “I don’t know”. HLE-MultiChoice uses the 513 text-only multiple-choice questions from cais/hle, scored as an exact letter match. When2Call uses NVIDIA’s multiple-choice test set after filtering 229 tasks. WildGuardTest keeps rows where all three annotators agreed on prompt harm, response harm and refusal, and drops questions that consistently trigger model refusals. Cost per million tasks is one pass through the index, divided by the combined task count, times one million.
driller
Published:
Source checked:
[142] Claude Max API credits used like Jev on three Japanese sets
The article scores jev-latest against Claude Haiku 5.5, Sonnet 5.5 and Opus 5.5 on MASSIVE ja-JP scenario labels (216 items), chABSA sentiment (210) and JGLUE JNLI v1.3 (210). After rewriting the chABSA and JNLI option text to match the annotation guides, Jev accuracy is 0.884, 0.910 and 0.838. Opus probabilities mode is 0.926, 0.881 and 0.914. Jev’s median latency is 0.16 to 0.17 seconds against 1.3 to 3.5 seconds for the Claude rows.
Published 10 October 2026, edited 11 October. Claude calls go through typesafe-ai/system-one-adapter-python with structured outputs. Haiku ran in discrete and probabilities modes; Sonnet and Opus ran in probabilities mode. Effort is each model’s default. Each class has the same count, seed 42, one run per setup. McNemar finds Opus ahead of Jev on JNLI (4 Jev-only correct, 20 Opus-only, p=0.002). The MASSIVE Opus gap is p=0.035 and the article says it is not significant after comparing ten pairs on one set. Cost per 1,000 items is $0.02 to $0.04 for Jev. The Claude API bill for the check is $14.41; Jev is about $0.03 on a separate bill. A cascade that sends low-confidence Jev items to Sonnet or Opus is a replay of these labels, and the author calls those accuracies optimistic.
driller (@patraqushe)
Published:
Source checked:
[141] Zenn post on Claude Max credits used like Jev
The post links a Zenn article that tries Claude Max plan API credits as a Jev-style classifier and compares jev-latest with Haiku 5.5, Sonnet 5.5 and Opus 5.5 on three Japanese tasks.
Posted 10 October 2026, the same day the article went up. The article was edited on 11 October. The post itself carries no scores.
Takuma Yoshida (@TakumaYoshidaUJ)
Published:
Source checked:
[207] 100 made-up refund emails: Clairic, Jev, Haiku
Takuma Yoshida reports 100 made-up refund emails scored with Clairic, Jev, and Claude Haiku. Jev plus Python is 78% right at ¥1.5 per 100. Clairic plus Jev is 68 to 77% at ¥4.9 to 8.8, which he says could be his setup. Haiku is 91% at ¥8.1. Keeping only the cases Jev is sure of, then doing the rest himself, moves 78% to 83%.
X post, 26 October 2026, linking the English write-up. The 83% line is the experiment-4 split that leaves high-confidence Jev answers in place and sends the rest to the author. Thresholds on that page were drawn after seeing the 100 results.
Takuma Yoshida
Published:
Source checked:
[208] Clairic and Jev on 100 return emails: four experiments
Four layouts sort 100 fictional Japanese return emails for a kitchenware shop. Claude Haiku 5.5 scores 91 of 100 in one call, at 3.0 seconds and ¥8.1 per 100. The six-step Jev layout without Clairic, experiment 4, scores 78 at 0.3 seconds and ¥1.5, using jev-1.14.0. In that layout Jev’s final choice matched a Python encoding of the nine rules on all 100 cases, so all 22 misses came from reading the email.
English write-up dated 26 October 2026. The emails and order records are fictional, written by Claude after the correct answer was fixed. The author is the one person in the person row. Text sent to Jev and Clairic is Japanese. Clairic is Cloudflare’s preview extraction model @cf/cloudflare/clairic as of 24 October 2026. Confidence thresholds such as 0.8 were drawn after seeing the 100 results. Layouts were chosen after seeing other layouts on the same 100 cases. Haiku’s cost is a list-price estimate inside a subscription. Yen figures use 150 yen to the dollar. UnlockJapan states it has no partnership with TypeSafe or Cloudflare.
John Siciliano and Knut Melvær
Published:
Source checked:
[200] Sanity’s 0.5-second docs agent with Jev, Haiku and gpt-oss-120b
The write-up times four decisions that used to sit on an LLM. Entry choice is 282 ms for Jev against 1,328 ms for Haiku. Model routing is about 300 ms against 818 ms. MCP choice is 170 ms against 829 ms. Speculation while the user types removes 1,828 ms at enter. The median full answer is 0.5 seconds. Three of four finish under 0.9 seconds.
Engineering post dated 24 September 2026. The answering model in the demo is openai/gpt-oss-120b on Cerebras. The authors score 20 of 22 answers as fully grounded in the docs, with the first token in about 350 ms. Routing made answers cheaper, about 4.5 times, and slower at the median, so they turned routing off for the 0.5-second demo. Combined entry, model, and MCP questions took about 340 ms. The post calls the numbers indicative: one week of testing on one corpus. Jev is called as jev-latest in the sample request, and the sample response logs jev-1.13.0.
Even Westvang, Rostislav Melkumyan and Knut Melvær
Published:
Source checked:
[199] How Sanity built the @s company agent harness
Sanity’s @s harness treats each agent as a Studio document and asks Jev which published agent should handle a Slack message. At the time of writing, Jev chose among 66 agents plus one option for questions about @s itself. Confidence below 0.4 returns the three most likely agents for the user to pick. The state is the message plus up to the last 20 messages in the thread.
Engineering post last edited 22 October 2026. Follow-ups stay with the current agent unless the new question belongs elsewhere. Watched channels get a yes-or-no on whether the agent should reply. About 42 of 61 hackathon agents worked fully or partly; most failures were missing access, not routing. HAL, a meeting-hygiene agent, answered everyone when its description said “workstreams,” and the team tightened that description. The post names no Jev version and prints no router accuracy. Inference goes through OpenRouter with zero data retention. The post links an earlier 0.5-second docs-agent write-up.
knut (@kmelve)
Published:
Source checked:
[197] Sanity’s internal harness routes agents with Jev
Knut Melvær writes that Sanity uses Jev as the router in an internal agent harness that lets anyone at the company spin up a bespoke agent. He names co-authors Rostislav Melkumyan (@rostimelk) and Even Westvang (@even) and links an engineering post.
X post, 22 October 2026. The post does not print a Jev version, a routing accuracy, or a latency. The linked post is the @s harness write-up.