OpenAI · Community reports
Jev + GPT: ClickHouse table ranking, Decision Index and a 0.5-second docs agent
ClickHouse ranks warehouse tables with Jev at 0.951 nDCG@10 in about 0.4 seconds. Sanity’s docs agent still answers with openai/gpt-oss-120b after Jev picks the knowledge-base entry.
Compiled by jevmodel.ai. We read the cited posts and documentation. The measurements stay with their authors. We have not rerun them.
Astra works on the prompt while Jev plays
paulwei tried Jev in Slay the Spire 2 after using GPT-6 Astra. The initial post describes Astra as capable but slow and reports about 0.7 seconds of deliberation for a Jev action. That 0.7 seconds comes from the author’s game setup. The post gives no full-agent latency and no result for other GPT models.
The follow-up adds an important failure: Jev’s first Ascension 10 attempt stopped at floor 6. The author says Astra then revised the prompts given to Jev, and the pair reached floor 17. The reported division of work is prompt improvement by Astra and repeated game decisions by Jev.
Sources:[3] paulwei[4] paulwei
A separate Codex experiment on the Mac desktop
Sac describes a Codex + Jev computer-use setup called Jev Use. A comparison video uses the same task, adding an event to the Mac calendar. Sac’s assessment is that the Jev version feels smoother while consuming roughly the same number of tokens.
The post leaves the GPT model behind Codex unnamed and publishes no repeated timing test. The report covers this desktop workflow. A speed or cost change for every Codex task is outside what the post measured.
Sources:[5] Sac
Choosing the Codex model for the next call
0xNatoshi’s jev-codex-router sends each Codex model call through four Choice questions: whether the call falls under a mandatory Astra policy, which of Luna, Terra, Sol or Astra is enough, how much thinking to allow, and how long to keep that route. Architecture, an independent final review, and risk-focused review can force Astra. The model that then runs still receives the full Codex request.
If Jev errors, the turn stays on Astra at medium effort. A local shadow file records the decision and still answers with Astra. The README’s historical simulation, about 60% lower than sending every turn to Astra, covers 237 turns under an older policy. The file says the figure comes from that simulation, and that the current policy has no matching figure. Later credit estimates use published rates. The file shows no account debit, and the author says the policy still needs outcome labels from real use.
Himanshu Kumar’s jev-model-router also has a jev-codex command. The README at commit 8d754fa marks that command experimental. It is a different project from the Luna, Terra, Sol and Astra router. That README includes no Codex quota.
Engin Diri’s jev-router, at commit f909310, is a third project. Its Codex profile maps the fast tier to Ollama Cloud glm-5.3-flash, the balanced tier to Ollama Cloud kimi-k2.7-code, and the frontier tier to gpt-6-astra. The trusted tier, after a message that contained a secret, is gpt-6-sol. The README says those OpenAI names come from Codex’s catalog and should be checked against the account. Full Codex sessions were not part of the live checks. The Claude side of the same proxy is on the Claude report.
Jev Router on DeepSWE, next to GPT-6 Astra
OpenRouter’s typesafe/jev-router uses Jev to pick a model and a reasoning effort for each request. Theo’s objection is that the prompt is the wrong input for that choice. Jev categorizes. It does not see the codebase, the tools, or the size of the change. A request to port a project to Rust could be a 100-line file or a million-line application, and the prompt does not say which.
He then ran the router on DeepSWE. During the bench he described it as basically a DeepSeek 4.1 Flash router. The finished note says he spent $1,000: performance was roughly the same as GPT-6 Astra on low, cost was slightly higher, and the run took almost 5 times as long. The next post says to use Astra on low. The posts publish no DeepSWE score, no task count, and no dollar gap beyond those words. The $1,000 is what the bench cost him, not a per-request price.
Christoph Görn’s Claude Code hook called the same OpenRouter alias and asked for a JSON size label. On the first 16 prompts the log names openai/gpt-6-luna five times, deepseek/deepseek-v4.1-flash three times, and moonshotai/kimi-k3 once. Seven calls returned no JSON. He writes that the alias answered with general-purpose models. Latency for that log, and an 8-prompt score, are on the Claude report.
Sources:[79] OpenRouter[80] Theo[81] Theo[82] Theo[83] Theo[98] Christoph Görn[99] Christoph Görn
A content check before a larger GPT model
Ali Alshehri describes a backend check on user fields that other people can see. The service had been calling gpt-4o-mini. After adding Jev, he says the service could call the model he names gpt luna 6 and still return responses 55% faster at 73% lower cost. A later reply says Jev answers within 200ms, and confidence under 80% is escalated to the language model. That reply does not repeat the model id.
The thread gives no request count, no time window, and no invoice. The 55% and 73% figures stay the author’s summary of that service.
Jev Choice and GPT-5.6 Luna on the same rubrics
Delip Rao and Chris Callison-Burch compare Jev Choice with GPT-5.6 Luna, Gemini 3.8 Flash and DeepSeek V4.1 Flash on nine panels and 5,003 pairs. Every judge reads the same criterion text. Accuracy separates in 8 of 27 paired comparisons. Jev’s leads fall mostly on binary criteria. Its deficits fall on graded criteria, chiefly against Gemini.
Across the nine panels, Jev Choice cost $0.063 and finished in 29 seconds of wall time. Luna cost 29 times as much and took 859 seconds. The language-model judges are called once per criterion, while one Jev request carries every criterion of a unit. Requests used the jev-latest alias. A same-day stability check logged jev-1.13.0, and the main runs did not log the version. On Jev’s most confident errors, 96.0% of the language-model verdicts repeat the wrong answer. A cascade replayed on the recorded verdicts gains at most 1.5 points over the best single judge when the threshold is fit on held-out units. The Gemini and DeepSeek rows are on the benchmark page.
Jev and GPT-5.6 Luna on the same shortlist
hev’s RESULTS.md, generated 17 September 2026, asks jev-1.13.0 for one relevance probability per document and compares that with a listwise GPT-5.6 Luna run. Both permute the same BM25 top-30 shortlist. nDCG@10 is 0.768 for 30 Jev probabilities in one call and 0.747 for Luna on SciFact’s 300 test queries, 0.358 and 0.355 on NFCorpus’s 323 queries, and 0.376 and 0.363 on FiQA’s 300 queries.
SciFact list-price cost for those runs is $0.18 and $0.71. Voyage rerank-3 scores 0.402 on FiQA, above both. The score ranks a shared shortlist. We have not rerun the queries. The full table, including the Claude rows, is on the benchmark page.
Sources:[75] hev
A 42-claim source check against GPT-5.4
Jamie Watters’s jev-bench asks whether a cited passage supports a sentence as written. The set has 24 constructed controls and 18 real published sentences. On all 42, Jev and GPT-5.4 both score 85.7%. On the hard 18, the schema-only run shows 77.8% for Jev and 66.7% for GPT-5.4. Schema-run cost per claim is $0.000025 versus $0.001289, and the recorded times are 0.23 seconds versus 1.01 seconds.
With a scratchpad first, GPT-5.4’s hard-set score is 61.1%, at $0.005450 and 3.77 seconds per claim. Watters reports McNemar p = 1.000 for that change on 18 items, and says the GPT-5.4 verdict call transcribes the scratchpad. Jev’s recall on unsupported claims is 78.9%, one item under the 80% bar written down before the run, so the write-up treats Jev as a pre-filter. There is one run per model. The table names jev-1.13.0, while the weaknesses section says the calls used the jev-latest alias. Frontier latency includes the OpenRouter hop, and the labels come from Watters’s own publishing pipeline.
A separate labeling post the same day includes GPT-5.6 Terra and Luna. On one 202-sentence article, both took more than 160 seconds, against 2.6 seconds for Jev labeling every sentence. That test’s marks are the author’s own, and the Gemini report records the rest of the comparison.
Jev 1.13.0 on the 231 public JevBench tasks
Zefan Cai’s Open-Jev note, at commit 3308a15, scores every available public JevBench task. There are 231: 72 original, 48 easy and 111 hard. Jev 1.13.0 is correct on 200. GPT-5.6 Luna, with reasoning effort set to none, is correct on 206. GPT-6 Astra, with reasoning effort low, is correct on 231. The easy tier is 48 of 48 for all three. On the original 72 the counts are 71 for Jev, 69 for Luna and 72 for Astra. On the hard 111 they are 81, 89 and 111. The note limits the score to these 231 public tasks. The other 303 of 534 were unavailable.
Open-Jev and Jev return native probabilities. The GPT rows return verbalized probabilities in constrained JSON. The adapters put candidates in different orders on 119 of 139 Choice tasks. One Jev probability vector was renormalized under the note’s rounding rule. From a single full-response observation per task, P50 is 291.3 ms for hosted Jev, 953.8 ms for Luna and 2,206.4 ms for Astra. The note says that sample is not a repeated latency test. Its tariff estimates for this mixture are $0.03842, $0.17914 and $8.81978 per 1,000 decisions, and it says those are not billing receipts. Brier means on the 231 returned vectors are 0.1811 for Jev, 0.2074 for Luna and 0.0085 for Astra. A cookbook snapshot uses the same public task count and stops early. Its Jev row is 116 correct, with tasks left unattempted.
GPT-5 nano writes the answer after Jev picks two passages
David Arias asked 50 questions about four 10-Ks. Code retrieves passages, one Jev call ranks 30 of them, and the language model reads the top two. With GPT-5 nano as that writer, Jev RAG scores 48 of 50. The same model, run as an agent that searches EDGAR and reads whole filings, scores 34 of 50. On that agent run the language model reads 64,634 tokens per question. The Jev RAG writer reads 815. The article’s DeepSeek V4 Flash row is 50 of 50 against 46 of 50. The three-writer table is on the benchmark page.
The Jev pipeline looks up the latest filing in code. Across the three agent runs, 21 of 29 misses opened the previous year’s filing. GPT-5 nano also asked a clarifying question four times instead of searching. A hand reading of the final answers changed 13 automatic grades, in both directions. The article dates the tests to 27 and 28 September 2026 and calls Jev through OpenRouter as jev-latest. The X write-up says that alias resolved to Jev 1.13.
Sources:[89] David Arias[90] David Arias
Jev 1.13.0 and GPT-6 Astra on the same judge contract
Yubo Li, Yidi Miao, Ramayya Krishnan and Rema Padman ask jev-1.13.0 and gpt-6-astra for a label and label probabilities, with no written rationale. Astra runs at low effort under a JSON schema. On their 400-pair RewardBench sample, 100 pairs from each of four categories, Jev scores 92.2% and Astra 93.5%. The paired difference, Jev minus Astra, is −1.25 points, interval −3.8 to 1.5. On 240 HaluEval judgments the scores are 87.5% and 86.7%. On 150 saved replies they are 94.0% and 96.7%, with macro-F1 0.923 and 0.947. An isolated panel of 120 judgments puts Jev at a 0.152 second median and $0.044 per 1,000, and Astra at 1.885 seconds and $12.182. Those fees use reported usage. They are not invoices.
JudgeBench’s 350-pair GPT-4o split separates them. Jev scores 78.6% and Astra 93.1%, a paired difference of 14.6 points, interval −18.9 to −10.3. Reasoning is 68.4% against 95.9% on 98 pairs, and coding is 76.2% against 97.6% on 42. On RM-Bench pairs where the rejected answer is the more elaborate one, Jev scores 74.8% and Astra 94.6%. When the two answers share a style, the scores are 84.0% and 93.3%. On 80 reference-free prose answers, Jev scores 52.5% and GPT-5.4 scores 55.0%, with an error-detection AUROC of 0.518 for Jev. The paper treats that workload as unsupported.
A teammate adjudicated every base item where Jev and Astra disagreed about correctness, without seeing the labels. On RewardBench the decision sides with Astra on 17 of 29 disagreements and with Jev on 5, and leaves 7 indecisive. The adjudicated difference is 3.0 points. On HaluEval it sides with Astra on 8 of 10. A second pass was an LLM. An author decided the 39 items where that pass and the human pass differed. A frozen two-order rule accepts Jev when its maximum label probability is at least 0.9 and otherwise takes Astra. On 510 later preference pairs it accepts 53.7%, scores 92.5% against Astra’s 93.1%, and uses 56.8% of Astra’s fee on reported usage, or 62.2% when missing usage is reserved. The paired change is −0.59 points. The paper’s closing line rounds that pair to 99% of Astra’s accuracy at 57% of the fee. The same threshold did not hold for every fallback: the GPT-5.6 policy at 0.7 loses 2.35 points. A single-order curve that picks the threshold on the scored items is a separate, post hoc table.
Sources:[102] Yubo Li, Yidi Miao, Ramayya Krishnan and Rema Padman
Jev filters scraped chunks before gpt-5.4 writes
Assaf Elovic’s GPT Researcher note, at commit 2906e632, saves 28 research tasks once and replays the filter. Twenty are SimpleQA questions and eight are open-ended. Search and scraping stay fixed. gpt-5.4 writes every report. Jev scores each 1,000-character chunk on a four-level usefulness rubric and keeps up to 10 chunks at or above 1.5. Embeddings keep the top 10 chunks with cosine similarity above 0.42. An independent gpt-5.4-mini call marks 73% of Jev’s chunks relevant and 46% of the embedding chunks. Blind comparisons, counted only when gpt-5.4 agrees in both orders, are 15 wins, 10 ties and 3 losses. The sign test on the 18 decided pairs is about p = 0.008. Cost per report is $0.115 and $0.117. Assaf’s post describes the precision gap as 59% more relevant context and the preference as 15 to 3.
SimpleQA is 19 of 20 for Jev, 18 of 20 for embeddings and 20 of 20 with no filter. The note says a difference of one question is noise. On the eight open-ended questions, the unfiltered report wins 6 of the decided comparisons with Jev. Skipping the filter costs $0.192 per report. Filter time for Jev is 2.4 seconds at the median with 32 calls in flight, and 1.7 seconds at the default of 64. The client sends jev-latest unless JEV_MODEL is set. The note does not record the resolved version. With a TypeSafe key, select.py uses Jev. Without one, it uses a BM25 keyword fallback: 19 of 20 on SimpleQA, precision 0.51, and 14 wins, 8 ties and 6 losses against embeddings, at $0.116. That sign test is about p = 0.12. Forcing Jev to fill all 10 slots, with no score floor, drops precision from 0.73 to 0.50.
A Glean citation check against GPT-5.6 Luna
Tony Gentilcore’s 28 September 2026 article holds one 1,448-word Glean answer fixed and runs a citation check three times. Jev takes 6.6 seconds and $0.014. GPT-5.6 Luna with no reasoning takes 81.3 to 85.5 seconds and $0.030 to $0.047. Luna at xhigh takes 227.4 to 259.7 seconds and $0.047 to $0.060. Jev’s time is sequential client wall time. The Luna rows add up model-call durations. Across 28 paragraphs, Jev changes its recall verdict on 1 of 28 and disagrees with itself on 2 of 84 run pairs. Luna with no reasoning changes 7 of 28 and disagrees on 15 of 84. At xhigh the counts are 5 of 28 and 10 of 84. Jev’s one change is whether a paragraph needed a citation. The article says the systems used different support criteria, and that the team did not collect independent human labels.
The same article reports three other checks. Query classification agrees with Glean’s production LLM on 66.8% of requests for zero-shot Jev, 35.9% for base Laya and 74.5% for a fine-tuned Laya. The text does not give the request count. Agreement here is with the production system. A routing replay of 751 golden entries and three experts is the check Gentilcore calls both more accurate and faster. The accuracy percentages sit in a chart. On 40 entries that actually transferred, the median speedup is 8.1 times. Jev adds a call when it does not transfer. Choice reranking on 4,855 captured queries assigns the same score to about 37 of 41 candidates. Breaking those ties with production order raises Recall@6 from 40.2% to 44.3%. The offline cost is about $0.00044 per query, at a p50 of 0.195 seconds. Gentilcore calls that a cheap baseline, and says it does not replace Glean’s production ranker.
Sources:[105] Tony Gentilcore
A 60-tick Bay Area window against gpt-4o
Daniel Liao’s jev-tower log, at commit 20dcbbb, replays 60 ticks of recorded Bay Area ADS-B. tower.mjs marks a pair as a hazard when the closest approach is inside 3 nautical miles and 1,000 feet within 120 seconds. Jev is called as jev-latest. A flag is a probability above 0.5. Beside gpt-4o, Jev averages 99 ms, with 385 true positives, 335 false positives and 0 misses. gpt-4o, at temperature 0, averages 2,356 ms, with 329 true positives, 32 false positives and 56 misses. gpt-4o-mini averages 2,242 ms, with 225 true positives, 22 false positives and 160 misses. The Jev run beside mini averages 103 ms and repeats the same 385, 335 and 0.
The README summary lists those OpenAI false positives as 1 and 0, and lists one Jev latency of 103 ms. The JSON logs show 32 and 22 false positives, and Jev averages of 99 ms and 103 ms on those two runs. The post’s 900 flights describe a live session. This comparison is the 60-tick window. The resolved Jev version is not in the logs. The hazard rule is the script’s geometry, which the README says is not a certified separation standard. The Claude Opus 5.5 log from the same window is in the Claude report.
OpenAI’s Decisions API on two early Jev checks
OpenAI’s 29 September 2026 post introduces a Decisions API powered by GPT-6 Luna. The caller supplies a question and the allowed answers, and the API returns one of those answers, for classification, routing, or an agent’s next action. The follow-up says the call can send text or images, with a support request and the teams that could take it as the example. Preview access is limited to selected API customers, and that post says a broader release is planned in the coming days. Dan Shipper’s Every article the same day expects a launch in a few weeks and says pricing has not been announced.
Every’s computer-task check was text only. Jack Cheng, a senior editor, found that Decisions picked the correct control on 76 of 78 scored steps, and Jev on 73 of 78. The typical response time was 230 milliseconds for Decisions and 500 for Jev. Kieran Klaassen, general manager of Cora, classified conversation threads. Shipper writes that the two were effectively tied on accuracy, with a median of 161 milliseconds for Jev and 309 for Decisions. The article’s opening says Decisions trailed Jev on Every’s broader evals. Those scores, the Jev version, the number of threads, and the list of computer tasks are not in the article.
Sources:[108] OpenAI Developers[109] OpenAI Developers[110] Dan Shipper
Picking a ledger account from a shortlist
Entendre asked Jev to pick a ledger account for the 118 tasks in Crypto Accounting Bench, from 2, 3, or 5 real accounts. Every shortlist contained the correct account, and the wrong options were other plausible accounts from the same chart. Reported accuracy is 51.7%, 47.5%, and 48.3%. Successful calls averaged 635 ms. On the five-account list, keeping answers with confidence of at least 0.75 raised accuracy to 61.5% and kept 39 of 118 tasks. The post says the highest percentages in the confidence charts come from the smallest samples. The post does not name a Jev version or count failed calls.
The same thread sets the five-account score of 48.3% next to 46.3% for GPT-5.6 Luna and 46.9% for DeepSeek V4 Flash, and the two-account score of 51.7% equal to a recorded Claude Opus 5 rate, with GPT-5.6 Sol at 51.4% and Claude Sonnet 5 at 49.4%. Those language-model percentages are account correctness when the models wrote full journal entries from the complete chart. Jev’s percentages are picks from a shortlist that already included the right account. The CAB paper’s Pass@3, a perfect rubric and every required gate on one of three attempts, is 47.46% for GPT-5.6 Luna, 50.00% for GPT-5.6 Sol, 45.76% for Claude Opus 5, 47.46% for Claude Sonnet 5, and 46.61% for deepseek-v4-flash-0731. The paper’s tasks show a median chart of 569 accounts. The public dataset card describes the same 118 full-entry tasks and does not include this Jev run.
Sources:[111] Entendre[112] Entendre[113] Entendre[114] Kareem Khattab, Omar Khattab and Mohamed Ibrahem[115] Entendre Finance
Banking77 subsets at 5, 10, 15 and 20 classes
Lucas Biagetti’s v1.2 brief, at commit efbfb42, reports zero-shot accuracy on Banking77 subsets of 5, 10, 15, and 20 classes. The manifest pins Jev to jev-1.13.0 and the OpenAI row to gpt-5-nano-2025-08-07 with reasoning effort set to minimal. Both rows use zero labeled examples. The campaign seed is 42. Each width holds out 40 examples per class, so the columns cover 200, 400, 600, and 800 examples. Jev scores 94.00%, 92.25%, 89.83%, and 86.12%. gpt-5-nano-2025-08-07 scores 92.00%, 88.25%, 87.33%, and 85.00%.
Jev is ahead of that OpenAI row at every width. MiniLM plus logistic regression, trained on 100 examples per class, scores 98.00%, 96.50%, 95.83%, and 92.38%, above both zero-shot rows. The brief calls the comparison exploratory. It uses one seed. Bootstrap intervals, from 2,000 gold-stratified resamples with seed 20260925, condition on the fitted models and do not estimate training-seed variability. The cohorts overlap across widths and must not be pooled. Jev is missing 1, 2, and 3 probability vectors at 10, 15, and 20 classes, so full-cohort calibration is unavailable there. OpenAI returned no probabilities. The brief makes no cross-provider cost or latency claim. Class definitions were assistant-reviewed, without a separate human validation. Changing the class count adds classes and examples, so the trend does not isolate a causal effect of cardinality. OpenRouter’s Banking77 run uses all 77 intents and 3,080 utterances. The paper in the next section uses 3,076 examples. Those three setups stay separate.
Sources:[116] Lucas Biagetti[117] Lucas Biagetti[118] Lucas Biagetti[87] Kenny Rogers[123] Tobias Deußer, Lorenz Sparrenberg and Rafet Sifa
jev-1.13.0 against Qwen3.8-27B and Gemma-4-E4B
Tobias Deußer, Lorenz Sparrenberg and Rafet Sifa evaluate jev-1.13.0 zero-shot on 37 datasets, 346,009 requests, in arXiv:2609.37647. The same requests are scored for Qwen3.8-27B and Gemma-4-E4B from the probability of the next token over option codes, with thinking off. Jev is ahead of Qwen on 27 datasets, tied on one, and behind on 9. Eleven of Jev’s leads have non-overlapping 95% bootstrap intervals. None of Qwen’s nine leads do. Jev is ahead of Gemma on all 37, and 31 of those gaps have non-overlapping intervals. On Banking77, 3,076 examples, accuracy is 79.7% for Jev, 77.4% for Qwen, and 67.0% for Gemma. On MMLU, 14,042 examples, the figures are 91.8%, 81.7%, and 67.4%. The evaluation split cost US$9.15. Mean client latency is 0.36 seconds at 32 concurrent requests. With the pilot and the extra analyses, the study cost US$9.40.
The authors froze one template per dataset after trying it on 20 development examples, with no in-context examples. Rotating the options leaves accuracy unchanged. Withholding the question drops accuracy near chance. They say that rules out shallow memorization, and that it does not rule out memorized question-answer pairs. Jev’s lower absolute scores include Emotion at 58.5%, SST-5 accuracy at 57.9%, GoEmotions macro-F1 of 0.243, AGB-DE F1 of 0.204, and UNFAIR-ToS micro-F1 of 0.499 at a 0.5 threshold. Tuning thresholds on training data raises the UNFAIR-ToS figure to 0.748. The same paper’s Spearman correlation on SST-5 is 0.851. On rubric dimensions, Qwen’s correlation is higher on 9 of 12, mostly by small margins. Jev leads on ToxiGen and on SummEval consistency and fluency. Gemma trails on every rubric dimension. Scoring the open models took 6.4 A40 hours for Gemma and 17.4 A100 hours for Qwen. The paper says those hours depend on the hardware and are not comparable to API prices. The repository at commit 6bbdeb3 pins jev-1.13.0 and links the responses at doi.org/10.5281/zenodo.23039006. The Jev response license allows research and evaluation and bars distillation and training a model to imitate Jev. Qwen3.8-27B and Gemma-4-E4B are the reference models in this section. The paper does not score the GPT models elsewhere on this page.
Sources:[123] Tobias Deußer, Lorenz Sparrenberg and Rafet Sifa[124] Tobias Deußer
GPT call shares in Arena’s Jev Router thread
Arena.ai’s thread on 7 October 2026 evaluates a product it calls Jev Router by @typesafeai, on more than 4,700 Agent Arena sessions. The opening post says the router most often picks DeepSeek V4.1 Flash, and that GPT-6.1 Sol and GPT-6 Luna are also frequent. The next post gives the shares it counted: DeepSeek V4.1 Flash 28.1%, GPT-6 Astra 23.5%, Sol 22.8%, and Luna 11.1%. It says the router preferred OpenAI models over Anthropic models. The opening name GPT-6.1 Sol and the follow-up name Sol are both in the thread. The thread does not say they are the same id.
The opening post says similar performance to DeepSeek V4.1 Flash (Max) costs 38% more, with median model-request latency 1.7 times higher, and that the router does not improve on the Pareto frontier. A later post gives end-to-end medians of 6.18 seconds for the router and 3.64 seconds for DeepSeek V4.1 Flash (Max), and P90 figures of 30.59 seconds and 14.26 seconds. If the router were on the leaderboard, Arena says it would show +3.6% net improvement at a median of $0.155 per task, below that DeepSeek model. Steerability is +10.0%, against +10.48 for Claude Opus 5.5 (High). Four of the five most-used models sit on the Pareto frontier. The fifth model is not named.
The thread prints no API model id, no code, and no session export. It does not say the calls used OpenRouter’s typesafe/jev-router. Theo’s DeepSWE note on that OpenRouter alias publishes no score. Christoph Görn’s hook log, 16 prompts with gpt-6-luna named five times, is a different sample.
Sources:[127] Arena.ai[79] OpenRouter[82] Theo[99] Christoph Görn
Vals: 390 of 400 claims, tied with Astra, Sol and Luna
Connor Frank’s Vals write-up, dated 6 October 2026, scores twelve systems on 400 claim-verification items. Each item is a claim plus a 10-K excerpt, labelled supported, contradicted, or insufficient_evidence. The excerpts come from 1,960 prose passages in the latest 10-Ks of 48 large US companies. Five construction rules, 80 items each, fix the answer before any system sees the item. Gemini 3.8 Flash wrote the 160 paraphrases and contradictions. A person audited every item. Jev 1.13.0 scores 0.975, which is 390 of 400. GPT-6 Astra, GPT-6.1 Sol and GPT-6 Luna also score 390. The twelve systems land between 0.953 and 0.995. Claude Sonnet 5.5 is the top scorer, eight items above Jev. All twelve score 1.00 on the 80 numeric-discrepancy items.
Jev costs $0.02 per 1,000 cases. That is 1/498th of Astra, 1/295th of Claude Opus 5.5, and 1/138th of Sonnet 5.5. GPT-6 Luna, the cheapest model that matches Jev’s score, costs $0.08 per 1,000 on a label-only call. Asking an LLM for probabilities as well raises Luna to $0.10 and Astra to $18.00. Jev’s output is already the distribution, so that extra charge is $0. The X thread rounds the label-only pair to about $0.025 against $12.29. At 32 judgments about one long document, Jev’s median latency is 194 times lower than Astra’s. From 1 to 64 judgments, Jev moves from 0.10 seconds to 0.12 seconds. Astra takes 38 seconds at 64. Isolated single-call median is 0.10 seconds, 0.11 seconds under load, with 8 Jev requests in flight against 4 per LLM.
A 1% error budget, fit on the first 30% of items and frozen, automates 95% of the held-out 70% at 1.6% error. Six of twelve systems exceed that budget, including Jev, Astra at 99% coverage and 1.5% error, Sol at 99% and 1.9%, and Luna at 97% and 1.9%. Sweeping the threshold over all 400 items would have let Jev automate 91% within the budget. Expected calibration error on this set is 0.011, the lowest of the twelve. The write-up says Jev’s confidence field tracks the probability distribution almost exactly. Native structured output, no JSON prompting. Intervals bootstrap whole filings. This claim set is a different task from David Arias’s 50-question 10-K reading test.
Seven political-science replications against GPT-6 Luna
Steven Denney and Matthew DiGiuseppe, arXiv:2610.06625v2, re-run seven published political-science tasks with Jev 1.13, GPT-6 Luna, and Qwen3.8-27B. The human-coded scores stay close across the three. Luna at OpenAI’s batch prices costs less on six of the seven primary tasks and about 3% more on the seventh. Against Luna’s standard prices, Jev is cheaper on six of seven. One-at-a-time medians, 200 timed items per application sent from the Netherlands on 29 and 30 September 2026, are 0.09 to 0.27 seconds for Jev and 0.75 to 1.04 seconds for Luna. Jev ran through OpenRouter except on the V-Dem recall task, which used TypeSafe’s endpoint.
On Gilardi, Alizadeh and Kubli’s relevance tweets, Jev scores 89.0% on the 2021 agreed sample against Luna’s 91.1%, and 76.2% on the 2023 sample against 84.2%. Both paired intervals exclude zero. On Ornstein, Blasingame and Truscott’s 907 Supreme Court tweets, Jev’s correlation with the experts’ mean is 0.740, Luna’s 0.816, Qwen’s 0.841, and the published GPT-4 figure 0.789. On Brandt and coauthors’ 37,709 Global Terrorism Database incidents, names-only accuracy is 0.715 for Jev and 0.709 for Luna, with macro F1 0.555 and 0.544. Mean correlation with V-Dem’s 2023 expert codes, 53 indicators in 171 countries, is 0.68 for Jev and 0.63 for Luna and for the published GPT-4o codes. On Di Leo and coauthors’ party-name comparisons, Jev’s Chapel Hill correlation averages 0.877 across eight election years, Luna 0.844, and the published GPT-3.5 figure 0.772. On Le Mens and Gallego’s manifesto sentences, Jev correlates 0.97 with the economic benchmark and 0.92 with the social benchmark; Luna 0.977 and 0.952. On 35 European Parliament speeches, Jev reaches 0.81 and Luna 0.947, below and above the published GPT-4o figure of 0.93.
When each question is asked once, Jev’s expected calibration error is lower than Luna’s token probabilities on all eight reading tasks with a categorical human label. On the party comparisons, 74% of Luna’s probabilities sit below 0.01 or above 0.99, against 18% for Jev. OpenAI returns at most five token candidates, and none when Luna reasons. Qwen’s calibration is lower than Jev’s on the four pairwise tasks and higher on several labelling and recall tasks. The V-Dem and party-name tasks give the model a name rather than a text, and the expert codes were public before these models were trained. The comments on the abstract list no public code archive.
Decision Index v1.0 next to GPT-6 Luna
Artificial Analysis launched Decision Index v1.0 on 13 October 2026. The launch article scores Jev 1.13 at 58, with GPT-6 Luna (Low) at 55. Luna’s median is 1.5 seconds per task and $131 per million tasks. Jev’s median is 0.13 seconds and $42. Claude Haiku 5.5 (Low), on the Claude report, scores 61 at 1.4 seconds and $184. Among decision models, GLiDE and Microsoft-Decision-1 sit at 54. Decider v1.1 27B has the lowest listed cost at $17 per million tasks. Celeris-1 Decision is the fastest at 0.06 seconds.
The index mixes four evaluations at 25% each. AA-Omniscience-Decision is 500 private five-option questions, with accuracy and non-hallucination each 12.5%. HLE-MultiChoice is 513 text-only multiple-choice items from Humanity’s Last Exam. When2Call is 3,423 NVIDIA tool-calling tasks. WildGuardTest is 835 safety rows scored as the average of three F1 figures. Language models answer with structured JSON that names one option and supplies no probability for the others. A decision model answers every question in one call. Artificial Analysis scores the highest-probability option, and takes yes at probability 0.5 or more.
On knowledge, Luna (Low) scores 66 accuracy on AA-Omniscience-Decision and Jev scores 35. Liquid d1 leads decision models there at 41, with a non-hallucination rate of 2. Jev’s non-hallucination rate is 35, behind Microsoft-Decision-1 at 40. On HLE-MultiChoice Jev scores 21, Luna (Low) 22, and every other decision model 15 or less. On When2Call Jev scores 81 and Luna (Low) 77. On WildGuardTest Jev scores 94 and Luna (Low) 89. The article says decision models are poorly calibrated on knowledge and reasoning, and relatively well calibrated on tool calling and classification. These figures are the launch snapshot. The live leaderboard can change.
Sources:[139] Artificial Analysis[138] Artificial Analysis[140] Artificial Analysis
ALFWorld and AndroidWorld, jev-1.13.0 against gpt-5.6-luna
Yuchen Ling, Shengcheng Yu, Zhenyu Chen and Chunrong Fang, arXiv:2610.13092v1, replace action selection with jev-1.13.0 in five agent families. Every LLM decision and every auxiliary generation call uses gpt-5.6-luna. Observations are text, matching Jev’s input. ALFWorld uses the 134 valid_unseen household tasks. AndroidWorld uses one fixed instance of each of 116 task types. Each configuration runs three times, 4,500 executions in all.
On ALFWorld, ReAct-J succeeds on 96.02% of tasks, 1.49 points under the original agent. Reflexion-J succeeds on 98.01%, 1.74 points under. An LLM-Choice control gives Luna the same candidate list and selection instructions as Jev; those gaps shrink to 1.00 and 0.00 points. On AndroidWorld, Jev trails T3A by 13.22 points, AutoDroid by 2.30, and FastAgent by 18.10. Both AutoDroid variants stay below 20% success. The 95% task-cluster intervals for the T3A and FastAgent drops sit entirely below zero. The AutoDroid interval includes zero. FastAgent-J executes one tool per step, while FastAgent-B can emit several.
Decision cost falls 66.1% to 78.4% and decision time 73.9% to 81.5%, from 4.13 to 8.13 seconds down to 1.04 to 1.75 seconds. Mean task cost falls in every family. ReAct’s task cost falls 50.9% and time 47.6%. Reflexion’s cost falls 32.6% and time 37.8%. T3A’s cost falls 17.1% and time 10.6%. AutoDroid’s cost falls 68.8% and time 5.7%. FastAgent’s cost falls 43.0% and its time rises 0.7%. Auxiliary reasoning and extra environment steps absorb part of that saving. T3A’s cost per delivered success rises 7.7%, from 68.70 to 73.97 mUSD, with a 95% interval that includes zero. FastAgent’s time per success rises from 239.4 to 377.8 seconds. Reflexion-J wins 383 of 402 first trials against 376 of 402, then recovers 11 of 19 initial failures against 25 of 26. Experimental code is at anonymous.4open.science/r/jev-emp-5BF4/.
Sources:[143] Yuchen Ling, Shengcheng Yu, Zhenyu Chen and Chunrong Fang
Arize: Jev 1.13 against OpenAI Decisions on gpt-6-luna
Jim Bennett’s Arize bake-off, posted 7 October 2026, runs eight decision models on 2,675 RAGTruth rows, 655 Phoenix evaluator cases, and 52 routing decisions. OpenAI Decisions uses gpt-6-luna. On RAGTruth, Jev 1.13 scores 85.5% balanced accuracy at a cutoff fit on half the data, against 81.5% for OpenAI. The 95% intervals touch, so Arize records a draw on accuracy and sends Jev through on speed and cost: 0.08 seconds against 0.16, and $0.045 per 1,000 against $0.090. An earlier Arize Opus 5 judge scored 86.6% at $14.30 per 1,000 on that hallucination task.
On the Phoenix suites Jev scores 91.1% and OpenAI 90.0%, with calibration error 0.023 against 0.042. Liquid d1 scores 87.6% with error 0.051. Arize calls the hosted-API final a draw and still advances Jev. On 52 routing cases Jev is 49 of 52 and OpenAI 48 of 52. Clef 27B is 50 of 52. OpenAI is the only model that does not accept Jev’s /v1/systemone layout. Arize wrote each case as labelled text sections and says a different layout could score differently.
A polarity check on the same 1,338 RAGTruth test rows asks either “Is every claim supported?” or “Does it contain an unsupported claim?”. Jev’s ROC AUC is 0.93 on both wordings. OpenAI is 0.90 and 0.89. Clef 27B is 0.92 and 0.91. Liquid d1 drops from 0.92 to 0.17. Kev 9B drops to 0.54. Accuracy wins in the bracket require non-overlapping 95% intervals. Cloud and laptop latencies are not comparable. This hallucination set is a different task from Vals’s 10-K claims.
Sources:[145] Jim Bennett[144] Arize AI
Sanity’s docs agent answers with gpt-oss-120b
John Siciliano and Knut Melvær’s docs-agent demo lets Jev pick the knowledge-base entry, the retrieval method, and, in one configuration, the answering model, while the user is still typing. The model that writes the answer in the 0.5-second demo is openai/gpt-oss-120b on Cerebras. The authors score 20 of 22 answers as fully grounded in the docs. The first token arrives in about 350 ms. The median full answer is 0.5 seconds, and three of four finish under 0.9 seconds.
The sample request sends jev-latest. The sample response logs jev-1.13.0. Routing to a smaller model cut the cost of those answers by about 4.5 times and added about half a second at the median, so they turned routing off for the speed demo. The post calls the numbers indicative: one week of testing on one corpus. Haiku timings for the same decisions are on the Claude report. The later @s harness uses Jev to pick among 66 internal agents and does not name gpt-oss-120b.
Sources:[200] John Siciliano and Knut Melvær[199] Even Westvang, Rostislav Melkumyan and Knut Melvær[197] knut
What a one-pass decision cannot combine
Wang Yang and coauthors, arXiv:2610.24929v1, ask what jev-1.13.0 cannot decide in a single forward pass. On a controlled log, Jev finds and selects records without error up to 20k tokens and 64 options. Once the answer depends on two or more updates of one user, accuracy falls, to 53 to 80% at sixteen updates, because similar records of other users interfere. A stale one-line summary of the state is followed even when the log contradicts it.
On 738 real items graded by how many thinking tokens Qwen3.8-27B needs, Jev answers 97.5% of the items that one forward pass of that model solves and 74% of the others. Accuracy falls from 99% on items that need fewer than 250 thinking tokens to 50% on items that need more than 4k. Confidence flags Jev’s own errors, AUROC 0.83 on the final-state items, and not errors inherited from a stale input, 0.53. An offered hand-off option is taken on 1.6% of the real-task items and on 3 of Jev’s 40 errors. Qwen3.8-27B is the reasoning reference on that paper. The paper does not score a GPT API model and does not score jev-1.14.0.
ClickHouse: ranking warehouse tables and labeling failures
Eduardo Vellasques, Lionel Palacin and Al Brown’s ClickHouse engineering post, dated 27 October 2026, puts Jev on two jobs inside the Agentic Analytics Benchmark. For each failed agent run, one System One call asks which of sixteen failure sub-modes fits, which schema-prompt rule would have prevented it, whether a schema hint would have helped, and whether the answer looks correct despite a fail verdict. On a four-model development-set run of 544 attempts, that labeler covers 277 failures. Wrong data selection is the largest bucket at 87 of 277, most of it entity resolution rather than wrong table. At a 0.65 threshold on “correct despite fail,” the post flags 18 answers the grader likely scored wrong.
For table ranking, the state is the question plus the warehouse schema, and the questions are one yes/no per table. On the 136-question development set over an 18-table warehouse, mean nDCG@10 is 0.951, recall@10 is 0.991, and for 84% of questions the top five tables already contain every table the reference SQL needs. Ranking takes about 0.4 seconds per question. A companion pilot on public text-to-SQL catalogs with 75 to 710 tables (250 cases) reaches 87% recall@10 and finds every required table in 73% of cases, at a p90 of 3.4 seconds and about $0.50 for the whole run.
Feeding the ranking back as an in-loop hint leaves accuracy flat across 20 models. On fired cells, the pooled net is +12 recoveries with a 95% interval from −18 to +42. The post keeps the labeler and the ranking, and treats the hint as a negative result. The harness ships as ClickHouse/data-agent-mnist at commit 403f3e7, Apache 2.0, with schema_retrieval.py and label_failure_modes.py under an optional Jev extra. The post does not name a Jev version.
Sources:[225] Eduardo Vellasques, Lionel Palacin and Al Brown[224] ClickHouse[226] ClickHouse
Jev was faster at the game, and weaker
paulwei says Jev was less capable at the game than Astra. The same thread that praises its speed also records an early failure. Reaching floor 17 after prompt revisions is how far that experiment got. The post includes no completed game and no measured success rate.
The reports describe two uses of Jev around OpenAI tools. One is repeated game decisions after prompt revisions. The other is desktop work alongside Codex. The two posts share no cost reduction, latency percentile, or accuracy score.
Sources:[4] paulwei[5] Sac
Questions about Jev + GPT / Codex
Which GPT version appears in the Jev game experiment?
paulwei names GPT-6 Astra. Sac’s separate desktop post names Codex and leaves the underlying GPT version unnamed.
Sources:[3] paulwei[5] Sac
Did Jev and GPT finish the Slay the Spire 2 run?
The cited follow-up reports reaching floor 17 on Ascension 10 after Astra revised Jev prompts. Floor 17 is the last floor in the post.
Sources:[4] paulwei
Does the Codex router’s 60% figure mean a smaller Codex bill?
The README says the simulation covers 237 turns under an older policy. The figure comes from that simulation. The current routing policy has no matching quota reading. Later reports estimate credits from published rates.
Sources:[40] 0xNatoshi
Is jev-codex the same project as the Luna, Terra, Sol and Astra router?
No. jev-codex is a command in Himanshu Kumar’s jev-model-router, and the README at commit 8d754fa marks it experimental. The Luna, Terra, Sol and Astra choices come from 0xNatoshi’s jev-codex-router.
Did Jev beat GPT-5.4 on the 42-claim source check?
On all 42 claims, Watters reports 85.7% for both. On the 18 real published sentences, the schema-only scores are 77.8% for Jev and 66.7% for GPT-5.4. Jev’s unsupported-claim recall is 78.9%, short of the 80% replacement bar set before the run.
Sources:[39] Jamie Watters
Did OpenRouter’s Jev Router beat GPT-6 Astra on DeepSWE?
Theo reports roughly the same DeepSWE performance as GPT-6 Astra on low, a slightly higher cost, and almost 5 times the runtime, after spending $1,000 on the bench. The posts do not publish the score or the task count. His follow-up says to use Astra on low.
Did typesafe/jev-router answer as Jev in the Claude Code hook?
Christoph Görn’s log for the first 16 prompts names gpt-6-luna five times, deepseek-v4.1-flash three times, and kimi-k3 once. Seven calls returned no JSON. He writes that the alias did not answer as TypeSafe’s Jev. Theo’s DeepSWE note is a separate run and does not publish that mix.
Sources:[99] Christoph Görn[81] Theo
Did Jev Choice match GPT-5.6 Luna as a rubric judge?
On nine panels and 5,003 pairs, Rao and Callison-Burch find a separated accuracy difference in 8 of 27 comparisons with Luna, Gemini 3.8 Flash, or DeepSeek V4.1 Flash. Luna cost 29 times Jev Choice’s $0.063 and took 859 seconds against Jev’s 29. A cascade on the recorded verdicts gains at most 1.5 points over the best single judge.
Did GPT-6 Astra score 231 of 231 on the full JevBench?
On the 231 public tasks, with reasoning effort low, the Open-Jev note records 231 of 231 for Astra. The same note says the benchmark has 534 tasks and the other 303 were unavailable. Jev 1.13.0 scores 200 of those 231, and GPT-5.6 Luna, with reasoning effort none, scores 206. Candidate order differs on 119 of 139 Choice tasks.
Sources:[92] Zefan Cai
Did GPT-5 nano beat Jev RAG on the 10-K questions?
With GPT-5 nano writing the answer, Jev RAG scores 48 of 50 and the agent scores 34 of 50. The Jev pipeline looks up the latest filing in code. Across three agent models, 21 of 29 misses opened the previous year’s filing. A hand reading changed 13 automatic grades.
Sources:[90] David Arias
Did Jev match GPT-6 Astra as a judge?
On the paper’s 400-pair RewardBench sample, Jev 1.13.0 scores 92.2% and GPT-6 Astra 93.5%. On 240 HaluEval judgments the scores are 87.5% and 86.7%. On JudgeBench’s 350 pairs they are 78.6% and 93.1%. A frozen cascade to Astra at probability 0.9 scores 92.5% against 93.1% on 510 pairs.
Sources:[102] Yubo Li, Yidi Miao, Ramayya Krishnan and Rema Padman
Did Jev’s chunk filter beat embeddings in GPT Researcher?
On 28 replayed tasks, 73% of Jev’s chunks were marked relevant against 46% for embeddings, and blind comparisons were 15 wins, 10 ties and 3 losses. Cost per report was $0.115 and $0.117. SimpleQA was 19 of 20 against 18 of 20. On the eight open-ended questions, the unfiltered report won 6 of the decided comparisons.
Did Glean find Jev more accurate than GPT-5.6 Luna on citations?
The article does not score accuracy. On one 1,448-word answer, Jev changed its citation-coverage verdict on 1 of 28 paragraphs across three runs. Luna with no reasoning changed 7, and Luna at xhigh changed 5. The systems used different support criteria, and there were no independent human labels.
Sources:[105] Tony Gentilcore
Did gpt-4o miss conflicts that Jev flagged in the Bay Area log?
In the 60-tick log beside gpt-4o, Jev misses 0 of the script’s hazards and flags 335 false positives. gpt-4o misses 56 and flags 32. A hazard is the script’s closest-approach rule, not a certified separation standard. The README table’s false-positive counts of 1 and 0 do not match these files.
Did OpenAI’s Decisions API beat Jev?
On Every’s text-only computer replay, Decisions picked the correct control on 76 of 78 scored steps and Jev on 73 of 78, with typical times of 230 and 500 milliseconds. On a conversation-classification test Shipper writes that the two were effectively tied on accuracy, and Jev’s median was 161 milliseconds against 309. Every says Decisions trailed on broader evals and does not publish those scores. Pricing is unannounced. The preview can take images. The computer replay was text only.
Did Jev beat GPT-5.6 Luna at picking a crypto ledger account?
Entendre’s five-account shortlist score for Jev is 48.3%, set next to 46.3% for GPT-5.6 Luna and 46.9% for DeepSeek V4 Flash. Jev chose from a shortlist that already contained the correct account. The post says the language models wrote full journal entries from the complete chart. The CAB paper’s Pass@3 on those full entries is 47.46% for Luna and 46.61% for deepseek-v4-flash-0731.
Sources:[113] Entendre[112] Entendre[114] Kareem Khattab, Omar Khattab and Mohamed Ibrahem
Did Jev beat gpt-5-nano-2025-08-07 on the Banking77 subsets?
On the zero-shot rows, jev-1.13.0 scores 94.00%, 92.25%, 89.83%, and 86.12% at 5, 10, 15, and 20 classes. gpt-5-nano-2025-08-07, with reasoning effort minimal, scores 92.00%, 88.25%, 87.33%, and 85.00%. Each class has 40 held-out examples. MiniLM plus logistic regression at 100 examples per class scores 98.00%, 96.50%, 95.83%, and 92.38%, above both zero-shot rows. The brief calls the comparison exploratory and uses one seed.
Are the paper’s Banking77 numbers the same experiment as the subset table?
No. The paper reports 79.7% for jev-1.13.0, 77.4% for Qwen3.8-27B, and 67.0% for Gemma-4-E4B on 3,076 Banking77 examples. Lucas’s table uses subsets of 5 to 20 classes, with 40 examples per class. OpenRouter’s run, 81.0% for Jev 1.13 against 84.4% for Claude Opus 5, uses 3,080 utterances and all 77 intents. The paper’s reference models are Qwen3.8-27B and Gemma-4-E4B.
Sources:[123] Tobias Deußer, Lorenz Sparrenberg and Rafet Sifa[117] Lucas Biagetti[87] Kenny Rogers
Did Arena’s Jev Router thread publish a DeepSWE score?
No. The 7 October 2026 thread reports Agent Arena figures for a product it calls Jev Router by @typesafeai, including 23.5% of model calls to GPT-6 Astra. It does not mention DeepSWE, and it prints no API model id. Theo’s DeepSWE note on OpenRouter’s typesafe/jev-router still publishes no score.
Sources:[127] Arena.ai[82] Theo
Did Jev match GPT-6 Astra on Vals’s claim check?
On 400 items, Jev 1.13.0, GPT-6 Astra, GPT-6.1 Sol and GPT-6 Luna each score 390. Claude Sonnet 5.5 scores eight items higher. Jev costs $0.02 per 1,000 cases, 1/498th of Astra on that write-up. The X thread rounds the label-only Astra cost to $12.29.
Sources:[133] Connor Frank[132] Vals AI
Did Jev cost less than GPT-6 Luna on the political-science replications?
At OpenAI’s batch prices, Luna costs less on six of the seven primary tasks and about 3% more on the seventh. Against Luna’s standard prices, Jev is cheaper on six of seven. One-at-a-time medians from the Netherlands are 0.09 to 0.27 seconds for Jev and 0.75 to 1.04 seconds for Luna.
Did Jev beat GPT-6 Luna on Artificial Analysis’s Decision Index?
On the 13 October 2026 launch snapshot, Jev 1.13 scores 58 and GPT-6 Luna (Low) scores 55. Luna is 1.5 seconds per task and $131 per million tasks; Jev is 0.13 seconds and $42. Luna leads on AA-Omniscience-Decision accuracy, 66 against 35. Claude Haiku 5.5 (Low) scores 61. The live leaderboard can change.
Did swapping Jev into Luna agents raise ALFWorld success?
No. ReAct-J succeeds on 96.02% of ALFWorld tasks, 1.49 points under the gpt-5.6-luna agent. Reflexion-J succeeds on 98.01%, 1.74 points under. An LLM-Choice control that keeps Luna and uses Jev’s candidate list shrinks those gaps to 1.00 and 0.00 points. On AndroidWorld, Jev trails T3A by 13.22 points and FastAgent by 18.10.
Sources:[143] Yuchen Ling, Shengcheng Yu, Zhenyu Chen and Chunrong Fang
Did OpenAI Decisions beat Jev on Arize’s RAGTruth run?
Jev 1.13 scores 85.5% balanced accuracy and OpenAI Decisions on gpt-6-luna scores 81.5%. The 95% intervals touch, so Arize records a draw on accuracy. Jev is faster and cheaper on that round: 0.08 seconds against 0.16, and $0.045 per 1,000 against $0.090. OpenAI received labelled text sections, not Jev’s named-field layout.
Sources:[145] Jim Bennett[144] Arize AI
Which model writes Sanity’s 0.5-second docs answers?
The demo uses openai/gpt-oss-120b on Cerebras after Jev chooses the knowledge-base entry. The authors score 20 of 22 answers as fully grounded. The median full answer is 0.5 seconds. Routing to a smaller model was turned off for that timing.
Did Jev track a user’s state after sixteen log updates?
On Yang and coauthors’ controlled logs, retrieval stayed perfect up to 20k tokens. Final-state accuracy after sixteen updates of one user fell to 53 to 80%. A stale summary was followed even when the log contradicted it. The model under test is jev-1.13.0.
Did ClickHouse’s table-ranking hint raise agent accuracy?
No. Across 20 models, the in-loop hint left accuracy flat. On fired cells the pooled net is +12 with a 95% interval from −18 to +42. Mean nDCG@10 for the ranking itself is 0.951 on the 136-question development set over an 18-table warehouse.
Sources:[225] Eduardo Vellasques, Lionel Palacin and Al Brown[224] ClickHouse
Sources and reading notes
The summaries below are paraphrases. Each link opens the original source, including its surrounding context.
paulwei (@coolish)
Published:
Source checked:
[3] Trying Jev after GPT-6 Astra in Slay the Spire 2
The author describes GPT-6 Astra as capable but slow in an earlier game run, then reports roughly 0.7 seconds of action deliberation with Jev.
Personal game experiment with a video. The 0.7 seconds comes from that one session.
paulwei (@coolish)
Published:
Source checked:
[4] Astra revises Jev prompts after a failed game run
The first Ascension 10 attempt reached floor 6. The author says Astra then iterated on Jev prompts and the pair reached floor 17, while describing Jev as weaker at the game than Astra.
Follow-up by the same author. The reported run stopped at floor 17.
Sac (@Saccc_c)
Published:
Source checked:
[5] Codex and Jev for adding a Mac calendar event
Sac describes a Codex + Jev computer-use experiment and compares adding the same Mac calendar event. The reported experience is smoother, with similar token consumption.
Qualitative report and comparison video. The GPT version is unnamed, and the post has no repeated latency measurement.
0xNatoshi
Source checked:
[40] Per-turn Codex model routing with Jev
The router asks Jev, on each Codex model call, whether Astra is mandatory, which of Luna, Terra, Sol or Astra to use, how much thinking to allow, and how long to keep the route. Architecture, independent final review and risk-focused review can force Astra. Errors fail open to Astra at medium effort. A shadow file logs the choice and Astra still answers.
Repository documentation read at commit 8701ef7. The figure of about 60% less than full Astra is a simulation of 237 turns under an older policy. The README says it was not read from a Codex quota, and that the current policy has no matching figure. Later credit estimates use published rates, with no account debit in the file. The author says outcome labels are still missing.
Himanshu Kumar
Source checked:
[78] jev-model-router README at commit 8d754fa
The proxy asks Jev which of the account’s Haiku, Sonnet or Opus models should handle a new Claude Code turn. Tool calls in that turn reuse the choice. A missing key, a timeout or an error continues on a safe model within about three seconds. The README marks jev-codex experimental and says the project is not affiliated with Anthropic or TypeSafe.
Checked at commit 8d754fa on 25 September 2026. The README says the path was exercised on Claude Code 2.1.282 with 94 tests. The screenshots were captured with scripts/fake_jev.py, a local stand-in. The README presents them as captures from that stand-in.
Engin Diri
Source checked:
[97] jev-router README and config at commit f909310
The proxy asks Jev once per new human message and, by default, only moves a session’s tier up. The Claude-only config maps the four Choice labels onto Haiku 4.5, Sonnet 5 and Opus 5.5. Codex’s frontier tier is gpt-6-astra and its trusted tier is gpt-6-sol. The package name is @ediri/jev-router.
Checked at commit f909310, release 1.6.0. config/anthropic-only.json at that commit sets policy.accept to 0.85, 0.6 and 0.3, sensitiveOverride to 0.7, claimGuard to 0.5, and idleResetMinutes to 10. The TypeSafe channel pins jev-1.13.0. src/jev.mjs sends one Choice plus two Noul guards. The README says the 58-prompt set has not been used to tune the bars, full Codex sessions were not in the live checks, and the offline suite has 176 tests against mocks. A 24 September 2026 check with Jev mocked used Claude Code 2.1.281. The unscoped npm name jev-router is a different project.
OpenRouter (@OpenRouter)
Published:
Source checked:
[79] OpenRouter announces typesafe/jev-router
OpenRouter introduced typesafe/jev-router as a cache-aware router that uses Jev to pick a model and a reasoning effort for each request, with the stated aim of balancing quality, speed and cost.
Product announcement. The post describes the router. It does not include a score, a task count, or a price for the routed models.
Theo (@theo)
Published:
Source checked:
[80] Why a prompt is a weak input for routing a coding agent
Theo writes that Jev categorizes and does not reason, and that a prompt alone does not show the codebase, the tools, or how large the change is. His example is a request to port a project to Rust, which could be a 100-line file or a million-line application.
Reply in the thread under OpenRouter’s Jev Router announcement. The argument is about what the router can see. It is not a benchmark score.
Theo (@theo)
Published:
Source checked:
[81] An in-progress look at Jev Router’s model mix
While the DeepSWE bench was still running, Theo wrote that Jev Router was so far basically a DeepSeek 4.1 Flash router.
Interim note, posted hours before the final DeepSWE comparison. The post does not give a percentage split across models.
Theo (@theo)
Published:
Source checked:
[82] DeepSWE run of Jev Router against GPT-6 Astra on low
Theo says he spent $1,000 and stayed up until 2am benchmarking Jev Router. On DeepSWE, performance was roughly the same as GPT-6 Astra on low, the router cost slightly more, and the run took almost 5 times as long.
Author-reported comparison with the OpenRouter router. The post gives no DeepSWE score, no task count, and no dollar gap beyond “slightly more” and the $1,000 spent on the bench.
Theo (@theo)
Published:
Source checked:
[83] Follow-up after the Jev Router bench
Immediately after the DeepSWE post, Theo’s update is to use Astra on low.
One-line follow-up in the same thread. It names the setup he would use instead of the router. It does not add a second measurement.
Christoph Görn (@goern)
Published:
Source checked:
[98] A Claude Code router timed against local Kev
Christoph Görn says a local Kev-4B router on an RTX 4090 was 10 to 15 times faster than the cloud sizer he called Jev, 100 to 180 ms against 700 to 1400 ms, and that Kev’s accuracy was 14%. The post links his write-up.
The 14% figure is Kev’s score in the write-up, on 8 prompts. The cloud side of that table is OpenRouter’s typesafe/jev-router alias, and the write-up says the log named other models. The 16-prompt latency average is a different sample.
Christoph Görn
Published:
Source checked:
[99] What answered the Claude Code sizing hook
The hook calls OpenRouter’s typesafe/jev-router and asks for a JSON size label. On the first 16 prompts the log counts 3 tiny, 1 everyday, 1 large, 4 left on the main session, and 7 errors. The answered_by field names gpt-6-luna five times, deepseek-v4.1-flash three times, and kimi-k3 once.
Read on 28 September 2026. The 9 successful calls average 2.99 seconds, from 1.26 to 4.41, and sizing cost about $0.0032. A separate 8-prompt table scores Kev-4B at 1 of 7 and the alias at 1 of 5, with latencies of 100 to 180 ms and 700 to 1400 ms. The author says the router code is in a private infra repository and that the router log has no successful Kev decision yet.
Ali Alshehri (@AliCantCode)
Published:
Source checked:
[84] A public-field check that moved from gpt-4o-mini to a Luna model
Ali says a service that used gpt-4o-mini could switch to the model he calls gpt luna 6 after Jev was added, with responses 55% faster and cost 73% lower.
The post names the two models and those two percentages. It does not give the request count, the time window, or an invoice.
Ali Alshehri (@AliCantCode)
Published:
Source checked:
[85] Escalating a content check when Jev’s confidence is under 80%
In the same thread, Ali says Jev answers within 200ms whether public user content is appropriate, and that confidence under 80% is sent on to the language model.
Reply describing the gate. It does not repeat the model id from the first post, and it does not publish the sample behind the 55% and 73% figures.
Delip Rao and Chris Callison-Burch
Published:
Source checked:
[86] Jev Choice against three flash rubric judges
The paper compares Jev Choice with GPT-5.6 Luna, Gemini 3.8 Flash and DeepSeek V4.1 Flash on nine panels and 5,003 pairs, using the same criterion texts. Accuracy separates in 8 of 27 paired comparisons. Across the panels, Jev Choice cost $0.063 and finished in 29 seconds. Luna, DeepSeek and Gemini cost 29, 66 and 325 times as much, and took 859, 6,298 and 950 seconds.
arXiv:2609.29769, submitted 24 September 2026. Requests used the jev-latest alias. A same-day stability check logged jev-1.13.0. The main runs did not log the version. LLM judges are called once per criterion. On Jev’s most confident errors, 96.0% of the LLM verdicts repeat the wrong answer. A cascade replayed on the recorded verdicts gains at most 1.5 points over the best single judge with cross-fitted thresholds, and at most 2.0 with oracle thresholds. Jev’s concurrency was lower than the LLM judges’, so the wall-time ratios understate a same-concurrency comparison. DeepSeek’s three failed calls on the binary panels were recorded as UNMET.
hev
Source checked:
[75] jev-1.13.0 reranking BEIR shortlists
The file records jev-1.13.0 on BM25 top-30 shortlists. On SciFact’s 300 test queries, 30 Nouls in one call score nDCG@10 of 0.768 at $0.18 list price. Claude Opus 5 at low effort scores 0.756, with 12 of 300 queries refused and scored as zero; substituting BM25 order on those 12 gives 0.778. GPT-5.6 Luna scores 0.747, 0.355 and 0.363 on SciFact, NFCorpus and FiQA.
Generated 2026-09-17. Costs are token usage at list price. Haiku 4.5 and Opus 5 appear on SciFact only. NFCorpus has 323 queries. We have not rerun the queries.
Jamie Watters
Source checked:
[39] 42-claim test of whether a cited passage supports a sentence
Watters scores 42 claim-and-passage pairs: 24 constructed controls and 18 real published sentences. The schema-only run reports 85.7% for Jev, GPT-5.4 and Claude Sonnet 5, and 81.0% for Gemini 3.1 Pro. On the hard 18, the README shows 77.8%, 66.7%, 66.7% and 55.6%. A scratchpad rerun of the frontier models leaves those hard-set scores no higher. Unsupported-claim recall for Jev is 78.9%, short of the 80% bar set before the run.
One run per model, labels from the author’s own pipeline, frontier calls through OpenRouter. The table names jev-1.13.0, and the known-weaknesses section says the calls used the jev-latest alias. Costs come from the repo’s price file. McNemar p-values on the hard-18 scratchpad change are 1.000, 0.219 and 1.000. At those p-values the README stops short of calling the drops significant. For GPT-5.4 and Sonnet 5, the verdict call transcribes a short scratchpad.
躺在废墟里 Lieflat (@lieflat_3)
Published:
Source checked:
[35] Labeling awkward metaphors in three AI-written articles
The author asked Jev, Claude Opus 5, Claude Opus 4.8, GPT-5.6 Terra, GPT-5.6 Luna and Gemini to find awkward metaphors in three AI-written articles. On one 202-sentence article, Jev labeled every sentence in 2.6 seconds. Opus 5 in non-reasoning mode took 8.2 seconds, Gemini 88.3 seconds, and Terra and Luna each took more than 160 seconds. Of 10 sentences with explicit comparison markers, Jev marked none and Opus 5 marked 5. The sentence Jev scored as the worst metaphor was a definition of the g factor. The author counts that sentence as a definition.
The labels are the author’s judgment. The post leaves Gemini’s version unnamed. Jev scored every sentence. The other models returned only the sentences they flagged. A repeated ruler metaphor received Jev scores of 0.32, 0.31 and 0.30.
Zefan Cai (@Zefan_Cai)
Published:
Source checked:
[91] Open-Jev 27B v1.1 on the public JevBench tasks
Zefan Cai reports Open-Jev 27B v1.1 at 197 of 231 public JevBench tasks, and 80 of 111 on the hard tier. Jev is 200 of 231 and 81 of 111. The post says the 231 tasks are a public subset of 534.
Announcement post. The GPT-5.6 Luna and GPT-6 Astra rows, timings and scoring rules are in the repository note at commit 3308a15, not in this post.
Zefan Cai
Published:
Source checked:
[92] Public JevBench note at Open-Jev commit 3308a15
The note scores six streams on all 231 available public tasks. Jev 1.13.0 is correct on 200. GPT-5.6 Luna, reasoning effort none, is correct on 206. GPT-6 Astra, reasoning effort low, is correct on 231. Easy tasks are 48 of 48 for each of those three. On the original 72 the counts are 71, 69 and 72. On the hard 111 they are 81, 89 and 111.
Read at commit 3308a15. The other 303 of 534 tasks were unavailable, and the note reports no 534-task score. Open-Jev and Jev return native probabilities. GPT returns verbalized probabilities in JSON. Candidate order differs on 119 of 139 Choice tasks. One Jev vector was renormalized. P50 full-response times, one observation per task, are 291.3 ms for Jev, 953.8 ms for Luna and 2,206.4 ms for Astra. The note says that sample is not a repeated latency test. Tariff estimates are $0.03842, $0.17914 and $8.81978 per 1,000 decisions, and the note says they are not billing receipts. Brier means are 0.1811, 0.2074 and 0.0085. Top-label ECE is 0.0318, 0.0932 and 0.0149. Open-Jev 27B v1.1 is 197 of 231, and its direct-execution timings are left out of the latency table. Upstream tasks are pinned at fstandhartinger/jevbench commit f8ce713.
Datawhale
Published:
Source checked:
[94] Jev Cookbook chapter 6 snapshot for 26 September 2026
The chapter records one run of five aliases on 231 planned public tasks. Kimi K3 has 139 valid answers and 139 correct. Jev has 120 valid and 116 correct, then leaves 110 tasks unattempted after a reservation stop. DeepSeek Flash has 110 valid and 110 correct, then stops after consecutive errors. Conditional accuracy uses only scorable answers.
Read at commit 546050a. snapshot.json in the same commit matches the counts. Recorded spend is $0.001986 for Jev, $0.011435 for Kimi K3 and $0.020312 for DeepSeek Flash. p50 latency is 0.77, 9.63 and 1.40 seconds. The aliases do not include a full version string. Hard-tier valid answers are 1, 29 and 2 of 111. The stored composite is highest for Jev, and the chapter says that score describes this incomplete run. Raw prompts and provider responses are not in the repository. This run is separate from the completed 231-task audit in the Open-Jev note.
David Arias (@Davidariasfin)
Published:
Source checked:
[89] Jev ranks 10-K passages before a language model answers
David Arias describes 50 questions about four annual reports. Jev ranks retrieved passages and a language model reads the top two. With DeepSeek Flash, that pipeline scores 50 of 50 against 46 of 50 for an agent that reads whole filings. The post says jev-latest resolved to Jev 1.13, and that search put the right passage first 40% of the time against 90% after Jev ranked the same candidates.
The X post is the write-up. The linked article has the per-model tables. The 40% figure matches the article’s keyword-plus-vector row, not every search method on that page.
David Arias
Published:
Source checked:
[90] 50 questions on four 10-Ks: Jev RAG and an agent loop
The article scores 50 analyst-style questions on the Apple, Microsoft, Nvidia and Amazon 10-Ks. Jev RAG with DeepSeek V4 Flash, thinking on, scores 50 of 50. The agent scores 46 of 50. The same passages sent to Qwen 3.7 Flash and GPT-5 nano score 49 and 48 of 50, against agent scores of 41 and 34. Code looks up the latest filing before Jev runs. A hand reading changed 13 automatic grades.
Read 28 September 2026. The page dates the tests to 27 and 28 September 2026. Jev is called as jev-latest through OpenRouter. The X post says that alias resolved to Jev 1.13. DeepSeek Flash runs on DeepSeek’s API. Qwen 3.7 Flash is qwen/qwen3.7-flash and GPT-5 nano is openai/gpt-5-nano, both through OpenRouter at provider defaults. For DeepSeek, the agent uses 4.7 calls, 82,908 tokens, 11.1 seconds and $0.0031 cached or $0.0133 uncached per question. Jev RAG uses 1 call, 840 tokens, 2.0 seconds and $0.0007. Keyword plus vector search is first on 20 of 50. Jev Choice is first on 45 of 50, and two shuffles score 46 and 44. One Jev call over about 12,000 tokens had a median of 0.5 seconds and cost about $0.0005.
Yubo Li, Yidi Miao, Ramayya Krishnan and Rema Padman
Published:
Source checked:
[102] Jev 1.13.0 as a judge next to GPT-6, Claude and Gemini
The paper scores jev-1.13.0 against hosted GPT, Claude and Gemini judges on the same Choice contract. On a 400-pair RewardBench sample Jev scores 92.2% and GPT-6 Astra 93.5%. Claude Sonnet 5’s primary scores are 91.2%, 88.9% and 85.8%, and four contract failures stay in that denominator. Gemini 3 Flash scores 92.8%, 76.0% and 83.3%. Gemini 3.1 Pro scores 94.5%, 87.4% and 87.1%.
arXiv:2609.26550v1, dated 22 September 2026. Jev is jev-1.13.0. GPT-6 Astra is gpt-6-astra at low effort with a JSON schema. Claude Sonnet 5 is claude-sonnet-5 and Gemini 3 Flash and 3.1 Pro are gemini-3-flash-preview and gemini-3.1-pro-preview, also at low effort. RewardBench here is 100 pairs from each of four categories, not the official subset weights. JudgeBench is the GPT-4o split. HaluEval is 240 evidence-grounded judgments. Existing-label accuracy is a fourth column on 150 replies: 94.0% for Jev, 94.7% for Sonnet 5, 95.3% for Flash, 94.7% for Pro and 96.7% for Astra. Invalid outputs count as errors in the primary accuracy. Sonnet 5 is valid on 1,308 of 1,312 items, all four failures HTTP 200 contract breaks. Table 6’s valid-only Sonnet 5 rates are 91.7%, 89.4% and 85.8%. Flash and Pro are valid on all 1,312. The isolated latency panel is 120 judgments, and the printed medians name Jev, GPT-4.1 mini and GPT-6. Fees use reported usage at collection-time prices and are not invoices. The single-order cascades read the threshold off the same items. The two-order policies were frozen earlier, on 510 extension pairs: Astra at 0.9 scores 92.5% against 93.1% at 56.8% of the fee; Sonnet 5 at 0.6 scores 88.0% against 89.2%; Flash at 0.6 scores 87.1% against 82.9%; Pro at 0.9 scores 89.6% against 90.4%. A human adjudicated disagreements with GPT-6. The second pass was an LLM, and an author resolved the 39 items where the two passes differed.
Assaf Elovic (@assaf_elovic)
Published:
Source checked:
[103] GPT Researcher replaces its embedding filter with Jev
Assaf Elovic says GPT Researcher replaced its embedding filter with Jev on 28 research tasks, that 73% of kept chunks were relevant against 46%, and that blind comparisons preferred the Jev reports 15 to 3 at the same cost. He says the project now runs on Jev by default.
Announcement post. The 15-to-3 figure omits the ties in the repository table. The task split, the writer model, the keyword fallback and the Jev alias are in the eval note at commit 2906e632.
Assaf Elovic
Published:
Source checked:
[104] GPT Researcher context-filter note at commit 2906e632
The note replays 28 saved research tasks, 20 from SimpleQA and 8 open-ended, and writes every report with gpt-5.4. Jev keeps 1,000-character chunks scoring at least 1.5 on a four-level usefulness rubric. Context precision is 0.73 against 0.46 for embeddings, and the blind comparison is 15 wins, 10 ties and 3 losses. Cost per report is $0.115 and $0.117.
Read at commit 2906e632. The precision judge is gpt-5.4-mini. A pairwise win counts only when gpt-5.4 prefers the same side in both orders. Jev’s filter time is 2.4 seconds at the median with 32 calls in flight, and 1.7 seconds at the default of 64. The client calls jev-latest unless JEV_MODEL is set, and the note does not record which version answered. select.py at the same commit uses Jev when a TypeSafe key is present and a BM25 keyword fallback otherwise. That keyword row is 19 of 20 on SimpleQA, precision 0.51, and 14 wins, 8 ties and 6 losses against embeddings, at $0.116. Its sign test is about p = 0.12. SimpleQA is 19, 18 and 20 of 20 for Jev, embeddings and no filter. The note says a one-question difference is noise. On the 8 open-ended questions, no filter wins 6 of the decided comparisons with Jev. Skipping the filter costs $0.192 per report.
Tony Gentilcore (@tonygentilcore)
Published:
Source checked:
[105] Four Glean checks, including a citation judge against GPT-5.6 Luna
Tony Gentilcore’s article names Aviral Singh, Manav Agrawal, and the accounts @eddiedzhou, @mr_cheu and @MatZhao. On one 1,448-word answer, Jev’s citation-coverage verdict changed on 1 of 28 paragraphs across three runs. GPT-5.6 Luna with no reasoning changed 7, and Luna at xhigh changed 5.
X article posted 28 September 2026. The citation timing is 6.6 seconds and $0.014 for Jev, against 81.3 to 85.5 seconds and $0.030 to $0.047 for Luna with no reasoning, and 227.4 to 259.7 seconds and $0.047 to $0.060 at xhigh. Jev’s time is sequential client wall time. The Luna times sum model-call durations. The article says the two systems used different support criteria and that there were no independent human labels. Query classification reports agreement with a production LLM, not a human label: 66.8% for Jev, 35.9% for base Laya and 74.5% for a fine-tuned Laya. The request count for that table is not in the text. A 751-entry routing replay uses three experts. The article says Jev was more accurate than the existing LLM router. The chart percentages are not transcribed here. The median speedup on 40 transfer calls is 8.1 times. Choice reranking on 4,855 queries gave identical scores to about 37 of 41 candidates. Using production order to break ties raised Recall@6 from 40.2% to 44.3%.
Daniel Liao (@Daniel_Bernath_)
Published:
Source checked:
[106] Jev judges aircraft pairs over the San Francisco Bay
Daniel Liao says a Jev judge watched about 900 live flights from an E2B sandbox, flagged risky pairs in 103 ms, and missed none. He says gpt-4o missed 56 in the same sky and Claude Opus 5.5 took 12 seconds to miss none.
The post links 7datumba/jev-tower. The 103 ms, 56 misses and 12 second figures match files in that repository. The false-positive counts and the fetch error are in those files, not in this post. The post’s 900 flights are not the 60-tick window used for the ladder.
Daniel Liao
Published:
Source checked:
[107] jev-tower logs at commit 20dcbbb
tower.mjs replays one 60-tick ADS-B window. A pair is a hazard when the closest approach is inside 3 nautical miles and 1,000 feet within 120 seconds. Jev, called as jev-latest, and the chat model both flag a pair when their probability is above 0.5. Beside gpt-4o, Jev averages 99 ms with 0 misses and 335 false positives. gpt-4o averages 2,356 ms with 56 misses and 32 false positives.
Checked at commit 20dcbbb. data/tower-gpt4o.json, data/tower-gpt4o-mini.json and data/tower-claude-opus-5-5.json are the three logs. Jev’s counts stay at 385 true positives, 335 false positives and 0 misses. Average latency on the three runs is 99, 103 and 119 ms. gpt-4o-mini averages 2,242 ms, with 160 misses and 22 false positives. claude-opus-5-5 averages 12,318 ms, with 0 misses and 174 false positives on the pairs it scored, and 378 true positives against Jev’s 385. The Opus log prints one fetch failure. The README table lists Jev at 103 ms and the OpenAI false positives as 1 and 0, which these files do not show. The resolved Jev version is not in the logs. The ground truth is the script’s geometry rule.
OpenAI Developers (@OpenAIDevs)
Published:
Source checked:
[108] Decisions API preview, powered by GPT-6 Luna
OpenAI’s post says the Decisions API, powered by GPT-6 Luna, takes a question and a fixed set of answers. The caller uses the returned selection to classify content, route a request, or choose an agent’s next action. The post says the API is in limited preview.
Announcement post on 29 September 2026. It does not state a price, a latency, or a comparison with Jev. Image input and the preview’s release plan are in the follow-up post.
OpenAI Developers (@OpenAIDevs)
Published:
Source checked:
[109] Decisions API accepts text or images in the preview
The follow-up says a Decisions API call can send text or images as context, for example a support request and the teams that could take it. The API returns a selection. Preview access is limited to selected API customers, and the post says a broader release is planned in the coming days.
Second post in the announcement thread. It links to OpenAI’s DevDay 2026 recap and does not give a price or a measured latency. The model name GPT-6 Luna is in the first post.
Dan Shipper
Published:
Source checked:
[110] Every’s early Decisions API checks against Jev
Dan Shipper reports two early checks of OpenAI’s Decisions API against Jev. On a text-only replay of computer tasks, Jack Cheng found 76 of 78 correct controls for Decisions and 73 of 78 for Jev, at typical times of 230 and 500 milliseconds. On conversation classification, Kieran Klaassen’s test was tied on accuracy, with medians of 161 and 309 milliseconds.
Every, 29 September 2026. The article says Decisions accepts images as well as text, and that this computer replay was text only. It says Decisions trailed Jev on Every’s broader evals and does not publish those scores, the Jev version, the thread count, or the task list. Pricing is unannounced. Shipper expects a launch in a few weeks. The post that names GPT-6 Luna is OpenAI’s, not this article.
Entendre (@EntendreAI)
Published:
Source checked:
[111] Jev picks a ledger account from two, three, or five options
Entendre tested Jev on Crypto Accounting Bench by asking it to choose among 2, 3, or 5 real accounts. Reported accuracy is 51.7%, 47.5%, and 48.3%. Successful calls averaged 635 ms. On the five-account list, confidence of at least 0.75 raised accuracy to 61.5% and covered 39 of 118 tasks.
First post in the 28 September 2026 thread. The post says the highest percentages in the confidence charts come from the smallest samples. It does not name a Jev version or say how many calls failed. The shortlist rules and the language-model comparison are in the next two posts.
Entendre (@EntendreAI)
Published:
Source checked:
[112] Every Crypto Accounting Bench shortlist contained the right account
Entendre says each option was a real ledger account, and the incorrect choices were plausible alternatives for that transaction. Every shortlist contained the correct account. The score is agreement with the journal entry the company recorded.
Second post in the thread. It points to the Crypto Accounting Bench write-up, the research paper, and a public dataset. The paper and dataset describe full journal entries. They do not contain this Jev shortlist run.
Entendre (@EntendreAI)
Published:
Source checked:
[113] Shortlist accuracy set beside full journal-entry rates
With five candidates, Entendre reports 48.3% for Jev, 46.3% for GPT-5.6 Luna, and 46.9% for DeepSeek V4 Flash. With two candidates, Jev’s 51.7% matches the recorded Claude Opus 5 rate in the post, above GPT-5.6 Sol at 51.4% and Claude Sonnet 5 at 49.4%.
Third post in the thread. Entendre says the language-model figures are account correctness when those models wrote full journal entries from the complete chart, while Jev received a shortlist that already contained the correct account. The paper’s Pass@3 table is a different metric.
Kareem Khattab, Omar Khattab and Mohamed Ibrahem
Published:
Source checked:
[114] Crypto Accounting Bench paper, arXiv:2609.14811
The paper evaluates full journal-entry reconstruction on 118 tasks from 7 organizations. Pass@3, a perfect rubric and every required gate on one of three attempts, is 47.46% for GPT-5.6 Luna, 50.00% for GPT-5.6 Sol, 45.76% for Claude Opus 5, 47.46% for Claude Sonnet 5, and 46.61% for deepseek-v4-flash-0731.
arXiv:2609.14811v1, submitted 13 September 2026. Mean Score is the paper’s primary metric. Pass@3 is the third column of Table 2 and matches Figure 3. The median chart is 569 accounts. The paper does not evaluate Jev. Entendre’s later shortlist percentages are not these Pass@3 figures.
Entendre Finance
Source checked:
[115] Public Crypto Accounting Bench dataset, 118 full-entry tasks
The dataset card describes 118 tasks whose target is the complete journal entry an organization posted, chosen from that organization’s chart of accounts. It repeats the paper’s best Mean Score of 77.43% and best Pass@3 of 56.78%. It does not include Entendre’s later Jev shortlist run.
Read 30 September 2026. The card says the public files are a transformed derivative and that the judge pipeline behind the paper’s headline numbers is not released. The deterministic scorer in the repository is described as a lower bound. No dataset revision hash is cited here.
Lucas Biagetti (@LucasBiagettiA)
Published:
Source checked:
[116] Banking77 text-classification benchmark updated with Jev
Lucas Biagetti says he updated his text-classification benchmark, which he now frames as a decision-model comparison, to include Jev and to fill some gaps. He writes that Jev wins zero-shot classification in his exploratory analysis.
Post on 1 October 2026. It does not give the table, the class counts, or the OpenAI model identifier. The reply in the thread attaches a chart. The numbers and the model pins are in the repository brief and manifest at commit efbfb42.
Lucas Biagetti
Published:
Source checked:
[117] Banking77 v1.2 brief, 56 quality conditions
The v1.2 brief reports accuracy for 14 method and budget variants at 5, 10, 15, and 20 Banking77 classes. Zero-shot Jev scores 94.00%, 92.25%, 89.83%, and 86.12%. The OpenAI zero-shot row scores 92.00%, 88.25%, 87.33%, and 85.00%.
Checked at commit efbfb42. The brief says each width uses 40 held-out examples per class, one seed, and that the cohorts overlap, so the widths are not independent samples. It reports 1, 2, and 3 missing Jev probability vectors at 10, 15, and 20 classes. It makes no cross-provider cost or latency claim. MiniLM plus logistic regression at 100 examples per class scores 98.00%, 96.50%, 95.83%, and 92.38%. The model identifiers are in the manifest, not in this brief.
Lucas Biagetti
Published:
Source checked:
[118] Banking77 v1.2 manifest: jev-1.13.0 and gpt-5-nano-2025-08-07
The manifest pins Jev to jev-1.13.0 and the OpenAI zero-shot row to gpt-5-nano-2025-08-07 with reasoning effort minimal. Both use zero labeled examples. The campaign seed is 42, the class counts are 5, 10, 15, and 20, and the test set has 40 examples per class.
Same commit as the v1.2 brief. The MiniLM row uses sentence-transformers/all-MiniLM-L6-v2 at revision 1110a243, frozen, with logistic regression. BERT is google-bert/bert-base-uncased for two epochs. The manifest does not store latency or a billed cost for the zero-shot rows.
Kenny Rogers
Published:
Source checked:
[87] Banking77 intents: Jev 1.13 and Claude Opus 5
OpenRouter ran the 3,080-utterance Banking77 test split through Jev 1.13 and Claude Opus 5. Accuracy is 81.0% and 84.4%, macro-F1 is 80.5% and 83.6%, median round trip is 175 ms and 2,266 ms, and the billed totals are $0.34 and $7.44.
Kenny Rogers, OpenRouter, 22 September 2026. Both models received the same one-line criteria, written from label names without looking at the test data. Opus ran at temperature 0 with reasoning off and a strict JSON enum. Eight requests were in flight. Latency is the client round trip through OpenRouter. The Opus system prompt was cached. The write-up says the uncached list rate would have been about $19 per 1,000 requests. Confidence at or above 0.99 covered 58% of utterances at 96.3% accuracy. Below 0.5 covered 3.5% at 29.6% accuracy. Sending confidence under 0.90 to Opus left 75.9% on Jev, at 84.0% accuracy and $0.69 per 1,000. That cascade table reuses the same 3,080 examples, and the write-up calls it an upper bound. Both models scored 0 of 40 on get_physical_card. One dataset, one afternoon. Opus was not run with reasoning on.
Tobias Deußer, Lorenz Sparrenberg and Rafet Sifa
Published:
Source checked:
[123] 37-dataset Jev evaluation, arXiv:2609.37647
The paper evaluates jev-1.13.0 zero-shot on 37 datasets, 346,009 requests. Against Qwen3.8-27B and Gemma-4-E4B scored on the same requests, Jev is ahead on 27 datasets, tied on one, and behind on 9, and ahead of Gemma on all 37. The evaluation split cost US$9.15.
arXiv:2609.37647v1, submitted 29 September 2026. Banking77 accuracy is 79.7%, 77.4%, and 67.0% on 3,076 examples. MMLU is 91.8%, 81.7%, and 67.4% on 14,042. Eleven of Jev’s leads over Qwen have non-overlapping bootstrap intervals, and none of Qwen’s nine leads do. Mean latency is 0.36 seconds at 32 concurrent requests. The whole study, including the pilot and extra analyses, is US$9.40. The reference models are not GPT, Claude, Gemini, Grok, Kimi, or DeepSeek. The harness and response archive are in the repository.
Tobias Deußer
Published:
Source checked:
[124] jev-benchmarking harness at commit 6bbdeb3
The README pins the evaluation to jev-1.13.0 and compares it with Qwen3.8-27B and Gemma-4-E4B on identical requests. It links the paper and the raw responses at doi.org/10.5281/zenodo.23039006. Offline scoring rebuilds each request and looks up the cached response.
Commit 6bbdeb3, 30 September 2026. The README says a dataset that changes on Hugging Face would no longer match the cache. LLM-AggreFact is gated. The open-model path needs a GPU. The response license allows research and evaluation use of the Jev responses and bars distillation and training a model to imitate Jev.
Arena.ai (@arena)
Published:
Source checked:
[127] Agent Arena thread on Jev Router
Arena evaluates a product it calls Jev Router by @typesafeai on more than 4,700 Agent Arena sessions. It says the router does not improve on the Pareto frontier: similar performance to DeepSeek V4.1 Flash (Max), 38% more cost, and 1.7 times the median model-request latency. A later post in the thread gives end-to-end medians of 6.18 seconds and 3.64 seconds.
Thread on 7 October 2026. The opening post names DeepSeek V4.1 Flash as the most picked model, with GPT-6.1 Sol and GPT-6 Luna also frequent, and steerability of +10% against +10.48 for Claude Opus 5.5 (High). The next posts give call shares of 28.1% DeepSeek V4.1 Flash, 23.5% GPT-6 Astra, 22.8% Sol and 11.1% Luna, a hypothetical leaderboard row of +3.6% net improvement at $0.155 per task, and P90 latency of 30.59 seconds against 14.26. The thread prints no API model id, no code, and no session export. It does not say the calls used OpenRouter’s typesafe/jev-router. Theo’s DeepSWE note and Christoph Görn’s 16-prompt log stay separate.
Connor Frank
Published:
Source checked:
[133] Independent evaluation of Jev on claim verification and LegalBench
Vals scores twelve systems on 400 claim-verification items built from 10-Ks and on a 396-question LegalBench slice. Jev 1.13.0 scores 0.975 on the claims, 390 of 400, tying GPT-6 Astra, GPT-6.1 Sol and GPT-6 Luna at $0.02 per 1,000 cases. On LegalBench it ranks last, with class-balanced accuracy 0.730 against 0.732 for Claude Haiku 4.5 and 0.932 for GPT-5.6 Terra.
Write-up dated 6 October 2026. Gemini 3.8 Flash wrote 160 paraphrases and contradictions; the labels come from the construction rule, and a person audited every item. LegalBench is 12 subtasks with 33 questions each, sampled from 1,808 available items, and the page says that slice is not representative of the full benchmark. At 32 judgments about one document, Jev’s median latency is 194 times lower than GPT-6 Astra’s and 241 times lower than Gemini 4 Argon’s. A 1% error budget, fit on the first 30% of claim items, automates 95% of the held-out 70% at 1.6% error. Native structured output, no JSON prompting. Intervals bootstrap whole filings and Bonferroni-correct eleven comparisons against Jev.
Vals AI (@ValsAI)
Published:
Source checked:
[132] Vals thread on a 400-item Jev claim check
The opening post says Vals tested TypeSafe’s Jev against 11 other models and matched GPT-6 Astra’s 97.5% accuracy on 400 claim-verification questions at about 1/500th the cost. A follow-up gives about $0.025 per 1,000 cases against $12.29 for Astra, and 194 times lower median latency at 32 judgments about one document.
Thread on 6 October 2026 linking the write-up. A later post in the thread says Jev ranked last on a 12-task LegalBench slice, and that a 1% error-budget threshold automated 95% of held-out claim-verification cases at 1.6% error. The write-up’s probability-priced table lists Jev at $0.02 per 1,000 cases and Astra at $18.00. The $12.29 figure is the thread’s Astra cost without that extra probability charge.
Steven Denney and Matthew DiGiuseppe
Published:
Source checked:
[134] Seven political-science replications, arXiv:2610.06625
The paper re-runs seven published political-science annotation and scaling tasks with Jev 1.13 against GPT-6 Luna and Qwen3.8-27B. Jev matches or comes close on the human-coded scores. Luna at OpenAI batch prices costs less on six of the seven primary tasks. One-at-a-time medians are 0.09 to 0.27 seconds for Jev and 0.75 to 1.04 seconds for Luna.
arXiv:2610.06625v2, 6 October 2026, 71 pages. v2 corrects author order in the metadata. Jev ran through OpenRouter except on the V-Dem recall task, which used TypeSafe’s endpoint. Qwen ran through OpenRouter on Parasail at FP8. Speed rows are 200 timed items per application, sent one at a time from the Netherlands on 29 and 30 September 2026. On Gilardi’s 2021 relevance tweets Jev scores 89.0% against Luna’s 91.1%; on the 2023 sample, 76.2% against 84.2%. Mean V-Dem correlation is 0.68 for Jev and 0.63 for Luna. On party-name comparisons, 74% of Luna’s probabilities sit below 0.01 or above 0.99, against 18% for Jev. The comments list no public code archive.
Artificial Analysis
Published:
Source checked:
[139] Artificial Analysis Decision Index v1.0 launch article
The 13 October 2026 launch article scores Jev 1.13 at 58 on Decision Index v1.0, ahead of GLiDE and Microsoft-Decision-1 at 54. Claude Haiku 5.5 (Low) scores 61 at 1.4 seconds per task and $184 per million tasks. GPT-6 Luna (Low) scores 55 at 1.5 seconds and $131. Jev’s median is 0.13 seconds and $42 per million tasks.
The index is an equal mix of AA-Omniscience-Decision (500 private five-option questions, accuracy and non-hallucination at 12.5% each), HLE-MultiChoice (513 text-only multiple-choice items), When2Call (3,423 NVIDIA tool-calling tasks after dropping 229 ambiguous ask-for-information items), and WildGuardTest (835 agreed rows, average of three F1 scores). Language models answer with structured JSON that names one option and supplies no probability. Jev scores 21 on HLE-MultiChoice, 35 accuracy and 35 non-hallucination on AA-Omniscience-Decision, 81 on When2Call, and 94 on WildGuardTest. Haiku 5.5 (Low) scores 24, 52, 73 and 93 on those four. Luna (Low) scores 22, 66, 77 and 89. The article says these are launch figures; the live leaderboard can change.
Artificial Analysis (@ArtificialAnlys)
Published:
Source checked:
[138] Artificial Analysis Decision Index launch thread
The opening post says Jev leads among decision models on Decision Index v1.0, and that language models such as Claude Haiku 5.5 still score higher at much higher cost and latency. The key-results post gives Jev 1.13 a score of 58, ahead of GLiDE and Microsoft-Decision-1 at 54, with Haiku 5.5 (Low) at 61, 1.4 seconds per task and $184 per million tasks against 0.13 seconds and $42 for Jev.
Thread on 13 October 2026 linking the launch article, methodology, and live leaderboard. Follow-up posts repeat the Pareto cost figures: Decider v1.1 27B at $17 per million tasks, Microsoft-Decision-1 at $31, and Jev 1.13 at $42. Celeris-1 Decision is 0.06 seconds per task. The thread says decision models are poorly calibrated on knowledge and reasoning, and relatively well calibrated on WildGuardTest and When2Call. Live scores can move after this launch snapshot.
Artificial Analysis
Published:
Source checked:
[140] Artificial Analysis Decision Index methodology
Version 1.0 weights four evaluations at 25% each. A decision model answers every question in one call. For a choice, Artificial Analysis scores the option with the highest probability. For a yes/no question it takes yes at probability 0.5 or more. Language models receive a fixed system prompt and must return JSON through structured outputs.
AA-Omniscience-Decision non-hallucination is 100 × (1 − hallucination rate), where hallucination is a wrong answer instead of “I don’t know”. HLE-MultiChoice uses the 513 text-only multiple-choice questions from cais/hle, scored as an exact letter match. When2Call uses NVIDIA’s multiple-choice test set after filtering 229 tasks. WildGuardTest keeps rows where all three annotators agreed on prompt harm, response harm and refusal, and drops questions that consistently trigger model refusals. Cost per million tasks is one pass through the index, divided by the combined task count, times one million.
Yuchen Ling, Shengcheng Yu, Zhenyu Chen and Chunrong Fang
Published:
Source checked:
[143] Jev action selection in ALFWorld and AndroidWorld, arXiv:2610.13092
The paper swaps gpt-5.6-luna action selection for jev-1.13.0 across five agent families, 4,500 executions, ALFWorld’s 134 valid_unseen tasks and AndroidWorld’s 116 task types, three runs each. ReAct-J succeeds on 96.02% of ALFWorld tasks, 1.49 points under the original agent. Reflexion-J succeeds on 98.01%, 1.74 points under. On AndroidWorld, Jev trails T3A by 13.22 points, AutoDroid by 2.30 and FastAgent by 18.10.
arXiv:2610.13092v1, 9 October 2026. All LLM decisions and auxiliary generation use gpt-5.6-luna. Jev accepts text-only observations. ALFWorld also has an LLM-Choice control that uses Luna with Jev’s candidate list; those gaps shrink to 1.00 and 0.00 points. Decision cost falls 66.1% to 78.4% and decision time 73.9% to 81.5%, from 4.13 to 8.13 seconds to 1.04 to 1.75 seconds. Mean task cost falls in every family. T3A’s cost per delivered success rises 7.7%, from 68.70 to 73.97 mUSD, with a 95% interval that includes zero. FastAgent’s time per success rises from 239.4 to 377.8 seconds. Experimental code is at anonymous.4open.science/r/jev-emp-5BF4/.
Jim Bennett
Published:
Source checked:
[145] Arize decision-model benchmark: Jev, OpenAI Decisions and six others
On 2,675 RAGTruth rows, Jev 1.13 scores 85.5% balanced accuracy at a tuned cutoff, against 81.5% for OpenAI Decisions on gpt-6-luna. Median latency is 0.08 seconds against 0.16, and cost per 1,000 is $0.045 against $0.090. The 95% intervals on accuracy touch, so Arize records a draw there and sends Jev through on speed and cost. On 655 Phoenix cases Jev scores 91.1% and OpenAI 90.0%. On 52 routing cases Jev is 49 of 52 and OpenAI 48 of 52.
Blog dated October 2026, posted on X on 7 October. OpenAI is the only model that does not accept Jev’s /v1/systemone layout; Arize wrote each case as labelled text sections and says a different layout could score differently. Accuracy wins require non-overlapping 95% intervals. Each model gets its own yes/no cutoff, fit on half the data and scored on the other half, same seed as Arize’s earlier Jev-as-a-judge work. An earlier Arize Opus 5 judge scored 86.6% balanced accuracy at $14.30 per 1,000. Reversing the hallucination question drops Liquid d1 from 0.92 ROC AUC to 0.17; Jev, Clef 27B and OpenAI stay near 0.93, 0.92 and 0.90 on both wordings.
Arize AI (@arizeai)
Published:
Source checked:
[144] Arize thread on eight decision models
The opening post says Arize benchmarked eight decision models on hallucination detection, 13 Phoenix evaluator suites, and agent routing. It writes that Jev, Clef 27B and Liquid sit within a point of each other on hallucination, and within the noise of an LLM judge that costs 60 to 400 times more.
Thread on 7 October 2026 linking the blog. A follow-up names Jev for a hosted API, Kev for self-hosting, and Clef 27B for open weights with a hosted option, and says to test the question as hard as the model. The thread itself does not print the RAGTruth percentages.
John Siciliano and Knut Melvær
Published:
Source checked:
[200] Sanity’s 0.5-second docs agent with Jev, Haiku and gpt-oss-120b
The write-up times four decisions that used to sit on an LLM. Entry choice is 282 ms for Jev against 1,328 ms for Haiku. Model routing is about 300 ms against 818 ms. MCP choice is 170 ms against 829 ms. Speculation while the user types removes 1,828 ms at enter. The median full answer is 0.5 seconds. Three of four finish under 0.9 seconds.
Engineering post dated 24 September 2026. The answering model in the demo is openai/gpt-oss-120b on Cerebras. The authors score 20 of 22 answers as fully grounded in the docs, with the first token in about 350 ms. Routing made answers cheaper, about 4.5 times, and slower at the median, so they turned routing off for the 0.5-second demo. Combined entry, model, and MCP questions took about 340 ms. The post calls the numbers indicative: one week of testing on one corpus. Jev is called as jev-latest in the sample request, and the sample response logs jev-1.13.0.
Even Westvang, Rostislav Melkumyan and Knut Melvær
Published:
Source checked:
[199] How Sanity built the @s company agent harness
Sanity’s @s harness treats each agent as a Studio document and asks Jev which published agent should handle a Slack message. At the time of writing, Jev chose among 66 agents plus one option for questions about @s itself. Confidence below 0.4 returns the three most likely agents for the user to pick. The state is the message plus up to the last 20 messages in the thread.
Engineering post last edited 22 October 2026. Follow-ups stay with the current agent unless the new question belongs elsewhere. Watched channels get a yes-or-no on whether the agent should reply. About 42 of 61 hackathon agents worked fully or partly; most failures were missing access, not routing. HAL, a meeting-hygiene agent, answered everyone when its description said “workstreams,” and the team tightened that description. The post names no Jev version and prints no router accuracy. Inference goes through OpenRouter with zero data retention. The post links an earlier 0.5-second docs-agent write-up.
knut (@kmelve)
Published:
Source checked:
[197] Sanity’s internal harness routes agents with Jev
Knut Melvær writes that Sanity uses Jev as the router in an internal agent harness that lets anyone at the company spin up a bespoke agent. He names co-authors Rostislav Melkumyan (@rostimelk) and Even Westvang (@even) and links an engineering post.
X post, 22 October 2026. The post does not print a Jev version, a routing accuracy, or a latency. The linked post is the @s harness write-up.
Wang Yang, Xinpeng Li, Yanyan Zhang, Chuang Ma, Shouren Wang, Jianzhe Wei, Liqin Ye, Xiaoran Cai, Mohsen Hariri, Vipin Chaudhary and Xiaotian Han
Published:
Source checked:
[210] What one-pass decision models cannot decide, arXiv:2610.24929
The paper evaluates jev-1.13.0 and 23 open Jev-style configurations. Retrieval and selection are perfect up to 20k tokens and 64 options. Final-state accuracy after sixteen updates of one user falls to 53 to 80%, because similar records of other users interfere. On 738 real items graded by how many thinking tokens Qwen3.8-27B needs, Jev answers 97.5% of the items that one forward pass of that model solves and 74% of the others.
arXiv:2610.24929v1, 18 October 2026. A stale one-line summary is followed even when the log contradicts it. Confidence flags Jev’s own errors (AUROC 0.83 on final-state items) and not errors inherited from a stale input (0.53). An offered hand-off option is taken on 1.6% of the real-task items and on 3 of Jev’s 40 errors. The 738-item set drops HelpSteer2 and 93 items whose thinking samples disagree with the gold label. Qwen3.8-27B is the reasoning reference, not a GPT API model. The paper does not score jev-1.14.0.
Eduardo Vellasques, Lionel Palacin and Al Brown
Published:
Source checked:
[225] Agentic Analytics Benchmark: ranking tables and labeling failures with Jev
On a 136-question development set over an 18-table warehouse, Jev ranks needed tables at mean nDCG@10 of 0.951, recall@10 of 0.991, and about 0.4 seconds per question, with every required table inside the top five for 84% of questions. A four-model development run of 544 attempts yields 277 labeled failures; wrong data selection is the largest bucket at 87 of 277. At a 0.65 threshold on “correct despite fail,” 18 answers look like grader errors. Feeding the ranking back as an in-loop hint leaves accuracy flat across 20 models: pooled net +12 on fired cells, 95% interval −18 to +42.
ClickHouse engineering blog dated 27 October 2026. A companion pilot on 250 public text-to-SQL cases with 75 to 710 tables reaches 87% recall@10 and finds every required table in 73% of cases, at a p90 of 3.4 seconds and about $0.50 for the run. The post does not name a Jev version. Code ships in ClickHouse/data-agent-mnist.
ClickHouse (@ClickHouseDB)
Published:
Source checked:
[224] ClickHouse thread on Jev table ranking
ClickHouse writes that it used TypeSafe’s Jev to label every failure on the Agentic Analytics Benchmark and to rank the tables each question needs, reporting 0.951 nDCG@10 in 0.4 seconds.
X post, 27 October 2026, linking the engineering blog. Detailed failure counts, the wide-catalog pilot, and the flat in-loop hint result sit in the article.
ClickHouse
Source checked:
[226] data-agent-mnist: optional Jev ranking and failure labeling
The harness README’s optional Jev section installs typesafe-sdk and points schema_retrieval.py, schema_retrieval_agent.py, and label_failure_modes.py at a TypeSafe key. Ranking scores each table with a yes/no judgment. The labeler reads a graded run and returns a failure sub-mode, a preventing rule, and a grader-error check. Apache 2.0.
Repository opened at commit 403f3e7 on 2 November 2026. The README says the tools are optional extras and name no fixed warehouse tables.