DeepSeek · Community reports
Jev + DeepSeek: Agent Arena routing, Vals costs and 10-K retrieval
Vals prices DeepSeek V4.1 Flash at $0.55 per 1,000 claim-verification cases with probabilities, against $0.02 for Jev 1.13.0. Arena still compares its Jev Router with DeepSeek V4.1 Flash (Max).
Compiled by jevmodel.ai. We read the cited posts and documentation. The measurements stay with their authors. We have not rerun them.
Moving intent recognition to Jev
LimboAI reports refactoring intent-recognition work in a project to use Jev alongside DeepSeek V4 Flash. The author is positive about the resulting speed and cost.
That is first-hand feedback about a division of work. The post includes no dataset, baseline, or billing totals, so it supports no percentage saving and no decisions-per-second figure.
Sources:[22] LimboAI
Pi, DeepSeek and Jev Ultrafast on 12306
Yanhua describes using Pi and DeepSeek with Browser Use’s Jev Ultrafast to look up train services on 12306. The account says the agent operated the site and organized the train results; a video accompanies the post.
The author calls the consumption low and gives no numerical cost and no DeepSeek version. The post is a search demonstration. It includes no ticket purchase or payment.
Sources:[23] Yanhua
How the browser project separates decisions from typing
Jev Ultrafast’s README explains the split: each observation produces an indexed table of browser elements. Jev selects an operation and a compatible target. A separate text model is called when the operation requires typing.
The text helper supports an OpenAI-compatible endpoint and documents configuration for DeepSeek, Gemini and GLM. Its checked default is Mercury 2.5. The post never names Mercury 2.5 as Yanhua’s text model. The project also requires an independent outcome check and lists browser features outside its current scope, including frames and uploads.
Sources:[24] Browser Use[23] Yanhua
A routing experiment shows why context matters
Jing Wang separately tested Jev as a router over a list containing DeepSeek, GLM, Kimi, GPT and Claude models. Adding Claude Opus 5 changed the selected model from DeepSeek Flash to Claude Sonnet 5, even though the task stayed the same.
Wang notes that the supplied context lacked model prices and performance details. The test includes no measured cheapest-model policy.
Sources:[21] Jing Wang
Sixty labeled emails, then a 10,000-email extrapolation
Matyas posted a table titled as an email-classifier benchmark of DeepSeek V4.1 Flash against Jev. Both sides score 60 of 60 on his labels. Median latency is 3.361 seconds versus 0.313 seconds. Average latency is 6.600 versus 0.348. The 95th percentile is 19.105 versus 0.465. Cost for those 60 calls is $0.01912 versus $0.00278. The table records zero API or schema errors on both sides. The table leaves the mailbox, the label rules, and the Jev version unnamed.
The post text then gives figures for 10,000 emails: about $3.19 versus $0.46, and about 9 hours 20 minutes versus 52 minutes, and calls the result 10.7 times faster and 85.5% cheaper. The 10.7 times matches the ratio of the two medians. The clock times match those medians multiplied by 10,000, and the dollar amounts match the table’s cost per 1,000 decisions multiplied by 10. The post shows no log of a 10,000-message run. V4.1 Flash, the name on this table, is a different label from the V4 Flash in LimboAI’s intent note.
Sources:[46] matyas[22] LimboAI
DeepSeek V4.1 Flash on the same rubric panels
Rao and Callison-Burch include DeepSeek V4.1 Flash as a third flash judge on the same nine panels and 5,003 pairs. Across the panels it costs 66 times Jev Choice’s $0.063 and takes 6,298 seconds, against 29 seconds for Jev. That is the longest of the three language-model runs. Three failed DeepSeek calls on the binary panels were recorded as UNMET.
The accuracy result is shared with Luna and Gemini: 8 of 27 paired comparisons separate, and a cascade on the recorded verdicts gains at most 1.5 points over the best single judge. DeepSeek was reached through OpenRouter. The cost table, including the lower concurrency on the Jev side, is on the benchmark page.
Jev sets the reasoning level for DeepSeek V4.1 Flash
pcparts001’s pi-jev-reasoning-router-lite, at commit 2215f7c, asks Jev one Choice before a Pi prompt: none, low, or high. The extension then changes only the thinking level of an allowlisted model. The README’s example is deepseek-v4.1-flash. The default route calls jev-latest on TypeSafe’s API, which the file equates with jev-1.13.0. A missing key, a timeout, or an unusable answer falls back to high. If the allowlist is empty, the extension does nothing.
The README’s 30-task benchmark is a pre-port run on a direct DeepSeek route, with always-high as the baseline. Corrected to the same max tokens, per-task cost is $0.2844 against $0.3352, a 15.2% reduction. Uncorrected, the pair is $0.2685 against $0.3193. Against always-low, routing costs $0.2845 against $0.2112, and answer quality is equal on that run. Spearman rho against difficulty is +0.504 (p = 0.0040, n = 30). The file says none was never returned, so the observed router was low versus high. It also limits those quality and cost figures to the direct DeepSeek route. The measured-results table lists decision latency of 409 to 856 ms, typically about 500 ms, added before the model request. A later provider table lists 156 to 432 ms on the TypeSafe route.
Sources:[88] pcparts001
Jev picks two 10-K passages, DeepSeek V4 Flash writes the answer
David Arias asked 50 questions about the Apple, Microsoft, Nvidia and Amazon 10-Ks. Code looks up the latest filing, keyword and vector search build a shortlist, and one Jev call ranks 30 passages. DeepSeek V4 Flash, with thinking on, then reads the top two and writes the answer. That pipeline scores 50 of 50. An agent using the same DeepSeek model to search EDGAR and read whole filings scores 46 of 50. The agent averages 4.7 model calls, 82,908 tokens, 11.1 seconds and $0.0031 cached or $0.0133 uncached per question. Jev RAG averages 1 call, 840 tokens, 2.0 seconds and $0.0007.
All four agent misses opened the previous year’s filing, even with the date in the prompt. Keyword plus vector search put a correct passage first on 20 of 50. Jev Choice did so on 45 of 50, and two shuffles of those candidates scored 46 and 44. A hand reading changed 13 automatic grades. The article calls Jev through OpenRouter as jev-latest. The X post says that alias resolved to Jev 1.13. One ranking call, about 12,000 tokens, had a median of 0.5 seconds and cost about $0.0005. The article treats four large-cap filings as too small to settle the comparison for other documents.
Sources:[89] David Arias[90] David Arias
An unfinished 231-task run under the DeepSeek Flash alias
The Jev Cookbook chapter at commit 546050a stores one snapshot, dated 26 September 2026, of 231 planned public tasks. The DeepSeek Flash alias returns 110 valid answers, all 110 correct, then stops. The file counts 54 errors and 67 tasks left unattempted. Fifty of the errors are empty probability parses, and the stop is 20 consecutive infrastructure errors. p50 and p95 on the timed requests are 1.40 and 2.09 seconds. Recorded spend is $0.020312. The alias does not say V4 or V4.1.
In the same file the Jev alias has 120 valid answers and 116 correct, with 110 tasks left after the run exceeded its reservation. Kimi K3 has 139 valid and 139 correct, with a row for every task and 92 errors. Conditional accuracy uses the scorable answers as the denominator. DeepSeek’s 110 of 110 leaves 67 tasks unattempted. The completed public-231 audit of Jev 1.13.0, 200 correct, is another run. The five-alias table is on the benchmark page.
A hook log that named deepseek-v4.1-flash
Christoph Görn’s Claude Code hook called OpenRouter’s typesafe/jev-router and asked for a JSON size label. On the first 16 prompts the answered_by field names deepseek/deepseek-v4.1-flash three times, openai/gpt-6-luna five times, and moonshotai/kimi-k3 once. Seven calls returned no JSON. He writes that the alias did not answer as TypeSafe’s Jev. The 9 successful calls averaged 2.99 seconds and cost about $0.0032 to size. That average is a different sample from the 700 to 1400 ms row in his 8-prompt table. The Claude report has both. This V4.1 Flash id is the string in the log. It is separate from the V4 Flash writer in the 10-K test and from the cookbook’s DeepSeek Flash alias.
A five-account shortlist beside DeepSeek V4 Flash
Entendre’s 28 September 2026 thread scores Jev at 48.3% when it picks among five real ledger accounts on the 118-task Crypto Accounting Bench. The post sets that figure next to 46.9% for DeepSeek V4 Flash and 46.3% for GPT-5.6 Luna. Jev chose from a shortlist that already contained the correct account. The language-model percentages are account correctness when those models wrote full journal entries from the complete chart.
The CAB paper’s Pass@3 on the full entries is 46.61% for deepseek-v4-flash-0731. Pass@3 counts a task when one of three attempts earns a perfect rubric and clears every required gate. The paper’s median chart size is 569 accounts. The public dataset does not include the Jev shortlist run. The paper’s model id is deepseek-v4-flash-0731. The post’s name for the comparison rate is DeepSeek V4 Flash. This is a different run from the V4 Flash writer in the 10-K test and from the cookbook’s DeepSeek Flash alias.
Sources:[111] Entendre[112] Entendre[113] Entendre[114] Kareem Khattab, Omar Khattab and Mohamed Ibrahem[115] Entendre Finance
Agent Arena: calling DeepSeek V4.1 Flash directly
Arena.ai’s thread on 7 October 2026 evaluates a product it calls Jev Router by @typesafeai, on more than 4,700 Agent Arena sessions. The opening post says similar performance to DeepSeek V4.1 Flash (Max) costs 38% more, with median model-request latency 1.7 times higher, and that the router does not improve on the Pareto frontier. A later post gives end-to-end medians of 6.18 seconds for the router and 3.64 seconds for DeepSeek V4.1 Flash (Max), and P90 figures of 30.59 seconds and 14.26 seconds. If the router were on the leaderboard, Arena says it would show +3.6% net improvement at a median of $0.155 per task, and would rank below DeepSeek V4.1 Flash. The thread does not print DeepSeek’s dollar cost or DeepSeek’s net-improvement percentage.
DeepSeek V4.1 Flash received 28.1% of model calls, the largest share in the follow-up. GPT-6 Astra received 23.5%, Sol 22.8%, and Luna 11.1%. The opening post’s frequent-choice list names GPT-6.1 Sol and GPT-6 Luna, and does not use the name Astra. Against this DeepSeek row, the router is ahead on steerability, +10.0% against −0.2%, and on praise versus complaint, +6.3% against +2.1%. DeepSeek is ahead on confirmed success, +8.3% against +2.2%, bash recovery, +7.8% against −0.7%, and tool hallucination, +0.4% against +0.2%. Arena’s closing post says calling DeepSeek V4.1 Flash directly gets similar performance at lower cost and latency. Four of the five most-used models sit on the Pareto frontier. The fifth is not named.
The thread prints no API model id, no code, and no session export. It does not say the calls used OpenRouter’s typesafe/jev-router. Theo’s DeepSWE note on that alias publishes no score, and during the bench he described the router as basically a DeepSeek 4.1 Flash router. Christoph Görn’s hook log names deepseek/deepseek-v4.1-flash on 3 of 16 prompts. Those three samples use different names and different sizes.
Sources:[127] Arena.ai[79] OpenRouter[81] Theo[82] Theo[99] Christoph Görn
Vals: DeepSeek V4.1 Flash on the 400-item claim check
Vals’s twelve-system claim check includes DeepSeek V4.1 Flash. The twelve systems score between 0.953 and 0.995 on 400 items. Jev 1.13.0 scores 0.975, 390 of 400, tying three GPT models. Probability-priced cost per 1,000 cases is $0.55 for DeepSeek, 81% above a label-only DeepSeek call, against $0.02 for Jev. On the 1% error budget, DeepSeek automates 96% of the held-out items at 0.8% error, inside the budget. Jev automates 95% at 1.6%. At many judgments about one document, Jev’s wall clock barely rises; the write-up plots DeepSeek in that sweep next to Jev, Haiku 4.5, Astra, and Argon. Arena’s Jev Router thread still uses DeepSeek V4.1 Flash (Max) as the comparison row on a different product and a different sample.
Sources:[133] Connor Frank[127] Arena.ai
Questions about Jev + DeepSeek
Which DeepSeek version do the Jev reports name?
LimboAI and David Arias both name DeepSeek V4 Flash. Arias also says thinking was on. Yanhua’s browser post names DeepSeek and leaves the version out. The cookbook snapshot uses the alias DeepSeek Flash and does not print V4 or V4.1. Matyas’s email table names V4.1 Flash. Christoph Görn’s hook log names deepseek/deepseek-v4.1-flash on three of 16 prompts sent to OpenRouter’s typesafe/jev-router. Arena’s thread names DeepSeek V4.1 Flash, and DeepSeek V4.1 Flash (Max) for the comparison row.
Sources:[22] LimboAI[23] Yanhua[90] David Arias[94] Datawhale[46] matyas[99] Christoph Görn[127] Arena.ai
What does DeepSeek do in the documented browser architecture?
The project supports configuring DeepSeek as the text helper for typing. Jev chooses an operation and an observed browser target. The checked repository default uses a different text model.
Sources:[24] Browser Use
Did the DeepSeek comparison sort 10,000 emails?
The table is 60 emails, 60 of 60 correct on the author’s labels for both models. The 10,000-email cost and time in the post scale the per-1,000 cost and the median latency. The post shows no log of a 10,000-message batch.
Sources:[46] matyas
Does the Pi extension’s 15.2% figure cover every DeepSeek session?
The README reports a 15.2% lower per-task cost on 30 pre-port tasks against always-high, after correcting to the same max tokens. Against always-low, routing cost more and answer quality was equal. The file limits those figures to the direct DeepSeek route.
Sources:[88] pcparts001
Did Jev RAG beat a DeepSeek agent on the 10-K questions?
On 50 questions about four 10-Ks, Jev RAG with DeepSeek V4 Flash scores 50 of 50 and the agent scores 46 of 50. Code looks up the latest filing before Jev runs. All four agent misses opened the previous year’s filing. A hand reading changed 13 automatic grades.
Sources:[90] David Arias
Did the DeepSeek Flash alias finish the 231 planned tasks?
In the 26 September cookbook snapshot it returns 110 valid answers, all correct, then stops with 67 tasks unattempted. Fifty errors are empty probability parses. The alias does not print a V4 or V4.1 version. Jev in that same file leaves 110 tasks unattempted.
Sources:[94] Datawhale
Did Jev beat DeepSeek V4 Flash at picking a crypto ledger account?
Entendre’s five-account shortlist score for Jev is 48.3%, set next to 46.9% for DeepSeek V4 Flash. Jev chose from a shortlist that already contained the correct account. The post says DeepSeek wrote full journal entries from the complete chart. The CAB paper’s Pass@3 for deepseek-v4-flash-0731 on those full entries is 46.61%.
Sources:[113] Entendre[112] Entendre[114] Kareem Khattab, Omar Khattab and Mohamed Ibrahem
Did Arena find that DeepSeek V4.1 Flash beat Jev Router?
Arena says the router ranks below DeepSeek V4.1 Flash (Max), at a router median cost of $0.155 per task, and that calling DeepSeek directly gets similar performance at lower cost and latency. The thread says the router costs 38% more. End-to-end medians are 6.18 seconds and 3.64 seconds. The thread does not print DeepSeek’s dollar cost or its net-improvement percentage, and it prints no API model id.
Sources:[127] Arena.ai
How did DeepSeek V4.1 Flash compare with Jev on Vals’s claim check?
Both sit in the 0.953 to 0.995 band on 400 items. Jev scores 0.975 at $0.02 per 1,000 cases. DeepSeek costs $0.55 per 1,000 when the call returns probabilities. On a 1% error budget DeepSeek automates 96% of held-out items at 0.8% error; Jev automates 95% at 1.6%.
Sources:[133] Connor Frank
Sources and reading notes
The summaries below are paraphrases. Each link opens the original source, including its surrounding context.
LimboAI (@limbopeng)
Published:
Source checked:
[22] Refactoring intent recognition with Jev and DeepSeek
LimboAI reports replacing intent-recognition work in a project with Jev alongside DeepSeek V4 Flash and describes a positive experience with speed and cost.
Qualitative personal feedback. No dataset, baseline, elapsed time or billing totals are given.
Yanhua (@yanhua1010)
Published:
Source checked:
[23] Searching train services with Pi, DeepSeek and Jev Ultrafast
Yanhua describes running Browser Use’s Jev Ultrafast with Pi and DeepSeek to look up train services on 12306 and organize the results.
Personal browser-automation demonstration. The post leaves out the DeepSeek version and any numerical cost, and it stops at the search.
Browser Use
Source checked:
[24] Jev Ultrafast action selection and text generation
The agent builds an indexed element table, asks Jev for an operation and compatible target, and calls a text model for TYPE_TEXT. The README describes an OpenAI-compatible helper configurable for Gemini, GLM and DeepSeek.
Project README. We did not rerun Yanhua’s setup. The checked README defaults to Mercury 2.5, requires independent outcome checks, and lists unsupported browser features.
Jing Wang (@jingwangtalk)
Published:
Source checked:
[21] A model-routing experiment sensitive to the candidate list
Wang experiments with a Jev router over DeepSeek, GLM, Kimi, GPT and Claude candidates. Adding Claude Opus 5 changes the selected model from DeepSeek Flash to Claude Sonnet 5. The author says price and performance information was missing from the supplied context.
A small exploratory test. A name on the candidate list shows what the router was offered. The test includes no finished integration for each name and no measured routing policy.
matyas (@Matyas_Jirsa)
Published:
Source checked:
[46] Sixty-email comparison with DeepSeek V4.1 Flash
The attached table compares DeepSeek V4.1 Flash and Jev on 60 emails. Both score 60/60. Median latency is 3.361 seconds versus 0.313 seconds, average latency 6.600 versus 0.348, and the 95th percentile 19.105 versus 0.465. Cost for the 60 is $0.01912 versus $0.00278, with zero API or schema errors on both sides.
The post text scales the same rates to 10,000 emails, about $3.19 versus $0.46 and about 9 hours 20 minutes versus 52 minutes. Those durations equal the median latency multiplied by 10,000. The post shows no log of that batch. The corpus, label rules, and Jev version are unnamed. DeepSeek V4.1 Flash is the image title. LimboAI’s note names V4 Flash, a different label.
Delip Rao and Chris Callison-Burch
Published:
Source checked:
[86] Jev Choice against three flash rubric judges
The paper compares Jev Choice with GPT-5.6 Luna, Gemini 3.8 Flash and DeepSeek V4.1 Flash on nine panels and 5,003 pairs, using the same criterion texts. Accuracy separates in 8 of 27 paired comparisons. Across the panels, Jev Choice cost $0.063 and finished in 29 seconds. Luna, DeepSeek and Gemini cost 29, 66 and 325 times as much, and took 859, 6,298 and 950 seconds.
arXiv:2609.29769, submitted 24 September 2026. Requests used the jev-latest alias. A same-day stability check logged jev-1.13.0. The main runs did not log the version. LLM judges are called once per criterion. On Jev’s most confident errors, 96.0% of the LLM verdicts repeat the wrong answer. A cascade replayed on the recorded verdicts gains at most 1.5 points over the best single judge with cross-fitted thresholds, and at most 2.0 with oracle thresholds. Jev’s concurrency was lower than the LLM judges’, so the wall-time ratios understate a same-concurrency comparison. DeepSeek’s three failed calls on the binary panels were recorded as UNMET.
pcparts001
Source checked:
[88] Pi extension that sets DeepSeek reasoning effort with Jev
The extension asks Jev whether the next Pi prompt needs none, low or high reasoning, then changes only that thinking level. The README’s example model is deepseek-v4.1-flash. On 30 tasks against always-high, the corrected per-task cost is $0.2844 versus $0.3352. Against always-low, routing costs $0.2845 versus $0.2112, with equal answer quality on that run.
Read at commit 2215f7c. The default route calls jev-latest on TypeSafe’s API, which the README equates with jev-1.13.0. The 30-task figures are from a pre-port benchmark on a direct DeepSeek route, and the README limits those quality and cost figures to the direct route. Spearman rho against difficulty is +0.504 (p = 0.0040, n = 30). The measured-results table lists decision latency of 409 to 856 ms. A missing key, a timeout, or a bad answer falls back to high. The README says none was never returned in that set, so the observed router was low versus high.
David Arias (@Davidariasfin)
Published:
Source checked:
[89] Jev ranks 10-K passages before a language model answers
David Arias describes 50 questions about four annual reports. Jev ranks retrieved passages and a language model reads the top two. With DeepSeek Flash, that pipeline scores 50 of 50 against 46 of 50 for an agent that reads whole filings. The post says jev-latest resolved to Jev 1.13, and that search put the right passage first 40% of the time against 90% after Jev ranked the same candidates.
The X post is the write-up. The linked article has the per-model tables. The 40% figure matches the article’s keyword-plus-vector row, not every search method on that page.
David Arias
Published:
Source checked:
[90] 50 questions on four 10-Ks: Jev RAG and an agent loop
The article scores 50 analyst-style questions on the Apple, Microsoft, Nvidia and Amazon 10-Ks. Jev RAG with DeepSeek V4 Flash, thinking on, scores 50 of 50. The agent scores 46 of 50. The same passages sent to Qwen 3.7 Flash and GPT-5 nano score 49 and 48 of 50, against agent scores of 41 and 34. Code looks up the latest filing before Jev runs. A hand reading changed 13 automatic grades.
Read 28 September 2026. The page dates the tests to 27 and 28 September 2026. Jev is called as jev-latest through OpenRouter. The X post says that alias resolved to Jev 1.13. DeepSeek Flash runs on DeepSeek’s API. Qwen 3.7 Flash is qwen/qwen3.7-flash and GPT-5 nano is openai/gpt-5-nano, both through OpenRouter at provider defaults. For DeepSeek, the agent uses 4.7 calls, 82,908 tokens, 11.1 seconds and $0.0031 cached or $0.0133 uncached per question. Jev RAG uses 1 call, 840 tokens, 2.0 seconds and $0.0007. Keyword plus vector search is first on 20 of 50. Jev Choice is first on 45 of 50, and two shuffles score 46 and 44. One Jev call over about 12,000 tokens had a median of 0.5 seconds and cost about $0.0005.
wang yiming (@wangyiming34110)
Published:
Source checked:
[93] A five-model snapshot of 231 planning tasks
The post points to chapter 6 of the Jev Cookbook and says that, on one snapshot of 231 planning tasks, Jev produced 120 valid answers at a p50 of 0.77 seconds, while Kimi produced 139 valid answers, the most of the five models.
The post gives valid-answer counts and one latency figure. Stop reasons, correct-answer counts and version strings are in the cookbook chapter at commit 546050a.
Datawhale
Published:
Source checked:
[94] Jev Cookbook chapter 6 snapshot for 26 September 2026
The chapter records one run of five aliases on 231 planned public tasks. Kimi K3 has 139 valid answers and 139 correct. Jev has 120 valid and 116 correct, then leaves 110 tasks unattempted after a reservation stop. DeepSeek Flash has 110 valid and 110 correct, then stops after consecutive errors. Conditional accuracy uses only scorable answers.
Read at commit 546050a. snapshot.json in the same commit matches the counts. Recorded spend is $0.001986 for Jev, $0.011435 for Kimi K3 and $0.020312 for DeepSeek Flash. p50 latency is 0.77, 9.63 and 1.40 seconds. The aliases do not include a full version string. Hard-tier valid answers are 1, 29 and 2 of 111. The stored composite is highest for Jev, and the chapter says that score describes this incomplete run. Raw prompts and provider responses are not in the repository. This run is separate from the completed 231-task audit in the Open-Jev note.
Zefan Cai
Published:
Source checked:
[92] Public JevBench note at Open-Jev commit 3308a15
The note scores six streams on all 231 available public tasks. Jev 1.13.0 is correct on 200. GPT-5.6 Luna, reasoning effort none, is correct on 206. GPT-6 Astra, reasoning effort low, is correct on 231. Easy tasks are 48 of 48 for each of those three. On the original 72 the counts are 71, 69 and 72. On the hard 111 they are 81, 89 and 111.
Read at commit 3308a15. The other 303 of 534 tasks were unavailable, and the note reports no 534-task score. Open-Jev and Jev return native probabilities. GPT returns verbalized probabilities in JSON. Candidate order differs on 119 of 139 Choice tasks. One Jev vector was renormalized. P50 full-response times, one observation per task, are 291.3 ms for Jev, 953.8 ms for Luna and 2,206.4 ms for Astra. The note says that sample is not a repeated latency test. Tariff estimates are $0.03842, $0.17914 and $8.81978 per 1,000 decisions, and the note says they are not billing receipts. Brier means are 0.1811, 0.2074 and 0.0085. Top-label ECE is 0.0318, 0.0932 and 0.0149. Open-Jev 27B v1.1 is 197 of 231, and its direct-execution timings are left out of the latency table. Upstream tasks are pinned at fstandhartinger/jevbench commit f8ce713.
Christoph Görn (@goern)
Published:
Source checked:
[98] A Claude Code router timed against local Kev
Christoph Görn says a local Kev-4B router on an RTX 4090 was 10 to 15 times faster than the cloud sizer he called Jev, 100 to 180 ms against 700 to 1400 ms, and that Kev’s accuracy was 14%. The post links his write-up.
The 14% figure is Kev’s score in the write-up, on 8 prompts. The cloud side of that table is OpenRouter’s typesafe/jev-router alias, and the write-up says the log named other models. The 16-prompt latency average is a different sample.
Christoph Görn
Published:
Source checked:
[99] What answered the Claude Code sizing hook
The hook calls OpenRouter’s typesafe/jev-router and asks for a JSON size label. On the first 16 prompts the log counts 3 tiny, 1 everyday, 1 large, 4 left on the main session, and 7 errors. The answered_by field names gpt-6-luna five times, deepseek-v4.1-flash three times, and kimi-k3 once.
Read on 28 September 2026. The 9 successful calls average 2.99 seconds, from 1.26 to 4.41, and sizing cost about $0.0032. A separate 8-prompt table scores Kev-4B at 1 of 7 and the alias at 1 of 5, with latencies of 100 to 180 ms and 700 to 1400 ms. The author says the router code is in a private infra repository and that the router log has no successful Kev decision yet.
Entendre (@EntendreAI)
Published:
Source checked:
[111] Jev picks a ledger account from two, three, or five options
Entendre tested Jev on Crypto Accounting Bench by asking it to choose among 2, 3, or 5 real accounts. Reported accuracy is 51.7%, 47.5%, and 48.3%. Successful calls averaged 635 ms. On the five-account list, confidence of at least 0.75 raised accuracy to 61.5% and covered 39 of 118 tasks.
First post in the 28 September 2026 thread. The post says the highest percentages in the confidence charts come from the smallest samples. It does not name a Jev version or say how many calls failed. The shortlist rules and the language-model comparison are in the next two posts.
Entendre (@EntendreAI)
Published:
Source checked:
[112] Every Crypto Accounting Bench shortlist contained the right account
Entendre says each option was a real ledger account, and the incorrect choices were plausible alternatives for that transaction. Every shortlist contained the correct account. The score is agreement with the journal entry the company recorded.
Second post in the thread. It points to the Crypto Accounting Bench write-up, the research paper, and a public dataset. The paper and dataset describe full journal entries. They do not contain this Jev shortlist run.
Entendre (@EntendreAI)
Published:
Source checked:
[113] Shortlist accuracy set beside full journal-entry rates
With five candidates, Entendre reports 48.3% for Jev, 46.3% for GPT-5.6 Luna, and 46.9% for DeepSeek V4 Flash. With two candidates, Jev’s 51.7% matches the recorded Claude Opus 5 rate in the post, above GPT-5.6 Sol at 51.4% and Claude Sonnet 5 at 49.4%.
Third post in the thread. Entendre says the language-model figures are account correctness when those models wrote full journal entries from the complete chart, while Jev received a shortlist that already contained the correct account. The paper’s Pass@3 table is a different metric.
Kareem Khattab, Omar Khattab and Mohamed Ibrahem
Published:
Source checked:
[114] Crypto Accounting Bench paper, arXiv:2609.14811
The paper evaluates full journal-entry reconstruction on 118 tasks from 7 organizations. Pass@3, a perfect rubric and every required gate on one of three attempts, is 47.46% for GPT-5.6 Luna, 50.00% for GPT-5.6 Sol, 45.76% for Claude Opus 5, 47.46% for Claude Sonnet 5, and 46.61% for deepseek-v4-flash-0731.
arXiv:2609.14811v1, submitted 13 September 2026. Mean Score is the paper’s primary metric. Pass@3 is the third column of Table 2 and matches Figure 3. The median chart is 569 accounts. The paper does not evaluate Jev. Entendre’s later shortlist percentages are not these Pass@3 figures.
Entendre Finance
Source checked:
[115] Public Crypto Accounting Bench dataset, 118 full-entry tasks
The dataset card describes 118 tasks whose target is the complete journal entry an organization posted, chosen from that organization’s chart of accounts. It repeats the paper’s best Mean Score of 77.43% and best Pass@3 of 56.78%. It does not include Entendre’s later Jev shortlist run.
Read 30 September 2026. The card says the public files are a transformed derivative and that the judge pipeline behind the paper’s headline numbers is not released. The deterministic scorer in the repository is described as a lower bound. No dataset revision hash is cited here.
Arena.ai (@arena)
Published:
Source checked:
[127] Agent Arena thread on Jev Router
Arena evaluates a product it calls Jev Router by @typesafeai on more than 4,700 Agent Arena sessions. It says the router does not improve on the Pareto frontier: similar performance to DeepSeek V4.1 Flash (Max), 38% more cost, and 1.7 times the median model-request latency. A later post in the thread gives end-to-end medians of 6.18 seconds and 3.64 seconds.
Thread on 7 October 2026. The opening post names DeepSeek V4.1 Flash as the most picked model, with GPT-6.1 Sol and GPT-6 Luna also frequent, and steerability of +10% against +10.48 for Claude Opus 5.5 (High). The next posts give call shares of 28.1% DeepSeek V4.1 Flash, 23.5% GPT-6 Astra, 22.8% Sol and 11.1% Luna, a hypothetical leaderboard row of +3.6% net improvement at $0.155 per task, and P90 latency of 30.59 seconds against 14.26. The thread prints no API model id, no code, and no session export. It does not say the calls used OpenRouter’s typesafe/jev-router. Theo’s DeepSWE note and Christoph Görn’s 16-prompt log stay separate.
OpenRouter (@OpenRouter)
Published:
Source checked:
[79] OpenRouter announces typesafe/jev-router
OpenRouter introduced typesafe/jev-router as a cache-aware router that uses Jev to pick a model and a reasoning effort for each request, with the stated aim of balancing quality, speed and cost.
Product announcement. The post describes the router. It does not include a score, a task count, or a price for the routed models.
Theo (@theo)
Published:
Source checked:
[81] An in-progress look at Jev Router’s model mix
While the DeepSWE bench was still running, Theo wrote that Jev Router was so far basically a DeepSeek 4.1 Flash router.
Interim note, posted hours before the final DeepSWE comparison. The post does not give a percentage split across models.
Theo (@theo)
Published:
Source checked:
[82] DeepSWE run of Jev Router against GPT-6 Astra on low
Theo says he spent $1,000 and stayed up until 2am benchmarking Jev Router. On DeepSWE, performance was roughly the same as GPT-6 Astra on low, the router cost slightly more, and the run took almost 5 times as long.
Author-reported comparison with the OpenRouter router. The post gives no DeepSWE score, no task count, and no dollar gap beyond “slightly more” and the $1,000 spent on the bench.
Connor Frank
Published:
Source checked:
[133] Independent evaluation of Jev on claim verification and LegalBench
Vals scores twelve systems on 400 claim-verification items built from 10-Ks and on a 396-question LegalBench slice. Jev 1.13.0 scores 0.975 on the claims, 390 of 400, tying GPT-6 Astra, GPT-6.1 Sol and GPT-6 Luna at $0.02 per 1,000 cases. On LegalBench it ranks last, with class-balanced accuracy 0.730 against 0.732 for Claude Haiku 4.5 and 0.932 for GPT-5.6 Terra.
Write-up dated 6 October 2026. Gemini 3.8 Flash wrote 160 paraphrases and contradictions; the labels come from the construction rule, and a person audited every item. LegalBench is 12 subtasks with 33 questions each, sampled from 1,808 available items, and the page says that slice is not representative of the full benchmark. At 32 judgments about one document, Jev’s median latency is 194 times lower than GPT-6 Astra’s and 241 times lower than Gemini 4 Argon’s. A 1% error budget, fit on the first 30% of claim items, automates 95% of the held-out 70% at 1.6% error. Native structured output, no JSON prompting. Intervals bootstrap whole filings and Bonferroni-correct eleven comparisons against Jev.