Jev + Gemini: sales-thread triage, a Jack & Jill screen and Argon

Business emails: Gemini had higher overall accuracy

Nikhil Mudholkar tested 1,565 German and English business emails across 10 categories. The set included 1,201 real emails and 364 AI-written examples for rare categories. All three models received the same email text and category instructions.

Reported overall accuracy was 96.4% for Jev, 97.5% for Gemini 3.5 Flash-Lite and 98.5% for Gemini 3.8 Flash. Equal weighting of categories widened the difference: 92.0%, 94.6% and 96.9%. Attachments were excluded. These scores describe that mixture of real and generated email, not every business inbox.

Sources:[9] nikhil mudholkar[10] nikhil mudholkar[11] nikhil mudholkar[14] nikhil mudholkar

Why the author still considered Jev useful

Mudholkar was interested in where Jev made mistakes. All 737 predictions with confidence at or above 99% matched the reference labels; below 70% confidence, nearly half were wrong. That observation gives a concrete reason to investigate a review or fallback path, without assuming those thresholds transfer to another dataset.

The author reports costs per 1,000 emails of $0.08 for Jev, $0.80 for Gemini 3.5 Flash-Lite and $1.79 for Gemini 3.8 Flash. He also notes that the dollar savings are small at normal inbox volumes. The experiment compares standalone models. It includes no measurement of a deployed Jev-to-Gemini handoff.

Sources:[12] nikhil mudholkar[13] nikhil mudholkar[9] nikhil mudholkar[2] TypeSafe AI

App reviews: faster labeling with a quality trade-off

Rahul Kumar’s Column Race evaluates sentiment, topic, bug reports and churn risk for 1,000 Android app reviews. Both lanes use batches of 20 and concurrency 8. The published run pair records Jev 1.13.0 at about 4.6 seconds and $0.023, versus Gemini 3.8 Flash at 18.8 seconds and $0.158.

The project’s quality checks favor Gemini slightly on sentiment correlation with star ratings: 0.822 versus Jev’s 0.803. Topic agreement is 84.4% and bug agreement is 92.6%. There are no human topic or bug labels, so those agreement figures only say the models matched each other. The project documents one recorded pair.

Sources:[15] Rahul Kumar[16] goodrahstar

Side-by-side tests, and a browser helper that can call Gemini

These two studies help identify work that might be routed or split. Neither measures a combined Jev + Gemini pipeline. Browser Use’s Jev Ultrafast README separately documents a text helper that can be configured for Gemini. In that architecture, Jev selects an operation and target, and a text model supplies text for typing.

The checked browser-agent configuration defaults to Mercury 2.5. The two classification studies are separate from that browser workflow, even where the browser README can be configured for Gemini.

Sources:[9] nikhil mudholkar[16] goodrahstar[24] Browser Use

Jev and Gemini on a product-configurator fan-out

On 25 September 2026 Nikhil Mudholkar tested Jev and Gemini on 300 customer requests against a fictional industrial catalogue. Eight linked dropdowns imply about 120 million combinations, of which 578,143 are valid. The agent asks later “what if” questions in the same call instead of waiting for each dropdown. He reports model calls falling from about 7 to 3. On the 270 requests he calls solvable, accuracy moves from 47.0% to 55.6% for Jev and from 52.2% to 62.6% for Gemini. Jev’s median time moves from 4.60 seconds to 1.89 seconds. The thread leaves out Gemini’s median time and both model versions.

Some fan-out answers met the catalogue and dropped a requirement the customer had stated. A later check, written on these same cases, stops 83 wrong Jev answers and 6 correct ones, against 72 wrong Gemini answers and 20 correct ones. Mudholkar describes that as about 14 mistakes stopped per correct answer lost for Jev, and 3.6 for Gemini. He also reports about one eleventh the cost per correct configuration once the check is included, without a price table. Gemini still leads on accuracy. He says the test is synthetic.

At about 300 questions in one call, Jev answered and Gemini rejected all 300 requests before generation. The input was about 55,000 tokens, inside a window of about 1 million. His configurator questions worked at 126 and failed at 127. Simpler yes/no questions reached 329. He says Jev answered 801 short questions in a separate email test. hev’s reply says their reranker sizing hit a batch-size limit too. That repository caps a call at about 30 passages and 32,000 tokens, which is a different limit from the 126/127 split.

Sources:[68] Nikhil Mudholkar[69] Nikhil Mudholkar[70] Nikhil Mudholkar[71] Nikhil Mudholkar[72] Nikhil Mudholkar[73] Nikhil Mudholkar[74] hev mind[76] hev

Metaphor labels, with Gemini left unnamed

Lieflat asked Jev, Claude Opus 5, Claude Opus 4.8, GPT-5.6 Terra, GPT-5.6 Luna and Gemini to mark awkward metaphors in three AI-written articles. The post leaves the Gemini version unnamed. On one article of 202 sentences, Jev labeled every sentence in 2.6 seconds. Gemini took 88.3 seconds. Opus 5 in non-reasoning mode took 8.2 seconds, and Terra and Luna each took more than 160 seconds.

The marks are the author’s. Of 10 sentences he counted as obvious metaphors, Jev marked none and Opus 5 marked 5. The sentence Jev scored as the most likely bad metaphor was a definition of the g factor. The author counts that sentence as a definition. A repeated ruler image scored 0.32, 0.31 and 0.30. Jev returned a score for every sentence. The other models returned only the sentences they flagged. The same 42-claim bench, on a different task, gives Gemini 3.1 Pro 81.0% overall and 55.6% on the 18 real sentences, unchanged when a scratchpad is allowed. That Gemini build is a preview, and the call went through OpenRouter.

Sources:[35] 躺在废墟里 Lieflat[39] Jamie Watters

People who sent the work back to another model

Melvyn writes that Jev did poorly on many of his tasks and that email classification was bad enough for him to switch that work back to Gemini. He gives no mailbox size, label set, or accuracy number.

The note answers Nevo David, who says classifiers built on Jev marked posts he wrote himself as AI-written, and gives no count. Serg, in the same thread, says the classifiers marked 70% of his posts as AI slop, also without saying how many posts that was. Nevo’s remark that Jev is 100x faster and cheaper has no test attached in the post.

Sources:[36] Melvyn[37] Nevo David[38] Serg

Stagehand act trials against Gemini 3.8 Flash

Kyle Jeong’s 17 September post shows one remote-browser task: an accessibility tree goes in as state, candidate actions go in as questions, and Jev picks the next step. He reports about $0.001 for that task. The thread uses jev-latest and leaves the page unnamed. It shows one trial.

His later Browserbase post describes the experimental Act flow. Jev classifies the instruction and picks a candidate. The post gives an acceptance threshold of 0.7, and Stagehand falls back to a language model when that pick is not accepted. He reports early-test median latency of 1.97 seconds to 0.46 seconds, and he writes that Jev-only computer-use agents were not ready for production in Browserbase’s trials.

The timings sit in an open pull request by miguelg719, read on 22 September 2026. The path is experimental and off unless experimentalJevAct or STAGEHAND_EXPERIMENTAL_JEV_ACT is set. Against Gemini 3.8 Flash, 40 act tasks score 39/40 on the baseline and 118/120 on three trials of the Jev arm. Median latency moves from 1.97 seconds to 0.46 seconds, and 4 of 147 acts call the language model. A 40-task breadth set that is not in the pull request moves from 35/40 to 39/40 and from 3.19 seconds to 0.77 seconds. Thresholds were tuned on these suites. With the language model switched off, act scores 27/40 and the breadth set scores 26/40. A companion observe pull request, also open and off by default, reports 9/12 to 10/12, with 11 of 16 observes answered by Jev in 0.18 seconds against a 1.99 second baseline. Its cache-check threshold of 0.35 is not covered by an end-to-end eval.

Sources:[41] Kyle Jeong[42] Kyle Jeong[43] miguelg719[44] miguelg719

Gemini 3.8 Flash on the same rubric panels

In the Rao and Callison-Burch rubric study, Gemini 3.8 Flash is one of three flash judges reading the same criterion texts as Jev Choice. Across nine panels, Gemini’s bill is 325 times Jev Choice’s $0.063, and its wall time is 950 seconds against Jev’s 29. On single panels the cost ratio runs from 121 on HealthBench, about 2 criteria per unit, to 770 on FED-Dialogue, with 10. The paper says Gemini is usually the best of the three language-model judges, and that Jev’s deficits on graded criteria are chiefly against Gemini.

Accuracy separates in 8 of the 27 paired comparisons, not in 8 comparisons against Gemini alone. Jev’s leads in that set fall mostly on binary criteria. A cascade that keeps Jev’s confident verdicts and replays the rest on the recorded language-model verdicts gains at most 1.5 points over the best single judge with cross-fitted thresholds. The language-model judges sent more concurrent requests than Jev, so the wall-time ratio is not a same-concurrency figure. The full cost table is on the benchmark page.

Sources:[86] Delip Rao and Chris Callison-Burch

EosHQ timed Jev against Gemini 3 Flash Preview

Kishore Mandyam’s post on 28 September 2026 links an EosHQ article about three decision steps: choosing a supplied label, choosing an Agent, and recognising an automation request. The comparator is Gemini 3 Flash Preview. The article does not name a Jev version. The measurements were collected that day, and the article says they are not a production latency guarantee.

Label selection is one observed request. The recorded call fell from 3.07 seconds to 0.455 seconds, and a separate timing of the Jev network request was 0.251 seconds. The article says the earlier Gemini path also dropped the supplied labels and wrote a rationale, so that pair includes a prompt and output change. Estimated model cost for 1,000 repetitions of the observed call is $1.0905 for Gemini and $0.0150 for the Jev path, from returned token counts and list prices.

Agent selection and automation detection used an isolated harness: six scenarios, three repetitions, and both providers, which the article counts as 72 synthetic tests across the two functions. The Jev path includes Gemini fallbacks. Agent selection moved from 3,380 ms to 1,616 ms, and the estimate per 1,000 from $1.293 to $0.526. Automation detection moved from 3,432 ms to 925 ms, and from $1.366 to $0.234. Jev-only requests averaged 365 ms and 350 ms. Fallbacks were 6 of 18 agent decisions and 3 of 18 automation decisions. The invoice-trigger example needed Gemini on all three repetitions and was slightly slower with Jev in front. An earlier skill-selection experiment, 2,636 ms to 477 ms, is a step those exact requests now skip. After that skip, one complete classification response moved from 3.29 seconds to 2.63 seconds. Gemini’s list price in the article is $0.50 per million input tokens and $3 per million output tokens, including thinking. Jev is $0.042 per million input tokens, with output free.

Sources:[100] Kishore Mandyam[101] EosHQ

Gemini 3 Flash and Gemini 3.1 Pro on the same judge contract

Yubo Li, Yidi Miao, Ramayya Krishnan and Rema Padman ask gemini-3-flash-preview and gemini-3.1-pro-preview for a label and label probabilities, at low effort under a JSON schema. Both return a valid contract on all 1,312 base items, so the valid-only rates match the primary rates. Flash scores 92.8% on the 400-pair RewardBench sample, 76.0% on JudgeBench’s 350 pairs, 83.3% on 240 HaluEval judgments, and 95.3% on 150 saved replies. Pro scores 94.5%, 87.4%, 87.1% and 94.7%. Jev 1.13.0 scores 92.2%, 78.6%, 87.5% and 94.0%. Macro-F1 on the existing labels is 0.939 for Flash, 0.923 for Pro and 0.923 for Jev.

The frozen Flash policy accepts Jev at a maximum label probability of 0.6. On 510 later preference pairs it accepts 90.8%, scores 87.1% against Flash alone at 82.9%, and the fee ratio is 0.281 on reported usage, 0.282 when missing usage is reserved. The cascade minus Flash is 4.1 points, interval 1.2 to 7.2. The Pro policy, at 0.9, accepts 53.7%, scores 89.6% against Pro alone at 90.4%, and the fee ratio is 0.594, or 0.635 with that reservation. The cascade minus Pro is −0.8 points, interval −1.8 to 0.2. These ratios are offline simulations. The paper’s printed latency medians on the 120-judgment panel name Jev, GPT-4.1 mini and GPT-6. EosHQ’s timings against Gemini 3 Flash Preview measure a different task and do not name a Jev version.

Sources:[102] Yubo Li, Yidi Miao, Ramayya Krishnan and Rema Padman

Gemini 4 Argon, announced without a Jev score

On 30 September 2026, Koray Kavukcuoglu announced Gemini 4 Argon. The post says the model is rolling out to trusted cyber defenders through Google’s Fairwind Program. Google says it is in the U.S. government’s voluntary pre-release access process, and that developers, enterprises, and consumers follow, starting with paid API customers and Google AI Ultra subscribers. The post gives no date and no API model id.

The introductory price is $2 per million input tokens and $10 per million output tokens. Cached input is priced at 95% off the input token price. After the introductory period, the post lists $4 per million input tokens and $20 per million output tokens. The output token limit is 1 million tokens, up from 64,000. The post describes that limit as room for one long trajectory. It does not state a separate context-window size.

Google reports 77.9% on DeepSWE v1.1 and calls that a new state of the art for real-world long-horizon software engineering. It reports 51.3% and first place on Zapier’s AutomationBench, 91.7% on LVBench, and a tie for first at 68% on the remediation portion of CWE-bench v1. It says Argon leads the Vals Index and does not print that score. The post does not mention Jev. Theo’s DeepSWE note on typesafe/jev-router publishes no score, so it is a separate report from the 77.9%.

Google also describes work inside the company. In one quantum example, Argon beat a published baseline by 40% in minutes. A team of Argon agents is credited with freeing more than 300 TiB of memory, with an estimate of 500 TiB to 1 PiB. On libgav1, the post says agents replaced 32,000 lines of SIMD in an existing Rust port and produced a memory-safe decoder 2.7 times faster than that port, with the same video output. For trusted defenders and Google’s own teams, the post says a version without cyber guardrails will be released. Through Wiz’s Scan for Good, it says the model found a critical vulnerability in healthcare software used by hospitals worldwide that earlier frontier models had missed. The post names no package, CVE, or patch.

Sources:[125] Koray Kavukcuoglu[82] Theo

One terminal post pairs Opus 5.5, Jev and Argon

Zorepad’s post on 1 October 2026 describes one terminal session. Claude Opus 5.5 on high effort plans and writes the code. Jev scores choices such as which file, which tool, and whether to retry or stop. Gemini 4 Argon is called three times and writes no code. The author places those Argon calls before the plan, when an error repeats, and before the session is done. Three agents run in parallel on medium effort. The author says the last 28 Jev decisions were 15 that ran in code and 13 sent back to Opus, and that Jev forked the task 2,395 times.

The task is a /search path whose p99 latency was 2.4 seconds, with one request firing 212 SQL queries. The author reports the p99 at 180 milliseconds afterward, 18 of 18 tests passing, and a pull request finished in 26 seconds for $0.47. The stated fix is 4 lines, after 581,000 tokens. The post attaches a 30-second video. It links no repository, log, invoice, or model identifier, and it does not say whether the Argon calls used Fairwind access. Google’s announcement the day before limits that rollout to trusted cyber defenders in the program.

Sources:[126] Zorepad[125] Koray Kavukcuoglu

A 100-turn routing bench against Gemma 4, not the Gemini API

David Chiu’s README at commit ed4a0ab, dated 4 October 2026, scores 100 turns: 20 conversations of 5 turns each. Every turn must get four decisions right: where the answer should come from, whether the scope changed, how to handle the request, and which source restriction carries forward. The bench does not retrieve documents or run tools. Hosted Jev 1.13.0 scores 61.4% on that all-four metric, with route 90.6%, scope change 85.8%, mode 91.6%, and restriction 84.4%. A self-hosted Gemma 4 31B call that writes one JSON object scores 77.0%, 90.0%, 87.0%, 92.0%, and 95.0%. Jev’s p50 is 749 ms and its p95 is 818 ms. The Gemma JSON row’s p50 is 2,293 ms and its p95 is 4,723 ms. The README says the 31B server also served live traffic, so that latency is only indicative.

Cygnet reads stock gemma-4-12B-it one token per question and scores 58.0%, with restriction 94.0%, route 79.0%, p50 265 ms, and p95 308 ms. The same readout on the 31B checkpoint scores 69.0%. Restriction stays at 95.0%. Scope change falls from 87.0% to 80.0%, and route from 90.0% to 86.0%. Each Cygnet row was run once. Cloudflare’s Clef 27B, served from the open weights with the model card’s systemone() function, scores 54.0%. Scope change is 86.0% and restriction 84.0%, close to Jev’s 85.8% and 84.4%. Route is 85.0% against Jev’s 90.6%. Clef was run twice and repeated the same answers. Clef-flash 9B scores 28.0%, with restriction at 51.0%. Cloudflare’s hosted API quota ran out partway through, so those rows are the self-hosted path. Ask-first accuracy is 43% for Gemma and for Clef, 31% for Jev, and the README says Cygnet on 12B does not ask first. Laya scores 0.0% and SemIf 24.0% on the all-four metric.

The original five systems were run 5 times. Clef-flash was run 5 times. The README says Cygnet was No. 1 on JevBench v1.5.4, and that those items are single-turn and about 700 input tokens, while this bench carries about 2,500 tokens of history. This Gemma 4 31B checkpoint is not Gemma-4-E4B in arXiv:2609.37647. None of these rows is a Gemini API model. Google’s Argon announcement still names no Jev score.

Sources:[131] David Chiu[123] Tobias Deußer, Lorenz Sparrenberg and Rafet Sifa[125] Koray Kavukcuoglu

203 sales threads against Gemini 3.8 Flash

Premsanth Rajamani’s Salesgear write-up, dated 30 September 2026, scores jev-1.13.0 and Gemini 3.8 Flash on 203 hand-written sales threads. The threads belong to a fictional seller with four products. Labels were fixed before either model saw them. The four categories are soft_deferral (61), hard_no (50), went_quiet (50), and not_sales (42). Sixty-eight threads are planned near-misses. Each thread goes to Jev as one request with six typed questions. Warmth, from 0 to 100, is computed in code from trigger clarity, seniority, and a come-back window. Date arithmetic stays in code. Gemini ran in JSON classification mode with low thinking. Claude helped draft the threads, so Gemini is the baseline.

results.json at commit 926e72e records Jev accuracy 0.9507, 193 of 203, and Gemini 0.9803, 199 of 203. On the 68 near-misses the figures are 0.8824 and 0.9412. Deferral recall is 0.9836 for both, 60 of 61. Precision at 10 is 0.9 for both, and both put thread m059, a travel out-of-office reply, in the top ten. Come-back window accuracy on real deferrals is 0.9180 for Jev and 0.7705 for Gemini. ECE is 0.0199 and 0.0230. Jev’s run cost $0.00906, $0.446 per 10,000 threads, median 0.346 seconds, 8 workers. Gemini’s run cost $0.25055, $12.34 per 10,000, median 2.188 seconds, 2 workers. The README’s 8-minute and 48-minute 10,000-thread estimates assume eight workers on both sides.

Jev missed 10 threads. Six were not sales mail. Three ended on the seller’s own follow-up. One real deferral was read as a no. Gemini missed four, including the same travel out-of-office and the same calendar decline. The write-up’s review rule sends answers under 75% confidence to a person: 15 threads, 7%, holding 6 of Jev’s 10 mistakes. The threads are synthetic. One run per model. One team wrote the labels. Mudholkar’s 1,565-email table and EosHQ’s Gemini 3 Flash Preview timings stay separate.

Sources:[135] Premsanth Rajamani[136] Salesgear[11] nikhil mudholkar[101] EosHQ

TypeSafe’s Jack & Jill screen in place of Gemini 3.1 Flash Lite

TypeSafe’s case study, dated 12 October 2026, says Jack & Jill replaced Gemini 3.1 Flash Lite on the initial scoring stage of a candidate-matching pipeline, within 10 days of the first test. Every open role pulled 6,000 candidates. A low-cost model scored that pool and kept a shortlist of 2,000 for a higher-power review. The quality test used 150 live roles, every past hiring-manager winner, and a sample of the remaining candidates weighted to represent the original 6,000. Jev kept 94.6% of those winners on the shortlist, against 93.9% for Gemini. TypeSafe reports a 90% confidence interval of −0.4 to +1.8 points and calls the gap not statistically significant. AUC is 0.933 against 0.924.

Median time to screen all candidates for a role is 10.3 seconds against 20.3. Cost per 1,000 candidates is $0.092 against $0.755. List prices in the table are Gemini $0.25 input and $1.50 output per million tokens, and Jev $0.042 input with output free. Savings at current volume are $265,000 a year. $500,000 is a projection that includes expected growth. Extra workflows, more than 15, sit outside those figures. One of them is a profile check that TypeSafe says dropped from about 3.6 seconds on Gemini to about 0.2 seconds. Matched roles at sign-up are billed at under $0.10 per candidate.

The page names no Jev version and links no public log. This is TypeSafe’s write-up of a customer test. It is a different Gemini pairing from Salesgear’s 203 synthetic threads and from Mudholkar’s email table.

Sources:[137] TypeSafe AI[135] Premsanth Rajamani[11] nikhil mudholkar

Vals: Gemini 4 Argon on the same 400 claims

Vals’s twelve-system claim check includes Gemini 4 Argon. On the 400 items the twelve systems land between 0.953 and 0.995, and Jev ties three GPT models at 390. Probability-priced cost per 1,000 cases is $6.46 for Argon against $0.02 for Jev. Asking Argon for probabilities cost 3% less than asking it for a label only. At 32 judgments about one long document, Jev’s median latency is 241 times lower than Argon’s. At 64 judgments Argon takes 65 seconds; Jev stays at 0.12 seconds. On the 1% error budget, Argon automates 100% of the held-out items at 0.8% error. Sweeping the threshold over all 400 items would have kept Argon, Opus 5.5, and Sonnet 5.5 at 100% coverage inside the budget. Google blocked Argon on one learned_hands_crime LegalBench item. Systems are scored on the items they answered. Koray Kavukcuoglu’s Argon announcement still names no Jev score.

Sources:[133] Connor Frank[125] Koray Kavukcuoglu

Questions about Jev + Gemini