Google · Community reports
Jev + Gemini: classification comparisons and routing considerations
Two published comparisons cover business emails and app reviews. Gemini leads on the reported quality checks; Jev costs less in those runs.
We checked the cited posts and documentation. Measurements are attributed to their authors and have not been independently rerun by this site.
Business emails: Gemini had higher overall accuracy
Nikhil Mudholkar tested 1,565 German and English business emails across 10 categories. The set included 1,201 real emails and 364 AI-written examples for rare categories. All three models received the same email text and category instructions.
Reported overall accuracy was 96.4% for Jev, 97.5% for Gemini 3.5 Flash-Lite and 98.5% for Gemini 3.8 Flash. Equal weighting of categories widened the difference: 92.0%, 94.6% and 96.9%. Attachments were excluded. These scores describe that mixture of real and generated email, not every business inbox.
Sources:[9] nikhil mudholkar[10] nikhil mudholkar[11] nikhil mudholkar[14] nikhil mudholkar
Why the author still considered Jev useful
Mudholkar was interested in where Jev made mistakes. All 737 predictions with confidence at or above 99% matched the reference labels; below 70% confidence, nearly half were wrong. That observation gives a concrete reason to investigate a review or fallback path, without assuming those thresholds transfer to another dataset.
The author reports costs per 1,000 emails of $0.08 for Jev, $0.80 for Gemini 3.5 Flash-Lite and $1.79 for Gemini 3.8 Flash. He also notes that the dollar savings are small at normal inbox volumes. The experiment compares standalone models; it does not measure a deployed Jev-to-Gemini handoff.
Sources:[12] nikhil mudholkar[13] nikhil mudholkar[9] nikhil mudholkar[2] TypeSafe AI
App reviews: faster labeling with a quality trade-off
Rahul Kumar’s Column Race evaluates sentiment, topic, bug reports and churn risk for 1,000 Android app reviews. Both lanes use batches of 20 and concurrency 8. The published run pair records Jev 1.13.0 at about 4.6 seconds and $0.023, versus Gemini 3.8 Flash at 18.8 seconds and $0.158.
The project’s quality checks favor Gemini slightly on sentiment correlation with star ratings: 0.822 versus Jev’s 0.803. Topic agreement is 84.4% and bug agreement is 92.6%. There are no human topic or bug labels, so those agreement figures cannot tell us which model is right. The project documents one recorded pair, not a broad model ranking.
Sources:[15] Rahul Kumar[16] goodrahstar
Comparison results and an actual integration are different evidence
These two studies help identify work that might be routed or split, but neither measures a combined Jev + Gemini pipeline. Browser Use’s Jev Ultrafast README separately documents a text helper that can be configured for Gemini. In that architecture, Jev selects an operation and target, and a text model supplies text for typing.
The checked browser-agent configuration defaults to Mercury 2.5. Configuration support for Gemini is not evidence that the two classification studies used it in this browser workflow.
Sources:[9] nikhil mudholkar[16] goodrahstar[24] Browser Use
Questions about Jev + Gemini
Did Jev beat Gemini on email accuracy?
No. Mudholkar reports 96.4% for Jev, 97.5% for Gemini 3.5 Flash-Lite and 98.5% for Gemini 3.8 Flash on that dataset. Jev’s lower reported cost and uncertainty signal motivated further interest.
Sources:[11] nikhil mudholkar[13] nikhil mudholkar[12] nikhil mudholkar
Are the app-review agreement scores accuracy scores?
No. They measure how often the models agree on topic and bug labels. The project has no human ground truth for those labels, and uses star ratings only as a weak proxy for sentiment.
Sources:[16] goodrahstar
Sources and reading notes
The summaries below are paraphrases. Each link opens the original source, including its surrounding context.
nikhil mudholkar (@nikhilmudholkar)
Published:
Source checked:
[9] Business-email classification against Gemini
Mudholkar compares Jev with Gemini on 1,565 German and English business emails in 10 categories. Gemini performs better on overall accuracy; the author is interested in Jev for its uncertainty signal.
An author-reported benchmark. It compares separate models, rather than measuring a combined Jev + Gemini pipeline.
nikhil mudholkar (@nikhilmudholkar)
Published:
Source checked:
[10] Email dataset composition and shared instructions
The dataset contains 1,201 real emails and 364 AI-written examples for rare categories, mostly in German. The three models receive the same email text and category instructions.
Methodology from the benchmark thread. The dataset is not entirely real correspondence.
nikhil mudholkar (@nikhilmudholkar)
Published:
Source checked:
[11] Email accuracy results for Jev and two Gemini models
Reported overall accuracy is 96.4% for Jev, 97.5% for Gemini 3.5 Flash-Lite and 98.5% for Gemini 3.8 Flash. Equal weighting of categories yields 92.0%, 94.6% and 96.9%, respectively.
Results against the author’s reference labels in this dataset. These are not general accuracy scores for the models.
nikhil mudholkar (@nikhilmudholkar)
Published:
Source checked:
[14] Attachments and synthetic examples in the email test
Attachments were excluded, and 364 emails were generated to match chosen labels. The author describes the lack of non-text input as a barrier to production use.
Limitations supplied by the experiment’s author. This source does not demonstrate processing email attachments.
nikhil mudholkar (@nikhilmudholkar)
Published:
Source checked:
[12] Where the email-classification mistakes occurred
The author reports that 737 predictions at 99% confidence or above matched the reference labels. Below 70% confidence, nearly half were wrong.
An observed relationship in one dataset, not a universal threshold or proof of error-free decisions.
nikhil mudholkar (@nikhilmudholkar)
Published:
Source checked:
[13] Reported cost per 1,000 business emails
The author reports $0.08 for Jev, $0.80 for Gemini 3.5 Flash-Lite and $1.79 for Gemini 3.8 Flash per 1,000 emails in these runs.
Workload-specific costs, not provider pricing or the cost of the whole 1,565-email dataset. The author notes that savings are small at ordinary inbox volumes.
TypeSafe AI
Source checked:
[2] Confidence, probabilities and fallback decisions
Choice and Score include confidence derived from their probability distributions. Noul does not have a separate confidence field. TypeSafe recommends choosing thresholds for the domain and consequences of an action.
Implementation reference. A confidence value is not a guarantee that an individual decision is correct.
Rahul Kumar (@rahulbuildsmore)
Published:
Source checked:
[15] Jev and Gemini labeling 1,000 app reviews
Kumar compares sentiment, topic, bug and churn labels on 1,000 app reviews. He reports Jev at 4.6 seconds and $0.023, versus Gemini 3.8 Flash at 18.8 seconds and $0.158.
The post links to a public project with recorded runs and measurement notes. It is a side-by-side comparison, not a combined workflow.
goodrahstar
Source checked:
[16] Column Race recorded runs and evaluation methodology
Both lanes process 20 reviews per request at concurrency 8. The recorded pair uses Jev 1.13.0 and Gemini 3.8 Flash. Sentiment correlation with star ratings is 0.803 and 0.822; topic agreement is 84.4% and bug agreement is 92.6%.
One recorded pair over 1,000 Android reviews. Stars are a weak sentiment proxy; topic and bug labels have no human ground truth. Agreement is not accuracy. Costs use token counts and list prices, not invoices.
Browser Use
Source checked:
[24] Jev Ultrafast action selection and text generation
The agent builds an indexed element table, asks Jev for an operation and compatible target, and calls a text model for TYPE_TEXT. The README describes an OpenAI-compatible helper configurable for Gemini, GLM and DeepSeek.
Project documentation, not a reproduction of Yanhua’s setup. The checked README defaults to Mercury 2.5, requires independent outcome checks, and lists unsupported browser features.