Independent community reports · Updated
Jev AI Model: Benchmarks, Workflows and Comparisons
Jev is TypeSafe’s decision model: you pass options or a scoring rubric, and it returns a typed answer with a probability or a confidence score. This site collects independent reports on how it behaves: benchmarks, image handling, and comparisons with GPT, Claude and Gemini.
jevmodel.ai is an independent reports site. It is not TypeSafe AI. Official product, docs, and console:typesafe.ai.
Compiled by jevmodel.ai from the cited posts and documentation. About this site.
Can Jev process images?
Hosted Jev takes text by default. On 14 October 2026 TypeSafe documented image files behind TypeSafe-Danger-Zone-Images: true, for jev-1.13.0 and jev-1.14.0. Community reports also OCR a screenshot first.
Read the images reportReports on Jev workflows and comparison tests with seven language-model families, plus image workflows that keep the pixels outside the Jev call. On 14 October 2026 TypeSafe shipped Jev 1.14.0 for Jev Day, with image input, a Box type, and 262k context behind danger-zone headers. The models page, accessed 26 October 2026, points both jev-latest and jev-preview at jev-1.14.0. ClickHouse’s 27 October Agentic Analytics post ranks warehouse tables with Jev at 0.951 nDCG@10 in about 0.4 seconds, labels 277 failures on a four-model development run, and finds that feeding the ranking back as an in-loop hint leaves accuracy flat across 20 models. Takuma Yoshida scores jev-1.14.0 at 78 of 100 fictional return emails against 91 for Claude Haiku 5.5. Sanity’s docs agent answers with gpt-oss-120b after Jev picks the knowledge-base entry. Lightpanda’s judge() scores a loaded page in about 0.2 seconds. UiPath shipped a TypeSafe connector in preview. The Jevtember page records Vercel’s September AI Gateway figures: about 13% of paid teams on day one, 35.3% two-week reach, and September’s highest request count. Community skills install Jev inside Muse and Muse Code; the cited Muse workflows date from 19 September to 1 October 2026 and name Muse or Muse Code. Artificial Analysis scores Jev 1.13 at 58 on Decision Index v1.0, with Claude Haiku 5.5 (Low) at 61 and GPT-6 Luna (Low) at 55. A Zenn check on MASSIVE, chABSA and JNLI finds Opus 5.5 ahead of jev-latest only on the inference set. Ling, Yu, Chen and Fang swap gpt-5.6-luna action selection for jev-1.13.0 across 4,500 ALFWorld and AndroidWorld runs; household success stays close and two mobile agents fall. Arize’s eight-model audition puts Jev 1.13 at 85.5% on RAGTruth against 81.5% for OpenAI Decisions on gpt-6-luna. Author-reported misses and version differences stay next to the numbers.
8 model reports · 234 source notes
- Choice
- Selects from options supplied by the application.
- Score
- Evaluates a question against a supplied rubric.
- Noul
- Returns a probability for a statement.
Sources:[1] TypeSafe AI
Jev vs Claude, GPT, Gemini and Other Models
A runtime handoff, a coding assistant, and a comparison test put two models next to each other in different ways. Each report says which of those the sources describe.
Jev + Images
Hosted Jev takes text by default. Image files need TypeSafe-Danger-Zone-Images on jev-1.13.0 and jev-1.14.0. The reports below also cover OCR first, filenames, pixel-color questions, and a drawing step gated by Jev.
Jev + GPT / Codex
ClickHouse ranks warehouse tables with Jev at 0.951 nDCG@10 in about 0.4 seconds. Sanity’s docs agent still answers with openai/gpt-oss-120b after Jev picks the knowledge-base entry.
Jev + Claude
Takuma Yoshida scores jev-1.14.0 at 78 of 100 fictional return emails against 91 for Claude Haiku 5.5. Haiku 5.5 (Low) scores 61 on Artificial Analysis’s Decision Index, 3 points above Jev 1.13.
Jev + Gemini
Salesgear’s 203-thread test scores Gemini 3.8 Flash at 199 of 203 and Jev 1.13.0 at 193. TypeSafe’s Jack & Jill case study reports replacing Gemini 3.1 Flash Lite on an initial candidate screen.
Jev + Grok
A 1,000-post selection workflow and a separate action-router setup, with source links and no inferred speed or cost claims.
Jev + Kimi
A measured email handoff to Kimi K3, a SemEval chart against Kimi-K2 Thinking, one snapshot where a Kimi K3 alias returned 139 valid answers of 231, and a hook log that named kimi-k3 once.
Jev + DeepSeek
Vals prices DeepSeek V4.1 Flash at $0.55 per 1,000 claim-verification cases with probabilities, against $0.02 for Jev 1.13.0. Arena still compares its Jev Router with DeepSeek V4.1 Flash (Max).
Jev + Muse Wonder
Meta released Muse Wonder on 12 October 2026. The cited Muse workflows install Jev as a typed judge inside Muse or Muse Code. Those posts and READMEs date from 19 September to 1 October 2026 and name Muse or Muse Code.
Jev Benchmarks: 43 Independent Tests
The cited authors measured these workloads. The tables keep their sample sizes, and a first-pass time stays separate from the full pipeline. We have not rerun the experiments.
Business-email classification
Nikhil Mudholkar: 1,565 emails across 10 categories, including 1,201 real emails and 364 AI-written examples.
Labeling 1,000 app reviews
Rahul Kumar’s Column Race: sentiment, topic, bug and churn labels for 1,000 Android reviews. Both lanes use batches of 20 at concurrency 8.
Email screening with a Kimi K3 fallback
Hassan: 100 emails, evenly split between legitimate and fraudulent examples. Jev sends 31 cases below 95% confidence to Kimi K3.
Does the cited passage support the sentence?
Jamie Watters, jev-bench: 42 labeled claims, of which 24 are constructed controls and 18 are real published sentences. One schema-only run per model on 20 September 2026, and a scratchpad rerun of the three frontier models on 21 September.
SemEval-2026 aspect sentiment
Eastwood: all ten SemEval-2026 DimABSA subtask 1 test sets. The aspect is given. Scores are official RMSE_VA, equal-weighted across the ten sets. Lower is a smaller error.
Stagehand act trials against Gemini 3.8 Flash
miguelg719, open pull request 2953, read 22 September 2026. Experimental Jev act path plus Gemini 3.8 Flash fallback, compared with the existing Gemini 3.8 Flash act path. Off unless experimentalJevAct is set.
Sixty-email classifier against DeepSeek V4.1 Flash
matyas: a 60-email table attached to the post. Both columns use the author’s labels. The corpus and the Jev version are unnamed.
Product-configurator fan-out, Jev and Gemini
Nikhil Mudholkar, 25 September 2026: 300 synthetic customer requests. Accuracy is on the 270 requests he calls solvable. Neither model version is named.
Reranking one BM25 shortlist with Jev, Claude and GPT-5.6 Luna
hev, RESULTS.md generated 17 September 2026, commit 1eb4726. jev-1.13.0 permutes the same BM25 top-30 shortlist as the other systems. SciFact and FiQA have 300 test queries. NFCorpus has 323. The score is nDCG@10.
Rubric judging against Luna, Gemini 3.8 Flash and DeepSeek V4.1 Flash
Delip Rao and Chris Callison-Burch, arXiv:2609.29769, submitted 24 September 2026. Nine panels, 5,003 pairs. Jev Choice, GPT-5.6 Luna, Gemini 3.8 Flash and DeepSeek V4.1 Flash read the same criterion texts. Language-model judges are called once per criterion.
Banking77 intents against Claude Opus 5
Kenny Rogers, OpenRouter, 22 September 2026. The Banking77 test split: 3,080 utterances, 40 for each of 77 intents. Same one-line criteria, written from the label names. Opus ran with reasoning off.
OpenRouter Jev Router on DeepSWE
Theo, 26 September 2026. One author-reported DeepSWE run of typesafe/jev-router against GPT-6 Astra on low. He spent $1,000 on the bench.
Reasoning effort for DeepSeek on 30 tasks
pcparts001, pi-jev-reasoning-router-lite at commit 2215f7c. A 30-task pre-port benchmark on a direct DeepSeek route. The extension asks Jev for none, low or high, then sets only the thinking level.
Public JevBench tasks against Luna and Astra
Zefan Cai, Open-Jev docs/jevbench-public.md at commit 3308a15. All 231 available public tasks: 72 original, 48 easy, 111 hard. Jev 1.13.0, GPT-5.6 Luna with reasoning effort none, and GPT-6 Astra with reasoning effort low. One pass, no retries.
Fifty 10-K questions: Jev RAG and an agent loop
David Arias, 27 and 28 September 2026. Fifty questions on the Apple, Microsoft, Nvidia and Amazon 10-Ks. Jev ranks a shortlist. The language model reads two passages. The agent searches EDGAR and can read whole filings.
Unfinished five-alias run on 231 planned tasks
Datawhale Jev Cookbook, chapter 6 at commit 546050a, snapshot dated 26 September 2026. One run. Five aliases. 231 planned public tasks. Conditional accuracy uses scorable answers only.
EosHQ decisions against Gemini 3 Flash Preview
EosHQ, 28 September 2026. Agent selection and automation detection are 18 Jev-enabled decisions each. Label selection is one observed request. The Jev path includes Gemini fallbacks. The article does not name a Jev version.
Claude Code hook log for the OpenRouter jev-router alias
Christoph Görn, 28 September 2026. Eight prompts from earlier sessions, two of each size. The cloud call is OpenRouter’s typesafe/jev-router, asked for a JSON size label. Kev-4B is a local model on an RTX 4090.
Jev 1.13.0 and hosted judges on one contract
Yubo Li, Yidi Miao, Ramayya Krishnan and Rema Padman, arXiv:2609.26550, 22 September 2026. Base-order accuracy. Invalid outputs count as errors. RewardBench is 400 pairs, JudgeBench 350, HaluEval 240.
GPT Researcher chunk filter before gpt-5.4
Assaf Elovic, eval note at commit 2906e632, 26 September 2026. 28 saved tasks, 20 from SimpleQA and 8 open-ended. gpt-5.4 writes every report. The client calls jev-latest. The note does not record the resolved version.
A 60-tick Bay Area conflict window
Daniel Liao, jev-tower at commit 20dcbbb. One recorded ADS-B window, 60 ticks. A hazard is a closest approach inside 3 nautical miles and 1,000 feet within 120 seconds. A flag is a probability above 0.5. Jev is called as jev-latest.
Citation-coverage changes on one Glean answer
Tony Gentilcore, 28 September 2026, with Aviral Singh and Manav Agrawal. One 1,448-word answer, checked three times. Jev’s time is sequential client wall time. The Luna times add up model-call durations.
Two early Decisions API checks against Jev
Dan Shipper, Every, 29 September 2026, reporting checks by Jack Cheng and Kieran Klaassen. OpenAI says the API uses GPT-6 Luna. The computer replay is text only. The article does not name a Jev version.
Ledger-account shortlists on Crypto Accounting Bench
Entendre, 28 September 2026. Jev chooses among 2, 3, or 5 real accounts on the 118-task CAB set. Every shortlist contains the correct account. The incorrect options are other plausible accounts from the same chart.
Banking77 subsets at four class counts
Lucas Biagetti, v1.2 brief at commit efbfb42. Zero-shot rows on held-out Banking77 subsets. Each width uses 40 examples per class and seed 42. The cohorts overlap. The OpenAI row is gpt-5-nano-2025-08-07 with reasoning effort minimal. Jev is jev-1.13.0.
jev-1.13.0 against Qwen3.8-27B and Gemma-4-E4B
Tobias Deußer, Lorenz Sparrenberg and Rafet Sifa, arXiv:2609.37647v1, submitted 29 September 2026. Zero-shot jev-1.13.0 on 37 datasets, 346,009 requests. Qwen3.8-27B and Gemma-4-E4B are scored on the same requests from next-token probabilities, with thinking off. They are the reference models on this card.
Gemini 4 Argon announcement figures
Koray Kavukcuoglu, 30 September 2026. Google’s announcement of Gemini 4 Argon. The post includes no Jev arm, no API model id, and no date past the Fairwind rollout to trusted cyber defenders.
Agent Arena thread on Jev Router
Arena.ai, 7 October 2026. A product the thread calls Jev Router by @typesafeai, measured on more than 4,700 Agent Arena sessions. The comparison row is DeepSeek V4.1 Flash (Max). The thread prints no API model id.
Tool or reply, with Sonnet 4.6 behind Jev
Josh Longenecker, Grounded AI, 8 October 2026. 200 items from GeneralFunctionCall-Test, 100 tool-call labels and 100 reply labels. The label is what Kimi K2-thinking did next. Jev is jev/jev-latest. The judge is bedrock/us.anthropic.claude-sonnet-4-6.
100-turn routing against Gemma 4, Cygnet and Clef
David Chiu, ywchiu/jev_benchmark at commit ed4a0ab, 4 October 2026. 20 conversations, 5 turns each. A turn counts only when route, scope change, mode and restriction are all correct. Nothing is retrieved and no tool runs. Gemma and Clef here are self-hosted. Jev is hosted Jev 1.13.0.
Vals 400-item claim verification
Connor Frank, Vals AI, 6 October 2026. 400 claim-verification items from 10-Ks of 48 companies, five construction rules of 80 items each. Twelve systems. Jev is jev-1.13.0. Gemini 3.8 Flash wrote 160 paraphrases and contradictions; labels come from the construction rule.
Vals 396-question LegalBench slice
Connor Frank, Vals AI, 6 October 2026. Twelve subtasks, 33 questions each, 396 questions sampled from 1,808 available items, fixed before any model was called. The page says the slice is not representative of the full LegalBench.
Seven political-science replications against GPT-6 Luna
Steven Denney and Matthew DiGiuseppe, arXiv:2610.06625v2, 6 October 2026. Seven published annotation and scaling tasks re-run with Jev 1.13, GPT-6 Luna, and Qwen3.8-27B. Human-coded scores are the reference.
203 sales threads, Jev against Gemini 3.8 Flash
Premsanth Rajamani, Salesgear, 30 September 2026. 203 hand-written synthetic threads, labels fixed before either model saw them. Jev is jev-1.13.0. Gemini is gemini-3.8-flash in JSON classification mode with low thinking. Commit 926e72e.
Jack & Jill initial screen, TypeSafe case study
TypeSafe AI, 12 October 2026. Offline test on 150 live roles. Gemini 3.1 Flash Lite against Jev on the initial scoring stage that cuts 6,000 retrieved candidates to a 2,000-person shortlist. No Jev version and no public log.
Artificial Analysis Decision Index v1.0
Artificial Analysis, 13 October 2026 launch article. Equal mix of AA-Omniscience-Decision (500 private questions), HLE-MultiChoice (513 text-only items), When2Call (3,423 tasks) and WildGuardTest (835 rows, average of three F1 scores). Language models answer with structured JSON. These are launch figures; the live leaderboard can change.
Three Japanese classification sets against Claude 5.5
driller, Zenn, 10 October 2026, edited 11 October. jev-latest against Claude Haiku 5.5, Sonnet 5.5 and Opus 5.5 through system-one-adapter. MASSIVE 1.1 ja-JP scenario labels, 18 classes, 216 items. chABSA sentiment, 3 classes, 210 items. JGLUE JNLI v1.3, 3 classes, 210 items. One run per setup, seed 42, equal counts per class.
ALFWorld and AndroidWorld agents, Jev against gpt-5.6-luna
Yuchen Ling, Shengcheng Yu, Zhenyu Chen and Chunrong Fang, arXiv:2610.13092v1, 9 October 2026. Five agent families, text-only observations, jev-1.13.0 for action selection, gpt-5.6-luna for every LLM decision and auxiliary call. ALFWorld 134 valid_unseen tasks. AndroidWorld 116 task types. Three runs, 4,500 executions.
Arize eight-model decision bake-off
Jim Bennett, Arize, 7 October 2026. Eight decision models. RAGTruth 2,675 rows. 13 Phoenix evaluator suites, 655 labelled cases. 52 routing decisions. Hosted APIs over the network; open models on a MacBook Pro M4 Max, 36 GB. Each model gets its own yes/no cutoff, fit on half the data and scored on the other half.
100 fictional return emails, jev-1.14.0 against Haiku 5.5
Takuma Yoshida, UnlockJapan, 26 October 2026. 100 made-up Japanese return emails for a fictional kitchenware shop. Six answers, nine policy rules applied from the top. Jev is jev-1.14.0. Haiku is claude-haiku-5-5, medium effort, one JSON call through the Claude Code CLI. Clairic is Cloudflare’s preview extractor @cf/cloudflare/clairic as of 24 October 2026.
Sanity docs-agent decision times against Haiku
John Siciliano and Knut Melvær, Sanity engineering, 24 September 2026. Four decisions on a documentation agent that answers from Sanity Knowledge Bases. Jev is called as jev-latest in the sample request; the sample response logs jev-1.13.0. The answering model in the 0.5-second demo is openai/gpt-oss-120b on Cerebras.
One-pass limits on state tracking and reasoning items
Wang Yang, Xinpeng Li, Yanyan Zhang, Chuang Ma, Shouren Wang, Jianzhe Wei, Liqin Ye, Xiaoran Cai, Mohsen Hariri, Vipin Chaudhary and Xiaotian Han, arXiv:2610.24929v1, 18 October 2026. jev-1.13.0 through the hosted API, plus 23 open Jev-style configurations. Controlled logs from 1k to 20k tokens. 738 real items after dropping HelpSteer2 and 93 disputed labels, graded by Qwen3.8-27B thinking tokens.
ClickHouse Agentic Analytics: table ranking and failure labels
Eduardo Vellasques, Lionel Palacin and Al Brown, ClickHouse, 27 October 2026. Jev ranks warehouse tables and labels agent failures on the Agentic Analytics Benchmark. Development set: 136 questions over an 18-table warehouse. Board: 201 held-out questions. Failure labels: four-model development run, 544 attempts, 277 failures. In-loop hint: 20 models. No Jev version named.
Questions about Jev
What is Jev?
Jev is TypeSafe’s decision model: you pass options or a scoring rubric, and it returns a typed answer with a probability or a confidence score. Choice selects an option, Score evaluates a rubric, and Noul returns a probability for a statement.
Sources:[1] TypeSafe AI[2] TypeSafe AI
Can Jev process images?
Hosted Jev takes text by default. On 14 October 2026 TypeSafe documented image files behind TypeSafe-Danger-Zone-Images: true, for jev-1.13.0 and jev-1.14.0. Box questions need jev-1.14.0 or jev-preview. Community write-ups also run OCR on a screenshot, then send the extracted text to Jev.
Sources:[34] TypeSafe AI[155] TypeSafe AI[156] TypeSafe AI[49] keno
How does Jev compare with GPT, Claude, and Gemini?
The reports keep each author’s test separate. Artificial Analysis scores Jev 1.13 at 58 on Decision Index v1.0, with Claude Haiku 5.5 (Low) at 61 and GPT-6 Luna (Low) at 55. Pairing notes for GPT, Claude, Gemini, Grok, Kimi, DeepSeek, Muse Wonder, and images sit on the model pages.
Sources:[139] Artificial Analysis
Is jevmodel.ai TypeSafe AI?
No. TypeSafe AI builds Jev and publishes the docs and console. This site reads public posts and documentation, then records the workflows and the limits of the reported results. TypeSafe does not edit these pages.
From the original discussions
All 234 sourcesThe cards paraphrase the linked posts.
Eduardo Vellasques, Lionel Palacin and Al Brown
Published:
Source checked:
[225] Agentic Analytics Benchmark: ranking tables and labeling failures with Jev
On a 136-question development set over an 18-table warehouse, Jev ranks needed tables at mean nDCG@10 of 0.951, recall@10 of 0.991, and about 0.4 seconds per question, with every required table inside the top five for 84% of questions. A four-model development run of 544 attempts yields 277 labeled failures; wrong data selection is the largest bucket at 87 of 277. At a 0.65 threshold on “correct despite fail,” 18 answers look like grader errors. Feeding the ranking back as an in-loop hint leaves accuracy flat across 20 models: pooled net +12 on fired cells, 95% interval −18 to +42.
ClickHouse engineering blog dated 27 October 2026. A companion pilot on 250 public text-to-SQL cases with 75 to 710 tables reaches 87% recall@10 and finds every required table in 73% of cases, at a p90 of 3.4 seconds and about $0.50 for the run. The post does not name a Jev version. Code ships in ClickHouse/data-agent-mnist.
Takuma Yoshida
Published:
Source checked:
[208] Clairic and Jev on 100 return emails: four experiments
Four layouts sort 100 fictional Japanese return emails for a kitchenware shop. Claude Haiku 5.5 scores 91 of 100 in one call, at 3.0 seconds and ¥8.1 per 100. The six-step Jev layout without Clairic, experiment 4, scores 78 at 0.3 seconds and ¥1.5, using jev-1.14.0. In that layout Jev’s final choice matched a Python encoding of the nine rules on all 100 cases, so all 22 misses came from reading the email.
English write-up dated 26 October 2026. The emails and order records are fictional, written by Claude after the correct answer was fixed. The author is the one person in the person row. Text sent to Jev and Clairic is Japanese. Clairic is Cloudflare’s preview extraction model @cf/cloudflare/clairic as of 24 October 2026. Confidence thresholds such as 0.8 were drawn after seeing the 100 results. Layouts were chosen after seeing other layouts on the same 100 cases. Haiku’s cost is a list-price estimate inside a subscription. Yen figures use 150 yen to the dollar. UnlockJapan states it has no partnership with TypeSafe or Cloudflare.
Even Westvang, Rostislav Melkumyan and Knut Melvær
Published:
Source checked:
[199] How Sanity built the @s company agent harness
Sanity’s @s harness treats each agent as a Studio document and asks Jev which published agent should handle a Slack message. At the time of writing, Jev chose among 66 agents plus one option for questions about @s itself. Confidence below 0.4 returns the three most likely agents for the user to pick. The state is the message plus up to the last 20 messages in the thread.
Engineering post last edited 22 October 2026. Follow-ups stay with the current agent unless the new question belongs elsewhere. Watched channels get a yes-or-no on whether the agent should reply. About 42 of 61 hackathon agents worked fully or partly; most failures were missing access, not routing. HAL, a meeting-hygiene agent, answered everyone when its description said “workstreams,” and the team tightened that description. The post names no Jev version and prints no router accuracy. Inference goes through OpenRouter with zero data retention. The post links an earlier 0.5-second docs-agent write-up.
Lightpanda (@lightpanda_io)
Published:
Source checked:
[201] Lightpanda puts Jev inside the browser as judge()
Lightpanda says it put Jev inside the browser binary and built judge() on top of it. The call asks predefined questions and reports whether the page has real content and what kind: article, listing, product, or docs. One call is about 0.2 seconds. A video of 30 sites runs in about 14 seconds total.
X post, 21 October 2026, with a follow-up that links the page-classify branch and a Python example. HTTP 200 pages can still be cookie walls, CAPTCHAs, logins, paywalls, error pages, empty searches, or pages still loading. The post prints no per-page accuracy and no Jev version.
The decision-models page puts Jev next to language models, OpenAI Decisions, and encoder classifiers. The Sage2 page records Levanto Labs’ 2 November 2026 launch: Extract, streaming Decide, Extract Bench v0.1 at 77.8 field accuracy, and early volumes from Normies and Perp Flow. The Jev for SEO page records Tenpoint Labs’ five classification jobs, Nico Gorrono’s live-site tests, jev-seo’s crawl audits, and Sam Underwood’s SERP Lens quality scores. The Composio Firewall page records the 21 October 2026 launch: session hooks that evaluate tool calls with Jev, before_execute and after_execute checks, and a Pro list price of $0.50 per 1M tokens evaluated. The Jevrywhere page follows Allie’s 30 October “put Jev everywhere” advice into Agent Tower UI slots, substring picks with jeveryword, line-by-line search, and Intelliprompter. The Jevtember page records Vercel’s September AI Gateway adoption figures and Diogo Almeida’s 18 October nickname post. The Jev Day 2026 page collects Jev 1.14.0, native images, Box, and 262k context. The workflow page covers ClickHouse’s warehouse table ranking and failure labels, Sanity’s @s router, Lightpanda’s page judge, UiPath’s TypeSafe connector, Zuplo’s AI Gateway listing, Decision Index comparisons, a Zenn cascade to Claude 5.5, Jev action selection inside Luna agents, Arize’s polarity check on yes/no judges, a 1% error-budget freeze, a sales-thread triage, a Gemini candidate screen, a difficulty tier before Claude, confidence-based handoffs, a judge cascade, context pruning, browser action selection, a 10-K passage rank, and Jev skills inside Muse and Muse Code. Image workflows that extract text first are in the Jev and images report. The Jev and Muse Wonder report covers those Muse skills.