We Ran Jev Against GPT-5 mini, Claude Haiku 4.5 and Gemini Flash on Live Crypto Order Flow. Here Is Who Actually Called the Next 15 Minutes (September 2026)

Jev is TypeSafe AI’s decision model: it takes a block of state plus typed questions and returns probabilities, never text. TypeSafe says it is calibrated, 20 to 200 times faster than an LLM and up to 400 times cheaper. Nobody outside TypeSafe had published an accuracy or calibration number when we started this, so we built our own test on the hardest kind of decision we could find: live crypto order flow on a token in the middle of a pump.

TL;DR: 320 fifteen-minute windows of AKE trades (745,354 trades on Gate and Kraken, September 16, 00:00 to September 19, 08:00 UTC), five questions per window, four models, same state, same questions, nothing after the window end. No model beat a coin flip on the 15-minute direction question, which is the honest headline. Best 15-minute Brier: Google Gemini 3.8 Flash at 0.2509. Best calibration (lowest ECE): Google Gemini 3.8 Flash at 0.029. Jev answered in a median 0.49 s and cost $0.039 per 1,000 decisions, 26x to 196x less than the LLMs and 5.6x to 28.7x faster. Speed and price are real. Alpha is not, for any of them.

The Headline Table

Brier score is the mean squared error of a probability forecast: 0 is perfect, 0.25 is what “always say 50%” scores, above 0.25 is worse than a coin flip. AUC is ranking power (0.5 = none). ECE is expected calibration error, the average gap between what a model said and what happened, in probability points; lower is better. Move accuracy is the share of windows where the model picked the right one of “up more than 3%”, “flat”, “down more than 3%” over the next hour. Risk rho is the Spearman correlation between the model’s drawdown score and the drawdown that actually followed.

Base rates for this sample: price was higher 15 minutes later in 52% of windows and higher 60 minutes later in 54%. Over the next hour the move was flat (within 3%) in 69% of windows, up more than 3% in 22% and down more than 3% in 9%.

Speed and Cost, Measured

All calls went through OpenRouter from Southeast Asia on September 19, 2026. Latency is wall-clock per call including network. Cost is OpenRouter’s own usage.cost field summed per model, so it includes the LLMs’ hidden reasoning tokens. One “decision” here is one call that answers all five questions.

Three things the pricing tables do not tell you. First, the LLMs’ input tokens are lower than Jev’s because Jev’s question rubrics are sent as structured criteria and billed as input, while the LLMs get a shorter system prompt; Jev still comes out 26x to 196x cheaper because output is free and there is no reasoning. Second, reasoning models needed a 1,500-token output budget and a low reasoning setting to return anything at all; at 400 tokens GPT-5 mini and Gemini returned empty bodies because their hidden reasoning consumed the budget. Third, all three LLMs ran with reasoning effort set to low, and that is where Claude Haiku 4.5’s 14-second median and $7.70 per 1,000 come from: it averaged about 1,417 output tokens per call, most of them thinking, against roughly 471 for GPT-5 mini and 158 for Gemini. We did not rerun Haiku with thinking off, so read its row as “Haiku with thinking on”, not as its floor. All of that is a real integration cost, and it is the kind of thing Jev’s fixed output shape removes. Full pricing arithmetic is in our Jev pricing guide.

Calibration: Does the Probability Mean Anything?

This is TypeSafe’s core claim, and the one that matters for automation. A model that says 0.7 should be right about 70% of the time. We bucketed every 15-minute direction forecast into five probability bins and compared the average forecast with the realised hit rate.

Read the bins with the window counts in mind; a bin with a dozen windows can swing 20 points on noise. The pattern that survives the noise: Google Gemini 3.8 Flash stayed closest to the diagonal (ECE 0.029), and TypeSafe Jev 1.13 came in at ECE 0.040. Jev’s forecasts clustered near 0.48; it rarely committed to a strong view on direction, which is what a calibrated model should do when the state carries little directional information.

The Regime Question: Where the Models Actually Disagree

“Accumulation, distribution, wash-like or quiet” has no ground truth, so we report what each model chose across the same 320 windows. It shows temperament more than skill.

How the Test Was Built

• Token. AKE (Akedo), picked from CoinGecko’s trending list on September 19 because it was up more than 150% in 24 hours and listed on five centralised exchanges plus PancakeSwap. Price in the sample ran from $0.0273 to a high of $0.0620.
• Data. Every public trade from the Gate and Kraken AKE spot markets, pulled from each venue’s public trade endpoint into one database. Kraken’s public history reaches back to September 16; Gate’s to September 17, 03:19 UTC, so the first 109 windows are Kraken-only and the remaining 211 carry both venues. KuCoin, MEXC, LBank and two PancakeSwap V3 pools were added to the collector at 08:00 UTC on September 19, after the last window in this sample; those venues only expose recent trades, so they will appear in the follow-up run, not here. Gate 5-minute candles supply the price labels.

• Windows. Every 15 minutes from September 16, 00:00 to September 19, 08:00 UTC: 320 windows. For each, the state is per-venue trade count, USD volume, taker buy share, average and largest ticket, the cross-exchange VWAP spread when both venues traded, volume versus the prior four hours, and returns over 15 minutes, 1, 4 and 24 hours. Nothing after the window end is in the state.
• Questions. The same five to every model: will price be higher in 15 minutes (yes/no probability), higher in 60 minutes (yes/no), the 60-minute move bucket (choice of three), the worst drawdown in the next hour (score on three levels), and the order-flow regime (choice of four). Jev got them as native noul, choice and score questions through OpenRouter’s decisions endpoint. The LLMs got the same definitions in a system prompt and were asked for JSON probabilities at temperature 0.
• Labels. From Gate 5-minute closes: price at window end versus 15 and 60 minutes later; the lowest low in the following hour for drawdown.
• Baselines. Always 50%; momentum (60% if the last 15 minutes were up, else 40%); and the window’s buy share used directly as the probability.
• Not included. Fees, slippage, or any trading. This is a forecasting test of a decision model, not a strategy.

What It Means

Direction is not forecastable from 15 minutes of tape, and none of the models pretended otherwise. No model beat a coin flip on the 15-minute direction question, which is the honest headline. That is the expected result on a liquid pump: the information in aggregate flow is priced within seconds. The value of the test is not that anyone found alpha. It is that a model that cannot find alpha should say so with probabilities near the base rate, and the calibration table shows which ones did.

Jev’s economics hold up on a real workload. 0.49 seconds median through a router from the far side of the planet, $0.039 per 1,000 five-question decisions, zero failed calls out of 320. The cheapest LLM in the test cost 26x more per decision and the slowest was 28.7x slower. If your pipeline runs a chat model to get a label, a bucket or a probability, those two numbers are the business case, and they replicate outside TypeSafe’s own benchmarks.

Where an LLM still earns its cost. The regime question is where the models diverge most, and it is also where a reasoning trace might matter to a human reading the output. Jev returns a probability and nothing else. If you need the why, you still need the LLM, and you pay 196x for it.

Limits of This Test

• One token, one pump, 320 windows, two exchanges. The confidence intervals on every number here are wide; treat differences under 0.01 Brier as noise.
• The first 109 windows see Kraken flow only. A multi-venue rerun with KuCoin, MEXC, LBank and PancakeSwap is collecting now.
• Fifteen-minute and one-hour horizons only. Longer horizons might separate the models, or might not.
• Labels come from one venue’s candles (Gate). A cross-venue VWAP label would shift edge cases.
• Jev was called with native typed questions; the LLMs with a JSON prompt. That is how each is meant to be used, but it is not identical framing.
• Each model was called once per window at temperature 0 (LLMs) or its default (Jev). No repeats, no ensembling.
• All speed and cost claims from TypeSafe remain vendor claims; the numbers above are ours, and the code is a few hundred lines of Python against public endpoints.

BetOnAI Verdict

• Use Jev for the decisions your stack makes thousands of times a day: routing, scoring, moderation, triage, cascade gates. On this workload it delivered the speed and cost TypeSafe advertises, with calibration in the same range as the LLMs.
• Do not use any of these models as a trading signal on short horizons. The tape does not carry enough directional information for a 15-minute call, and the models were honest about that.
• If you want to reproduce this, the recipe is above. Start with one question, one token and a week of windows, and watch the calibration table before the accuracy number.

The cost side of this decision lives in our Jev pricing guide; the routing fees in the OpenRouter pricing guide; the wider small-model price war in AI API pricing war 2026; and what to charge for building this kind of pipeline in the AI automation rate card.

Frequently Asked Questions

Did Jev beat the LLMs on this benchmark?

On the 15-minute direction question the best Brier score was Google Gemini 3.8 Flash at 0.2509, against Jev’s 0.2540; the differences between the four are within noise for 320 windows. Jev won clearly on latency (0.49 s median) and on cost ($0.039 per 1,000 decisions), and had the second-lowest calibration error of the four.

Can Jev predict crypto prices?

No, and neither can the LLMs on this data. Fifteen minutes of aggregate order flow on a liquid token carries almost no directional information once it is public. A calibrated model answers near the base rate in that situation, which is what the calibration table shows.

How was Jev called?

Through OpenRouter’s decisions endpoint (POST /api/alpha/decisions, model typesafe/jev-1.13) with the window state as a JSON string and five native questions: two noul (yes/no), two choice and one score. The response carries a probability per option and a confidence value.

How were the LLMs called?

Through OpenRouter chat completions at temperature 0 with a system prompt defining the same five questions and a JSON-only answer format. All three ran at low reasoning effort with a 1,500-token output budget; with a 400-token budget GPT-5 mini and Gemini 3.8 Flash returned empty bodies. Haiku 4.5 spent about 1,417 output tokens per call, mostly thinking, which is why it was the slowest and most expensive row.

What does a Brier score of 0.25 mean?

It is the score of a model that always says 50%. Lower is better. A model below 0.25 has some skill on this question; a model above it would have been better off saying 50% every time.

Is this financial advice?

No. It is a benchmark of a decision API on a live data feed. Nothing here was traded, and nothing here should be.

Sources

• TypeSafe AI, Introducing System One Models and Jev: https://typesafe.ai/blog/introducing-system-one-models-and-jev
• TypeSafe docs, API reference and confidence: https://docs.typesafe.ai/api, https://docs.typesafe.ai/confidence
• OpenRouter, Decisions API reference: https://openrouter.ai/docs/api/api-reference/alphadecisions/submit-a-decisions-questions-and-answers-request
• OpenRouter, Jev 1.13 model page: https://openrouter.ai/typesafe/jev-1.13
• Public trade endpoints used: Gate.io API v4 spot/trades, Kraken public Trades, KuCoin market/histories, MEXC api/v3/trades, LBank v2 trades, GeckoTerminal pool trades.
• CoinGecko trending, September 19, 2026: https://www.coingecko.com/en/highlights/trending-crypto
• Brier score and calibration definitions: https://en.wikipedia.org/wiki/Brier_score