Jev limitations and open questions

A constrained answer can still be wrong

Jev returns decisions within a specified interface. That helps software consume a result, but does not establish that the selected category matches reality. Treat a valid response as a judgment to evaluate, not as proof about the world.

TypeSafe describes calibration as a property of groups of predictions. A reported probability does not guarantee the correctness of one particular decision. Calibration explanation

Consider a wrong but valid support route. Parsing succeeds and the queue exists, yet the customer still waits for the wrong team. Measure that failure directly instead of counting successful API responses as successful outcomes.

Input and output boundaries

The documented interface accepts text, including structured text in JSON, rather than raw images, audio, or video. It does not write free-form replies, code, or reasoning explanations. System One capabilities

If your workflow starts with a scan or a recording, another component must prepare the text. Review errors introduced there: a missing negation in a transcript can change the eventual decision even if every later component works as designed. If you need an explanation for a user, retain inspectable evidence and design that explanation separately; do not fabricate a rationale from a probability.

Version-specific issues to account for

TypeSafe’s Jev 1.13 known-limitations page, itself reviewed September 16, 2026, reports weaknesses with literal wording, numerical precision, date comparison, indirect reasoning, irrelevant context, hostile input, and conflicting instructions. It gives two context constraints: 64k tokens across state and all questions, and 32k for state plus the longest question. These are version-specific vendor statements, not our measurements.

Our practical response is to keep exact calculations and date ordering in code, identify the relevant records explicitly, and write questions whose criteria agree with their instructions. Larger input capacity does not mean that filling it with unrelated material helps.

Do not interpret a fractional Score as an exact measurement. Score is a weighted position across the rubric’s levels; different distributions can produce the same value. Read the distribution alongside the score. Score interpretation

Hostile input and action boundaries

Text inside a document, email, or retrieved passage may be trying to influence the system. TypeSafe’s RAG cookbook explicitly warns that its injection filter is not a security boundary; a passage below its threshold can still reach the generator.

Keep permissions and execution checks outside the model. For example, an instruction in an email must not grant access to another account, and a suggested action must still satisfy your application’s authorization rules. Decide how to handle a timeout or an ambiguous answer before enabling actions. A human review queue is useful only if someone owns it and can inspect the evidence.

Community observations need careful labels

Two community developers reported that Choice label names or insertion order could affect probability outputs on jev-1.13.0. The discussion included repeated requests and a follow-up on reranking data. See the initial report and follow-up report, posted September 17, 2026.

These are self-reports, not independent proof. We have not rerun the experiments or verified the external dataset. They do not establish which representation is more accurate, a universal preference for the first option, or that Choice cannot rank useful results. No verified official explanation or fix was observed in the captured thread. The links may require community access, and the research did not cover every discussion.

Our resulting evaluation advice is to version the whole question definition: model, instructions, primitive, labels, descriptions, and order. Recheck decisions when that definition changes. Do not assume that changing a yes/no task from Noul to Choice preserves identical probability values.

Access, versions, and operating assumptions

A moving alias may select a newer model without an application change, and early-access rate limits can change. The Models page is the reference for current versions and limits. Log the resolved version and revisit thresholds before adopting an update.

This guide does not establish an uptime guarantee, an enterprise data-handling agreement, a retention policy for your account, or suitability for regulated decisions. Confirm requirements with TypeSafe before submitting sensitive material. Missing evidence here means unverified, not unsupported.

The site’s 12 reading languages are a publishing feature. They do not establish Jev’s supported languages or equivalent quality across languages. Include the languages, domain terms, and ambiguous cases your users actually produce in your own evaluation.

A sensible boundary for an initial trial

Choose a narrow, reversible recommendation with a visible review path. Keep examples where the answer is missing, unclear, contradictory, or outside your categories. Record incorrect decisions and unresolved cases, then decide whether the improvement justifies operating the extra component.

No independent API runs, reliability measurements, or quality benchmarks were performed for this edition. Read our evidence policy, inspect the documented cases, or return to the integration walkthrough.