Can Jev Hallucinate? What 'Can't Hallucinate' Really Means
Last checked · Independent guide, not affiliated with TypeSafe AI
Jev cannot hallucinate in the narrow sense TypeSafe means: it can only return one of the answers you defined, so it never invents a label, a field or a malformed value. It can still pick the wrong answer, sometimes with high confidence. Calibrated probabilities make those mistakes easier to catch, but they do not remove them.
“Can’t hallucinate” was one of the launch claims that drew the most pushback on Hacker News. Both sides have a point, because the word means two different things.
What TypeSafe is claiming
Section titled “What TypeSafe is claiming”TypeSafe ties the claim to type safety. With an LLM, the answer is free text: the model can invent a category you never offered, return malformed JSON, or call a tool that does not exist. Your code has to parse and validate every response.
With Jev, the possible answers are fixed in the request. A Choice can only return one of your option keys; a Noul returns a number between 0 and 1; a Score returns a position on your levels. TypeSafe says there is no way for the output to break that contract, and its launch post notes that this 0% is a guarantee of the schema rather than a measured rate.
In that sense the claim holds: Jev cannot produce an answer that does not exist.
What critics mean
Section titled “What critics mean”Most people use “hallucination” for a different failure: a confident answer that is wrong. By that definition Jev can hallucinate. It can choose billing when the ticket is technical, or give 0.95 to a statement that is false. Commenters on Hacker News pointed out that an LLM limited to a fixed set of labels also cannot produce an invalid label, so the guarantee comes from the interface, not from some property unique to the model.
TypeSafe’s own documentation agrees with the underlying point. It says calibration is measured across groups of predictions and does not guarantee that any individual answer is correct, and its list of Jev 1.13 weak spots includes literal reading, math, dates and adversarial text, all ways to be wrong without being malformed.
So what do you actually get?
Section titled “So what do you actually get?”| Failure | LLM | Jev |
|---|---|---|
| Invents an option you did not offer | Possible | Not possible |
| Returns malformed or unparseable output | Possible | Not possible |
| Picks the wrong option | Possible | Possible |
| Tells you when it is unsure | Unreliable self-reports | Calibrated probabilities on every answer |
The practical gain is the last row. Because Jev returns a probability for every answer, you can decide what to do with uncertain cases instead of discovering the mistake later.
How to handle Jev’s mistakes
Section titled “How to handle Jev’s mistakes”- Threshold on confidence. Act automatically only when a Choice or Score has high confidence, or a Noul is far from 0.5. Route the middle band to a person. See Confidence.
- Give it an “other” option. A Choice without a way out has to pick something. TypeSafe recommends adding
otherornone of the abovewhen your list might not cover every input. - Ask narrow questions. Several simple questions combined in code are easier to get right, and to debug, than one broad question.
- Measure on your own data. Label a few dozen real examples and check the error rate before automating. See how to benchmark Jev.
The one-line answer
Section titled “The one-line answer”Jev will never make up an answer that is not on your list. It can still choose the wrong one, but it tells you how sure it is, which is what makes those mistakes manageable.