All benchmarks · Models for decisions
JevOss
An open test suite that checks how changes to a question affect a model that chooses among fixed answers.
JevOss starts with questions whose answers are known and changes one thing at a time. It reorders the options, adds an instruction that the model should ignore, inserts about 600 words of unrelated text, rewrites a yes-or-no question or repeats the same request. It then counts how often the answer changes.
The suite reports accuracy and checks whether the model’s probabilities match how often it is right, a measure called calibration. The Brier score measures errors in those probabilities; lower scores mean smaller errors. It works with any server that accepts the Jev API request format (POST /v1/systemone), including JevAlt and Kev-4B.
- Checks
- 5: option order, hidden instructions, unrelated text, yes-or-no formats, repeated requests
- Commands
- eval, probe, calibrate, compare, ask
- Needs
- Python 3.11 or newer and a running decision-model server
- Version
- 0.1.0, released October 1, 2026
- Licence
- Apache-2.0
Results
| Model | Answers changed |
|---|---|
| Kev-4B | 36.0% |
| Deem-4B | 14.0% |
| Karar-4B | 19.0% |
| Wähler-4B | 17.5% |
Limits
After about 600 words of unrelated text are added, Kev-4B loses 5.4 accuracy points, versus 17.4 points for Deem-4B.
Run it
The default server address is http://127.0.0.1:8000. Start a decision-model server with jevalt serve, then run the test commands below.
pip install "jevoss[suites] @ git+https://github.com/mertkayacs/jevoss"
jevoss eval typed-decisions
jevoss probe typed-decisions --limit 100Cite
@software{kaya2026jevoss,
author = {Mert Kaya},
title = {JevOss: Open Evaluation Toolkit for Jev-Type Decision Models},
year = {2026},
license = {Apache-2.0},
url = {https://github.com/mertkayacs/jevoss}
}