[All benchmarks](https://eschatialabs.com/benchmarks/) · Models for decisions

# JevOss

An open test suite that checks how changes to a question affect a model that chooses among fixed answers.

  [Project site](https://jevoss.mertkayacs.com)

JevOss starts with questions whose answers are known and changes one thing at a time. It reorders the options, adds an instruction that the model should ignore, inserts about 600 words of unrelated text, rewrites a yes-or-no question or repeats the same request. It then counts how often the answer changes.

The suite reports accuracy and checks whether the model’s probabilities match how often it is right, a measure called calibration. The Brier score measures errors in those probabilities; lower scores mean smaller errors. It works with any server that accepts the Jev API request format (POST /v1/systemone), including JevAlt and Kev-4B.

    Checks 5: option order, hidden instructions, unrelated text, yes-or-no formats, repeated requests

  Commands eval, probe, calibrate, compare, ask

  Needs Python 3.11 or newer and a running decision-model server

  Version 0.1.0, released October 1, 2026

  Licence Apache-2.0

## Results

      Hidden instructions: share of 200 test cases where the added instruction changed the answer Model Answers changed Kev-4B 36.0% Deem-4B 14.0% Karar-4B 19.0% Wähler-4B 17.5%

### Limits

After about 600 words of unrelated text are added, Kev-4B loses 5.4 accuracy points, versus 17.4 points for Deem-4B.

## Run it

The default server address is http://127.0.0.1:8000. Start a decision-model server with jevalt serve, then run the test commands below.

```bibtex
pip install "jevoss[suites] @ git+https://github.com/mertkayacs/jevoss"
jevoss eval typed-decisions
jevoss probe typed-decisions --limit 100
```

## Cite

```bibtex
@software{kaya2026jevoss,
  author = {Mert Kaya},
  title = {JevOss: Open Evaluation Toolkit for Jev-Type Decision Models},
  year = {2026},
  license = {Apache-2.0},
  url = {https://github.com/mertkayacs/jevoss}
}
```

## Links

- [Project site](https://jevoss.mertkayacs.com)

- [Code (GitHub)](https://github.com/mertkayacs/jevoss)

- [Documentation](https://mertkayacs.github.io/jevoss/)

Canonical: https://eschatialabs.com/benchmarks/jevoss/