[All benchmarks](https://eschatialabs.com/benchmarks/) · Models for decisions

# jevalt-bench

3,116 written situations in English, Turkish and German for testing models that choose among fixed answers and give a probability for each answer.

  [Dataset (Hugging Face)](https://huggingface.co/datasets/mertkayacs/jevalt-bench)

1,600 downloads on Hugging Face, as of October 18, 2026.

Each row describes a situation, such as an email, a policy or a chat message. The questions ask the model to choose an option, give a rating or estimate the probability of yes.

The test reports accuracy. It also checks how closely the model’s probabilities match how often it is right, a measure called calibration. The Brier score measures errors in those probabilities; lower scores mean smaller errors.

    Rows 3,116: English 1,277, Turkish 1,019, German 820

  Questions 8,434

  Format JSON Lines, one subset per language

  Licence Rows written for the project: Apache-2.0. Converted rows keep their source licence: MASSIVE CC BY 4.0, Open-Jev CC0, PAWS-X free to use with credit to Google, typed-decisions Apache-2.0.

  Released October 1, 2026

## Results

      Accuracy and probability error (Brier score) on questions kept apart from training Language Kev-4B JevAlt Brier, Kev-4B Brier, JevAlt English 84.7% Deem-4B 94.7% 0.256 0.091 Turkish 87.1% Karar-4B 96.8% 0.214 0.058 German 81.1% Wähler-4B 92.0% 0.298 0.119

### Limits

The test questions and training examples were produced by the same process, so these scores apply to JevAlt’s own task. On the separate German news dataset 10kGNAD, Kev-4B reaches 65.3% accuracy, versus 62.5% for Wähler-4B.

## Run it

Load one language with the Hugging Face datasets library. To test your own model, connect JevOss to a server that accepts the Jev API, the request format shared by these decision models.

```bibtex
from datasets import load_dataset

test = load_dataset("mertkayacs/jevalt-bench", "en", split="test")
print(test[0])
```

## Cite

```bibtex
@techreport{kaya2026jevaltreport,
  author = {Kaya, Mert},
  title = {JevAlt: Technical report},
  institution = {Eschatia Labs},
  year = {2026},
  url = {https://eschatialabs.com/research/jevalt-report/},
  note = {Technical report, not peer-reviewed}
}
```

## Links

- [Dataset (Hugging Face)](https://huggingface.co/datasets/mertkayacs/jevalt-bench)

- [Code (GitHub)](https://github.com/mertkayacs/jevalt)

- [Project site](https://jevalt.mertkayacs.com)

- [Technical report](https://eschatialabs.com/research/jevalt-report/)

Canonical: https://eschatialabs.com/benchmarks/jevalt-bench/