Mert Kaya
MenuClose

All benchmarks · Models for decisions

jevalt-bench

3,116 written situations in English, Turkish and German for testing models that choose among fixed answers and give a probability for each answer.

Dataset (Hugging Face)

1,600 downloads on Hugging Face, as of October 18, 2026.

Each row describes a situation, such as an email, a policy or a chat message. The questions ask the model to choose an option, give a rating or estimate the probability of yes.

The test reports accuracy. It also checks how closely the model’s probabilities match how often it is right, a measure called calibration. The Brier score measures errors in those probabilities; lower scores mean smaller errors.

Rows
3,116: English 1,277, Turkish 1,019, German 820
Questions
8,434
Format
JSON Lines, one subset per language
Licence
Rows written for the project: Apache-2.0. Converted rows keep their source licence: MASSIVE CC BY 4.0, Open-Jev CC0, PAWS-X free to use with credit to Google, typed-decisions Apache-2.0.
Released
October 1, 2026

Results

Accuracy and probability error (Brier score) on questions kept apart from training
LanguageKev-4BJevAltBrier, Kev-4BBrier, JevAlt
English84.7%Deem-4B 94.7%0.2560.091
Turkish87.1%Karar-4B 96.8%0.2140.058
German81.1%Wähler-4B 92.0%0.2980.119

Limits

The test questions and training examples were produced by the same process, so these scores apply to JevAlt’s own task. On the separate German news dataset 10kGNAD, Kev-4B reaches 65.3% accuracy, versus 62.5% for Wähler-4B.

Run it

Load one language with the Hugging Face datasets library. To test your own model, connect JevOss to a server that accepts the Jev API, the request format shared by these decision models.

from datasets import load_dataset

test = load_dataset("mertkayacs/jevalt-bench", "en", split="test")
print(test[0])

Cite

@techreport{kaya2026jevaltreport,
  author = {Kaya, Mert},
  title = {JevAlt: Technical report},
  institution = {Eschatia Labs},
  year = {2026},
  url = {https://eschatialabs.com/research/jevalt-report/},
  note = {Technical report, not peer-reviewed}
}