All benchmarks · AI task assistants
Tholos-Bench
160 tasks for testing small AI assistants, scored by whether their completed work matches the request.
170 downloads on Hugging Face, as of October 18, 2026.
Each test creates a Tholos workspace where AI assistants use tools, tables and notes, then gives one assistant a task. A run passes only when the finished workspace meets every check, such as adding the correct row, assigning the correct teammate and avoiding forbidden tools.
The 14 categories cover table edits, task handoffs, approvals, writing conflicts, research on saved web pages, instructions hidden in pages or notes, and tasks where the correct action is to leave the workspace unchanged.
- Scenarios
- 160 in 14 categories
- Language
- English
- Format
- One JSON file per scenario, also available as JSON Lines on Hugging Face
- Settings
- Temperature 0 (the same answer on every run), a context limit of 16,384 text units (tokens), fixed clock
- Licence
- Apache-2.0
- Released
- October 1, 2026, with Tholos 0.1.0
Results
| Model | File | Passed (of 160) | Hidden instructions (of 12) |
|---|---|---|---|
| Qwen3.5-2B | 1.28 GB | 97 | 4 |
| MiniCPM5-2B | 1.56 GB | 112 | 8 |
| Tholos-2B | 1.56 GB | 137 | 10 |
| Llama 3.2 3B Instruct | 2.02 GB | 37 | 1 |
| Granite 4.2-3B | 2.24 GB | 129 | 7 |
| Qwen3.5-4B | 2.74 GB | 141 | 6 |
Limits
Tholos-2B was trained from MiniCPM5-2B on recorded examples of this task format, and it is the only model in the table with that training. On the same ordinary processor (CPU) with llama.cpp, Tholos-2B passes 134 of 160 tasks, versus 117 for MiniCPM5-2B. Compare scores only when the hardware and software setup is the same.
Run it
Start a model server, install Tholos and run all 160 tasks. Add --only and a category name to run one category.
llama-server -hf mertkayacs/Tholos-2B-GGUF:Q4_K_M --jinja -c 16384 -t 4 -a tholos-2b --host 127.0.0.1 --port 8080
uv tool install git+https://github.com/mertkayacs/tholos
tholos bench --base-url http://127.0.0.1:8080/v1 --model tholos-2bCite
@misc{kaya2026tholosbench,
author = {Kaya, Mert},
title = {Tholos-Bench: Workspace Tasks for Small AI Assistants},
publisher = {Eschatia Labs},
year = {2026},
url = {https://huggingface.co/datasets/mertkayacs/tholos-bench}
}