[All benchmarks](https://eschatialabs.com/benchmarks/) · AI task assistants

# Tholos-Bench

160 tasks for testing small AI assistants, scored by whether their completed work matches the request.

  [Dataset (Hugging Face)](https://huggingface.co/datasets/mertkayacs/tholos-bench)

170 downloads on Hugging Face, as of October 18, 2026.

Each test creates a Tholos workspace where AI assistants use tools, tables and notes, then gives one assistant a task. A run passes only when the finished workspace meets every check, such as adding the correct row, assigning the correct teammate and avoiding forbidden tools.

The 14 categories cover table edits, task handoffs, approvals, writing conflicts, research on saved web pages, instructions hidden in pages or notes, and tasks where the correct action is to leave the workspace unchanged.

    Scenarios 160 in 14 categories

  Language English

  Format One JSON file per scenario, also available as JSON Lines on Hugging Face

  Settings Temperature 0 (the same answer on every run), a context limit of 16,384 text units (tokens), fixed clock

  Licence Apache-2.0

  Released October 1, 2026, with Tholos 0.1.0

## Results

      One graphics card (a Kaggle T4), llama.cpp, files for running each model on your own computer (GGUF) compressed with Q4_K_M to about 4 bits per weight, and answers limited to a fixed JSON structure Model File Passed (of 160) Hidden instructions (of 12) Qwen3.5-2B 1.28 GB 97 4 MiniCPM5-2B 1.56 GB 112 8 Tholos-2B 1.56 GB 137 10 Llama 3.2 3B Instruct 2.02 GB 37 1 Granite 4.2-3B 2.24 GB 129 7 Qwen3.5-4B 2.74 GB 141 6

### Limits

Tholos-2B was trained from MiniCPM5-2B on recorded examples of this task format, and it is the only model in the table with that training. On the same ordinary processor (CPU) with llama.cpp, Tholos-2B passes 134 of 160 tasks, versus 117 for MiniCPM5-2B. Compare scores only when the hardware and software setup is the same.

## Run it

Start a model server, install Tholos and run all 160 tasks. Add --only and a category name to run one category.

```bibtex
llama-server -hf mertkayacs/Tholos-2B-GGUF:Q4_K_M --jinja -c 16384 -t 4 -a tholos-2b --host 127.0.0.1 --port 8080
uv tool install git+https://github.com/mertkayacs/tholos
tholos bench --base-url http://127.0.0.1:8080/v1 --model tholos-2b
```

## Cite

```bibtex
@misc{kaya2026tholosbench,
  author = {Kaya, Mert},
  title = {Tholos-Bench: Workspace Tasks for Small AI Assistants},
  publisher = {Eschatia Labs},
  year = {2026},
  url = {https://huggingface.co/datasets/mertkayacs/tholos-bench}
}
```

## Links

- [Dataset (Hugging Face)](https://huggingface.co/datasets/mertkayacs/tholos-bench)

- [Code (GitHub)](https://github.com/mertkayacs/tholos/tree/main/tholos/bench)

- [Project site](https://tholos.mertkayacs.com)

- [Technical report](https://eschatialabs.com/research/tholos-2b-report/)

Canonical: https://eschatialabs.com/benchmarks/tholos-bench/