Dataset · LocalLLaMA
Typed probabilistic decisions benchmark
LocalLLaMA/typed-decisions
Benchmark: the model receives an unstructured state and answers five typed questions with probabilities.
What's inside
A dataset for evaluating typed probabilistic decisions. The model receives a single fragment of an unstructured state and simultaneously answers five typed questions; each answer is a probability distribution rather than a single class. The schema follows the System One primitives (noul, choice, score) from TypeSafe AI, and the string is reproduced via an API with such a shape. The benchmark is independent of TypeSafe and does not reproduce their Jev model; the language is English, and the size is 1–10k records.
What you can do with it
- Testing LLMs on probabilistic typed decisions
- Evaluating API compatibility with the System One schema
- Comparing models in text-classification with distributions
- Tasks
- Text classification
- Languages
- English
- Size
- 1K–10K rows
- Downloads this month
- 23.1K
- Likes
- 278
Models trained on it
- bartowski/frontier-infra_jebadiah-9b-v2-GGUFImage text → text 5.1K
- mradermacher/lazydecider-0.8b-GGUF 1.5K
- mradermacher/jebadiah-9b-v2-GGUF 1.2K
- mrm8488/embeddinggemma-2-ft-typed-decisionText classification 1.1K
- manjunathshiva/opendecider-nanoZero shot classification 1.1K
- mradermacher/jebadiah-4b-v2-GGUF 1K
- mradermacher/OpenJev-8B-GGUF 974
- mradermacher/jebadiah-27b-GGUF 926
How to load it
from datasets import load_dataset
ds = load_dataset("LocalLLaMA/typed-decisions")