Skip to content

Dataset · LocalLLaMA

Typed probabilistic decisions benchmark

LocalLLaMA/typed-decisions

Benchmark: the model receives an unstructured state and answers five typed questions with probabilities.

What's inside

A dataset for evaluating typed probabilistic decisions. The model receives a single fragment of an unstructured state and simultaneously answers five typed questions; each answer is a probability distribution rather than a single class. The schema follows the System One primitives (noul, choice, score) from TypeSafe AI, and the string is reproduced via an API with such a shape. The benchmark is independent of TypeSafe and does not reproduce their Jev model; the language is English, and the size is 1–10k records.

What you can do with it

  • Testing LLMs on probabilistic typed decisions
  • Evaluating API compatibility with the System One schema
  • Comparing models in text-classification with distributions
Tasks
Text classification
Languages
English
Size
1K–10K rows
Downloads this month
23.1K
Likes
278

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("LocalLLaMA/typed-decisions")