Dataset · K-and-K
Knights and knaves logic puzzles for LLM evaluation
K-and-K/knights-and-knaves
English-language benchmark with logic puzzles for evaluating language model reasoning skills.
What's inside
The dataset contains text logic puzzles of the knights and knaves format in English, divided into train and test splits (for example, the 2ppl split). The total size is between 1 and 10 thousand examples. The authors also mention a complicated version of the dataset to check whether the model actually reasons or simply memorizes answers.
What you can do with it
- Evaluating LLM capability for multi-step logical reasoning
- Testing models for overfitting and simple memorization of test data
- SFT fine-tuning of models to improve logic puzzle solving skills
- Tasks
- Question answering
- Languages
- English
- Size
- 1K–10K rows
- Downloads this month
- 3K
- Likes
- 40
Models trained on it
How to load it
from datasets import load_dataset
ds = load_dataset("K-and-K/knights-and-knaves")