Skip to content

Dataset · K-and-K

Knights and knaves logic puzzles for LLM evaluation

K-and-K/knights-and-knaves

English-language benchmark with logic puzzles for evaluating language model reasoning skills.

What's inside

The dataset contains text logic puzzles of the knights and knaves format in English, divided into train and test splits (for example, the 2ppl split). The total size is between 1 and 10 thousand examples. The authors also mention a complicated version of the dataset to check whether the model actually reasons or simply memorizes answers.

What you can do with it

  • Evaluating LLM capability for multi-step logical reasoning
  • Testing models for overfitting and simple memorization of test data
  • SFT fine-tuning of models to improve logic puzzle solving skills
Tasks
Question answering
Languages
English
Size
1K–10K rows
Downloads this month
3K
Likes
40

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("K-and-K/knights-and-knaves")