Skip to content

Dataset · maveriq

Challenging BIG-Bench Hard benchmark for language models

maveriq/bigbenchhard

A set of 1k–10k challenging English text tasks where LLMs historically lagged behind humans.

What's inside

A collection of 1k to 10k challenging English test tasks extracted from the well-known BIG-Bench benchmark. The dataset focuses on tasks (question-answering, text and token classification) where language models in base mode performed worse than the average human evaluator.

What you can do with it

  • Evaluate the reasoning and logic limits of large language models (LLMs)
  • Test the effectiveness of prompting techniques (e.g., few-shot or chain-of-thought) on complex tasks
Tasks
Question answering, Token classification, Text classification
Languages
English
Size
1K–10K rows
Downloads this month
987
Likes
47

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("maveriq/bigbenchhard")