Dataset · maveriq
Challenging BIG-Bench Hard benchmark for language models
maveriq/bigbenchhard
A set of 1k–10k challenging English text tasks where LLMs historically lagged behind humans.
What's inside
A collection of 1k to 10k challenging English test tasks extracted from the well-known BIG-Bench benchmark. The dataset focuses on tasks (question-answering, text and token classification) where language models in base mode performed worse than the average human evaluator.
What you can do with it
- Evaluate the reasoning and logic limits of large language models (LLMs)
- Test the effectiveness of prompting techniques (e.g., few-shot or chain-of-thought) on complex tasks
- Tasks
- Question answering, Token classification, Text classification
- Languages
- English
- Size
- 1K–10K rows
- Downloads this month
- 987
- Likes
- 47
Models trained on it
- davidschulte/ESM_maveriq__bigbenchhard_multistep_arithmetic_two 18
- davidschulte/ESM_maveriq__bigbenchhard_tracking_shuffled_objects_three_objects 10
- davidschulte/ESM_maveriq__bigbenchhard_temporal_sequences 9
- davidschulte/ESM_maveriq__bigbenchhard_tracking_shuffled_objects_seven_objects 7
- davidschulte/ESM_maveriq__bigbenchhard_movie_recommendation 7
- davidschulte/ESM_maveriq__bigbenchhard_causal_judgement 7
- davidschulte/ESM_maveriq__bigbenchhard_logical_deduction_seven_objects 6
- davidschulte/ESM_maveriq__bigbenchhard_salient_translation_error_detection 6
How to load it
from datasets import load_dataset
ds = load_dataset("maveriq/bigbenchhard")