Skip to content

Dataset · opencompass

NeedleBench benchmark for long text fact retrieval

opencompass/NeedleBench

A test suite from OpenCompass with 1 to 10 thousand text tasks to evaluate LLM retrieval and reasoning over large documents.

What's inside

The dataset is part of the OpenCompass project and contains between 1 and 10 thousand text examples for question-answering tasks. It includes a series of test scenarios: single-needle retrieval, multi-needle retrieval, and multi-hop reasoning based on several fragments. Specific language is not specified in the metadata, but the description is provided in English.

What you can do with it

  • Verifying single-fact extraction accuracy from long context (Needle-in-a-Haystack)
  • Evaluating LLM capability to synthesize information from multiple parts of a large document
  • Comparing long-context models within the OpenCompass ecosystem
Tasks
Question answering
Size
1K–10K rows
Downloads this month
6.8K
Likes
7

How to load it

from datasets import load_dataset

ds = load_dataset("opencompass/NeedleBench")