AutoMathText-2.5 math and STEM corpus with 2T tokens
Cleaned bilingual (EN/ZH) corpus exceeding 2 trillion tokens on mathematics, code, and reasoning for LLM training.
- Text generation
- Question answering
- English
- Chinese
Hugging Face datasets with plain-language descriptions: what's inside and what models to train on them. 4,788 in the catalog so far, with new ones added every hour.
Answered by Gemini 4 · the chat is kept only in this tab
Cleaned bilingual (EN/ZH) corpus exceeding 2 trillion tokens on mathematics, code, and reasoning for LLM training.
English video QA dataset for evaluating multimodal neural network video understanding in real time.
The largest open corpus of over 1 million Swiss court decisions for legal NLP tasks.
Real multilingual user conversations with ChatGPT featuring demographics and prompt metadata.
79.6 million Wikipedia-based text examples for end-to-end training of small reasoning models.
A multimodal dataset with schematic images and questions to test VLM engineering reasoning.
Over 33 thousand open legal articles and publications in Traditional Chinese for fine-tuning language models.
About 200k English school math word problems with answers generated by GPT-4-Turbo.
Collection of 48,284 English books from Project Gutenberg, prepared for generative model training.
Q&A based on historical US Treasury bulletins to evaluate AI agents' ability to analyze tables, charts, and text.
Over 720,000 PII-clean posts spanning humor, prose, news, and tech.
A large multilingual audio dataset (10M-100M entries) with pseudo-labels for European speech recognition and translation.
Updated data for LLM pretraining: math, code, English Common Crawl, and synthetic data.
Processed UltraFeedback: 64k prompts with model answers and GPT-4 scores aggregated into preference pairs.
Large raw archive of successful and failed web agent actions in English and Korean for subsequent filtering.
Classic English benchmark for testing model fact-answering capability without source context.
Verse-aligned Bible texts for machine translation of low-resource languages.
Around 900k competitive math problems with Chain of Thought solutions.
290 billion tokens of English and Chinese math data across three tiers for LLM pre-training.
Test set of complex search queries and a fixed corpus of 100 thousand web documents for fair agent comparison.
3,763 long captioned videos with questions to evaluate long-context multimodal models.
A massive corpus of digitized historical US newspapers in English for training language models.
Evaluation set of images and English text across 10 tasks and 30 subtasks for testing U-MLLMs.
A text corpus of 10 to 100M records for pre-training models in English, French, and West African languages.