Dataset · ReactiveAI
English and Polish LLM pre-training corpus
ReactiveAI/Beta-Pre-Train-Corpus
A massive cleaned text and tabular corpus (100M-1B records) featuring web text, science, math, and code.
What's inside
A pre-training corpus for RxT-Beta models compiled from open datasets (including FineWeb-Edu, MegaMath Web Pro, and GitHub Code). Contains English and Polish texts, science and math sections, and multi-language programming code filtered by length from 512-1024 to 2048 tokens. Total size ranges from 100 million to 1 billion examples.
What you can do with it
- Pre-training language models with English and Polish language support
- Fine-tuning models to improve math, science, and programming skills
- Size
- 100M–1B rows
- Downloads this month
- 23.7K
- Likes
- 0
Models trained on it
How to load it
from datasets import load_dataset
ds = load_dataset("ReactiveAI/Beta-Pre-Train-Corpus")