Skip to content

Dataset · ReactiveAI

English and Polish LLM pre-training corpus

ReactiveAI/Beta-Pre-Train-Corpus

A massive cleaned text and tabular corpus (100M-1B records) featuring web text, science, math, and code.

What's inside

A pre-training corpus for RxT-Beta models compiled from open datasets (including FineWeb-Edu, MegaMath Web Pro, and GitHub Code). Contains English and Polish texts, science and math sections, and multi-language programming code filtered by length from 512-1024 to 2048 tokens. Total size ranges from 100 million to 1 billion examples.

What you can do with it

  • Pre-training language models with English and Polish language support
  • Fine-tuning models to improve math, science, and programming skills
Size
100M–1B rows
Downloads this month
23.7K
Likes
0

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("ReactiveAI/Beta-Pre-Train-Corpus")