Dataset · coral-nlp
154.56B Tokens of Open German Text
coral-nlp/german-commons
A large corpus of openly licensed German texts from 41 sources for training language models.
What's inside
Combines German-language texts and tabular data from 41 different sources under open licenses. The total volume is 154.56 billion tokens (10 million to 100 million records), with an included Bloom filter compatible with DOLMA. The corpus is designed for training German text generation models.
What you can do with it
- Pretraining and fine-tuning German LLMs
- Filtering and deduplication of corpora using the included bloom_filter.bin
- Research in German text generation
- Tasks
- Text generation
- Languages
- German
- Size
- 10M–100M rows
- Downloads this month
- 3.6K
- Likes
- 42
Models trained on it
How to load it
from datasets import load_dataset
ds = load_dataset("coral-nlp/german-commons")