Dataset · soberania
Largest cleaned Portuguese corpus for LLMs
soberania/jabuticaba
669 GB of cleaned and deduplicated Portuguese texts (over 139 billion tokens) for LLM training.
What's inside
A massive corpus from the SoberanIA project totaling 669 GB, containing over 139 billion tokens (calculated using OpenAI's tiktoken tokenizer) and between 100 million and 1 billion entries. The Portuguese texts have been cleaned and deduplicated and are approved for commercial use among others. Table and text modalities are specified in the metadata.
What you can do with it
- Pre-training or continued pre-training of language models in Portuguese
- Training fill-mask language models and embeddings
- Building localized commercial LLMs for the Portuguese-speaking market
- Tasks
- Text generation, Fill mask
- Languages
- Portuguese
- Size
- 100M–1B rows
- Downloads this month
- 467
- Likes
- 25
- Access
- on request on Hugging Face
How to load it
from datasets import load_dataset
ds = load_dataset("soberania/jabuticaba")