Skip to content

Dataset · soberania

Largest cleaned Portuguese corpus for LLMs

soberania/jabuticaba

669 GB of cleaned and deduplicated Portuguese texts (over 139 billion tokens) for LLM training.

What's inside

A massive corpus from the SoberanIA project totaling 669 GB, containing over 139 billion tokens (calculated using OpenAI's tiktoken tokenizer) and between 100 million and 1 billion entries. The Portuguese texts have been cleaned and deduplicated and are approved for commercial use among others. Table and text modalities are specified in the metadata.

What you can do with it

  • Pre-training or continued pre-training of language models in Portuguese
  • Training fill-mask language models and embeddings
  • Building localized commercial LLMs for the Portuguese-speaking market
Tasks
Text generation, Fill mask
Languages
Portuguese
Size
100M–1B rows
Downloads this month
467
Likes
25
Access
on request on Hugging Face

How to load it

from datasets import load_dataset

ds = load_dataset("soberania/jabuticaba")