Dataset · meaecosystem
Large text corpus focused on the Indonesian language
meaecosystem/pustaka
Filtered multilingual web corpus ranging from 1 to 10 billion records for training language models.
What's inside
An open text corpus from MEA Ecosystem with a volume of 1 to 10 billion examples, created with a primary focus on the Indonesian language (English, Japanese, Chinese, and Korean are also present). Data was gathered from web crawling, encyclopedias, news, forums, and slang dictionaries. All texts underwent cleaning with language detection and document length filtering.
What you can do with it
- Pretraining and fine-tuning LLMs on Indonesian and Asian languages
- Adapting tokenizers and language models to colloquial vocabulary
- Tasks
- Text generation
- Languages
- Indonesian, English, Japanese, Chinese, Korean
- Size
- 1B–10B rows
- Downloads this month
- 67.8K
- Likes
- 13
How to load it
from datasets import load_dataset
ds = load_dataset("meaecosystem/pustaka")