Skip to content

Dataset · meaecosystem

Large text corpus focused on the Indonesian language

meaecosystem/pustaka

Filtered multilingual web corpus ranging from 1 to 10 billion records for training language models.

What's inside

An open text corpus from MEA Ecosystem with a volume of 1 to 10 billion examples, created with a primary focus on the Indonesian language (English, Japanese, Chinese, and Korean are also present). Data was gathered from web crawling, encyclopedias, news, forums, and slang dictionaries. All texts underwent cleaning with language detection and document length filtering.

What you can do with it

  • Pretraining and fine-tuning LLMs on Indonesian and Asian languages
  • Adapting tokenizers and language models to colloquial vocabulary
Tasks
Text generation
Languages
Indonesian, English, Japanese, Chinese, Korean
Size
1B–10B rows
Downloads this month
67.8K
Likes
13

How to load it

from datasets import load_dataset

ds = load_dataset("meaecosystem/pustaka")