Skip to content

Dataset · BAAI

BAAI Chinese and English industry text corpus

BAAI/IndustryCorpus

A large cleaned text corpus of 100 million to 1 billion records for training industry LLMs.

What's inside

A large-scale Chinese and English text dataset from the Beijing Academy of Artificial Intelligence (BAAI). To create it, the authors applied 22 specialized industry data processing operators. The project solves the problem of a lack of high-quality, deep domain information when training enterprise language models.

What you can do with it

  • Pre-training and fine-tuning domain-specific LLMs
  • Adapting generative models for corporate and industrial tasks
  • Building text generation systems with deep subject matter expertise
Tasks
Text generation
Languages
Chinese, English
Size
100M–1B rows
Downloads this month
7.7K
Likes
61

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("BAAI/IndustryCorpus")