Dataset · BAAI
BAAI Chinese and English industry text corpus
BAAI/IndustryCorpus
A large cleaned text corpus of 100 million to 1 billion records for training industry LLMs.
What's inside
A large-scale Chinese and English text dataset from the Beijing Academy of Artificial Intelligence (BAAI). To create it, the authors applied 22 specialized industry data processing operators. The project solves the problem of a lack of high-quality, deep domain information when training enterprise language models.
What you can do with it
- Pre-training and fine-tuning domain-specific LLMs
- Adapting generative models for corporate and industrial tasks
- Building text generation systems with deep subject matter expertise
- Tasks
- Text generation
- Languages
- Chinese, English
- Size
- 100M–1B rows
- Downloads this month
- 7.7K
- Likes
- 61
Models trained on it
How to load it
from datasets import load_dataset
ds = load_dataset("BAAI/IndustryCorpus")