Skip to content

Dataset · Skywork

Large Chinese web corpus for LLMs

Skywork/SkyPile-150B

Large filtered and deduplicated Chinese web text dataset for language model pre-training.

What's inside

SkyPile-150B is a large-scale Chinese text collection gathered from public web pages. The authors applied strict filtering, deduplication, and sensitive data removal, using fastText and BERT to weed out low-quality content. According to metadata, the size ranges from 1M to 10M records; the exact volume is not specified in the provided excerpt.

What you can do with it

  • Pre-training Chinese LLMs
  • Fine-tuning models on Chinese web texts
  • Evaluating filtering and deduplication pipelines
Tasks
Text generation
Languages
Chinese
Size
1M–10M rows
Downloads this month
14.8K
Likes
421

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("Skywork/SkyPile-150B")