Skip to content

Dataset · OpenLLM-France

Multilingual text corpus for pre-training Lucie-7B LLM

OpenLLM-France/Lucie-Training-Dataset

Tens of billions of text examples in five European languages and code for pre-training base language models.

What's inside

The corpus contains 10 to 100 billion records in English, French, German, Spanish, and Italian, as well as examples in various programming languages. The data is gathered from web pages, video subtitles, research articles, digital books, newspapers, and magazines, some of which were processed via OCR. The dataset was used to pre-train the Lucie-7B base model.

What you can do with it

  • Pre-training multilingual base LLMs
  • Fine-tuning models to improve knowledge of European languages and code
  • Researching filtering and balancing of heterogeneous text corpora
Tasks
Text generation
Languages
English, French, German, Spanish, Italian, Code
Size
10B–100B rows
Downloads this month
12.9K
Likes
40

Models trained on it

How to load it

from datasets import load_dataset

ds = load_dataset("OpenLLM-France/Lucie-Training-Dataset")