Skip to content

Dataset · AgentPublic

Open government data corpus with 380 billion tokens

AgentPublic/open_government

Massive text and table corpus from open government archives of the USA, France, and EU for language model pre-training.

What's inside

Contains about 380 billion tokens (10 to 100 million records) in text and table formats. Materials are taken from open government programs of the USA, France, European, and international organizations, including 16 finance and law collections. Languages include English, French, Bulgarian, Croatian, Czech, and Danish.

What you can do with it

  • Pre-training large language models on official, legal, and financial texts
  • Building RAG systems for government and legal document search
  • Training models to handle tabular data from the public sector
Tasks
Text generation
Languages
English, French, Bulgarian, Croatian, Czech, Danish, Dutch, Estonian and 4 more
Size
10M–100M rows
Downloads this month
3.6K
Likes
7

How to load it

from datasets import load_dataset

ds = load_dataset("AgentPublic/open_government")