Dataset · AgentPublic
Open government data corpus with 380 billion tokens
AgentPublic/open_government
Massive text and table corpus from open government archives of the USA, France, and EU for language model pre-training.
What's inside
Contains about 380 billion tokens (10 to 100 million records) in text and table formats. Materials are taken from open government programs of the USA, France, European, and international organizations, including 16 finance and law collections. Languages include English, French, Bulgarian, Croatian, Czech, and Danish.
What you can do with it
- Pre-training large language models on official, legal, and financial texts
- Building RAG systems for government and legal document search
- Training models to handle tabular data from the public sector
- Tasks
- Text generation
- Languages
- English, French, Bulgarian, Croatian, Czech, Danish, Dutch, Estonian and 4 more
- Size
- 10M–100M rows
- Downloads this month
- 3.6K
- Likes
- 7
How to load it
from datasets import load_dataset
ds = load_dataset("AgentPublic/open_government")