Dataset · SciPhi
Corpus and embeddings for agent search
SciPhi/AgentSearch-V1
Over 50 million documents and 1 billion passages with jina-v2-base embeddings for training search.
What's inside
The English-language AgentSearch-V1 corpus includes over 50 million documents and upwards of 1 billion passages, sourced from Arxiv, Wikipedia, Project Gutenberg, and filtered Creative Commons data. Embeddings (over 1 billion) were generated using the jina-v2-base model. Metadata lists the size as 10k–100k, but the card claims a billion-scale; the format is not detailed in the card.
What you can do with it
- Training retrieval models or agentic search
- Building indices and evaluating RAG systems
- Fine-tuning embeddings or rerankers on passages
- Tasks
- Text generation
- Languages
- English
- Size
- 10K–100K rows
- Downloads this month
- 3K
- Likes
- 92
How to load it
from datasets import load_dataset
ds = load_dataset("SciPhi/AgentSearch-V1")