Skip to content

Dataset · SciPhi

Corpus and embeddings for agent search

SciPhi/AgentSearch-V1

Over 50 million documents and 1 billion passages with jina-v2-base embeddings for training search.

What's inside

The English-language AgentSearch-V1 corpus includes over 50 million documents and upwards of 1 billion passages, sourced from Arxiv, Wikipedia, Project Gutenberg, and filtered Creative Commons data. Embeddings (over 1 billion) were generated using the jina-v2-base model. Metadata lists the size as 10k–100k, but the card claims a billion-scale; the format is not detailed in the card.

What you can do with it

  • Training retrieval models or agentic search
  • Building indices and evaluating RAG systems
  • Fine-tuning embeddings or rerankers on passages
Tasks
Text generation
Languages
English
Size
10K–100K rows
Downloads this month
3K
Likes
92

How to load it

from datasets import load_dataset

ds = load_dataset("SciPhi/AgentSearch-V1")