Dataset · stai-tuebingen
SmolLM text novelty evaluation corpus
stai-tuebingen/faiss-smollm
Pretrain corpus and FAISS tutorial for measuring text query novelty for SmolLM and SmolLM2 models.
What's inside
Includes text pretraining data for SmolLM and SmolLM2 models ranging from 1B to 10B in size. It features a pipeline for generating embeddings and searching for similar documents using FAISS with optional ColBERTv2 reranking.
What you can do with it
- Evaluating text query novelty against the SmolLM pretrain corpus
- Setting up similar document retrieval pipelines with FAISS and ColBERTv2
- Size
- 1B–10B rows
- Downloads this month
- 19.9K
- Likes
- 0
How to load it
from datasets import load_dataset
ds = load_dataset("stai-tuebingen/faiss-smollm")