Skip to content

Dataset · stai-tuebingen

SmolLM text novelty evaluation corpus

stai-tuebingen/faiss-smollm

Pretrain corpus and FAISS tutorial for measuring text query novelty for SmolLM and SmolLM2 models.

What's inside

Includes text pretraining data for SmolLM and SmolLM2 models ranging from 1B to 10B in size. It features a pipeline for generating embeddings and searching for similar documents using FAISS with optional ColBERTv2 reranking.

What you can do with it

  • Evaluating text query novelty against the SmolLM pretrain corpus
  • Setting up similar document retrieval pipelines with FAISS and ColBERTv2
Size
1B–10B rows
Downloads this month
19.9K
Likes
0

How to load it

from datasets import load_dataset

ds = load_dataset("stai-tuebingen/faiss-smollm")