Skip to content

Dataset · enguyen

SmolLM corpus chunks and FAISS search indices

enguyen/smollm-chunked

SmolLM/SmolLM2 text fragments and FAISS vector indices for evaluating LLM generation novelty.

What's inside

Inside are chunked text data from the SmolLM and SmolLM2 corpora, as well as ready-to-use FAISS search indices. The dataset size ranges from 100 million to 1 billion records. The materials were gathered as part of research into language model generation novelty through the lens of semantic similarity.

What you can do with it

  • Evaluating the novelty of LLM-generated texts and finding matches with the training set
  • Testing and benchmarking semantic search systems on large volumes
  • Exploring semantic similarity in pre-training data
Tasks
Text retrieval
Size
100M–1B rows
Downloads this month
20.4K
Likes
1

How to load it

from datasets import load_dataset

ds = load_dataset("enguyen/smollm-chunked")