Dataset · enguyen
SmolLM corpus chunks and FAISS search indices
enguyen/smollm-chunked
SmolLM/SmolLM2 text fragments and FAISS vector indices for evaluating LLM generation novelty.
What's inside
Inside are chunked text data from the SmolLM and SmolLM2 corpora, as well as ready-to-use FAISS search indices. The dataset size ranges from 100 million to 1 billion records. The materials were gathered as part of research into language model generation novelty through the lens of semantic similarity.
What you can do with it
- Evaluating the novelty of LLM-generated texts and finding matches with the training set
- Testing and benchmarking semantic search systems on large volumes
- Exploring semantic similarity in pre-training data
- Tasks
- Text retrieval
- Size
- 100M–1B rows
- Downloads this month
- 20.4K
- Likes
- 1
How to load it
from datasets import load_dataset
ds = load_dataset("enguyen/smollm-chunked")