QueryLM 3.3B
A base model with built-in search support
2 min readupdated
QueryLM 3.3B (DCLM-Edu) is a 3.3-billion parameter language model trained from scratch on web texts with embedded search block snippets. During pre-training, search queries and relevant document excerpts were added to the text at key points, teaching the model to independently formulate search queries and use retrieved information for text generation. The model architecture follows the nanochat style (decoder, 32 layers, 4,095-token context). It is useful for researchers and developers studying the integration of search engines into language models during the pre-training stage.
Key features
- Trained on the DCLM-Edu corpus with embedded search query and response blocks.
- Based on a nanochat-style decoder architecture with a hidden size of 2,816 and 22 attention heads.
- Ships with custom loading code (`modeling_nanochat.py`) that requires the `trust_remote_code=True` parameter.
- Serves as a research artifact for studying models capable of searching for information during generation.
What you can do
- Set up a generation loop with an external search engine (for example, via BM25 or a search API) so the model requests up-to-date data.
- Integrate the model into custom text processing pipelines using the `transformers` library.
- Experiment with generating search queries inside a special token ``.
Example
An example of using the model to generate a search query and a response based on it:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "Kausp11/querylm-dclm-3.3b"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, torch_dtype=torch.bfloat16).eval()
Q_OPEN = tok.convert_tokens_to_ids("")
print("Query token ID:", Q_OPEN)
Read more
DetailsParameters: 3.3B reported (5.5B stored, including embedding values).
Tokenizer: SmolLM2 with four additional tokens for working with the search environment.
Limitations: Base model (simply continues text, no chat format or safety instructions), works only in English.
Requirements: `transformers` with support for executing remote code.










Comments