BabyLM Dense Baseline
A Compact Language Model for Research
2 min readupdated
The 47-million-parameter BabyLM Dense Baseline model was created for research into training language models on limited amounts of text. It generates text continuations or can be used to evaluate probabilities using a built-in masked prediction method. The architecture relies on 10 independent decoder blocks with rotary positional embeddings and pre-RMS normalization. The model features a mixed training strategy combining causal and masked language generation and serves as a baseline for future architectural experiments. It is useful for researchers and developers studying efficient model training on small text corpora.
Key features
- 47 million parameters and a compact size for quick deployment
- Trained on the 10-million-word BabyLM Strict-Small English corpus
- Utilization of causal prediction and masked next-token prediction (MNTP)
- Compatibility with the standard Llama implementation in the transformers library
What you can do
- Generate text based on a short English prompt
- Use the built-in helper to calculate log probabilities of masked tokens
- Conduct your own research on training models with limited text
Example
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "AwakeningOS/babylm-dense-baseline-47m"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo).eval()
prompt = [tok.bos_token_id] + tok.encode("When the rain stopped,", add_special_tokens=False)
ids = torch.tensor([prompt])
with torch.inference_mode():
out = model.generate(ids, attention_mask=torch.ones_like(ids),
max_new_tokens=20, do_sample=False)
print(tok.decode(out[0].tolist(), skip_special_tokens=True))
Read more
DetailsBabyLM NLP average result: 52.7745
Average across six zero-shot benchmarks: 50.8106
Average across seven supervised tasks: 64.5581
Details: uses an architecture with a block width of 384 and SwiGLU layers, along with a 32,768-element Unigram tokenizer.










Comments