RetNet-0.5B-16k
experimental long-context research export
2 min readupdated
An experimental 0.5B-parameter language model built on the RetNet architecture, trained on sequences up to 16,384 tokens. While it shows stronger long-document modeling, it comes with a major caveat: needle retrieval across large contexts performs no better than random guessing. The release is aimed at researchers studying alternative architectures and long-sequence processing without standard attention.
Key features
- 504 million parameters across 20 layers and 22 heads
- Trained at a 16k context length on roughly 0.85B tokens
- Significant metric gains on the long tail of 16k documents compared to the base version
- Uses a dedicated chunkwise path to handle long sequences efficiently
What you can do
- Run the model in research scripts to experiment with the RetNet architecture
- Test long-text processing using the RETNET_FAST_CHUNKWISE=1 environment variable
- Use the code to analyze generation metrics and benchmark chunkwise path performance
Example
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "Ayaka157/RetNet-0.5B-16k"
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, torch_dtype="float32").eval()Read more
DetailsBenchmarks: 16k within-doc long-tail improved to 2.5942 (down from 3.2841 for the base model), while needle@16k remained at random chance (0.0978).
Hardware requirements: 1.88 GiB file size (fp32); training required 2×RTX PRO 6000 GPUs with 83.1 GiB peak memory usage.
Limitations: not suitable for RAG or information retrieval; use the v21.3-chat version for chat tasks.










Comments