Meridian21Lab/concept-memory-140m-11.5bt
a compact language model with concept memory for long texts
2 min readupdated
This new 141.5-million-parameter base model (not fine-tuned for chat) was trained on 11.5 billion tokens using a unique concept memory mechanism. As it reads, the model groups previously seen text into small token clusters ("concepts" of roughly 8 tokens each) and stores their summaries, attending only to the ones most relevant for predicting the next word. Even though each local layer only sees the last 128 tokens, the model handles contexts up to 64k tokens (around 50,000 words). Developed as part of the MrCogito project, it is built for efficient long-text compression and reasoning.
Key features
- Top long-context performance in its class: Outperforms similarly sized models on 64k-token documents, achieving 0.795 bits per byte.
- Full-history utilization: Prediction quality steadily improves as context grows all the way up to 61k tokens.
- High data efficiency: Scores strongly on grammar (80.56% on BLiMP) despite a modest training budget of just 11.5B tokens.
- Consumer-friendly hardware: Processes a full 64k-token document in a single pass on a standard RTX 3090 with 24 GB VRAM.
What you can do
- Research long-text processing: Test how the model handles full-length books, research papers, and reports in experimental setups.
- Build into analysis pipelines: Use it as an efficient base model for projects that need to retain large amounts of context on limited hardware.
- Explore memory architectures: Experiment with conceptual text compression mechanisms and benchmark different long-context approaches.
Example
Sample generation (temperature 0.7, top-p 0.9):
The capital of France is Paris, the second largest city in the world, with a population of 7.5 million people. It is known for its rich history and culture, as well as its iconic landmarks such as the Eiffel Tower, Louvre Museum, and Notre-Dame Cathedral. …
Read more
DetailsBenchmarks: BLiMP (grammar) — 80.56%, ARC-Easy — 48.95%, long context (64k) — 0.795 bits per byte.
Hardware requirements: a single RTX 3090 (24 GB VRAM) to run 64k-token documents.
Limitations: research release, base model (not instruction-tuned), trained on a single seed.
Training details: 141.5M parameters, 11.5B public tokens, training cost around $320 (roughly 40 hours on a 4-GPU node).










Comments