TinyForge-Vision-222M
a compact multimodal model for visual question answering
2 min readupdated
TinyForge-Vision-222M is an ultra-lightweight 222.1M-parameter VLM (vision-language model that understands images) from developer Juanillaberia. It connects the google/siglip-base-patch16-224 image encoder (85.8M) and the HuggingFaceTB/SmolLM2-135M language model (134.5M) via a pixel-shuffle projector (compressing 2×2 patches into a single token). Built as a research and educational reproduction of nanoVLM v0.1 and trained on the Cauldron dataset, it answers short questions about images and describes their contents in English, though it trails much larger VLMs by a wide margin. It is a solid fit for developers and students studying multimodal architecture and training from scratch.
Key features
- Compact footprint: just 222.1M parameters total (85.8M vision encoder, 1.8M projector, and 134.5M language model).
- Image compression into 49 tokens of dimension 576, fed into the language model ahead of the text tokens (up to 79 text tokens).
- Full unfreezing during training on 1.65M examples across 45 subsets of the Cauldron dataset at 224×224 resolution.
- Strict prompt format:
Question: {question} Answer:with no trailing space after the colon, matching the nanoVLM v0.1 codebase.
What you can do
- Get short answers to simple visual questions or detailed scene descriptions in English.
- Use the checkpoint and open repository as a hands-on reference for assembling and training a VLM from scratch.
- Run experiments on prompt formatting and fine-tuning ultra-compact multimodal models.
Example
from tfv.models.vlm import VisionLanguageModel
model = VisionLanguageModel.from_pretrained("Juanillaberia/TinyForge-Vision-222M")
Read more
DetailsMetrics: 27.6% ± 1.1 on the MMStar benchmark (1,500 multiple-choice questions; compared to 35.3% for the published nanoVLM-222M — the gap comes down to the missing space after Answer: in the v0.1 code).
Training: 5 epochs (32,330 steps, batch size 256, bf16) took 250 minutes on a single NVIDIA A100 80 GB in Google Colab; the best checkpoint was selected by MMStar score (~1.9 epochs).
Limitations: English only; resizing images to 224×224 without preserving aspect ratio hurts small text reading, documents, charts, and object counting.
Inference: runs through the custom
tinyforge-vision repository code rather than the standard transformers library.









Comments