Inkling-2-Small-Vision
standalone vision encoder from Inkling 2 Small
2 min readupdated
This repository packages the vision tower (image encoder) and projector extracted from the multimodal Inkling 2 Small model by Thinking Machines. It takes an image, passes it through 23 transformer blocks, compresses the tokens, and outputs 4608-dimensional visual embeddings already aligned with the language backbone. Under the hood is Meta's PE-Lang L architecture (vit_pe_lang_large_patch14_448 in timm), fine-tuned alongside the language model, weighing in at around 0.29B parameters. Unlike a full VLM, this release strips out the text model to offer a standalone vision module, preserving the original BF16 weights with verified exact tensor parity. It is built for multimodal developers who need a drop-in vision backbone or a clean reference implementation for vLLM.
Key features
- Meta and Thinking Machines foundation: The vision tower is initialized from
vit_pe_lang_large_patch14_448_timmand fine-tuned as part of Inkling 2 Small. - Parameter-free token compression: 2x2 spatial patch grouping merges 1024 tokens of width 1024 into 256 tokens of width 4096 without any trainable weights.
- Included projector: A two-layer MLP (
Linear(4096, 4096)→ GELU →Linear(4096, 4608)) maps visual features directly into the LLM latent space. - Bit-exact weight parity: Every tensor is verified via
torch.equal, and the forward pass is validated against the reference vLLM implementation.
What you can do
- Extract visual features: Run the model in
image-feature-extractionpipelines to generate dense embeddings from 640×640 images. - Build custom VLMs: Attach the pre-aligned vision tower and projector to a compatible language model or use them as a base for multimodal experiments.
- Test and debug inference: Use the module and the included
inkling2_vision.pyscript to verify vision encoder compatibility when porting to inference engines like vLLM.
Example
The repo includes a ready-to-use inkling2_vision.py loader (built on timm) and an examples/inference.py script for feature extraction. Feeding an image at its native 640×640 px resolution produces a 32×32 patch grid (patch size 20), which maps to 1024 tokens of width 1024, compresses via 2×2 grouping to 256 tokens of width 4096, and exits the projector as 256 visual tokens of width 4608.
Read more
DetailsParameters: 0.29B (420 tensors in
model.safetensors and 2 tensors in projector.safetensors, BF16)Tower architecture: 23 blocks, hidden size 1024, 16 attention heads, patch size 20, 2D RoPE, exact GELU, LayerScale
Native resolution: 640 px (32 × 32 patch grid), projector output — 256 × 4608
License: Apache-2.0










Comments