Skip to content
Other

Inkling-2-Small-Vision

standalone vision encoder from Inkling 2 Small

2 min readupdated

This repository packages the vision tower (image encoder) and projector extracted from the multimodal Inkling 2 Small model by Thinking Machines. It takes an image, passes it through 23 transformer blocks, compresses the tokens, and outputs 4608-dimensional visual embeddings already aligned with the language backbone. Under the hood is Meta's PE-Lang L architecture (vit_pe_lang_large_patch14_448 in timm), fine-tuned alongside the language model, weighing in at around 0.29B parameters. Unlike a full VLM, this release strips out the text model to offer a standalone vision module, preserving the original BF16 weights with verified exact tensor parity. It is built for multimodal developers who need a drop-in vision backbone or a clean reference implementation for vLLM.

Key features

  • Meta and Thinking Machines foundation: The vision tower is initialized from vit_pe_lang_large_patch14_448_timm and fine-tuned as part of Inkling 2 Small.
  • Parameter-free token compression: 2x2 spatial patch grouping merges 1024 tokens of width 1024 into 256 tokens of width 4096 without any trainable weights.
  • Included projector: A two-layer MLP (Linear(4096, 4096) → GELU → Linear(4096, 4608)) maps visual features directly into the LLM latent space.
  • Bit-exact weight parity: Every tensor is verified via torch.equal, and the forward pass is validated against the reference vLLM implementation.

What you can do

  • Extract visual features: Run the model in image-feature-extraction pipelines to generate dense embeddings from 640×640 images.
  • Build custom VLMs: Attach the pre-aligned vision tower and projector to a compatible language model or use them as a base for multimodal experiments.
  • Test and debug inference: Use the module and the included inkling2_vision.py script to verify vision encoder compatibility when porting to inference engines like vLLM.

Example

The repo includes a ready-to-use inkling2_vision.py loader (built on timm) and an examples/inference.py script for feature extraction. Feeding an image at its native 640×640 px resolution produces a 32×32 patch grid (patch size 20), which maps to 1024 tokens of width 1024, compresses via 2×2 grouping to 256 tokens of width 4096, and exits the projector as 256 visual tokens of width 4608.

Read moreDetails
Parameters: 0.29B (420 tensors in model.safetensors and 2 tensors in projector.safetensors, BF16)
Tower architecture: 23 blocks, hidden size 1024, 16 attention heads, patch size 20, 2D RoPE, exact GELU, LayerScale
Native resolution: 640 px (32 × 32 patch grid), projector output — 256 × 4608
License: Apache-2.0
Discuss Telegram 0 7

Will it run?

Runs in BF16

Fits without quantisation.

  • BF16weights 0.5 GB · ready files1.1 GB
  • FP8 / INT8recommendedweights 0.3 GB0.8 GB
  • 4-bitweights 0.2 GB0.7 GB

Tip: go 8-bit.FP8 / INT8 is within about a percent of full precision in quality, takes half the memory and runs faster on RTX 40/50 and H100. Full precision only pays off for fine-tuning.

What to get for this model

Massed ComputeOpen Massed Compute

Your own GPU: FP8 / INT8 — from RTX 2060 · BF16 — from RTX 2060

Massed Compute hourly prices, November 2026. The discounted price applies with the coupon at checkout.

Estimate: weights plus working memory. Long videos, high resolutions and long contexts need more.

Run it on Massed Compute

The script downloads the model to the rented machine. How to run it is in the model card.

  1. 1. Rent a machine with a fitting GPU

    Open Massed Compute
  2. 2. Paste the script into the machine's terminal

    Connect over SSH (the Massed Compute panel shows the command) or open a terminal on the machine's desktop.

    HF_TOKEN / CIVITAI_TOKENThe script starts with token lines: paste yours between the quotes. The Hugging Face token is for gated models, the Civitai one for Civitai models.

    # immanuelpeter/Inkling-2-Small-Vision
    mkdir -p ~/models
    curl -LsSf https://astral.sh/uv/install.sh | sh
    source "$HOME/.local/bin/env"
    uv tool install "huggingface_hub[cli]"
    hf download immanuelpeter/Inkling-2-Small-Vision --local-dir ~/models/Inkling-2-Small-Vision
    # The model is in ~/models. How to run it is in the model card (link above).
    Add more LoRAs to Inkling-2-Small-Vision
  3. 3. Use it

    The model is in ~/models. How to run it is in the model card (link above).

Stop the machine in the panel when you are done: it bills by the minute while it runs.

How to speed it up

Picked for this model type. Choose your hardware above to see what fits you exactly.

  • FlashAttention 44.0.0b40from November 18, 2026new release

    fastest exact attention on B200

    Kernels for Blackwell (beta). Exact attention without quantisation — fine for everything, LLMs included.

    How to enable: pip install flash-attn-4; vLLM and SGLang pick it up themselves.

    Earlier generations (2)
    • FlashAttention 3 — The Hopper (H100/H200) version with FP8.
    • FlashAttention 2 — The standard for RTX 30/40/50 and A100: exact attention without the memory cost of long contexts.
  • torch.compile2.15.0from October 28, 2026new release

    10–30% for free

    Compiles the model for your GPU. Slower first run, faster afterwards.

    How to enable: model = torch.compile(model); in ComfyUI the TorchCompileModel node.

Model family

New posts about versions of Inkling-2-Small (quants, finetunes, LoRAs) will show up under the bell.

Quantised versions 12All on Hugging Face
Fine-tunes 9All on Hugging Face
LoRAs and adapters 4All on Hugging Face

About the model

immanuelpeter/Inkling-2-Small-Vision
Author
immanuelpeter
Category
Other
Released
Posted

On NeyroLearn

Hugging Face

70

Ask AI about this model

Answered by Gemini 4 · the chat is kept only in this tab

Comments