Inkling 2 Small (MXFP8)
a 594B multimodal MoE model with a 1M context window
2 min readupdated
Inkling 2 Small is a multimodal model from Thinking Machines that accepts text and images (from 40px to 4096px) to generate text responses. Built on a 48-layer transformer with an MoE architecture (Mixture-of-Experts, a sparse setup where 8 out of 384 modules activate per token to speed up inference), only 17B of its 594B total parameters run at any given time. The model features a 1M-token context window and hybrid attention combining local and global layers. Stored in the MXFP8 format, these weights are geared toward developers building AI agents, retrieval-augmented generation (RAG) systems, and coding assistants.
Key features
- 594B / 17B MoE architecture: 48 decoder layers routing 8 of 384 experts per token, paired with a built-in vision encoder for processing text and images together.
- Up to 1M-token context: Holds massive volumes of text and visual data in memory within a single request.
- MXFP8 format: An 8-bit floating-point variant that scores 1–2 percentage points higher on benchmarks compared to the base NVFP4 release.
- Broad framework support: Works out of the box for local inference via SGLang, vLLM, TokenSpeed, and Unsloth, as well as fine-tuning with Tinker, Miles, and SkyRL.
What you can do
- Build autonomous AI agents: Power systems that rely on external tool-use and complex instruction following.
- Run a coding assistant: Generate, analyze, and debug code across multiple programming languages while keeping large codebases in context.
- Create multimodal RAG services: Analyze long documents, diagrams, and images inside a context window of up to 1M tokens.
Example
You can test the model in the interactive Tinker Playground, call it via API providers (NVIDIA, Fireworks, Together, Baseten, Nebius, Modal, Cloudflare), or deploy it locally using recipes for vLLM, SGLang, TokenSpeed, and Unsloth—passing in UTF-8 text alongside images ranging from 40×40 to 4096×4096 pixels.
Read more
DetailsBenchmarks: AA Index (v5.0) — 30.6, Vals Index (v2.2) — 56.1%, FrontierCode v2.0 — 25.4%.
Hardware requirements: The MXFP8 version requires a GPU cluster with at least 800 GB of total VRAM (such as 4× NVIDIA B300 or 8× NVIDIA H200).
Authors' note: The model was trained with NVFP4 as its base precision (which requires 500+ GB VRAM), and the authors consider MXFP8's 1–2 percentage point benchmark gain negligible for most practical workloads.










Comments