ORB planner
a compact AI model for generating sound specs from text
2 min readupdated
This 5.10M-parameter decoder-only transformer turns a single-line prompt (such as a description of weather or sound effects) into a structured JSON object (Spec). Rather than synthesizing raw audio directly, it hands the output off to a deterministic DSP renderer in the ORB repo, which compiles the format into a 48 kHz WAV file. Trained from scratch on 300k synthetic pairs without pretrained weights or external audio datasets, it is built for UI developers, game devs, and sound designers who need programmatic control over sound effect generation.
Key features
- Compact size — only about 5.10M parameters with a 1024-token context window.
- Strict grammar — grammar-masked decoding guarantees a valid JSON config every time without failures.
- Wide sound palette — supports 47 sound roles (from clicks and whooshes to footsteps and lightning strikes), 13 textures (wind, rain, city noise, and more), and 10 moods.
- Agent integration — the repository includes an MCP server so you can expose sound generation as agent tools.
What you can do
- Generate sound effects — create custom UI pops, whooshes, and ambient textures from text prompts for apps and games.
- Plug into pipelines — use the model as an audio planner and tweak parameters via a structured JSON object.
- Connect AI agents — hook up the MCP server to let AI agents automate soundscape creation.
- Procedural rendering — pair the generator with static DSP algorithms to get clean 48 kHz audio without neural audio synthesis artifacts.
Example
from huggingface_hub import hf_hub_download
from orb.api import load_planner, generate, save_wav
path = hf_hub_download("marcosmaceo/orb-planner", "planner.safetensors")
hf_hub_download("marcosmaceo/orb-planner", "config.json")
tok, planner = load_planner(path)
out = generate(
"6.00-second bright cue, wind bed, with whoosh @1.00 0.90s -6dB L2 #2a",
planner=(tok, planner),
)
save_wav(out["audio"], "whoosh.wav")
print(out["spec"])Read more
Details• Accuracy on grammar test prompts: 100% (500/500 and 2000/2000 samples).
• Accuracy on free-form text prompts: 58.4% (custom data) and 70.0% (blind set).
• Decoding errors: 0.
• Training: 40k steps on a single H100 GPU (300k synthetic rows dataset).
• License: Apache-2.0.










Comments