LFM3-Audio-2B-Full-Duplex
real-time full-duplex voice AI
3 min readupdated
An adaptation of the base LiquidAI/LFM3-Audio-2B audio model by MAdel121 that turns it into a full-duplex voice conversationalist with an 80 ms processing step. Unlike the original model, which strictly takes turns, this version listens and speaks simultaneously: it yields when interrupted, doesn't get derailed by brief user acknowledgments, and drops in natural "uh-huh" and "yeah" cues on its own without external VAD (voice activity detection), ASR (speech recognition), or TTS (text-to-speech) modules. It relies on a lightweight LoRA adapter (a small add-on to the base model, 68 MB) and a separate 91K-parameter Conv1D head to generate backchannel reactions. Built for researchers and local voice assistant developers who need natural, low-latency conversation, including on Apple Silicon.
Key features
- 80 ms streaming mode: takes in a
16 kHzuser audio stream and generates24 kHzresponse audio viaLiquidCodecinside a single continuous conversation. - Resilient to backchanneling: window-based training lets the model keep speaking through brief user interjections like "yeah, right" in
93.3%of cases, compared to58.3%with a frame-by-frame approach. - Isolated reactions: a separate Conv1D head plays back 3 pre-encoded "uh-huh/yeah" audio clips while bypassing the main model's KV cache, avoiding context pollution and keeping future responses on track.
- Base knowledge preserved: self-distillation using the model's own responses with a KL anchor eliminated persona leakage and retained roughly
95%of the base model's accuracy on spoken QA (29–30 / 120vs.31 / 120).
What you can do
- Run a local browser voice chat: test live conversation and barge-in behavior on headphones via the built-in
eval.liveweb interface. - Process pre-recorded audio: run
.wavand.flacfiles offline througheval.demoto evaluate turn-taking timing and interruption handling. - Embed in a research assistant prototype: use the out-of-the-box
80 msruntime as a compact alternative to heavy cascades of separate ASR and TTS models.
Example
# Запуск живого голосового чата в браузере (требуются наушники)
python -m eval.live serve --out talks/local --key live
# Офлайн-тест на аудиофайле
python -m eval.demo run --champion --in user_input.wav --out demo_out
Read more
DetailsBenchmarks (Full-Duplex-Bench v1.0, 727 clips): pause TOR of
0.044 (synthetic) and 0.116 (CANDOR), interruption relevance of 4.38 / 5, and a median interruption stop time of 1.22 s.Hardware and speed: runs locally in
float32 on Apple Silicon (MPS) at 59 ms per 80 ms step; session length is capped at roughly 3 minutes due to KV cache growth.Training and size: the LoRA (rank 16,
68 MB) was trained in 45 GPU minutes, and the Conv1D head weighs 1.4 MB; requires the LiquidAI/LFM3-Audio-2B weights and Python 3.12.Limitations and license: non-commercial research only (LFM Open License v1.0 + non-commercial datasets); wired headphones are mandatory in the browser since echo cancellation is disabled.










Comments