SpeechLLM ASR: English speech recognition powered by Villanova-2B
1 min readupdated
This is a multimodal speech language model (SpeechLLM) for automatic English speech recognition, built on the SLAM-LLM framework. The architecture combines a frozen FAMA encoder, a linear projector, and the Villanova-2B model with LoRA (a lightweight adapter for the base model). The checkpoint contains trained weights totaling 14.16 million parameters for the projector and adapters. The model outperforms Whisper large-v3-turbo on the VoxPopuli dataset and efficiently handles audio recordings longer than 30 seconds.
Key features
- Uses a combination of the FAMA encoder and the Villanova-2B language model.
- Achieves a WER of 6.55% on the VoxPopuli English test set.
- Capable of processing long audio fragments without truncation.
- Trained using the SLAM-LLM framework on NVIDIA A100 GPUs.
What you can do
- Recognize English speech with high accuracy in audio projects and services.
- Run decoding or fine-tuning using the scripts from the Villanova-Speech-LLM repository.
Example
hf download SpeechTek/speechllm_fama_meusli_villanova_asr_en_v1 model.pt --local-dir checkpoints/speechllm_fama_meusli_villanova_asr_en_v1Read more
Details• Architecture: FAMA encoder + Villanova-2B (LLM) with LoRA
• Training: VoxPopuli English, NVIDIA A100 64 GB
• Metrics: WER 6.55% on VoxPopuli English test
• License: CC BY 4.0










Comments