Whisper Small Multilingual CS
a model for mixed speech recognition
2 min readupdated
A fine-tuned version of the Whisper Small model by Jevvellabs, optimized for processing sentences that mix German, English, Spanish, and Polish (code-switching). The model maintains high accuracy in monolingual automatic speech recognition (ASR) and does not generate false silence hallucinations. It was trained on Whisper Small using LoRA adapters (small add-ons to the base model) on the decoder while freezing the audio encoder. Useful for developers of voice assistants, translators, and transcription systems for multilingual users.
Key features
- A 41.4% reduction in word error rate (WER) on code-switched text compared to the base model.
- Improved recognition of technical terms and specific names.
- Preservation of accuracy on monolingual data without significant regressions.
- Support for Hugging Face Transformers formats and a ready-to-use whisper.cpp GGML F16 artifact.
What you can do
- Recognize audio recordings with frequent switching between German, English, Spanish, and Polish within a single dialogue.
- Run transcription on CPUs and mobile devices using the ready GGML file via whisper.cpp.
- Integrate the model into Python speech processing pipelines using the standard transformers library.
Example
import torch
from transformers import pipeline
asr = pipeline(
"automatic-speech-recognition",
model="Jevvellabs/whisper-small-cs-de-en-es-pl",
dtype=torch.float16 if torch.cuda.is_available() else torch.float32,
device=0 if torch.cuda.is_available() else -1,
)
result = asr("mixed_de_en_es_pl_audio.wav")
print(result["text"])Read more
DetailsBenchmarks: WER on code-switching dropped from 31.62% to 18.52%
Term recognition accuracy increased to 77.08%
Hardware requirements: standard for the Whisper Small model, GGML F16 file available (465 MB)
License: MIT










Comments