Whisper-large-v3-turbo Korean Dysarthria LoRA
Adapter for Impaired Korean Speech Recognition
2 min readupdated
This lightweight PEFT LoRA module (a small add-on on top of the base model) was trained on a Korean corpus combining audio from speakers with dysarthria (articulation disorders) and typical speech. The base openai/whisper-large-v3-turbo model often struggles with non-standard pronunciation or "forgets" standard speech patterns when fine-tuned narrowly. Joint training solves this problem, improving recognition accuracy for both speakers with speech impairments and everyday users. It is a solid fit for developers building voice assistants, inclusive communication tools, and transcription services.
Key features
- Optimized for Korean speech affected by motor and neurological articulation disorders.
- Cuts Character Error Rate (CER) by 86.7% on dysarthria benchmarks.
- Boosts accuracy on typical speech as well, without degrading baseline performance.
- Trains just 6.44% of the base model's parameters (around 55.7M trainable weights).
What you can do
- Plug the adapter into your audio pipeline to transcribe challenging or slurred speech accurately.
- Build accessible tools and captioning systems for people with speech impairments.
- Merge the weights (merge_and_unload) into the base Whisper model for faster production inference.
- Test transcription on Korean lectures and audio recordings using the transformers Python library.
Example
import torch
import torchaudio
from transformers import WhisperForConditionalGeneration, WhisperProcessor
from peft import PeftModel
base_model_id = "openai/whisper-large-v3-turbo"
adapter_id = "PL-HFT/whisper-large-v3-turbo-ko-dysarthria-lora"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.float16 if device == "cuda" else torch.float32
processor = WhisperProcessor.from_pretrained(base_model_id, language="Korean", task="transcribe")
base_model = WhisperForConditionalGeneration.from_pretrained(base_model_id, torch_dtype=dtype)
model = PeftModel.from_pretrained(base_model, adapter_id).eval().to(device)
waveform, sr = torchaudio.load("sample.wav")
if sr != 16000:
waveform = torchaudio.functional.resample(waveform, sr, 16000)
inputs = processor.feature_extractor(waveform.mean(dim=0).numpy(), sampling_rate=16000, return_tensors="pt")
input_features = inputs.input_features.to(device=device, dtype=dtype)
predicted_ids = model.generate(input_features, language="Korean", task="transcribe", temperature=0.0)
print(processor.batch_decode(predicted_ids, skip_special_tokens=True)[0])Read more
DetailsBenchmarks: 1.17% overall CER (down from 2.90% on the base model), and 0.59% on dysarthric speech (down from 4.45%).
Hardware: trained for 10 hours on 1x NVIDIA A100 (80GB VRAM).
Architecture: PEFT LoRA (r=64, lora_alpha=128), 16 kHz mono audio up to 29.5 seconds.
Limitations: not intended for clinical diagnosis.










Comments