AI GitHub
Whisper for multilingual speech recognition
Transcribe or translate speech with OpenAI's published Whisper models.
Key Takeaways
- The reference implementation runs an encoder-decoder model on audio segments.
- Choose a model size and task based on language, hardware and latency.
- Transcribe a short clip with noise and unusual names.
- Whisper can produce convincing text where audio is weak or silent.
How it works
The reference implementation runs an encoder-decoder model on audio segments. Model and task settings determine whether it transcribes source speech or translates supported inputs into English.
What you need
Choose a model size and task based on language, hardware and latency. Use the same recording and decoding settings when comparing a model with another implementation.
What to try first
Transcribe a short clip with noise and unusual names. Compare two model sizes with the same audio and inspect errors around pauses and speaker changes.
Include a silence-only sample and a phrase with a non-English proper name. These two cases help expose hallucination and transcription errors that a clean sample misses.
Where to look for failures
Whisper can produce convincing text where audio is weak or silent. Check accuracy against the recording and verify timestamps before treating them as editing boundaries.
Related Franklin pages
Use the same source audio and decoding assumptions for transcription tests. Measure performance and listen to the mistakes, especially around silence, names and language changes.
- Run compatible Whisper models through a native inference engine. whisper.cpp for local Whisper transcription
- Run Whisper recognition through CTranslate2 with configurable compute precision. faster-whisper for CTranslate2 transcription
For an application where the quality of recording and handling audio matters, see Franklin’s Open Mic AI profile. Open Mic AI
Project documentation
Try it for your workflow
Explore the project’s documentation for setup and usage, or save this page to a collection for later.
Read the project documentation ↗