SenseVoice for multilingual speech and audio events

How it works

The model produces speech text along with tags in supported workflows. Applications decide how to parse and use those tags in transcripts or audio analysis.

What you need

Choose the inference workflow for the supported task and language mix. Decide which tags become metadata and which parts of the output should be shown as transcript text.

What to try first

Run a short multilingual clip and a clip with background events. Inspect the raw output and test the tag-removal or preservation path used by your application.

Check one clip with music but no speech. An application that parses annotation tags needs a sensible result when there is no transcript to display.

Where to look for failures

Event and emotion labels are predictions from audio. Avoid interpreting them as reliable evidence of a person's intent or emotional state.

Related Franklin pages

Use the same source audio and decoding assumptions for transcription tests. Measure performance and listen to the mistakes, especially around silence, names and language changes.

• Transcribe or translate speech with OpenAI's published Whisper models. Whisper for multilingual speech recognition
• Run compatible Whisper models through a native inference engine. whisper.cpp for local Whisper transcription

For an application where the quality of recording and handling audio matters, see Franklin’s Open Mic AI profile. Open Mic AI

Project documentation

• Current project README
• README at the reviewed revision