Fast track — streaming Zipformer
Emits partial text continuously, ~1.8s behind the speaker. That delay is the model's own lookahead window, not overhead that can be optimised away. This is the track you actually read while someone is still talking.
BUILD LOG / DESKTOP AUDIO
Realtime Japanese subtitles on Windows
A desktop app that listens to whatever Windows is playing, prints furigana-annotated Japanese at the bottom of the screen within ~1.8 seconds, overwrites it with a higher-accuracy pass the moment the sentence ends, and puts a Chinese translation underneath. This page is the engineering write-up: the architecture, the measured numbers, and the three failure modes that cost the most time.
STATUS
Working, in daily personal use, not publicly distributed. There is no download link on this page because there is nothing honest to link to yet. What is published here is the part that transfers: the architecture and the measurements.
The problem: subtitles for audio nobody subtitled
Raw Japanese video, a live meeting, an interview — the cases where you need subtitles are exactly the cases where none exist. Cloud captioning solves the accuracy half and fails the other two: latency you feel in a conversation, and sending your meeting audio to someone else's server. So recognition runs locally. That constraint is what forces every interesting decision below.
Architecture: two tracks, because speed and accuracy are different problems
A single model cannot be both fast and right. A streaming model must emit words before the sentence exists, so it guesses; an offline model sees the whole utterance and does far better, but only after the speaker stops. Running one means choosing which half of the job to fail. So both run at once on the same audio, and the accurate one overwrites the fast one in place.
System audio (WASAPI loopback) → two recognizers in parallel → transparent overlay
Final track — VAD + SenseVoice
A voice-activity detector cuts the audio at 0.6s of silence and hands each complete utterance to an offline model. It lands 1.2–1.5s after the sentence ends and replaces the fast track's guess. Nearly ten accuracy points separate the two.
Translation — fully non-blocking
Each finalised line goes out to a cloud translator on its own worker. Median round trip is roughly 0.7–1.0s, and none of it is allowed to touch the audio thread — see the first lesson below for what happens when it does.
Measured, not estimated
Measured 2026-08-01 against 605 human-authored reference subtitles from a 47-minute podcast — human-authored, not the platform's auto-captions, which would only measure agreement with another ASR. Accuracy is character error rate over kana.
| Metric | Measured | Note |
|---|---|---|
| Subtitle coverage, both tracks merged | 100.00% | Not one of the 605 reference lines had both tracks silent at the same time |
| Recognition accuracy, final track | 90.3% | 9.7% kana CER over the full 47 minutes |
| Recognition accuracy, fast track | 80.6% | Was 72.6% before the endpointing fix — one bug was worth eight points |
| Audio frames dropped | 0.0% | Two earlier versions dropped 48% and 10.9%; both causes were synchronous work on the capture thread |
| Time to first word | ~1.8s | The streaming model's inherent lookahead |
| Time to corrected line | 1.2–1.5s after the sentence ends | Structurally governed by the 0.6s silence threshold |
| Compute headroom | RTF 0.16–0.23 | Both tracks running together; median 2–3ms per audio block |
Three failure modes worth stealing
None of these are specific to subtitles. All three are the shape of bug that AI-assisted development produces most readily, because none of them raise an error.
1. Nothing that scales with input size may run on the capture thread
WASAPI hands you a ring buffer: whatever you do not collect in time is overwritten by the next audio, silently. Version one waited synchronously for a translation HTTP call and lost 48% of the audio. A later version inlined the offline decode and lost 10.9%. Neither ever threw. The rule that came out of it: the capture thread may only copy bytes and hand them off — network calls, model inference and file writes all belong on a worker.2. reset() does not mean "back to a clean state"
Resetting the recogniser between sentences left enough residue to trap the model in a loop: it emitted nothing, the endpointer read the empty output as silence, fired every 2.4 seconds, and reset again. A real person spoke for eighteen seconds and not one character appeared. The fix is conditional, not blanket: keep resetting on a normal sentence boundary to preserve left context, and construct a fresh stream only when the result came back empty. Replacing every reset with a fresh stream fixes the hang and costs accuracy.3. An "initialised" flag must be the last step, never a by-product
Model init assigned the model handle partway through, then threw. The guard clause read that handle, concluded initialisation was done, and cached a half-dead object forever. A completion flag has to mean everything is ready and the thread is alive — not that we got far enough to assign one field.
Stack
Every layer picked for one reason: recognition stays on the machine.
- Language
- Python 3.14
- GUI
- PyQt6 + Fluent Widgets
- Overlay
- PyQt6-WebEngine — HTML ruby for furigana
- Audio capture
- soundcard (WASAPI loopback)
- ASR
- sherpa-onnx — streaming Zipformer + SenseVoice
- Furigana
- fugashi + unidic
- Translation
- Google / MiniMax, cloud, off-thread
Why furigana is rendered in a browser engine
Reading speed was the whole point, and ruby text — small kana printed above the kanji — is the one typographic feature that makes unfamiliar Japanese scannable rather than parseable. Getting ruby right in a native widget toolkit is a project of its own; HTML has had a ruby element for two decades. So the subtitle layer is a transparent, click-through browser view. The unglamorous choice was the fast one.