How We Achieve 1.2% WER: Online Learning & Continual Optimization

Most speech-to-text services ship a model and call it done. We treat accuracy as a living system that improves every day through online learning, intelligent audio pre-processing, and continual benchmarking against the best models available.

The Problem With Static Models

Traditional STT pipelines train a model on a fixed dataset, deploy it, and move on. The model never sees new accents, new vocabulary, or new acoustic conditions after deployment. Error rates stagnate. Users adapt to the model's weaknesses instead of the model adapting to them.

DictatorFlow takes a fundamentally different approach: continual online learning. Our models improve in production, every single day, without manual retraining cycles.

Gold Standard: Oracle Model

To optimize, you need ground truth. We use the lowest WER model available as our gold-standard oracle. Every audio segment that flows through our pipeline gets a reference transcription from the oracle, which we treat as the target label.

This creates a continuous stream of labeled data without human annotation. We compare our production model's output against the oracle reference, compute per-segment WER, and feed discrepancies back into the training loop. The result: our models converge toward oracle-level accuracy while running at a fraction of the cost and latency.

Pipeline:
  Audio In -> Pre-Augmentation -> DictatorFlow Model -> Output
                                       |
                                  Oracle Model
                                       |
                                  WER Comparison
                                       |
                                 Online Update Loop

Pre-Augmentation Pipeline

Before audio hits our model, it passes through a pre-augmentation stage that dramatically improves recognition accuracy:

• 1. Respeed normalization -- Audio is time-stretched to a consistent speaking rate. Fast talkers and slow talkers produce equivalent acoustic features.
• 2. Lossy compression artifacts removal -- We detect and compensate for codec artifacts from Opus, AAC, and low-bitrate sources. A dedicated denoising pass restores frequency content lost to compression.
• 3. Dynamic range compression -- Whispered segments get boosted; shouted segments get attenuated. The model sees consistent amplitude regardless of mic distance or vocal intensity.
• 4. Noise profiling & subtraction -- We build a real-time noise profile of the first 500ms of silence and subtract it from the entire segment. Background noise from keyboards, fans, and office chatter disappears.

Multi-Model Throughput Optimization

We don't run a single model. DictatorFlow dynamically routes audio to the optimal model based on complexity, language, and latency requirements. Our routing layer evaluates:

• Audio complexity score -- Simple, clean dictation goes to our fastest lightweight model (sub-50ms). Noisy, multi-speaker audio routes to our high-accuracy heavy model.
• Language detection -- Specialized models for high-traffic languages (EN, ES, ZH, JA) outperform general multilingual models by 0.3-0.5% WER.
• Latency budget -- Streaming mode uses a different model architecture than batch mode, optimized for time-to-first-token rather than overall WER.

The Online Learning Loop

Every 6 hours, we run an optimization cycle:

1. Collect all production audio segments and their oracle reference transcriptions from the last window.
2. Compute per-segment WER and identify regression clusters -- groups of similar audio where our model underperforms.
3. Generate targeted training batches from these regression clusters with augmented variants (pitch shift, speed variation, noise injection).
4. Fine-tune the production model with a small learning rate, using elastic weight consolidation to prevent catastrophic forgetting.
5. A/B test the updated model against the current production model on a held-out validation set.
6. If WER improves without latency regression, promote the new model to production. Otherwise, discard and retry with adjusted hyperparameters.

Audio Co-Processing

Our Zig-based audio engine handles the heavy lifting before audio reaches the ML models. Written in Zig for zero-overhead memory management and SIMD-accelerated DSP, the co-processor handles:

• Opus/WebM decoding at native speed with zero-copy buffer management
• Resampling to 16kHz mono via polyphase FIR filters
• Voice Activity Detection to strip silence and reduce model input by 30-60%
• Spectral gating for real-time noise suppression without ML overhead

This pre-processing alone accounts for a 0.2-0.4% WER improvement over feeding raw audio directly to the model.

What's Next

We're working on per-user adaptation -- lightweight LoRA layers that learn individual speaking patterns and vocabulary over time. Early tests show another 0.1-0.3% WER improvement for repeat users. Combined with our continual learning pipeline, we expect to push below 1.0% WER within the next quarter.