Local ASR Engine Deployment & Systems Optimization

TL;DR

Built a complete offline ASR system from scratch: a pure Rust MLX engine for GPU-accelerated inference, a Python orchestration layer for audio preprocessing and punctuation restoration, and cross-platform GUIs (Gradio WebUI on Mac, PyQt-Fluent on Windows). Key metrics: 76ms TTFT, 0.013× RTF, constant 883MB peak memory for arbitrary-length audio, 200 QPS sustained throughput.

System Architecture

┌─────────────────────────────────────────────────────────────┐
│                     Client Layer                             │
│  ┌──────────────────┐     ┌──────────────────────────────┐  │
│  │  Mac WebUI        │     │  Windows GUI                 │  │
│  │  (Gradio)         │     │  (PyQt-Fluent-Widgets)       │  │
│  └────────┬─────────┘     └──────────────┬───────────────┘  │
│           │                              │                   │
│           └──────────┬───────────────────┘                   │
│                      ▼                                       │
│  ┌──────────────────────────────────────────────────────┐   │
│  │              Python Orchestration Layer                │   │
│  │  • Audio decode (pydub) → 16kHz mono WAV              │   │
│  │  • BPE dedup + disfluency smoothing                   │   │
│  │  • Punctuation restoration (CT-PUNC)                  │   │
│  │  • Async QThread dispatch (PyQt)                      │   │
│  └──────────────────────┬───────────────────────────────┘   │
│                         │ subprocess / FFI                   │
│                         ▼                                    │
│  ┌──────────────────────────────────────────────────────┐   │
│  │           Rust MLX Compute Engine                     │   │
│  │  ┌────────────┐  ┌───────────┐  ┌────────────────┐  │   │
│  │  │ Mel Frontend│  │ SAN-M     │  │ CIF Predictor  │  │   │
│  │  │ 80-bin LFR  │→ │ Encoder   │→ │ (non-AR)       │  │   │
│  │  │ 7/6 stacking│  │ 50 layers │  │ parallel decode│  │   │
│  │  └────────────┘  └───────────┘  └────────────────┘  │   │
│  │                     Metal GPU (Apple Silicon)          │   │
│  └──────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────┘

The system is split into three layers by language boundary:

The key insight: Rust handles the hot path (inference loops, FFT, tensor ops), Python handles everything else (audio format conversion, punctuation model, UI glue). The boundary is a single subprocess.run() call — the Python layer invokes cargo run --release --example transcribe and reads stdout.

Non-Autoregressive ASR: Why Paraformer?

Traditional ASR (Whisper, Deepgram) is autoregressive: it generates one token at a time, each depending on the previous. For a 30-second clip with 500 tokens, that’s 500 sequential inference steps.

Paraformer is non-autoregressive: it predicts all tokens in a single forward pass using a CIF (Continuous Integrate-and-Fire) predictor to determine token boundaries, then a bidirectional decoder generates all tokens simultaneously.

Autoregressive (Whisper):
  token[0] → token[1] → token[2] → ... → token[499]
  500 × 50ms = 25,000ms (25s for 30s audio)

Non-Autoregressive (Paraformer):
  audio → [Encoder] → [CIF: find boundaries] → [Decoder: all tokens at once]
  400ms total (one forward pass)

The tradeoff: non-AR models can’t use language model beam search, so they rely on the CIF mechanism for alignment quality. Paraformer’s SAN-M encoder with FSMN memory blocks compensates for this.

Key Metrics

Cross-Platform Clients

Mac (Gradio WebUI):

Mac: Gradio WebUI — Python + Rust backend, audio upload → real-time transcription

Python Gradio app. Audio upload → Rust engine → punctuation restore → formatted output. Simple, functional, runs locally.

Windows (PyQt-Fluent-Widgets):

Windows: Native PyQt GUI — QThread async dispatch, Fluent Design

Native Windows GUI with Fluent Design. Uses QThread to keep the UI responsive during inference — the heavy MLX compute runs in a background thread, dispatching results back to the main event loop via signals.

What’s Next

The deep-dive posts cover the three core engineering challenges:

1. Dynamic Tensor Chunking: Constant Memory for Arbitrary-Length Audio — How 30-second tensor slicing keeps peak memory at 883MB regardless of input duration
2. Heterogeneous Backend: Python + Rust MLX Architecture — Why Rust for compute, Python for orchestration, and how they communicate
3. Dynamic Padding: 200 QPS with Zero Compute Waste — Eliminating Whisper-style 90% padding waste for short audio