Heterogeneous Backend: Python + Rust MLX Architecture for ASR

Project: This post is part of the Local ASR Engine project — a heterogeneous Rust+Python speech recognition system.

TL;DR

The compute engine is pure Rust (MLX bindings for Metal GPU), while audio preprocessing and punctuation use Python. The two communicate via subprocess — Python calls cargo run --release --example transcribe and reads stdout. This gives Rust’s memory safety and GPU access on the hot path, while Python’s ecosystem handles audio formats and NLP post-processing.

Why Not Pure Python?

Python ASR works fine — FunASR’s Python API is mature. But:

1. GIL contention: Python’s Global Interpreter Lock prevents true parallelism. In a multi-worker server, threads can’t overlap inference.
2. Memory overhead: Python objects have 40-80 bytes of overhead per object. For 10K tensor elements, that’s significant.
3. No Metal access: Python MLX bindings exist but add a layer of indirection. Rust MLX bindings call Metal directly.

Why Not Pure Rust?

Rust is great for compute, terrible for:

1. Audio format decoding: WAV is simple; AAC, MP3, OGG need codec libraries. Python’s pydub wraps ffmpeg — reimplementing this in Rust is possible but not worth it.
2. NLP post-processing: Punctuation restoration uses FunASR’s CT-PUNC model, which has a Python API. Rewriting the tokenizer and inference in Rust would take weeks.
3. Prototyping speed: Gradio gives a working UI in 50 lines. Rust GUI libraries (iced, egui) are powerful but slower to iterate.

The Split

┌─────────────────────────────────────────────┐
│              Python Layer                    │
│                                              │
│  1. Audio decode (pydub → ffmpeg)            │
│     "Convert any format to 16kHz mono WAV"  │
│                                              │
│  2. Punctuation restoration (CT-PUNC)        │
│     "Add ，。？！ to raw token stream"       │
│                                              │
│  3. BPE dedup + disfluency smoothing         │
│     "com @ @ pu @ @ ting → computing"        │
│                                              │
│  4. UI (Gradio / PyQt)                       │
│     "File picker, output display"            │
└──────────────────┬──────────────────────────┘
                   │
                   │  subprocess.run([
                   │    "cargo", "run", "--release",
                   │    "--example", "transcribe",
                   │    "--", audio_path, model_dir
                   │  ])
                   │
                   ▼
┌─────────────────────────────────────────────┐
│              Rust Layer                      │
│                                              │
│  1. Load WAV (raw f32 samples)              │
│                                              │
│  2. Resample to 16kHz (if needed)           │
│                                              │
│  3. Mel frontend (80-bin FFT + LFR)         │
│     Pure Rust, Metal-accelerated FFT         │
│                                              │
│  4. SAN-M Encoder (50 layers)               │
│     Self-attention + FSMN memory blocks      │
│     All tensor ops on Metal GPU              │
│                                              │
│  5. CIF Predictor                            │
│     Continuous integrate-and-fire            │
│     Determines token boundaries              │
│                                              │
│  6. Bidirectional Decoder                    │
│     Parallel token generation                │
│     Output: token IDs → stdout              │
└─────────────────────────────────────────────┘

The Communication Protocol

The interface between Python and Rust is deliberately simple: a single subprocess call with stdout parsing.

# Python side (app.py)
def process_audio(audio_path, is_save):
    # 1. Decode audio to 16kHz mono WAV
    audio = AudioSegment.from_file(audio_path)
    audio = audio.set_frame_rate(16000).set_channels(1)
    temp_wav = tempfile.NamedTemporaryFile(suffix=".wav", delete=False)
    audio.export(temp_wav.name, format="wav")

    # 2. Invoke Rust engine
    cmd = [
        "cargo", "run", "--release", "--example", "transcribe",
        "--", temp_wav.name, MODEL_DIR
    ]
    result = subprocess.run(cmd, capture_output=True, text=True)

    # 3. Parse output
    for line in result.stdout.split('\n'):
        if "Transcription:" in line:
            raw_text = line.split("Transcription:", 1)[1].strip()
            break

    # 4. Post-process in Python
    raw_text = re.sub(r'\s*@\s*@\s*', '', raw_text)  # BPE cleanup
    raw_text = disfluency_smooth(raw_text)             # Remove repetitions
    punc_model = get_punc_model()                       # Lazy-load CT-PUNC
    final_text = punc_model.generate(input=raw_text)[0]['text']

    return final_text

// Rust side (examples/transcribe.rs)
fn main() {
    let audio_path = std::env::args().nth(1).unwrap();
    let model_dir = std::env::args().nth(2).unwrap();

    // Load audio (pure Rust, no Python)
    let (samples, sample_rate) = load_wav(&audio_path).unwrap();
    let samples = resample(&samples, sample_rate, 16000);

    // Load model (one-time cost)
    let mut model = load_model(&format!("{}/paraformer.safetensors", model_dir)).unwrap();
    let (addshift, rescale) = parse_cmvn_file(&format!("{}/am.mvn", model_dir)).unwrap();
    model.set_cmvn(addshift, rescale);

    // Transcribe (all on Metal GPU)
    let audio_array = mlx_rs::Array::from_slice(&samples, &[samples.len() as i32]);
    let token_ids = model.transcribe(&audio_array).unwrap();

    // Decode tokens → text
    let vocab = Vocabulary::load(&format!("{}/tokens.txt", model_dir)).unwrap();
    let text = vocab.decode(&token_ids_vec);

    println!("Transcription: {}", text);
}

Why Subprocess Instead of FFI?

The obvious question: why not use PyO3 or rust-cpython for direct function calls?

For ASR, the inference itself takes 50-400ms. The 10ms subprocess overhead is 2-20% of total latency — acceptable for a local tool. The benefits outweigh the cost:

• Process isolation: If Rust panics, Python survives. If Python OOMs, Rust survives.
• No build complexity: No PyO3, no maturin, no special build flags. Just cargo build --release.
• Debugging: cargo run output goes to stderr, Python logs go to stdout. No interleaving.

76ms TTFT Breakdown

For a short command (< 3s audio):

The Python overhead (30ms) is 39% of total TTFT. For a production system, you’d keep the Rust engine warm (no subprocess startup) and eliminate the pydub step — bringing TTFT under 50ms.

Lessons Learned

1. Subprocess is underrated — for local tools, the simplicity of process isolation beats FFI’s performance. The 10ms overhead is noise compared to model inference.
2. Python’s ecosystem is irreplaceable — audio format handling, NLP post-processing, UI frameworks. Rust can’t match this breadth yet.
3. Rust’s strength is the hot path — FFT, tensor ops, memory management. These are exactly where Python is slow.

Previous: Dynamic Tensor Chunking — How memory stays constant at 883MB.

Next: Dynamic Padding: 200 QPS with Zero Compute Waste → How short audio gets 9× less compute than Whisper.

🔙 Back to Project Overview