Dynamic Padding: 200 QPS with Zero Compute Waste for Short Audio

Project: This post is part of the Local ASR Engine project — a heterogeneous Rust+Python speech recognition system.

TL;DR

Whisper pads all audio to 30 seconds before inference — a 1-second command gets 29 seconds of silence. This wastes 90% of compute. By dynamically padding each batch to its actual maximum length, a 4-worker system sustains 200 QPS for short audio with zero padding waste.

The Whisper Padding Problem

Whisper’s architecture requires fixed-length input: all audio is padded to 30 seconds. For a 1-second voice command:

Actual audio:  [speech 1s]
Whisper input: [speech 1s] [padding 29s]  ← 97% of compute is silence

In a smart home scenario (10-50 commands per minute), most audio is 0.5-3 seconds. Whisper wastes 90%+ of every inference on padding.

This isn’t just inefficient — it’s a throughput killer. At 200 QPS with 30s padding, you’d need 200 × 30s = 6,000 seconds of compute per second — 100× more GPU than available.

The Solution: Pad to Actual Length

Instead of padding everything to 30 seconds, pad each batch to the maximum actual length in that batch:

Fixed padding (Whisper-style):
  Batch: [1s, 2s, 3s, 1s]
  Pad to: 30s each
  Total compute: 4 × 30s = 120s
  Waste: 120 - 7 = 113s (94% waste)

Dynamic padding:
  Batch: [1s, 2s, 3s, 1s]
  Pad to: 3s (max in batch)
  Total compute: 4 × 3s = 12s
  Waste: 12 - 7 = 5s (42% waste)

  Savings: 10× less compute

Implementation

The key change is in the batching logic. Instead of a global 30-second target, compute per-batch padding:

/// Batch transcribe with dynamic padding
pub fn transcribe_batch_dynamic(
    model: &mut Paraformer,
    batch: &[Vec<f32>],  // List of audio chunks (variable length)
) -> Result<Vec<Vec<i32>>> {
    // Find the longest audio in this batch
    let max_len = batch.iter().map(|a| a.len()).max().unwrap();

    // Pad each audio to max_len (not to 30s)
    let padded: Vec<Array> = batch.iter().map(|audio| {
        let mut padded = vec![0.0f32; max_len];
        padded[..audio.len()].copy_from_slice(audio);
        mlx_rs::Array::from_slice(&padded, &[max_len as i32])
    }).collect();

    // Stack into batch tensor and run inference once
    let batch_tensor = mlx_rs::stack(&padded, 0)?;
    let all_tokens = model.transcribe_batch(&batch_tensor)?;

    Ok(all_tokens)
}

The critical detail: one inference call per batch, not one per audio. The batched tensor is [batch_size, max_len] — all audios are processed simultaneously on the GPU.

Throughput Math

For a 4-worker system handling 1-second commands:

The last row assumes the Rust engine processes a 4-audio batch in ~20ms (measured on M3 Max). With 4 workers, each handling 50 batches/second: 4 × 50 × 4 = 800 QPS theoretical, 200 QPS sustained (accounting for scheduling overhead).

Concurrency Architecture

                    ┌──────────────┐
                    │  Request Q   │
                    │  (200 QPS)   │
                    └──────┬───────┘
                           │
                    ┌──────▼───────┐
                    │   Dispatcher │
                    │  (round-robin)│
                    └──┬───┬───┬───┘
                       │   │   │
              ┌────────┘   │   └────────┐
              ▼            ▼            ▼
        ┌──────────┐ ┌──────────┐ ┌──────────┐
        │ Worker 1 │ │ Worker 2 │ │ Worker N │
        │ (Rust)   │ │ (Rust)   │ │ (Rust)   │
        └──────────┘ └──────────┘ └──────────┘
             │            │            │
             └────────────┼────────────┘
                          ▼
                    ┌──────────┐
                    │  Results  │
                    └──────────┘

Each worker is a separate Rust process (via subprocess). They don’t share memory — the model weights are loaded independently in each process. This wastes memory (4 × 800MB = 3.2GB for model weights) but:

1. No GIL: Python’s GIL doesn’t affect Rust processes
2. No lock contention: Each worker has its own Metal GPU context
3. Fault isolation: One worker crashing doesn’t affect others

Dynamic Padding vs Static Batching

When Dynamic Padding Doesn’t Help

• Uniform-length audio: If all inputs are exactly 30s, dynamic padding = static padding
• Very long audio: For 10-minute files, chunking handles the splitting; padding is irrelevant
• Batch size 1: No batching opportunity — each request is padded to its own length (which is correct)

Lessons Learned

1. Padding is a hidden tax — Whisper’s 30s padding is invisible in benchmarks but devastating for real-time workloads
2. Dynamic padding is simple — the implementation is ~10 lines of code. The hard part was realizing it was needed.
3. Concurrency ≠ parallelism — 4 workers don’t mean 4× throughput if they share a GPU. On Apple Silicon, Metal contexts are per-process, so each worker gets full GPU access.

Previous: Heterogeneous Backend: Python + Rust MLX Architecture — How the two languages communicate.

🔙 Back to Project Overview