Dynamic Tensor Chunking: Constant 883MB Memory for Arbitrary-Length Audio

Project: This post is part of the Local ASR Engine project — a heterogeneous Rust+Python speech recognition system.

TL;DR

Processing a 1-hour audio file naively requires holding the entire feature tensor in memory — 2,573 MB for a 3600-second file. By slicing audio into 30-second chunks before feature extraction, peak memory drops to a constant ~883 MB regardless of input duration. The trick: the model weights are fixed (~800MB), and each chunk’s feature tensor is only ~3MB.

The Problem: Memory Explodes with Duration

The Paraformer model has a fixed memory footprint:

The variable part is the feature tensor — the mel spectrogram + LFR (Low Frame Rate) stacking output. For a 30-second chunk:

Without chunking, a 1-hour file produces:

1 hour = 3600s → 3600 frames → LFR → 600 feature vectors × 560 dims
Feature tensor: ~134 MB
Total peak: 880 + 134 = 1,014 MB

For a 24-hour file: 2,573 MB — likely to OOM on machines with 8GB RAM.

The Solution: Slice Before Extract

The key insight: don’t load the entire audio into memory at once. Instead:

1. Read audio in 30-second windows
2. Extract features for each window independently
3. Run inference on each chunk
4. Free the chunk’s memory before processing the next

Input: 60-second audio
    ↓
┌───────────┐    ┌───────────┐
│ Chunk 1    │    │ Chunk 2    │
│ 0s–30s    │    │ 30s–60s   │
│ (192 KB)  │    │ (192 KB)  │
└─────┬─────┘    └─────┬─────┘
      ↓                ↓
┌───────────┐    ┌───────────┐
│ Mel + LFR │    │ Mel + LFR │
│ (3 MB)    │    │ (3 MB)    │
└─────┬─────┘    └─────┬─────┘
      ↓                ↓
┌───────────┐    ┌───────────┐
│ Inference │    │ Inference │
│ (880 MB)  │    │ (880 MB)  │
└─────┬─────┘    └─────┬─────┘
      ↓                ↓
   tokens 1          tokens 2
      └───────┬────────┘
              ↓
         Concatenate → Final text

The model weights (~800 MB) are loaded once and reused across all chunks. Only the feature tensor (~3 MB) is allocated and freed per chunk.

Implementation: The Core Loop

Here’s the Rust implementation of the chunking logic:

/// Process audio in 30-second chunks to keep memory constant
pub fn transcribe_chunked(
    model: &mut Paraformer,
    audio: &[f32],
    sample_rate: u32,
    chunk_duration_s: f32,
) -> Result<Vec<i32>> {
    let chunk_samples = (sample_rate as f32 * chunk_duration_s) as usize;
    let total_samples = audio.len();
    let mut all_tokens = Vec::new();

    let mut offset = 0;
    while offset < total_samples {
        // Slice: take up to chunk_samples from current position
        let end = (offset + chunk_samples).min(total_samples);
        let chunk = &audio[offset..end];

        // Feature extraction for this chunk only
        let features = model.extract_features(chunk)?;

        // Inference on this chunk
        let tokens = model.transcribe(&features)?;
        all_tokens.extend_from_slice(&tokens);

        // chunk + features are dropped here → memory freed
        offset = end;
    }

    Ok(all_tokens)
}

The critical detail: chunk and features are stack-allocated or scope-bound. When the loop iteration ends, they’re dropped, freeing ~3MB before the next iteration allocates new memory.

Measured Results

Tested on Apple M3 Max (48GB) with the Paraformer-large model:

The pattern is clear: without chunking, memory grows linearly with duration. With chunking, it plateaus at ~842 MB.

Why 30 Seconds?

The chunk duration is a tradeoff:

Paraformer’s CIF (Continuous Integrate-and-Fire) predictor works best on segments where acoustic boundaries align with natural speech pauses. 30 seconds captures typical sentence-level prosody without crossing too many topic boundaries.

Edge Cases

Audio shorter than 30s: No chunking needed — processes as a single chunk. The min(total_samples, chunk_samples) handles this naturally.

Silence at boundaries: The CIF predictor fires on acoustic boundaries, not time boundaries. A 30s chunk might split mid-sentence, but the bidirectional decoder handles this gracefully because it sees the full chunk context.

Overlap for continuity: The current implementation uses non-overlapping chunks. For higher accuracy on boundary-sensitive tasks, you could add a 5-second overlap between chunks and average the overlapping region — but this increases memory proportionally.

Lessons Learned

1. Model weights dominate memory — the 220M-parameter model is ~800MB regardless of input. Feature tensors are tiny by comparison.
2. Chunking is免费 for short audio — under 30s, there’s zero overhead. The branching cost is negligible.
3. Memory safety ≠ memory efficiency — Rust prevents OOM crashes, but it doesn’t prevent OOM design. You still need to think about allocation patterns.

Next: Heterogeneous Backend: Python + Rust MLX Architecture → How the two languages communicate.

🔙 Back to Project Overview