24 kHz in. 12.5 Hz out.

The numbers describe different clocks#

A speech model takes in 24,000 waveform samples every second. Its tokenizer emits 12.5 frames in that same second. Each frame might contain eight codebook indices. The language model might make one large temporal update per frame, then several smaller predictions to fill in those indices.

Every number in that description can be correct at once. The engineering mistakes begin when we call all of them “tokens per second.”

Waveform sampling controls which frequencies the input can represent. Codec frame rate controls the spacing between learned representations. Codebook count and vocabulary size control their discrete capacity. The model's sequence layout determines how quickly those representations consume its context. Streaming introduces another set of clocks: buffering, lookahead, computation, and playback.

This article follows one running example: 24 kHz mono audio, 12.5 codec frames per second, and eight codebooks with 2,048 entries each. It explains the arithmetic, then connects it to selected published architectures. The animations are original Manim illustrations; the latency examples are assumed budgets, and the comparison table contains reported configurations. We have not run a comparative model benchmark for this article.

Keep the units attached. “12.5 Hz” is a rate, so a finite tensor still contains a whole number of frames. A two-second interval gives us 25 frame intervals and avoids a misleading drawing of half a token.

1. What 24 kHz preserves#

A continuous signal contains variations over time. Sampling records its value at regularly spaced instants. With a waveform sample rate of f_s, the ideal Nyquist boundary is f_s / 2: the signal must be band-limited below that boundary for unambiguous reconstruction under the sampling theorem's assumptions.

16,000 samples/s  →   8,000 Hz Nyquist boundary
22,050 samples/s  →  11,025 Hz Nyquist boundary
24,000 samples/s  →  12,000 Hz Nyquist boundary
48,000 samples/s  →  24,000 Hz Nyquist boundary

This gives 24 kHz audio room to represent higher-frequency content than 16 kHz audio. It also gives a waveform encoder half as many input samples as 48 kHz audio of the same duration. That is a sample-count comparison; the reduction in total inference time depends on the rest of the network.

The extra bandwidth can matter to the sound of a voice. Speech is more than its fundamental frequency and low-frequency vowels. Consonants, breath, and other details can carry higher-frequency energy. Listening research has demonstrated sensitivity to changes in high-frequency vocal energy, although its value varies with the listener and the task. Monson and colleagues, high-frequency energy in speech and singing

Why an anti-alias filter matters#

Take a 9 kHz cosine and sample it directly at 16 kHz. At integer sample index n, the two expressions below are equal:

cos(2π × 9,000 × n / 16,000)
cos(2π × 7,000 × n / 16,000)

Their frequencies add to the sampling rate; the cosine's periodicity and symmetry make the sample values coincide. The 9 kHz signal therefore aliases to 7 kHz. At 24 kHz sampling, 9 kHz lies below the 12 kHz Nyquist boundary.

A proper conversion from 24 kHz to 16 kHz filters out frequencies above the target passband before downsampling. Practical filters need a transition band, so “up to 8 kHz” is an ideal boundary rather than a promise of a perfectly flat response all the way to it. Once filtering removes that information, upsampling the result back to 24 kHz cannot recover the original high-frequency detail.

There is a related implementation trap: changing an audio file's sample-rate metadata does not resample its waveform. Playing 24,000 stored samples at 16,000 samples per second changes the duration to 1.5 seconds and lowers the pitch. A resampler creates the appropriate new sequence of samples while preserving the intended duration.

For a streaming encoder, preserve the resampler's filter state across chunks. Otherwise each transport packet can behave like a new recording boundary. Log the actual sample count and sample rate at the model boundary; a correct-looking file header is insufficient evidence that the waveform has been converted correctly.

2. From waveform samples to codec frames#

A neural codec usually turns short-range waveform patterns into a sequence of latent vectors. Its encoder reduces temporal resolution as it increases the amount of information represented at each latent step. Quantization then makes those representations discrete, and a decoder maps them back into a waveform.

For a regular temporal grid, the sample rate, frame rate, and effective hop satisfy:

H = f_s / F
frame interval = H / f_s = 1 / F
 
H = 24,000 / 12.5 = 1,920 samples
frame interval = 1,920 / 24,000 = 0.08 s = 80 ms

In an encoder built from strided layers, the effective hop often follows from the product of their strides. A hypothetical stack with strides 4 × 5 × 6 × 16 has an overall factor of 1,920. Real architectures can insert resampling or other operations, so inspect the complete path into the quantizer before inferring a rate from a partial list of layers.

Hop, receptive field, and finite clips#

The hop tells us how far the encoder advances between outputs. The receptive field tells us which input samples can influence one output. Those quantities can be very different. A frame produced every 80 ms may summarize a much longer stretch of audio.

A causal encoder uses current and past input. An encoder with lookahead also depends on future input, which has to arrive before the frame can be completed in a live stream. A decoder may likewise use neighboring latent frames and overlapping synthesis operations. Thinking of each frame as an independent 80 ms audio file misses those dependencies.

For a finite recording, padding, convolution boundary rules, and flushing can change the exact output length. duration × F is the rate-based expectation; the implementation decides how an incomplete interval at the end is handled. Save the original length, inspect the produced tensor shape, and trim reconstructed padding according to the codec's contract. Do not silently drop the final partial chunk.

The useful debugging question is therefore concrete: given N input samples, how many complete code frames does this implementation emit, when does it emit the first one, and how many reconstructed samples are valid after flushing?

3. What a codebook index actually represents#

Suppose the encoder produces a latent vector z_t for frame t. A codebook is a learned table of vectors. Vector quantization selects an entry from that table and transmits its index. At reconstruction time, the receiver looks up the same vector.

That index is a category label. Index 302 is not necessarily acoustically closer to index 303 than to index 1,900. The geometry lives in the learned vectors, rather than in the numerical order of their IDs. Interpolating the integer IDs has no general acoustic meaning.

A common code tensor layout is [batch, codebooks, frames]. In that layout, one two-second example at our running configuration has shape [1, 8, 25]: 25 temporal steps and 200 integer indices. Libraries can use a different axis order, so check the API before treating a dimension as time.

A single codebook has limited capacity. Ordinary residual vector quantization, or RVQ, uses several codebooks in sequence. Each stage approximates what earlier stages have not yet represented:

r_0 = z_t
i_k = argmin_j ||r_(k-1) - E_k[j]||²
q_k = E_k[i_k]
r_k = r_(k-1) - q_k
 
quantized latent = q_1 + q_2 + ... + q_K
transmitted representation = (i_1, i_2, ..., i_K)

Here E_k is codebook k, q_k is its chosen vector, and r_k is the residual after subtracting it. Training learns the encoder, the codebook representations, and the decoder together or in coordinated stages. The exact distance, projections, update rules, and loss functions vary by codec. EnCodec is one published example of a neural audio codec using RVQ. EnCodec paper

The animation uses two dimensions so the geometry fits on a page. Actual learned representations are higher-dimensional, and their loss functions can prioritize perceptual or linguistic properties that a two-dimensional Euclidean diagram cannot capture. More nominal capacity also does not guarantee that training uses it well: some entries can be used rarely, while frequent patterns dominate others.

Semantic and acoustic information can be organized differently#

A codec intended for a speech language model may train part of its representation to carry phonetic or linguistic structure. That makes the training objective relevant to what an index preserves.

Mimi's split RVQ is a useful specific example. Its semantic branch uses a single quantizer trained with a WavLM-derived target; a separate seven-level acoustic RVQ runs in parallel, and their outputs are summed. It is therefore more specific than the ordinary serial eight-stage RVQ pictured above. Moshi, semantic-acoustic quantization

These objectives encourage useful structure without making the first codebook a dictionary of words or making the remaining codebooks a clean list of pitch, accent, and emotion. To understand what a representation retains, examine its training objective and evaluate the property of interest.

4. Convert capacity into bits#

There are two different compression calculations hiding in our example.

The encoder reduces 24,000 sample positions per second to 12.5 frame positions per second, a temporal factor of 1,920. Storage compression requires comparing the number of bits used by the input and output representations.

A codebook with 2,048 entries has log2(2,048) = 11 bits of index capacity. With eight such indices per frame:

bits per frame = 8 × 11 = 88 bits
codec bitrate  = 12.5 × 88 = 1,100 bits/s
               = 1.1 kbps
 
16-bit mono PCM at 24 kHz = 24,000 × 16
                          = 384,000 bits/s
 
nominal compression ratio = 384,000 / 1,100
                          ≈ 349.09

The codec is lossy. A smaller representation does not preserve every PCM sample exactly; the decoder reconstructs plausible audio from the information the representation retained and the structure learned during training.

For unequal vocabulary sizes, independently packing each index into a fixed-width field gives:

fixed-width bits per frame = sum(ceil(log2(V_k)))
fixed-width bitrate       = F × sum(ceil(log2(V_k)))

The ceiling matters when a codebook size is not a power of two. For example, 2,016 possible entries still require an 11-bit independent binary field. A dense block representation can approach the joint combinatorial capacity, while entropy coding can exploit a nonuniform and predictable index distribution. Neither calculation automatically gives the size of a file on disk.

Why the saved tensor can be larger#

An implementation may store every index as a 16-bit or 64-bit integer for convenience. At 100 indices per second, a raw uint16 array already uses 1,600 bits per second before metadata, even though those values fit into 11 bits apiece. A serializer has to pack them to realize the 1,100-bit figure.

Headers, timestamps, channel information, padding, transport framing, error protection, and any separately supplied conditioning also count toward the actual payload. For very short clips, fixed overhead can dominate. State whether a reported rate means nominal audio-code capacity, packed bytes, or bytes sent over the network.

Channel count belongs in this accounting too. Two independent mono streams double their combined payload. A joint stereo codec can organize channels differently, so its rate should come from its actual representation rather than from an assumption that every stereo design is two mono codecs.

5. The language model decides how much context it uses#

Our codec emits 12.5 × 8 = 100 indices per second. What does the language model do with them?

In a flattened sequence, the model places each index at its own position. One frame occupies eight consecutive positions. If all 2,048 available positions belong to audio, the sequence holds 256 complete frames:

complete frames = floor(C / K) = floor(2,048 / 8) = 256
audio duration  = 256 / 12.5 = 20.48 seconds

In a frame-wise hierarchy, the large temporal model advances once per frame. A smaller predictor handles the codebooks associated with that step. A budget of 2,048 temporal positions now represents:

audio duration = C / F = 2,048 / 12.5 = 163.84 seconds

The resulting audio representation has the same number of indices in both arrangements. The models organize their computation and their persistent temporal history differently.

Moshi explicitly separates a temporal transformer from a smaller depth transformer over codebooks. Qwen3-TTS's 12.5 Hz path uses another hierarchical arrangement: its backbone predicts the first codebook, then a multi-token prediction module generates the residual codebooks. These are concrete reasons to distinguish indices per second from backbone positions per second. Moshi, hierarchical modeling, Qwen3-TTS technical report

Parallel and delayed schedules#

A model can also predict several codebooks with parallel output heads. This changes which dependencies are modeled explicitly within a frame. A delayed schedule offsets codebook streams across temporal steps, allowing some dependencies to pass through earlier positions. The startup and drain portions of that schedule add padding and can affect when a complete frame becomes available.

These choices form an architecture, not a formatting detail. Codebook predictions may be sequential, parallel, or partly both; an implementation can combine temporal grouping with delays. Read the generation loop and attention masks alongside the paper's diagram.

Text and speaker streams require the same care. A serialized text prompt uses positions that are no longer available to audio. A simultaneous aligned text stream may share the temporal grid while adding work within each step. Two speakers might occupy aligned streams or a serialized sequence. Multiplying by the number of streams is correct for some layouts and wrong for others.

What shorter sequences do to memory#

For a transformer storing ordinary key and value caches, a useful capacity estimate for one sequence is:

KV bytes ≈ 2 × layers × KV heads × head dimension
           × cached positions × bytes per element

The factor of two represents keys and values. Consider an illustrative model with 32 layers, eight KV heads, head dimension 128, and two-byte cache elements. It stores 131,072 bytes, or 128 KiB, per cached position. At 2,048 positions, that is 256 MiB for the KV tensors of one sequence.

For the same 20.48 seconds of audio, flattening our eight-codebook representation uses 2,048 positions; frame-wise processing uses 256 temporal positions. Holding that hypothetical temporal transformer fixed, its temporal KV storage changes from 256 MiB to 32 MiB. The depth predictor, weights, activations, allocation overhead, and any additional streams require their own accounting.

This does not establish an eightfold runtime improvement. Output heads, depth prediction, memory traffic, kernel efficiency, and waveform decoding all cost time. Dense attention's pairwise work can fall faster than sequence length, but that is only one term in the total workload. Measure the actual model at the concurrency and context lengths you intend to serve.

6. Why lowering the frame rate can hurt#

Fewer frames make the temporal modeling problem shorter. Each frame then needs to account for a longer span of evolving sound. A vowel transition, a brief consonant, and a pause may all compete for the representation's capacity.

One response is to increase the number or size of codebooks. Another is to strengthen the decoder or give it text and speaker conditioning. A third is to accept a different reconstruction trade-off. These choices redistribute information and computation through the system.

NanoCodec provides a useful controlled comparison within one study. At roughly 1.1 kbps, its authors tested 25, 12.5, and 6.25 frames per second. They observed worse intelligibility at 6.25 and proposed a connection to packing multiple phonemes into one frame. Their 6.25 Hz setting used more codebooks to maintain the bitrate. NanoCodec, frame-rate ablation

That result should guide investigation without turning into a universal minimum frame rate. Speech sounds have different durations; speaking rate and language change their distribution. Learned frames need not align with phoneme boundaries, and a sufficiently informative representation can describe multiple events within one interval.

TaDiCodec illustrates how the decoding problem can change. It reports a 6.25 Hz, single-codebook representation and a 0.0875 kbps audio-code rate, using a diffusion decoder conditioned on text and prompts. Those conditions provide information beyond the audio indices, and the code bitrate excludes that separate information. Its results cannot be reduced to a claim that every ordinary codec should work at the same rate. TaDiCodec

There is evidence in the other direction too. GLM-TTS increased its speech-token rate from 12.5 to 25 Hz while also expanding its vocabulary and changing other components, reporting improvements in pronunciation and expressive events. Because several factors changed together, that report does not isolate the causal contribution of frame rate. GLM-TTS, speech tokenizer

The question for a deployment is whether its operating point preserves the distinctions listeners need while meeting the system's compute and timing constraints. A fluent sentence can still contain the wrong name, number, or negation. Perceptual quality and linguistic fidelity deserve separate measurements.

7. Selected configurations, with their units intact#

The following configurations show the range of designs. The bitrate column describes nominal audio-code rates; it does not normalize decoder cost, side information, training data, or reconstruction quality. A model family can offer several settings.

Mimi's entries have 2,048 choices. NanoCodec's cited setting uses 2,016. Qwen's 16 codebooks have 2,048 entries each, giving 12.5 × 16 × 11 = 2,200 bits/s. WavTokenizer's selected setting uses a 4,096-entry codebook; EnCodec's uses 1,024-entry codebooks. These parameters account for different bitrates even when two configurations emit similar numbers of indices. Moshi/Mimi, NanoCodec, Qwen implementation, Qwen sample-rate configuration, TaDiCodec, WavTokenizer, EnCodec implementation

The Qwen model name says “12Hz”; the reported operating rate is 12.5 frames per second. Such naming shortcuts are another reason to use the configuration's actual numbers in a capacity calculation.

When comparing semantic tokenizers with complete waveform codecs, check both ends of the path. The input used to extract semantic features can have a different sample rate from the final synthesized waveform. The table above deliberately focuses on selected waveform-codec configurations whose rates can be stated together.

8. Frame spacing does not determine first-audio latency#

At 12.5 frames per second, the codec grid advances every 80 ms. A streaming application still has to acquire or generate the required information, run the relevant networks, deliver the result, and schedule playback.

For a simple serial capture-to-reconstruction path, an accounting model is:

first-playout delay = initial audio accumulation
                   + required future lookahead
                   + computation on the critical path
                   + transport and playout buffering

Define the measurement boundary before using this equation. Accumulation here starts with the first captured sample. Lookahead means additional future input beyond that accumulation window. Counting it twice would overstate the delay. In a steady pipeline, stages can overlap; the first-output critical path and the long-run throughput are different measurements.

For N frames per chunk, the chunk covers N / F seconds of audio. Four frames at 12.5 Hz cover 320 ms. Batching can improve compute efficiency while delaying the first output; its effect has to be evaluated against the application's timing needs.

Text-to-speech has a different starting point. If the source text is already available, it need not wait to capture 80 ms of a microphone signal before predicting its first audio frame. It still pays for text processing, autoregressive prediction, any depth or diffusion steps, waveform decoding, and buffering. A codec's input accumulation budget cannot simply be copied into a TTS latency claim.

Fast enough on average can still underrun#

Real-time factor is compute time divided by the duration of the audio produced. Taking 20 ms to generate 80 ms of audio gives an RTF of 0.25. That indicates faster-than-real-time throughput under the measured conditions, but it does not prove that the first packet arrives quickly or that every later packet meets its playback deadline.

A system can have a low average RTF and still stutter because of a long scheduling pause, an unusually expensive decoding step, or network jitter. Measure first audio, inter-chunk gaps, and underruns alongside throughput. Report distributions at realistic concurrency, including cold-start conditions if users experience them.

For conversational agents, endpointing and language-model response time sit above this codec-level budget. Our voice-agent latency article examines that wider turn. Keep the boundary explicit when comparing numbers: a codec frame interval, a TTS time to first audio, and a caller's wait after finishing a sentence are different observations.

9. A small calculator you can audit#

The arithmetic fits in a short Python function. It takes vocabulary sizes individually, uses exact rational frame rates, and counts complete frames when a flattened context ends partway through a codebook group.

from fractions import Fraction
 
def budget(sample_rate, frame_rate, vocab_sizes, audio_positions):
    rate = Fraction(str(frame_rate))
    if sample_rate <= 0 or rate <= 0 or audio_positions < 0:
        raise ValueError("Rates must be positive; positions nonnegative")
    if not vocab_sizes or any(v < 2 for v in vocab_sizes):
        raise ValueError("Each codebook needs at least two entries")
    books = len(vocab_sizes)
    bits = sum((v - 1).bit_length() for v in vocab_sizes)
    return {
        "samples_per_hop": Fraction(sample_rate) / rate,
        "frame_ms": 1000 / rate,
        "indices_per_second": rate * books,
        "fixed_width_bits_per_second": rate * bits,
        "flattened_seconds": Fraction(audio_positions // books) / rate,
        "framewise_seconds": Fraction(audio_positions) / rate,
    }

Call it with the running example:

result = budget(24_000, "12.5", [2_048] * 8, 2_048)
for name, value in result.items():
    print(f"{name}: {float(value):g}")
 
# samples_per_hop: 1920
# frame_ms: 80
# indices_per_second: 100
# fixed_width_bits_per_second: 1100
# flattened_seconds: 20.48
# framewise_seconds: 163.84

The function assumes a fixed frame rate and an audio-only position budget. It does not simulate convolution padding, text allocation, a delayed schedule, or transport overhead. A noninteger samples_per_hop is a prompt to inspect the implementation's time grid and resampling path, rather than to round silently and let timestamps drift.

The article source bundle includes this calculator, the Markdown, the five Manim scenes, caption cues, and rendering instructions. The diagrams use constructed signals and vectors, so reproducing them requires no model checkpoints or private audio.

10. What to measure before choosing a tokenizer#

Start with the intended workload. A live conversation, an audiobook, a noisy transcription stream, and voice conversion stress different parts of the representation. Keep the source audio, evaluation splits, and inference conditions controlled when comparing candidates.

Verify the representation first. Record input and output sample rates, channel handling, effective hop, codebook count, vocabulary sizes, padding rules, and any side information. Inspect the actual shapes emitted for short clips and for a long stream. A mismatch here can invalidate later quality and capacity measurements.

Measure reconstruction and generation separately. A codec can reconstruct ground-truth codes well while a language model struggles to generate the right code distribution. Evaluate original audio against codec reconstruction, then evaluate generated speech against the intended text and voice. This separates bottlenecks that require different fixes.

Listen for the distinctions that matter. Include fast speech, names, numbers, negation, code-switching, quiet endings, interruptions, and the recording conditions present in the product. Character or word error rates help expose intelligibility failures. Speaker similarity and listening judgments answer different questions. A single perceptual score cannot stand in for all three.

Account for the full decoder. Include conditioning extraction, depth prediction, diffusion iterations where applicable, waveform synthesis, and resampling in the compute budget. Record device, precision, batch size, concurrency, context length, chunk size, and cache policy. Otherwise a rate comparison can hide most of the actual work.

Exercise the streaming contract. Feed uneven transport chunks, terminate on an incomplete codec interval, flush, and check the valid output length. Measure the delay at your real application boundary and inspect tail behavior. Repeatedly restarting a stateful encoder or decoder can introduce boundary artifacts that an offline whole-file test never exposes.

These measurements belong in a repeatable release evaluation; see our notes on evals as a release gate. They turn a tokenizer choice into an engineering decision with observable failure modes.

For the running configuration, the facts now fit together: 24,000 samples per second, an 80 ms hop, 12.5 frames per second, 100 indices per second, and a nominal 1.1 kbps code stream. How much speech the language model remembers depends on its sequence layout. How quickly a listener hears it depends on the entire streaming path.