Audio

RVQ codebooks

Quantizes audio into discrete tokens through several rounds of vector lookup, each round encoding the previous round’s error.

First page of SoundStream: An End-to-End Neural Audio CodecPopularized inJul 2021SoundStream: An End-to-End Neural Audio CodecZeghidour et al. · arXiv 2107.03312 ↗
RVQ: audio → discrete tokens, round by roundtarget≈ reconcodebook 1: coarse shape+ codebook 2: the residual+ codebook 3: nearly exact · a few tokens per frame

Residual vector quantization turns continuous audio features into discrete tokens a language model can predict: the first codebook approximates the vector, the second encodes the leftover error, and so on. More codebooks mean higher fidelity at more tokens per second. Neural audio codecs and most modern TTS/music models are built on this.

Adoption over time

Share of new models that have included RVQ codebooks over time.

1%202220232024202520261%20222023202420252026

See it in real models

Open any of these on hfviewer to find this block in the interactive architecture graph.

facebook/wav2vec2-base architecture graphfacebook/wav2vec2-base↓ 3.3M · ♡ 124Open in visualizer Qwen/Qwen3-TTS-12Hz-1.7B-Base architecture graphQwen/Qwen3-TTS-12Hz-1.7B-Base↓ 3.0M · ♡ 484Open in visualizer Qwen/Qwen3-TTS-12Hz-1.7B-CustomVoice architecture graphQwen/Qwen3-TTS-12Hz-1.7B-CustomVoicetext-to-speech · ↓ 2.2M · ♡ 2kOpen in visualizer facebook/wav2vec2-xls-r-300m architecture graphfacebook/wav2vec2-xls-r-300m↓ 2.2M · ♡ 131Open in visualizer ResembleAI/chatterbox architecture graphResembleAI/chatterboxtext-to-speech · ↓ 2.1M · ♡ 2kOpen in visualizer facebook/musicgen-medium architecture graphfacebook/musicgen-mediumtext-to-audio · ↓ 1.9M · ♡ 164Open in visualizer Qwen/Qwen3-TTS-12Hz-0.6B-CustomVoice architecture graphQwen/Qwen3-TTS-12Hz-0.6B-CustomVoicetext-to-speech · ↓ 1.5M · ♡ 175Open in visualizer kyutai/mimi architecture graphkyutai/mimifeature-extraction · ↓ 858k · ♡ 321Open in visualizer k2-fsa/OmniVoice architecture graphk2-fsa/OmniVoicetext-to-speech · ↓ 849k · ♡ 1kOpen in visualizer fishaudio/s2-pro architecture graphfishaudio/s2-protext-to-speech · ↓ 531k · ♡ 1kOpen in visualizer Qwen/Qwen3-TTS-12Hz-0.6B-Base architecture graphQwen/Qwen3-TTS-12Hz-0.6B-Basetext-to-speech · ↓ 494k · ♡ 276Open in visualizer Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign architecture graphQwen/Qwen3-TTS-12Hz-1.7B-VoiceDesigntext-to-speech · ↓ 429k · ♡ 387Open in visualizer

Browse all 52 models with this in the catalog →

Related concepts

Mel spectrogram front-end Convolutional subsampling Conformer block

hfviewer renders the full architecture of 3,500+ Hugging Face models as interactive graphs. Hover any block to see what it does, with this model’s real numbers.

Browse all model graphs →