U-Codec: Ultra Low Frame-rate Neural Speech Codec for Fast High-fidelity Speech Generation

📄 Report · Code · 🤗 Hugging Face

We propose U-Codec, an Ultra low frame-rate neural speech Codec that achieves high-fidelity reconstruction and fast generation via an extremely frame-rate at 5Hz (5 frames per second). Extreme compression at 5Hz typically leads to severe intelligibility and spectral detail loss, we overcome this by integrating a Transformer-based inter-frame long-term dependency module and systematically optimizing residual vector quantization (RVQ) depth and codebook size. Moreover, we apply U-Codec into a large language model (LLM)-based auto-regressive TTS model, which leverages global and local hierarchical architecture to effectively capture dependencies across multi-layer tokens. We extend its application to from 3 RVQ at 50Hz up to 32 RVQ layers at 5Hz. Experimental results demonstrate that U-Codec improves LLM-based TTS inference speed by around 3 x over high-frame-rate codecs, while preserving similarity and naturalness. These results validate the feasibility of using highly compressed 5Hz discrete tokens for fast, high-quality speech synthesis.

Fig.1 Reconstruct quality on varying the frame rate of different codecs, where bubble size indicates the number of RVQ layers. Compared to previous systems, such as EnCodec, Mimi, and DAC operating at higher frame rates, our proposed codec achieves competitive PESQ performance under ultra low frame rates (5Hz).

Fig.2 Architecture and training of our U-Codec at 5Hz. The encoder (left) consists of convolutional layers followed by a Transformer to capture long-term dependencies. Latent features are quantized through factorized residual vector quantization (FRVQ) and optimized for high-fidelity reconstruction. The decoder (right) mirrors the encoder to synthesize the output waveform.