XCodec2 by HKUST Audio: A Powerful Speech Tokenizer for LLM-Based Speech Synthesis

by

The rapid evolution of audio language models (ALMs) and large language model (LLM)-based speech synthesis has created the need for more advanced speech tokenization systems. Traditional neural audio codecs were primarily designed for compression efficiency, but modern speech AI systems require semantic awareness, multilingual support, and seamless integration with transformer-based architectures.

One of the most promising solutions in this space is XCodec2 by HKUST Audio. Released on Hugging Face, XCodec2 is a speech tokenizer optimized for LLaMA-based speech synthesis and large-scale audio language modeling. It introduces a streamlined vector quantization approach while maintaining high-quality speech reconstruction.

This article explores XCodec2’s architecture, features, installation process, use cases, licensing considerations, and why it is becoming a foundational component in next-generation speech AI systems.

What Is XCodec2?

XCodec2 is a neural speech tokenizer that converts raw waveform audio into discrete vector quantization (VQ) codes. These codes can then be used as input tokens for transformer-based speech models, similar to how text tokens are used in natural language processing.

Unlike traditional codecs focused purely on bitrate reduction, XCodec2 emphasizes semantic preservation. This makes it particularly well-suited for speech language models and text-to-speech (TTS) systems built on LLaMA-style architectures.

XCodec2 is closely associated with the research paper LLaSA: Scaling Train-Time and Inference-Time Compute for LLaMA based Speech Synthesis, which presents a framework for scaling speech synthesis using LLaMA-based models.

Core Features of XCodec2

1. Single Vector Quantization

XCodec2 uses a single vector quantizer rather than multi-stage quantization stacks. This simplifies the architecture while maintaining expressive capability for speech representation.

2. 50 Tokens Per Second

The model generates speech tokens at a rate of 50 tokens per second. This balance enables:

  • Efficient transformer modeling
  • Lower sequence length compared to frame-based systems
  • Improved training scalability

3. Multilingual Semantic Support

XCodec2 supports multilingual speech tokenization. It preserves phonetic and semantic information across languages, making it ideal for:

  • Multilingual TTS
  • Cross-lingual voice cloning
  • Speech-based foundation models

4. High-Quality Speech Reconstruction

Despite focusing on semantic representation, XCodec2 maintains high reconstruction fidelity. Decoded audio retains natural prosody and clarity, especially for speech sampled at 16kHz.

5. Large Model Capacity

With approximately 0.8 billion parameters (F32 tensor type), XCodec2 has the capacity required for large-scale speech modeling tasks.

Technical Specifications

FeatureSpecification
Model NameHKUSTAudio/xcodec2
Parameter Count~0.8B
Input Sample Rate16kHz only
Token Rate50 tokens/sec
FrameworkPyTorch
LicenseCC-BY-NC-4.0
Model TypeSpeech Tokenizer

The 16kHz constraint ensures consistent training alignment but requires preprocessing if working with higher sample rates.

Installation and Setup

To begin using XCodec2, create a dedicated Python environment:

conda create -n xcodec2 python=3.9
conda activate xcodec2
pip install xcodec2

The maintainers recommend:

  • xcodec2==0.1.5 for inference and LLaSA fine-tuning
  • xcodec2==0.1.3 for more stable codec training alignment

Basic inference example:

import torch
import soundfile as sf
from xcodec2.modeling_xcodec2 import XCodec2Model

model = XCodec2Model.from_pretrained("HKUSTAudio/xcodec2")
model.eval().cuda()

wav, sr = sf.read("test.wav")
wav_tensor = torch.from_numpy(wav).float().unsqueeze(0)

with torch.no_grad():
    vq_code = model.encode_code(input_waveform=wav_tensor)
    recon_wav = model.decode_code(vq_code).cpu()

sf.write("reconstructed.wav", recon_wav[0, 0, :].numpy(), sr)

The model currently supports single-input inference. For batch inference or large-scale code extraction, the official repository provides extended utilities.

Relationship to LLaSA

XCodec2 plays a foundational role in the LLaSA framework. LLaSA scales both training-time and inference-time compute for LLaMA-based speech synthesis. By converting audio into discrete tokens, XCodec2 enables speech to be modeled similarly to text within large language models.

This approach unlocks:

  • Unified speech-text modeling
  • Scalable training on tokenized speech datasets
  • Long-context speech generation
  • Multi-speaker synthesis

The associated Llasa collection on Hugging Face provides models trained on 160,000 hours of tokenized speech data.

Use Cases

1. Speech Language Models

XCodec2 enables the creation of speech-native language models where speech tokens replace text tokens.

2. LLaMA-Based Text-to-Speech

Because it is optimized for transformer architectures, XCodec2 integrates naturally with LLaMA-style decoders.

3. Multilingual TTS Systems

Its multilingual semantic support makes it ideal for systems supporting Chinese, English, Japanese, Korean, and other languages.

4. Research in Audio Tokenization

Researchers exploring token efficiency, semantic compression, and neural speech modeling can leverage XCodec2 as a strong baseline.

Licensing Considerations

XCodec2 is released under the CC-BY-NC-4.0 license. This means:

  • Attribution is required
  • Commercial use is not permitted without permission

Organizations planning commercial deployment must carefully review licensing implications before integration.

Strengths and Limitations

Strengths

  • Designed specifically for speech LLMs
  • Strong semantic preservation
  • Scalable token generation
  • Active research backing
  • High model capacity

Limitations

  • Non-commercial license
  • Large memory footprint
  • 16kHz-only input
  • Single-input inference limitation

How XCodec2 Compares to Traditional Codecs

Traditional codecs like EnCodec focus on bitrate compression. XCodec2 shifts the focus toward semantic richness and compatibility with transformer models.

Rather than minimizing kilobits per second, it optimizes for token efficiency within large language modeling pipelines. This makes it more suitable for AI-driven speech generation rather than bandwidth-limited communication.

Future Outlook

As audio language models grow in popularity, speech tokenizers will become as critical as byte-pair encoders were for text models. XCodec2 represents a major step toward scalable speech foundation models.

Future directions likely include:

  • Higher sample rate support
  • Commercial licensing paths
  • Improved batch inference APIs
  • Integration with multimodal LLM systems

With the release of the LLaSA framework and growing community adoption, XCodec2 is positioned as a key infrastructure layer for speech-native AI systems.

Conclusion

XCodec2 by HKUST Audio is a powerful and research-driven speech tokenizer designed for LLaMA-based speech synthesis and large-scale audio language modeling. By combining single vector quantization, multilingual semantic support, and transformer compatibility, it bridges the gap between neural audio codecs and modern speech foundation models.

Although its non-commercial license may limit enterprise deployment, it stands out as a leading solution for academic research and advanced speech AI development. As speech-native LLMs continue to expand, XCodec2 is likely to remain a cornerstone technology in this rapidly evolving field.

Leave a Comment