If you've followed image generation over the past three years, you know the playbook: train a denoising model in a compressed latent space, condition it on text or other signals, and iteratively refine noise into structured output. The same playbook is now driving the most impressive results in audio generation. But audio has its own set of challenges that require meaningful architectural adaptations.
This primer walks through how latent diffusion works for audio, what's different from the image domain, and where the open problems are.
Why Latent Space?
Raw audio is absurdly long. A 3-minute song at CD quality (44.1kHz, 16-bit stereo) is about 32 million samples. Training a diffusion model directly on sequences this long is computationally intractable because the attention mechanism in a transformer scales quadratically with sequence length, and even with efficient attention variants, millions of tokens are beyond reach.
The solution is to compress the audio first. A neural audio codec (EnCodec, DAC, SoundStream) encodes raw audio into a much shorter sequence of discrete or continuous tokens. A typical codec operating at 50Hz with 8 residual quantization layers turns our 3-minute song into roughly 24,000 tokens, a reduction of over 1,000x. The diffusion model operates in this compressed space.
After generation, a decoder converts the latent representation back to raw audio. The quality of this roundtrip (encode, generate in latent space, decode) depends critically on the codec's ability to preserve perceptually important information during compression.
The Denoising Process
Diffusion models learn to reverse a noise-adding process. During training, you take a clean audio latent, add Gaussian noise at a randomly sampled timestep, and train the model to predict either the noise that was added or the clean signal that should result. At inference time, you start with pure noise and iteratively denoise, with each step producing a slightly cleaner signal.
The mathematical framework is well-established. The forward process is a fixed Markov chain that gradually corrupts the signal. The reverse process is a learned model, typically a transformer, that estimates the denoising step at each noise level. The training objective reduces to a weighted regression loss at each timestep.
For audio, the key difference from image diffusion is the temporal dimension. Images are 2D spatial grids; audio latents are 1D temporal sequences (or 2D if you include the codebook dimension). The temporal structure means that long-range dependencies matter more, as musical phrases, harmonic progressions, and rhythmic patterns create structure across thousands of tokens that the model needs to maintain coherent.
The Backbone: Diffusion Transformers (DiT)
Early audio diffusion models used U-Net architectures adapted from image generation. Recent work has shifted to Diffusion Transformers (DiTs), standard transformer architectures trained with the diffusion objective. The shift mirrors what happened in image generation, where DiTs (as used in DALL-E 3 and Stable Diffusion 3) proved more scalable and performant than U-Nets.
For audio, DiTs have a specific advantage: transformers with attention naturally handle long-range dependencies, which are critical for musical coherence. A U-Net's receptive field is limited by its depth and kernel size; a transformer's attention can, in principle, attend to any position in the sequence.
In practice, full attention over audio sequences is still expensive. Most implementations use some form of efficient attention (windowed, dilated, or linear attention) to manage computational cost. The art is in balancing the attention window size against the need for long-range coherence.
Conditioning
The power of diffusion models comes from conditioning, providing additional information that guides the generation. For text-to-audio, this means conditioning on text embeddings from a language model. For singing voice synthesis, you might condition on pitch contours, lyrics, speaker embeddings, and a reference audio clip simultaneously.
Conditioning in DiTs is typically implemented via cross-attention (the conditioning signal attends to the noisy latent) or adaptive layer normalization (AdaLN, where the conditioning modulates the normalization parameters). AdaLN is the current preferred approach; it's simpler, more parameter-efficient, and empirically works as well or better than cross-attention for most conditioning signals.
Classifier-free guidance (CFG) amplifies the effect of conditioning at inference time by interpolating between conditioned and unconditioned predictions. Higher CFG values produce output that more closely matches the conditioning signal, at some cost to diversity and naturalness. The optimal CFG value is application-dependent and is one of the key hyperparameters to tune.
Audio-Specific Challenges
Temporal coherence. A generated audio clip needs to be coherent over its full duration; a pop song has verse-chorus structure, a speech clip has sentence-level prosody. Maintaining this coherence over thousands of generation steps is harder than maintaining spatial coherence in an image.
Pitch precision. In music, pitch errors of 20-50 cents (fractions of a semitone) are clearly audible. This is a much tighter precision requirement than image generation faces for any analogous attribute. Models need to learn to control pitch with high accuracy, which often requires explicit pitch conditioning rather than relying on the model to learn pitch control implicitly.
Multi-scale structure. Audio has structure at multiple time scales: individual samples (microseconds), phonemes (tens of milliseconds), words (hundreds of milliseconds), phrases (seconds), and sections (tens of seconds). The latent representation and the model architecture need to capture all of these scales simultaneously.
Evaluation. Evaluating generated audio is harder than evaluating generated images. Human preference is the gold standard, but it's expensive and slow. Automated metrics (FAD, FID adapted for audio, PESQ, STOI) capture different aspects of quality and don't always correlate with human judgment. The field needs better evaluation methodology.
The Current State
Latent diffusion for audio is roughly where latent diffusion for images was in late 2022: the core approach works, quality is impressive but inconsistent, and there's a clear path to substantial improvement through scaling and architectural refinement.
The best current systems can generate coherent audio clips of 30-90 seconds with good quality. Full-song generation (3-5 minutes) with maintained structure is an active research goal. Conditioning fidelity (how precisely the output matches the conditioning signal) is improving rapidly, driven by better conditioning architectures and training strategies.
If you're building in this space, the key technical choices are: codec selection (which determines your latent space), backbone architecture (DiT variant and attention mechanism), and conditioning strategy (what signals you provide and how they're injected). Getting these right is the difference between a demo and a product.