Current audio generation models produce impressive output for 30 to 90 seconds. Push past that, and things fall apart. Melodies lose coherence. Rhythmic patterns drift. Song structure (verse, chorus, bridge) dissolves into aimless wandering. The model forgets where it's been and where it's going.
The bottleneck is attention. And how the field solves it will determine whether AI audio remains a short-form novelty or becomes a tool for full-length production.
The Quadratic Wall
Standard self-attention scales quadratically with sequence length. Double the sequence, quadruple the compute and memory. For text, this is manageable; a long document might be a few thousand tokens. For audio latents, a 3-minute song compressed through a neural codec at 50Hz is roughly 9,000 tokens. A 5-minute song is 15,000. Full attention over these lengths is computationally brutal, and the memory requirements exceed what most training setups can handle.
The result is that most audio models are trained on short clips, typically 10 to 30 seconds, and struggle to generalize to longer durations at inference time. They've never seen long-range musical structure during training, so they can't reproduce it during generation.
Windowed and Sparse Attention
The most straightforward approach is to limit the attention window. Rather than attending to every position in the sequence, each token attends only to its local neighborhood, say 512 or 1024 tokens in each direction. This reduces the complexity to linear in sequence length and makes long sequences tractable.
The tradeoff is obvious: local attention can't capture long-range dependencies. A chorus that should mirror an earlier chorus, a melodic callback to the intro, a dynamic arc that builds over minutes: these require attending to positions far outside any reasonable window size.
Sparse attention patterns attempt to split the difference. Techniques like dilated attention (attending to every Nth position), strided attention (attending to fixed stride positions at multiple scales), and random attention (attending to a random subset of positions) maintain some long-range connectivity while keeping computation manageable.
In practice, sparse attention works better than pure local attention but still struggles with the specific kind of long-range structure that music requires. Musical structure isn't random or evenly distributed. It's hierarchical (notes within beats within bars within phrases within sections) and repetitive (the chorus recurs, the verse pattern repeats). Generic sparse patterns don't capture this structure efficiently.
Hierarchical Approaches
A more promising direction mirrors the hierarchical structure of music itself. Rather than generating a flat sequence of tokens, these approaches generate at multiple levels of abstraction.
At the highest level, a model generates a structural plan: the sequence of sections, their approximate durations, key centers, and energy profiles. At the next level, a model fills in each section, conditioned on the structural plan and on neighboring sections. At the lowest level, a model generates the fine-grained audio details within each sub-section.
This cascaded approach means that no single model needs to attend over the full song length. The high-level model operates on a short sequence (one token per section, perhaps 8-16 tokens total). The mid-level model operates on section-length sequences (a few hundred tokens). Only the low-level model deals with dense audio tokens, but it only needs to generate short segments because the broader context is provided by the higher levels.
The challenge is training these systems end-to-end and ensuring coherence across the levels. Errors at the structural level propagate downward; a bad section plan produces a bad song regardless of how good the low-level generation is. And the interfaces between levels need careful design to avoid discontinuities.
State Space Models
The most architecturally novel approach comes from state space models (SSMs), specifically Mamba and its variants. SSMs process sequences with linear complexity and constant memory per step, making them theoretically ideal for long sequences. Unlike attention, which computes pairwise relationships between all positions, SSMs maintain a compressed hidden state that's updated incrementally.
Early experiments applying SSMs to audio generation have shown mixed results. They handle long sequences efficiently, but the compressed state loses fine-grained information about specific earlier positions. For audio, this means they can maintain general stylistic consistency over long durations but struggle with exact repetition, such as reproducing a specific melody or rhythmic pattern from earlier in the piece.
Hybrid architectures that combine SSMs for long-range context with local attention for fine-grained detail are an active research direction. The SSM provides the "memory" of the overall piece, while local attention handles the precise, position-specific relationships that musical coherence requires.
Memory and Retrieval
Another approach borrows from the retrieval-augmented generation (RAG) paradigm in NLP. Rather than forcing the model to maintain all context in its hidden state or attention window, explicitly store previously generated segments in an external memory and retrieve relevant segments when generating new ones.
For music, this is particularly natural. When generating a second chorus, the model retrieves the first chorus from memory and uses it as a reference, ensuring melodic and lyrical consistency without needing to attend over the entire intervening sequence. The retrieval can be learned (the model decides what to retrieve) or rule-based (always retrieve the most structurally similar previous section).
The appeal is that this cleanly separates the "memory" problem from the "generation" problem. The generation model can focus on producing high-quality audio in a short window, while the memory system handles long-range consistency. This modular approach is easier to debug and improve incrementally than end-to-end long-context generation.
What's Missing
All of these approaches make progress on the length problem, but none fully solve it. The core difficulty is that musical coherence operates simultaneously at multiple time scales: sample-level continuity, beat-level rhythm, phrase-level melody, section-level structure, song-level arc, and current architectures handle some scales better than others.
The most likely near-term solution is a combination: hierarchical generation for structural planning, efficient attention (sparse or SSM-based) for medium-range coherence, and retrieval-augmented memory for exact repetition. No single mechanism handles all the requirements, but together they cover the space.
Full-song generation with intentional, coherent structure is probably 12 to 18 months away for constrained genres and conditions. The general case (any genre, any structure, with the kind of intentional arc that a human songwriter creates) is further out. But the architectural ideas are in place. The remaining work is engineering, not fundamental research.