Speech synthesis is, for most practical purposes, solved. Modern TTS systems produce output that's difficult to distinguish from recorded human speech. You can clone a voice from a few seconds of audio and generate fluent, natural sentences in that voice. The remaining challenges are edge cases: unusual prosody, extreme emotions, noisy reference audio.
Singing is a different world. And the gap between speech and singing synthesis is not a matter of degree; it's a difference in kind.
The Precision Problem
In speech, pitch is loosely controlled. Your voice rises and falls over a range of about one octave, following broad intonational patterns. A pitch error of a semitone is barely noticeable. Timing is fluid, and words can stretch and compress without sounding wrong.
In singing, pitch must be controlled with extreme precision across two to three octaves. An error of 20 cents (a fifth of a semitone) is clearly audible as "out of tune." Timing is locked to a musical grid. A note that arrives 50 milliseconds late sounds wrong in a way that the same delay in speech never would.
This precision requirement propagates through every component of a synthesis system. The acoustic model needs finer pitch resolution. The vocoder needs to handle a wider frequency range without artifacts. The duration model needs to respect musical meter, not just linguistic rhythm. Every module that works fine for speech needs to be significantly more capable for singing.
The Dynamic Range Problem
Speech operates in a relatively narrow dynamic range. You might whisper or shout, but the variation between your quietest and loudest moments in conversation is modest. Singing spans an enormous dynamic range, from a barely audible pianissimo to a full-power belt, and the transitions between dynamics carry enormous musical meaning.
A crescendo building into a chorus, a sudden drop to a whisper, the dynamic arc of a phrase: these aren't ornamental. They're core to the musical expression. A synthesis model that flattens dynamics or misplaces dynamic transitions produces output that sounds technically correct but emotionally dead.
Modeling dynamics in singing requires understanding musical context that goes far beyond the current timestep. The model needs to know where it is in the phrase, in the section, in the song, and how the dynamic trajectory serves the musical intent. Current models handle this poorly compared to other aspects of synthesis.
The Vocal Technique Problem
Speech uses a small subset of the vocal tract's capabilities. Singing exploits the full range: chest voice, head voice, falsetto, mixed voice, vocal fry, breathy tone, belt, twang, vibrato, tremolo, melisma, portamento, vocal runs, growl, scream, whistle register.
Each of these techniques involves different physiological configurations. Vibrato is a periodic oscillation of pitch, typically 5-7 Hz, with an amplitude of 30-100 cents. Belting involves high subglottal pressure with a relatively low larynx position. Falsetto engages a fundamentally different mode of vocal fold vibration than chest voice.
A model trained primarily on speech data has never encountered most of these techniques. It doesn't know what vibrato is, let alone how to produce it naturally. And the techniques interact. A singer might belt with vibrato while transitioning into a melismatic run, all in the span of two seconds. The combinatorial space of vocal techniques in singing dwarfs anything in speech.
The Register Transition Problem
Every singer has register transitions, points in their range where they shift from one mode of phonation to another (chest to head voice, head voice to falsetto). These transitions are unique to each singer. A trained vocalist smooths these transitions; an untrained one has audible "breaks."
For voice cloning, this creates a particularly nasty problem. A three-second reference clip of speech captures a person's voice in one register, in a narrow pitch range, with limited dynamic variation. It tells you almost nothing about how that person sounds an octave higher, or in head voice, or at full volume. Extrapolating from a speech reference to a singing performance requires the model to hallucinate information it fundamentally doesn't have.
This is why singing voice cloning from a speech reference typically works in one part of the range and fails in others. The cloned voice sounds right in the pitch range near the reference and increasingly wrong as you move away from it. Solving this requires either much longer reference audio that captures the singer's full range, or models with strong learned priors about how human voices behave across registers.
The Expressiveness Gap
Listen to a great vocal performance and try to catalog everything that makes it expressive. The way a vowel bends into pitch. The slight breathiness at the start of a phrase. The way the singer pushes ahead of or behind the beat. The micro-variations in vibrato rate and depth across different notes. The specific timbral shift when they move from intimate to powerful.
These details are what separate a good vocal performance from a generic one. They're also extremely difficult to model. Current synthesis systems tend to produce output that's in the right ballpark of expressiveness but lacks the specific, idiosyncratic details that make a performance feel human. The result is output that sounds "correct" but not "alive."
The challenge is that expressiveness is high-dimensional and context-dependent. The same note might be sung with dozens of different expressive colorings depending on the lyrical content, the emotional arc, the genre conventions, and the individual singer's style. Capturing this requires models that understand musical context at a depth that current architectures struggle with.
Where Things Stand
Despite all of these challenges, singing voice synthesis has made meaningful progress. Modern systems can produce clean, on-pitch singing with natural timbre for constrained use cases. Solo vocals in mainstream pop genres, with moderate pitch ranges and standard vocal techniques, are approaching usable quality.
The frontier challenges (extreme ranges, complex technique, fine-grained expressiveness, and faithful voice cloning across the full singing range) remain active research problems. Progress on these fronts will require either architectural innovations specifically targeting singing (rather than adapting speech models) or significantly larger and more diverse training datasets of singing data.
The gap between speech and singing synthesis is real, and it matters. Singing is one of the most complex and expressive uses of the human voice. Synthesizing it convincingly is a harder problem than most of the AI community has appreciated. The teams that take it seriously, rather than treating it as an incremental extension of speech, will be the ones that make real progress.