Where the name comes from
In machine learning, "zero-shot" means doing a task for something the model saw zero examples of in training. A zero-shot voice model was trained on thousands of other voices, never on yours, and still speaks in yours, because it learned in general how voices differ and how to follow a reference. The opposite is fine-tuning, where a model is trained further on recordings of one particular voice.
How it works
- The reference clip is analysed. The model draws a compact description of the voice from it, sometimes called a speaker embedding, and may also use a few seconds of the clip directly as a prompt to continue from.
- The text is turned into tokens, small units the model works with.
- Speech is generated token by token, conditioned on both the text and the voice.
- Those tokens become audio in the reference voice.
In Chatterbox, the model abcVoiceover uses, the voice description comes from the whole clip, while the parts that copy the finer detail use its first 10 seconds, and its first 6 seconds as a prompt. That's one reason longer samples don't give better clones: past about 10 seconds, extra audio adds little, and the opening seconds matter most.
Zero-shot vs fine-tuned cloning
| Zero-shot | Fine-tuned | |
|---|---|---|
| Recording needed | Seconds | Minutes to hours |
| Training | None | Required, per voice |
| Ready | Immediately | After training |
| Try a different sample | Swap the clip | Train again |
| Resemblance | Good, very dependent on the clip | Often closer, with lots of clean audio |
| Data held about the voice | The clip | The clip and a trained model |
For most creative work, the speed and flexibility of zero-shot cloning outweigh the extra resemblance a fine-tuned model can reach: a cast of ten characters is ten short recordings, and a character who isn't working is a new recording away from a fix.
Getting the best from a zero-shot model
- Start speaking straight away. The first seconds carry the most weight, so a long silence, a cough or "is this on?" at the start costs you.
- 5 to 15 seconds is plenty. abcVoiceover accepts 3 seconds to 10 minutes, but a clean 10 seconds beats a noisy minute.
- Clean audio. Echo and noise are copied as faithfully as the voice.
- The right performance. The model imitates the sample's energy and pace in every line.
- The right language. A sample in the language the voice will speak gives the most natural accent.
The full checklist is in how to record a voice sample.
Speaking other languages
Because a zero-shot model separates voice from text, it can make a voice speak a language that isn't in its sample, a cross-lingual clone. The sample's accent tends to carry over, which can be charming for a character and a problem for native-sounding narration. A sample recorded in the target language avoids it. See the language pages.
Why it raises the stakes for consent
If seconds of audio are enough, then almost anyone's voice is enough: a voicemail, a video, a speech. That makes consent the line that matters. abcVoiceover's terms only allow your own voice or one whose owner has agreed. See the consent template.