How it works, in brief
A voice cloning model is trained on a very large amount of speech from many different speakers. In the process it learns two things that are usually tangled together: what is being said, and how it sounds, meaning the qualities that make one person's voice different from another's. Give the trained model a short recording of a new speaker, and it captures the "how it sounds" part; give it text, and it supplies the "what is said". The result is new speech that combines the two.
The detail varies from model to model. Many current systems, including the open-source Chatterbox model that abcVoiceover uses, turn text into a sequence of speech tokens, guided by the reference recording, and then turn those tokens into audio. How AI voice generators work goes step by step.
Two kinds of voice cloning
| Zero-shot cloning | Trained (fine-tuned) cloning | |
|---|---|---|
| Audio needed | One short clip, a few seconds | Usually minutes to hours |
| Time before it's ready | Immediately | Minutes to hours of training |
| Changing the voice | Swap the clip | Train again |
| Resemblance | Good, and depends heavily on the clip | Often closer, with enough clean data |
abcVoiceover uses zero-shot cloning: one 5 to 15 second recording per voice, no training, ready at once. See what is zero-shot voice cloning?
What a clone copies
- Timbre: the basic colour of the voice, deep or light, warm or bright.
- Accent: carried over even into other languages.
- Pace and energy: a fast, lively sample gives fast, lively speech.
- The recording itself: echo, background noise and microphone character come along too, which is why a clean sample matters so much.
What it can't do
- Perform what isn't in the sample as convincingly as what is. A calm sample makes a less convincing shout.
- Reproduce every quirk of a person's speech: their particular laugh, the way they say one word.
- Read every word right: names and rare words can be mispronounced and need respelling.
- Make non-speech sounds well: sobs, gasps, laughter and screams are hard for most models.
- Speak languages it wasn't trained on. Chatterbox speaks 23.
What people use it for
Audio dramas and podcasts, game dialogue, animation and video narration, audiobooks, courses and training, product videos, accessibility, and localization into other languages, among many others. The use cases cover each in detail. What these have in common is a script, a voice someone has agreed to lend, and many lines to make.
The responsibility that comes with it
The same technology that lets a performer license their voice, or a creator narrate every video in their own voice, can be misused to impersonate people, fake recordings or run scams. Responsible use rests on three rules: clone a voice only with its owner's consent, tell your audience when voices are AI-generated, and never use a clone to deceive. See using AI voices responsibly and is AI voice cloning legal?
How abcVoiceover does it
- Zero-shot cloning with Chatterbox, an open-source model, running on a server the abcVoiceover team operates.
- 23 languages, chosen voice by voice.
- Every take checked with Whisper speech recognition, and remade if it skips words or runs on.
- An inaudible watermark added by Chatterbox to the audio it generates.
- Your recordings stay yours: not sent to outside AI companies, not used to train models.