1. Cleaning up the text
Before anything is generated, the text is tidied into a form the model knows well. In Chatterbox, the open-source model abcVoiceover uses, that includes:
- Punctuation: ellipses, colons, semicolons and spaced dashes become commas; a missing full stop is added at the end.
- Lower case: the text is lowercased, which is why capitals don't add emphasis or spell out acronyms.
- Language-specific steps: Japanese kanji are converted to kana with a dictionary, Korean syllables are split into their letters, and Chinese characters are split into words and encoded by their shapes.
That's why the way a script is written matters so much. See pacing and pauses and numbers and abbreviations.
2. Tokens
The cleaned text is split into tokens, small pieces such as letters, parts of words or whole words, each with a number the model works with. Speech is represented the same way: as speech tokens, each standing for a tiny slice of sound. Generating speech becomes predicting the right sequence of speech tokens for the text.
3. The reference voice
For voice cloning, the model also takes in the reference recording. It draws a compact description of the voice from it, and uses the start of the clip as an example to continue from. Every speech token it predicts is conditioned on both the text and that voice. See zero-shot voice cloning.
4. Generating speech tokens
The model predicts speech tokens one after another, each time choosing from many possible next tokens. How it chooses is where the settings come in:
| Setting in abcVoiceover | What it is | What it changes |
|---|---|---|
| Expressiveness | Temperature | How adventurous each choice is: varied or predictable. |
| Guidance strength | Classifier-free guidance weight | How strongly the result is pushed towards the text and reference. |
| Emotion intensity | An "exaggeration" control the model takes as input | How dramatic the delivery is. |
| Min probability, Top P | Limits on which tokens may be chosen | How safe or surprising the choices can be. |
| Repetition penalty | A penalty on repeating recent tokens | Too high and the voice keeps talking past the end of the line. |
Every slider is explained in voice settings explained.
5. From tokens to sound
A second network turns the speech tokens into audio in the reference voice: first into a detailed picture of the sound, then into a waveform you can play. Chatterbox's audio comes out at 24 kHz, plenty for speech. It also adds an inaudible watermark that detection tools built for it can find later.
6. Long text
Models work best on a few sentences at a time. abcVoiceover splits a long line at sentence endings into pieces of about 20 seconds of speech, generates each in the same voice, and joins them with a short pause. See narrating long text.
7. Checking the result
Generation is a sequence of choices, so occasionally a piece skips a word or rambles on. abcVoiceover checks every piece with Whisper, a speech recognition model: it compares what it hears with the text, and a piece that's missing words or runs far past its text is generated again, up to three tries. What it can't judge is taste, so the final listen is still yours.
Why the same line comes out differently
Because each token is chosen with a little randomness, two takes of the same line are never identical, just as a human actor never reads a line the same way twice. That's a feature when you want options, and the reason saved lines lock their take. See picking the best take.