Following my Croatian RAG LLM, I decided to build a small text-to-speech (TTS) model specifically for Croatian. The goal was real-time performance on a smartphone CPU.

I used distillation: I had a large TTS model (HiggsAudio v3) read Croatian scripts to create training data, then trained a small model from scratch on that data. This time, I first ran a pilot with 7.5 hours of data and compared how the same sentence changed as training progressed.

Here is the conclusion up front.

• With 32 million parameters (18.3M Matcha-TTS + 13.5M Vocos), it genuinely speaks Croatian
• After 50 minutes of training (5,000 steps), it was already reading the script correctly. Further improvements were mainly in reducing noise.
• On 14 evaluation sentences, the final character error rate was 0.9% (10 of 14 sentences were exact matches)
• Even on one CPU thread, it generated one second of audio in 0.09 seconds (i7-13700K)

Demo: Let It Speak

I put the final 30,000-step version of the model from this article in the Croatian TTS Playground. Enter any Croatian text and it will read it aloud on the spot.

• Choose from 50 example sentences (greetings, travel, hospitals, and more), or enter up to 1,000 characters of your own
• You can adjust speaking speed, the amount of intonation, and the number of synthesis steps. With a fixed seed, the same sentence produces the same audio every time
• Synthesis runs entirely on the server CPU (ONNX Runtime; no GPU or external API)
• It cannot read numbers, so they need to be written out as words, such as “dvadeset pet (25)”

What I Did

1. Create scripts: I had a local Gemma 4 31B write Croatian everyday-conversation lines. I randomly specified the situation, conversation partner, emotion, and length—for example, “talking with an elderly woman at a market” or “being scared by a strange sound in the middle of the night”—and generated 22 lines at a time. The speaker was set as a single woman in her thirties. I mixed in 23 emotions, including neutral, joy, anger, and whispering.

2. Turn them into audio: I had HiggsAudio v3 read the scripts in the voice of a 16-second reference clip. Adding emotion tags such as <|emotion:anger|> made it appropriately angry or cheerful. I generated 4,500 clips in about five hours.

3. Check them automatically: I transcribed the clips with Whisper large-v3 and compared them against the scripts, filtering out misreadings. I also used a speaker-recognition model to measure vocal similarity to the reference clip and filtered out samples where the voice had drifted. 4,393 of 4,500 clips passed (97.6%). After trimming leading and trailing silence, this came to 7.46 hours.

4. Train the small model: I trained Matcha-TTS—a flow-matching TTS approach—on an RTX 4090 for five hours, or about 30,000 steps. For the component that converts back to an audio waveform, the vocoder, I used the publicly available Vocos as-is.

Croatian is written largely as it is pronounced, so I used the characters directly rather than converting them to phonetic symbols. However, lj, nj, and dž are each one sound, so I treated each as a single character.

Comparing Audio Throughout Training

I synthesized a sentence that was not in the training data—“Molim Vas, možete li mi reći koliko košta ova haljina?” (“Excuse me, how much is this dress?”)—using models at different points in training.

Step 5,000 (about 50 minutes after training began)

Step 10,000 (about 1 hour 40 minutes)

Step 15,000 (about 2 hours 20 minutes)

Step 20,000 (about 3 hours 10 minutes)

Step 30,000 (about 5 hours, final)

For reference, here is one training-data sample created by HiggsAudio, the teacher model (a different sentence: “Samo polako hodajte, nema potrebe da žurimo kroz ovaj mrak.”).

When transcribed by Whisper, this sentence matched the script character for character from step 5,000 through the end. In other words, it could already say the right words from the beginning.

The difference is audio quality. To my ear, step 5,000 has a layer of rough noise, which is greatly reduced by step 10,000. This becomes clear when the same audio is shown as a spectrogram, with time on the horizontal axis and frequency on the vertical axis.

The horizontal bands that follow pitch—the harmonics—are blurred at step 5,000, with a hazy fill between them. That is the part that sounds like noise. From step 10,000 onward, the bands are crisp and the gaps become dark. As training advances, the bands also curve smoothly up and down, making the movement in pitch, or intonation, appear more distinct.

Looking at the Numbers

I transcribed 14 evaluation sentences, all outside the training data, with Whisper large-v3 and measured character error rate (CER). They range from short greetings to monologues of around 16 seconds.

Steps	Training time	Character error rate	Exact match
5,000	About 50 minutes	1.1%	9 / 14
10,000	About 1 hour 40 minutes	3.4%	8 / 14
15,000	About 2 hours 20 minutes	0.7%	11 / 14
20,000	About 3 hours 10 minutes	0.9%	11 / 14
30,000	About 5 hours	0.9%	10 / 14

• The worse score at step 10,000 was almost entirely because one sentence, “Bok!” (“Hi!”), was transcribed as “Vok.” Since one of three characters differs, that one sentence has a 33% error rate. To the ear, step 10,000 was far cleaner than step 5,000. Character error rate only measures whether it is being read correctly, so audio quality ultimately has to be judged by listening.
• The remaining errors at the end were đ and dž in “Đak je pojeo džem” (“The student ate jam”). These are infrequent sounds even in Croatian, and were transcribed as “Žak” and “Džak.”

CPU Speed

I measured it on an i7-13700K using unoptimized PyTorch. These are averages over three runs generating four sentences totaling 47.6 seconds of audio. RTF is “generation time ÷ audio duration”; lower is faster.

Threads	Steps	Matcha	Vocos	Total RTF
1	2	0.042	0.025	0.067
1	4	0.067	0.025	0.092
4	4	0.032	0.010	0.042
8	4	0.028	0.007	0.036

“Steps” is the number of iterations used to turn noise into audio during synthesis; the samples above were generated with four steps. Even one thread can generate one second of audio in 0.09 seconds. A smartphone CPU will be slower, but it should still fit within real time—though I have not measured it on an actual device yet, so that is only a prediction.

How Many Hours of Data Does TTS Need?

Since I got this far with 7.5 hours, I looked into common benchmarks.

• LJSpeech, a widely used English single-speaker TTS dataset, contains about 24 hours. The official Matcha-TTS model is trained on it as well
• Japanese JSUT and Chinese CSMSC contain about 10 hours each and are commonly used as single-speaker corpora
• For training from scratch, around one hour is a lower bound, though pronunciation errors can increase depending on the model
• When fine-tuning a pretrained model, one report found that 30 minutes produced results comparable to training from scratch for 27 hours

I also found a study that trained Matcha-TTS entirely on synthetic speech (Shanghai Jiao Tong University, December 2025). It found that greater text diversity, clean noise-free audio, and consistent speaking style all improve training. Since synthetic data is noise-free and consistent in speaking style, it may be easier to train on than real recordings for the same amount of audio. That would fit with how far this project got in 7.5 hours.

Things That Went Wrong

• My PC froze: Two instances of the 31B model for script creation accidentally loaded onto the GPU simultaneously, overflowing VRAM. After that, I made “only one model on the GPU at a time” a rule.
• Gemma 4 overthinking: Roughly two-thirds of its output was “thinking.” Disabling it made generation about three times faster.
• HiggsAudio hallucinations: In an early test, it spontaneously continued after the sentence with “Hvala što pratite kanal” (“Thank you for watching the channel”), producing an 81-second clip. It seems to have learned a YouTube sign-off. I began filtering out clips whose length was far from the expected value.
• Matcha-TTS would not run on Windows: Some C extensions would not build, so I rewrote those parts in Python using numba.