Following on from the Croatian speech synthesis model. This time, I had one small model speak three languages: Croatian, Japanese, and English. Then I compared how much difference there was with 10, 5, and 3 hours of training data per language.

The conclusion first.

• In the end, there was almost no difference whether each language had 10, 5, or 3 hours of data. The best character error rate was 0.7% for all three (28 out of 32 sentences matched perfectly)
• Training progressed almost identically as well. In every case, Japanese and English could be read almost exactly as scripted after 3,000–5,000 steps, and Croatian after 10,000 steps
• Only the 3-hour version retained a tendency for a while to misread “Bok! (Hi!)” as “Blok.” It took a little longer to settle into its best state
• There was no difference even with sentences full of technical terms. I tested this because I expected the 10-hour version to have a stronger vocabulary, but all three were almost the same
• The three models made mistakes on the same words in the same way for technical terms. In English, the cause was not the model but the component that converts text into phonetic symbols (the dictionary)

What I did

Models: The same Matcha-TTS as last time (about 18.3 million parameters), plus Vocos (about 13.5 million parameters) to convert it back into waveforms. The speaker was one man in his twenties.

How I made it support three languages: I used phonetic symbols shared across all three languages (IPA) as the model input. Sounds that are the same across languages, such as “ʦ” and “ʨ,” share the same symbol. To specify which language to speak, I used the number originally meant to switch speakers as a “language number.”

• Croatian: Since it is a language that is pronounced almost exactly as written, I simply replace each character one-to-one with a phonetic symbol
• Japanese: I used OpenJTalk to extract pronunciations and accents, then added markers for where pitch rises and falls
• English: I converted it to phonetic symbols with g2p_en (the CMU Pronouncing Dictionary and a small model that estimates pronunciations for words not in the dictionary)

Training data: Audio of the same male speaker reading dialogue lines in the three languages. After using speech recognition to compare it with the scripts and remove misread lines, the amounts were 9.4 hours of Croatian, 9.7 hours of Japanese, and 8.3 hours of English (about 27 hours total). I call this the “10-hour version.”

From there, the “5-hour version” was made by randomly selecting exactly 5 hours from each language (15 hours total), and the “3-hour version” by selecting 3 hours each (9 hours total, about 5,800 clips). All other settings—the model size, batch size, and 60,000 steps—were identical. Since the number of steps was the same, less data meant seeing the same audio more often (the 3-hour version saw it about three times as often as the 10-hour version).

Training used a single RTX 4090 and took 6 to 8.5 hours per run.

Comparing voices during training

I synthesized the following sentences, which were not included in the training data, with models at different points during training.

• English: Tomorrow we're going to the seaside, and the day after we'll visit my grandma in Split.
• Japanese: 明日は海に行って、あさってはおばあちゃんの家に遊びに行く予定です。
• Croatian: Sutra idemo na more, a prekosutra ćemo posjetiti baku u Splitu.

From left to right: the 10-hour version, 5-hour version, and 3-hour version.

Step	10-hour version	5-hour version	3-hour version
1,000	English Japanese Croatian	English Japanese Croatian	English Japanese Croatian
2,000	English Japanese Croatian	English Japanese Croatian	English Japanese Croatian
3,000	English Japanese Croatian	English Japanese Croatian	English Japanese Croatian
4,000	English Japanese Croatian	English Japanese Croatian	English Japanese Croatian
5,000	English Japanese Croatian	English Japanese Croatian	English Japanese Croatian
10,000	English Japanese Croatian	English Japanese Croatian	English Japanese Croatian
20,000	English Japanese Croatian	English Japanese Croatian	English Japanese Croatian
30,000	English Japanese Croatian	English Japanese Croatian	English Japanese Croatian
40,000	English Japanese Croatian	English Japanese Croatian	English Japanese Croatian
50,000	English Japanese Croatian	English Japanese Croatian	English Japanese Croatian

By ear, all 1,000-step models were still unclear. But in English, you could already mostly tell what was being said. Japanese varied by version: at 1,000 steps, the 5-hour version transcribed this sentence perfectly, word for word. Meanwhile, the 10-hour version produced “Please try taking care of it,” while the 3-hour version produced “With tomorrow’s medic...”—entirely different sentences.

At 3,000–5,000 steps, all models could read both Japanese and English as written in the script. Croatian lagged slightly behind: the first word, “Sutra” (“tomorrow”), continued to be heard as “Sve,” “Straj,” “Satra,” and so on until around 5,000–10,000 steps. After 20,000 steps, it was almost impossible to hear any difference among the three.

By the Numbers

I transcribed evaluation sentences (14 Croatian, 9 Japanese, and 9 English sentences, all outside the training data) with Whisper large-v3 and measured their character error rate (CER) against the scripts. For Japanese, I converted the text to katakana readings before comparing it, so spelling variants in kanji—such as “あさって” and “明後日”—would not count as errors.

First, here is the average across all 32 sentences in the three languages. Numbers in parentheses indicate the number of exact matches.

Steps	10-hour version	5-hour version	3-hour version
1,000	43.8% (2)	41.9% (2)	43.8% (1)
2,000	16.6% (7)	17.7% (12)	17.6% (9)
3,000	7.7% (16)	6.2% (16)	9.9% (14)
4,000	5.9% (21)	3.7% (19)	2.7% (25)
5,000	2.1% (23)	3.3% (20)	4.5% (23)
10,000	1.4% (24)	1.1% (23)	1.7% (24)
15,000	0.8% (26)	0.7% (26)	2.0% (24)
20,000	0.7% (27)	0.8% (26)	0.9% (25)
30,000	0.7% (28)	0.8% (26)	1.7% (26)
40,000	1.2% (25)	1.7% (26)	0.7% (26)
45,000	3.9% (25)	0.7% (28)	0.7% (27)
50,000	0.7% (28)	0.7% (26)	0.7% (28)
60,000	1.8% (26)	0.8% (27)	1.9% (25)

Broken down by language, the results are as follows (Croatian / Japanese / English).

Steps	10-hour version	5-hour version	3-hour version
5,000	2.8 / 1.2 / 1.8%	6.5 / 1.2 / 0.3%	2.0 / 1.8 / 11.1%
10,000	2.2 / 1.2 / 0.3%	1.7 / 1.2 / 0.3%	2.9 / 1.2 / 0.3%
20,000	0.8 / 1.2 / 0.0%	0.9 / 1.2 / 0.0%	0.8 / 1.8 / 0.3%
30,000	0.7 / 1.2 / 0.0%	1.1 / 1.2 / 0.0%	3.2 / 1.2 / 0.0%
50,000	0.7 / 1.2 / 0.0%	0.9 / 1.2 / 0.0%	0.7 / 1.2 / 0.0%

• The best results were the 10-hour version at 30,000 and 50,000 steps, the 5-hour version at 45,000 steps, and the 3-hour version at 50,000 steps. All reached an overall 0.7%, with 28 of 32 sentences matching exactly—all three arrived at the same point
• The points where the numbers jump are almost always differences of a single word. For example, “Bok!” (“Hi!”) was transcribed as “Bog,” and “Hi!” as “Bye.” With short sentences of one or two words, a difference of just one character can make the error rate jump dramatically
• In the 3-hour version, Croatian was in the 3% range from 15,000 to 35,000 steps. Looking at the details, almost all of that came from one sentence where “Bok!” was read as “Blok.” It was corrected by 50,000 steps. With less data, it seems to take longer for quirks in short sentences to disappear
• The 1.2% for Japanese comes entirely from one sentence: every time, “しーっ” in “しーっ、静かに。” (“Shh, be quiet.”) was transcribed as “し.” It is an issue of how much of the long vowel to write; when you listen, it clearly says “しーっ”
• Most of the remaining Croatian errors are transcription-side quirks, such as writing “prekosutra” (“the day after tomorrow”) as the two words “preko sutra”

The differences among 10 hours, 5 hours, and 3 hours were within the margin of error. At least for “whether it can read the script as written,” three hours per language was sufficient. However, the 3-hour version took longer to stabilize, so it is better to train it for at least 40,000 steps.

Comparing Sentences Packed with Technical Terms

Even if less data is enough for everyday conversation, perhaps more data would be stronger when dealing with difficult words that rarely appear in the training data. With that in mind, I created ten sentences packed with technical terms from medicine, IT, law, chemistry, finance, and other fields, and had the models read them in all three languages. I used the 50,000-step version of each model.

1. Because a myocardial infarction is suspected, please urgently prepare coronary angiography and thrombolytic therapy.
   • We suspect a myocardial infarction, so please prepare coronary angiography and thrombolytic therapy immediately.
   • Sumnjamo na infarkt miokarda, zato odmah pripremite koronarnu angiografiju i trombolitičku terapiju.
2. This algorithm has an architecture that combines a convolutional neural network with a recurrent neural network.
   • This algorithm combines a convolutional neural network with a recurrent architecture.
   • Ovaj algoritam kombinira konvolucijsku neuronsku mrežu s rekurentnom arhitekturom.
3. The defendant’s attorney argued for acquittal on appeal on the grounds that the evidence was obtained unlawfully.
   • The defendant's attorney argued for acquittal on appeal, citing the unlawful acquisition of evidence.
   • Branitelj optuženika zatražio je oslobađajuću presudu u žalbenom postupku zbog nezakonito pribavljenih dokaza.
4. In photosynthesis, the electron transport chain operates in the thylakoid membrane of chloroplasts, synthesizing adenosine triphosphate.
   • In photosynthesis, the electron transport chain in the thylakoid membrane of the chloroplast produces adenosine triphosphate.
   • U fotosintezi lanac prijenosa elektrona u tilakoidnoj membrani kloroplasta proizvodi adenozin-trifosfat.
5. Quantum key distribution using entanglement can, in principle, detect eavesdropping.
   • Quantum key distribution based on entanglement can, in principle, detect any eavesdropping.
   • Kvantna distribucija ključeva temeljena na spregnutosti u načelu može otkriti svako prisluškivanje.
6. In response to rising inflation, the central bank decided to raise its policy interest rate and implement quantitative tightening.
   • Responding to rising inflation, the central bank decided to raise interest rates and begin quantitative tightening.
   • Zbog rasta inflacije središnja banka odlučila je povisiti kamatne stope i započeti kvantitativno pooštravanje.
7. For patients with knee osteoarthritis, intra-articular hyaluronic acid injections are used in combination with physiotherapy.
   • Patients with knee osteoarthritis receive intra-articular hyaluronic acid injections combined with physiotherapy.
   • Pacijenti s osteoartritisom koljena dobivaju intraartikularne injekcije hijaluronske kiseline uz fizioterapiju.
8. This bridge supports a reinforced concrete deck with prestressed concrete girders.
   • This bridge supports a reinforced concrete deck on prestressed concrete girders.
   • Ovaj most nosi armiranobetonsku ploču na prednapregnutim betonskim nosačima.
9. The cryptocurrency exchange verifies the integrity of its distributed ledger using a consensus algorithm.
   • The cryptocurrency exchange verifies the integrity of its distributed ledger with a consensus algorithm.
   • Mjenjačnica kriptovaluta provjerava cjelovitost distribuirane glavne knjige algoritmom konsenzusa.
10. Overuse of antibiotics leads to the emergence of multidrug-resistant bacteria and increases the risk of hospital-acquired infections.

• Excessive use of antibiotics leads to multidrug-resistant bacteria and increases the risk of nosocomial infections.
• Prekomjerna uporaba antibiotika dovodi do multirezistentnih bakterija i povećava rizik od bolničkih infekcija.

Sentence	10-hour version	5-hour version	3-hour version
1. There is a suspected myocardial infarction, so please urgently prepare for coronary angiography and thrombolytic therapy.	English Japanese Croatian	English Japanese Croatian	English Japanese Croatian
2. This algorithm combines convolutional neural networks and recurrent networks.	English Japanese Croatian	English Japanese Croatian	English Japanese Croatian
3. The defendant’s attorney argued for acquittal on appeal on the grounds that the evidence had been illegally obtained.	English Japanese Croatian	English Japanese Croatian	English Japanese Croatian
4. In photosynthesis, the electron transport chain operates in the thylakoid membranes of chloroplasts, synthesizing adenosine triphosphate.	English Japanese Croatian	English Japanese Croatian	English Japanese Croatian
5. Quantum-entanglement-based cryptographic key distribution can detect eavesdropping in principle.	English Japanese Croatian	English Japanese Croatian	English Japanese Croatian
6. In response to rising inflation, the central bank decided to raise its policy rate and implement quantitative tightening.	English Japanese Croatian	English Japanese Croatian	English Japanese Croatian
7. For patients with knee osteoarthritis, we use a combination of intra-articular hyaluronic acid injections and physical therapy.	English Japanese Croatian	English Japanese Croatian	English Japanese Croatian
8. This bridge supports a reinforced-concrete deck slab with prestressed-concrete girders.	English Japanese Croatian	English Japanese Croatian	English Japanese Croatian
9. Cryptocurrency exchanges verify the integrity of distributed ledgers using consensus algorithms.	English Japanese Croatian	English Japanese Croatian	English Japanese Croatian
10. The overuse of antibiotics promotes the emergence of multidrug-resistant bacteria and increases the risk of hospital-acquired infections.	English Japanese Croatian	English Japanese Croatian	English Japanese Croatian

Results.

	Croatian	Japanese	English	Exact matches out of 30 sentences
10-hour version	0.6%	1.5%	4.6%	17
5-hour version	0.3%	1.0%	3.9%	20
3-hour version	0.5%	1.0%	4.1%	18

The results did not show that the 10-hour version was stronger. All three were about the same, and the 5-hour version was actually slightly better. To be safe, I also evaluated the 10-hour and 5-hour versions using the models at 60,000 steps: the 10-hour version scored 1.9% and the 5-hour version 2.0%, reversing the result. In other words, the difference was just random variation, and it is impossible to say that any one is better. Only the 3-hour version used the CPU for synthesis and transcription because the GPU was occupied with other work. Its conditions were therefore slightly different.

What was interesting was the nature of the errors.

• For the English word “cryptocurrency,” all three models were transcribed as sounding like “Kripke-Kunershi,” “Kripke-Kinoshi,” or “Kripik-Innershi”
• “intra-articular,” “multidrug-resistant,” and “angiography” also broke down in the same way across all three models
• The Japanese term “coronary angiography” was transcribed as “〜増え” by all three, and “床版” as “障板” by all three
• For the Croatian word “pooštravanje” (tightening), every model was transcribed as “poštravanje”

Models trained on different amounts of data made the same kinds of mistakes on the same words. For English, the cause was clear: it was not the model but the part that converts text into phonetic symbols.

These models do not read text directly; they read phonetic symbols. In English, g2p_en estimates pronunciations for words that are absent from its dictionary. When I checked, “cryptocurrency” had been converted into phonetic symbols resembling “kuripakyunāshi,” while “angiography” had been converted into phonetic symbols with an n inserted in the middle. The models merely turn the phonetic symbols they receive into speech, so they make mistakes in the same places whether the training data contains 10 hours or 3 hours.

Japanese is slightly different. OpenJTalk produced the correct readings: “床版（ショーバン）” and “冠動脈造影（カンドーミャクゾーエー）.” “床版” becoming “障板” is simply a different kanji with the same reading, so the pronunciation itself is correct. Whether “造影” becoming “増え” was because Whisper mapped an unfamiliar word to a familiar one, or because the model’s pronunciation was indistinct, cannot be distinguished with this evaluation method. Still, all three models made the same mistake here as well.

In other words, with this setup, the handling of English technical terms is determined by the pronunciation dictionary, not the amount of training data—at least in this case. If you want it to handle technical terms better, adding words to the dictionary is faster than increasing the data.

Summary

• A single small model (about 32 million parameters in total) could speak three languages
• With 10, 5, or 3 hours of training data per language, reading accuracy ultimately ended up almost the same. The 3-hour version took a little longer to settle down
• There was no difference even for technical terms. The errors occurred on the pronunciation-dictionary side and cannot be fixed by increasing the amount of data

Creating data takes both time and money, so it seems reasonable to start with three hours per language and add more only if needed. Next, based on these results, I plan to increase the number of languages to ten. However, this test measured only “what it says.” For aspects that cannot be measured by character error rate—such as vocal naturalness and the range of emotion—more data may have an advantage. I want to assess that by listening.