Anomaly in my trained models

[kirys]

Hi all I'm trying to create checkpoints for the Italian language using MLS as a dataset.
I've noticed a strange anomaly, my models seems unable to say short sentences, while they works well with long ones (longer than 5 or 6 words).

The models are here -> https://huggingface.co/kirys79/piper_italiano

Do you have an hint on the possible cause, and how to fix it?
Thank you
K.

[periloso]

@kirys, I, too, am training a set of Italian voices and, having stumbled across your models, I can tell you that I don't have a single suggestion as to the possible actual cause of the problem you are noticing: I think there are many reasons for it.
I'll write here what I noticed so that, should other developers stumble upon your models (just like the ones provided by rhasspy), they will know how to fix them.

1. Your JSON configuration files for your models are incorrect: the model quality is set to the value "High"; however, the capital "H" in "High" may have led you to train the models with the medium quality (as I'll briefly explain in the next point);
2. I noticed that the file size of your models "seemed a bit weird": even though your model files stated "High" quality, the file size - both for the onnx files, as well as the checkpoints - had the same file sizes of any "medium" model: leonardo-epoch=2024-step=996300.onnx is 63.5MB, and leonardo-epoch=2024-step=996300.ckpt is 846MB. If you look at the lessac checkpoint epoch=2218-step=838782.ckpt, you will notice that the file size is 998MB and, by looking at my own fine-tuned onnx models, I can definitely say that the onnx files should have the file size of 109MB (this also agrees to what happens with the lessac model, btw). This means that your models have actually been trained with the "medium" model.

These issues alone, however, have probably only degraded the quality of your models from high to medium, but I don't think they're the actual cause why your models are currently unable to generate short sentences. So, while making my own tests, I also analyzed the dataset you tried to use, and I noticed what follows, but -- First things first: it is not your fault! (😅)

1. If you take a look at the MLS dataset that you're using, you will notice that the whole dataset does not have any punctuation mark. There is hardly any comma (,), colon (:), semicolon (;), as well as any exclamation mark (!) and question mark (?). Moreover, if you hear the audio transcription of any audio, you will easily notice that the transcript does NOT match the actual audio. This is a huge issue, that causes any model that is trained with that "stuff" to misbehave.
2. Each .wav file is meant to contain only one single sentence. If you look at the transcription provided inside the dataset, you will notice that many files have multiple sentences in them. This is exactly what causes the major problems that you noticed in your training: the model learns to play some "gibberish stuff" before the actual sentence.
3. The dataset lacks short, medium, and long sentences: during the training phase, you need to make the model know that sentences can be short, medium, or long — and you should also show that, some of them may be complex and have semicolons, colons, hyphens (—)... and, possibly, ellipsis (... or …). From what I saw, while espeak-ng DOES support ellipsis, the piper phonemizer DOES NOT, and simply ignores "that symbol", leading to some more misbehaviors. I've not found a solution to this last issue yet.

TL;DR
There are several issues that are causing your models (that are being trained with the italian MLS dataset) to misbehave, but they can all be "easily" fixed: you just have to perform the transcription by yourself, and shouldn't use the transcription provided by the dataset authors — since what they provided with the dataset is completely wrong.

I'm sure that, if you get past this point, you will probably also notice some other problems with the piper's phonemizer for the italian language; this, however, would be a different topic, and it would probably need to be addressed in a different issue.

PS: If you're still interested in training and/or finetuning piper TTS models, feel free to let me know 😊

[kirys]

Thank you! those are some great insight about the issue. I was suspecting i had to "rework" the dataset, I'm still interested in doing some Italian voices, but editing the dataset by hand is quite a burden and I had no free time for doing it right now. I'll try to do something this summer (august).
Thank You again

[periloso]

@kirys
My suspicion that there might have been an issue with the dataset arose for a very simple reason: after searching online, anyone who was using the Italian datasets — both foreign and Italian users, and both the MLS and the LibriVox datasets — noticed that all the generated models did not follow punctuation properly: they all showed the "emphatic pauses", very often at the beginning of the sentences.
Since emphatic pauses are typically caused by the model that is learning these pauses within the prosody (because the pauses weren't indicated with punctuation), I took a look at the dataset transcription and realized that yes, the dataset is fine and is definitely a great start but, before going ahead with the training, the punctuation definitely needs to be added to the transcription.

I did a quick proof of concept for one of my clients (with a voice that my client intends to use - that is not free to use) and, after fine-tuning the voice with just 15 minutes of audio that I manually labeled correctly, I can definitely confirm that the training process works perfectly; this is an example link, with your comment just to let you hear it 😉.

Thank you so much for your testing and for your initial models: it is thanks to you that I understood how to proceed, and what I needed to check for this problem! 😉😊

[kirys]

finetune from what model?

[periloso]

It’s nothing more than an extremely basic finetuning of the lessac high model; the audio dataset I used was a 22.05khz sample I got from my client (30 minutes total; 25 minutes used).
To finetune the model, I stabilized the generator and the discriminator very quickly by using a larger batch size; then, when the losses started to get lower, I decreased the batch size to 1/3, resumed from the best checkpoint, and I simply kept checking whether the output kept sounding 'right'.
I initially chose to use a larger batch size to avoid making the model too unstable during the early stages of training, as this could have risked preventing proper convergence. Once I was confident that the model was converging correctly, I reduced the batch size, and continued the finetuning process.

To clarify, when I said 'extremely basic finetuning', I meant that the audio transcription is still definitely a draft, and there's still much to be done there. However, since I was asked to provide a very quick and cost-effective proof-of-concept, I wanted to give a brief idea of what we could achieve with Piper without even getting into the transcription refining process – so we would be able to compare the results with different solutions.

However, based on some brief tests I conducted earlier, I firmly believe that without further refining our transcription, the medium model would still have produced the same – or very close – results.
The training process took about ~12 hours on a 3090; however, I also had some long sentences (~28s) in the dataset, and this has certainly slowed down the whole process.

[kirys]

Thanks for your hits I hope to be able to soon making it works

[rmcpantoja]

Hi.
In summary: mls is not a good dataset for speech synthesis in none language. Reasons was already explained.
There is high quality material from a single speaker through LibriVox in Italian (see Divina Commedia for example). I have been working in a TTS dataset based on this book, but I have it paused for now.
Cheers.

[kirys]

I'll take a look at librivox too, but I don't think the Divina Commedia could be a good text due to the old Italian forms and the bad habit of "dantisti" to read it in a way too emphatic tone.

[periloso]

Hi, @rmcpantoja, and thank you for your time and patience! 😉
I did take a look at both the datasets (the LibriVox, as well as the MLS - which also uses part of the LibriVox) and, just like any other dataset, I believe that they could both be fine - but "it depends on what you're aiming for".
If you want to use your model for a PBX system or a realtime chat, I'm pretty sure that none of those datasets will suit you well, because those books must be played, and not read conversationally; therefore, the dataset won't include the output you're aiming for.
If instead, like you rightfully pointed out, you're aiming for a more classical tone, I believe that the Divina Commedia (and many more books as well) will surely be the best option you have.

I simply wanted to tell @kirys, and any other italian developer that is working on fine-tuning his own model, the reason why we don't have a working model yet: we just need to transcribe carefully the audio by hand, and we should not rely on the text provided within the datasets themselves (or, at least, not before checking them out!).
Thank you as well for your insights! 😊

[kirys]

Thanks

[TikoTako]

Its weeks that i'm working with LibriVox audiobooks as base for a an italian dataset and let me tell you, you can get an "ok" voice but not a production-ready one.
The main problem is that you don't have any way to fix these:

• sentences duration distribution
• less used phonemes distribution
• breathing (this is a massive problem with auto-segmentation)
• inconsistency of speech (like when some reader "imitate" the speech of the characters)
• inconsistency of transcription, unless you have tens of ppl you cant check if the transcriptions are correct, even if you use some SST.

This is an example of the audio segmetns distribution, most sentences are in the 2-3 sec range.
In this case the TTS model learn to generate short sentences and basically become crappier with longer ones, also the more epochs you run the higher is the risk to overfit the right side because the samples are way too low.

This is an example of breathing, i tried with silero vad, ten-vad, ffmpeg, all of them fail to split the sentences because the breathing is too strong.

[periloso]

The Riccardo voice is (arguably?) one of the worst voices of the dataset and, if I had to choose, I'd certainly choose to stay away from it. With that said, I'm using a different approach for my client's voice data.

I am not splitting the files into many thousands files using whisper or silero and, instead, I'm using audacity. From the whole wav file, I am creating the labels whenever there's more than 1 second of silence, and then I save the label file somewhere. It's also a good idea to check when and where these markers end; breathing is not necessarily a problem, since it leads to a more natural output - but only if they appear near a punctuation mark (humans do breathe!).

Then, with the wav file and the label file, I've written a script that extracts each one of these small segments, and I am letting whisperx transcribe it.
The resulting transcription, associated with the start and end timestamp of each segment, is then saved to a new label file, allowing me to check within audacity whether the label reflects what's being spoken — and, usually, the answer is "no": all whisper models (I'm using large-v2) do not handle semicolons, colons, and they very often skip commas, so you have to either add them manually, or accept that the model will add random "emphatic pauses" due to your incorrect transcription.
To help me with the transcription, I'm also using a spellchecker (using hunspell from python) to include notes about words that have been either written or spoken incorrectly. If I have doubts, or that sentence is too arguable, I simply discard it.

Then, using a draft model that uses the same voice (trained with just a few epochs) , I check whether espeak-ng is writing the correct phonemes or not — and, again, the answer here is very often "NO". When there's a mistake, I add an exception to the it_extra file, check if the draft voice sounds like the voice I'll be training and move on.

By doing it like this, YES, you will certainly need some time, but you'll be able to create a voice without a huge corpus (my client's budget is limited, after all).

How are you approaching to your transcription/segmentation, instead? Would you like some help with the process?

[TikoTako]

i'm using the raw audiobooks with single speaker, so is 5 speakers with hundreds files.
I did something similar but with ffmpeg, find silences, invert list, whisper on that.
i tried v3-large and v3-large-trurbo, the difference in speed is huge but the quality id say is the same because both fail on some words.
the breathing, yes is ok for a pause but only if isnt audible, if you put a comma on the breathing the model will learn that.

This is another voice with other audiobooks, the raw distribution looks better but it have alot less segments

I also made this, to see the distribution based on the TTS target type

Sadly i had to put all on hold because i don't have an ac and is too hot to do the training atm (almost 35°c in the room)

[periloso]

Well, I'll admit that I tried with ffmpeg (I was tempted by chatgpt to do that) but, as usual, it didn't go very well: chatgpt is more about creativity than actual productivity. however... at least I tried 😂
My client asked me for another example, so I'm training a new model (this one was also started with lessac to speed up the process since it's still a PoC).

After completing the high-quality model, I still wasn't sure whether there was a real difference between the medium and high versions for our specific use case: so, this time I'm targeting the medium model, so I can also speed up the process in case the client requires more samples (let's hope he does not).

Since I already have the pipeline set up, if you want and once I'm done with the client, I can take a look at the dataset and transcription you're using: we could try to continue together (my workstation will be available while I wait for the client response).
If you're willing to share the transcriptions and the URLs of the datasets you're using (either on GitHub or via Gist), as soon as I finish with the client, I'd be happy to take a look at them 😊

[nshmyrev]

You can get much better dataset with synthetic speech from 11labs than with audiobook