question about audio VAE

[nzhang258]

Thanks for the excellent open-source work!

I have a question regarding the audio VAE in this project:

1. What is the purpose of the audio VAE?
   From the dataset preparation part, it seems that the audio VAE is not involved in encoding or decoding the audio data. Could you clarify its role in the overall architecture or training pipeline?（https://github.com/FunAudioLLM/ThinkSound/blob/master/data_utils/extract_training_audio.py#L80）

2. Why is the conventional VAE approach (mel-spectrograms → VAE → vocoders) not used?
   It appears that the project does not follow the commonly used pipeline where audio is first converted into mel-spectrograms, then encoded by a VAE, and finally reconstructed by a vocoders. What were the considerations behind this design decision?

I would really appreciate it if you could provide some clarification on this. Thanks again for your great work!

[onestardao]

your questions around the role of the audio VAE (why it seems not to directly encode/decode audio, and why the design diverges from the common mel-spectrogram → VAE → vocoder pipeline) line up with ProblemMap No.4 — symbolic drift.
the architecture is technically valid, but once the VAE is repurposed away from its conventional audio reconstruction role, the semantics drift: it’s no longer clear what “representation” the VAE is anchoring, and follow-up design choices (like bypassing mel-specs) feel inconsistent.

we’ve catalogued 16 such failure modes with reproducible fixes. if you’d like the full list (with matching repair strategies), just let me know — it saves time versus debating whether the VAE is “involved enough” in the pipeline.

[nzhang258]

@onestardao Thanks for your fantastic explanation, and "Symbolic Drift" is a great term for it. I'd love to see the full list of 16 failure modes and fixes you've catalogued.
That would be immensely helpful. Thanks!