torch.cuda.OutOfMemoryError: CUDA out of memory

[zhuxiyulu]

Thank you for this work. I'm trying to train the flow model. I tried the following configurations, but an OOM error occurred when the model training was running at some step of epoch 0. The torch.cuda.OutOfMemoryError: CUDA out of memory occurs in decoder phase flow model. I checked that the memory of the gpu is large enough, but the gpu memory keeps increasing with the step.

ex.1 examples/music_generation/conf/inspiremusic.yaml batch_type: 'dynamic' max_frames_in_batch: 10000 or 2000
ex.2 examples/music_generation/conf/inspiremusic.yaml batch_type: 'static' batch_size: 2

Could you please tell me the correct configuration for training the flow model？

[iris2c]

Thank you for this work. I'm trying to train the flow model. I tried the following configurations, but an OOM error occurred when the model training was running at some step of epoch 0. The torch.cuda.OutOfMemoryError: CUDA out of memory occurs in decoder phase flow model. I checked that the memory of the gpu is large enough, but the gpu memory keeps increasing with the step.

ex.1 examples/music_generation/conf/inspiremusic.yaml batch_type: 'dynamic' max_frames_in_batch: 10000 or 2000 ex.2 examples/music_generation/conf/inspiremusic.yaml batch_type: 'static' batch_size: 2

Could you please tell me the correct configuration for training the flow model？

You may try batch_type: 'dynamic' max_frames_in_batch: 8000, for 80GB memory

[zhuxiyulu]

Thank you for this work. I'm trying to train the flow model. I tried the following configurations, but an OOM error occurred when the model training was running at some step of epoch 0. The torch.cuda.OutOfMemoryError: CUDA out of memory occurs in decoder phase flow model. I checked that the memory of the gpu is large enough, but the gpu memory keeps increasing with the step.
ex.1 examples/music_generation/conf/inspiremusic.yaml batch_type: 'dynamic' max_frames_in_batch: 10000 or 2000 ex.2 examples/music_generation/conf/inspiremusic.yaml batch_type: 'static' batch_size: 2
Could you please tell me the correct configuration for training the flow model？

You may try batch_type: 'dynamic' max_frames_in_batch: 8000, for 80GB memory

@iris2c Thanks.
Actually, my gpu memory is 80G. I have tried this configuration batch_type: 'dynamic' max_frames_in_batch: 8000, but there are still OOM errors during training. Because the memory keeps increasing with the step. I can't train the flow model in a complete epoch. Do you have any good solutions to this problem.

[zhuxiyulu]

Thank you for your reply. @iris2c

I have a few questions about training the flow model. Could you help me clarify them.
Q1. Is it necessary to filter the acoustic_token field by min_acoustic_length and max_acoustic_length during flow model training?

Q2. I'm a little curious about the implementation of line 151 of InspireMusic/inspiremusic/dataset/processor.py，
Why not filter the data through the following code：
if new_sample_frames > max_length or new_sample_frames < min_length or target_sample_frames < min_acoustic_length or target_sample_frames > max_acoustic_length:

[iris2c]

Thank you for your reply. @iris2c

I have a few questions about training the flow model. Could you help me clarify them. Q1. Is it necessary to filter the acoustic_token field by min_acoustic_length and max_acoustic_length during flow model training?

Q2. I'm a little curious about the implementation of line 151 of InspireMusic/inspiremusic/dataset/processor.py， Why not filter the data through the following code： if new_sample_frames > max_length or new_sample_frames < min_length or target_sample_frames < min_acoustic_length or target_sample_frames > max_acoustic_length:

Based on our experience, filtering out the audio samples which is too long will help to avoid OOM, because during training we usually use dynamic batch, if some of the sample length is too long, it will easily cause OOM problem.