Can the copy of large files be done in parallel across ranks?

[jprorama]

We have been using dsync to migrate data from an old file system to new file system (GPFS->CephFS). The data migration works well and is meeting most of our needs.

In our data sets we have many small files (>95% of file count) and a smaller number of large files (>95% storage used). The dsync operations generally work well and perform the tree walks efficiently. The data copy performance is ultimately limited by the file sizes.

Something we have noticed, however, is that we can get into a very long tail during our batches. The throughput starts strong but then trails off to a trickle for a long time. The next batch then picks up again where the throughput starts good but then trails off again with a long tail.

We suspect this is due a a large file existing in the batch and that the rank processing the file hasn't finished, leaving the other ranks idle awaiting the next batch. It seems that the file list can be shared across ranks but that a file action (copy) is only carried out by a single rank.

Is our intuition correct?

If so, is there a way to improve the copy portion of the dsync? One solution could be to copy file data in parallel, assigning portions of a data transfer to idle ranks. This would enable all ranks to contribute to the completion of the transfer of the large file in that batch. Another option might be to allow the idle ranks to start on the next batch, avoiding a stall due to lack of work.

We'd be interested in your feedback on this assessment and suggestions for improvement.

Thanks.

[jbd]

Hello,

I'm not an mpifileutils developer but I share your intuition.

Having idle ranks is not ideal, and it would be great if they could pickup work for the next batch. I've got a feeling that it would not be trivial to implement. Same for the single rank copy. I'm using -mca mpi_yield_when_idle on the mpirun command line to make those "idle" rank cpu ressources available, but that's all.

I'm proposing the introduction of a -V/--batch-volume that will limit the amount of data per batch. It can be used in combination of --batch-files options.

I'm in the process of evaluating #655 on real data to see if it could useful or not.

[jbd]

After some thoughts, I understand that this PR is not useful and won't help at all. I think you already identified the issue here: in an ideal world, we should not have idle ranks.

From my understanding, no idle ranks means either:

• ranks can pick up work in the next batches
  or
• split large file copy across ranks

[jbd]

When using dsync, I can see all MPI processes issuing 4MB (default chunksize) pwrite64 syscall, so my guess would be that large file copy are correctly splitted across ranks.

It's not yet clear to me why I can have idle ranks on the tail of batches.