Radiance Field Loss / Many worlds

[ziyi-zhang]

This PR introduces new training modes that promote radiance surface representations.
For research code used in the paper, please check this repo.

This PR should not change any existing behavior unless users explicitly enable the new training modes.

New features

[x] Add RFL and a relaxed variant in JIT training mode.
[x] Support surface rendering in JIT rendering mode.
[x] Non-JIT mode RFL training will not be supported as it is significantly slower.

Performance

• Rendering FPS is significantly improved---about 2× faster. This applies both for RFL training, and when post-processing a converged NeRF scene with RFL for 500 iterations.
• Training speed with default batch size (2^18):
  • With NeRF training: INGP launches ~5k rays per iteration, underutilizing the GPU in JIT mode and causing a ~80% slowdown compared to non-JIT (numbers vary a lot depending on the scene).
  • With RFL training: INGP launches up to 30k rays per iteration. The increased ray count does not lead to a notable increase in iteration time---thanks to the new loop-based megakernel implementation. At this ray count, the non-JIT version becomes ~60% slower.

[ziyi-zhang]

Hi @Tom94,

This is ready in my opinion.
Two quick notes:

• I tried to inject as little change as possible -- it should not change any existing INGP behavior.
• The videos in the SIGGRAPH talk are generated with this code.

[Tom94]

Awesome, thank you very much!

Did you want to add some info about the new modes to the readme and link to your paper? I think it'd be great if users could discover these significant improvements easily.

We should also consider making RflRelax the default -- could you remind me whether you saw any scenes where it hurt the reconstruction at ~equal training time more than it helped?

[ziyi-zhang]

Thanks for the feedback. I have handled all small issues and added more comments.

I have set Rfl-Relax as default. To ensure performance remains at least as good as NeRF, I used a very conservative training strategy:

• First 15k iterations: standard NeRF training
• Next 15k iterations: relaxed mode
• Final 5k iterations: NeRF refinement

MipNeRF360 tests still show ~2x rendering speedup vs NeRF. The conservative approach costs us ~20% performance compared to full RFL, but good enough.

We could definitely make it more robust and squeeze out more performance. But IMO this hybrid reconstruction approach still needs more research and has many unexplored potentials, so probably I shouldn't overdo it for now.

[Tom94]

Thanks again -- it's great to have the radiance field loss be officially part of instant NGP!

I'll be doing some testing in the coming days and then put more info into the readme + probably push a release. :)

[phongnhhn92]

Hi @ziyi-zhang and @Tom94 do you guys have an instructions on how to turn on radiance field loss (RFL) that can matches the settings in the paper ?
It seems like there is a relaxed variant of RFL, I would like to try it out as well but dont know where to start. Also I wonder what is the advantage of RflRelax over Rfl