Consistent world coordinates when training with different subsets of frustums

[NicolasAraya932]

Summary

I am training a model on a dataset with camera frustums around an object, specifically the indoor bonsai n24 MipNeRF360 sparse dataset (see: https://nerfbaselines.github.io/mipnerf360-sparse).

When I train on the full set of frustums and then train a second model on a subset of those frustums (e.g., I delete roughly half of them), the world coordinate system changes between runs.

Because of this, I cannot directly match:

• Radiance values
• Intermediate features / latent codes

between the "full frustums" model and the "subset frustums" model, even though they are supposed to represent the same physical scene.

Screenshots illustrating the misalignment (same point set, different runs):

Full frustums model

Subset frustums model

---

Setup

• Repo / framework: nerfstudio (using the COLMAP dataparsers)
• Dataset: MipNeRF360 sparse indoor bonsai n24
• Task: Object-centric NeRF / radiance field, cameras around the bonsai

Experiments:

1. Experiment A (Full)

   • Use the full bonsai n24 camera set / frustums.
   • Train a NeRF model.

2. Experiment B (Subset)

   • Start from the same dataset but delete ~50% of the frustums (subset of the original cameras).
   • Train another NeRF model.

Then I:

• Query radiance / intermediate features from both models at what should be the “same” 3D locations, or
• Visualize samples / points in world space.

The resulting coordinate frames are different (visually shifted / recentered).

---

Observed behavior

• Training on the full frustum set vs. a subset of frustums produces different world coordinate systems.
• The scene appears to be:
  • Recentered, and/or
  • Translated / rotated

Based on my visualizations, I do not observe any rescaling of the scene (and it's confirmed in the last topic).
The issue seems to be purely a change in origin and/or orientation, not a change in scale.

As a result:

• The same physical point on the bonsai has different coordinates in the two models.
• I cannot directly align radiance values or latent features from the full-frustums model and the subset-frustums model.

---

What I tried

I was looking into the code to find where exactly the scene center is defined / fixed.

I found the function in nerfstudio/cameras/camera_utils.py:

def auto_orient_and_center_poses(
    poses: Float[Tensor, "*num_poses 4 4"],
    method: Literal["pca", "up", "vertical", "none"] = "up",
    center_method: Literal["poses", "focus", "none"] = "poses",
) -> Tuple[Float[Tensor, "*num_poses 3 4"], Float[Tensor, "3 4"]]:

---

Verification that the scenes are the same (up to a rigid transform)

To check whether the two trainings (full frustums vs subset of frustums) were actually learning the same geometry or something different, I did the following:

• Used a semantic variation of NeRF to obtain features / point samples from both models.
• Aligned the two 3D point sets using:
  • PCA (to roughly align principal axes),
  • RANSAC (to robustly estimate a candidate rigid transform), and
  • ICP (Iterative Closest Point) to refine the alignment.

After this pipeline, the two scenes match well under a rigid SE(3) transform (rotation + translation, no scaling).

So:

• The underlying reconstructed scene is essentially the same.
• The difference between the full-frustum and subset-frustum runs is a global rigid transform
• What I would like is for this transform to be the identity (i.e., the same canonical world frame), so I don’t need to do PCA + RANSAC + ICP every time just to compare radiance / features across runs.

Pose correction pipeline

Although this pose-correction pipeline works as a workaround, it adds extra complexity.
What I really need is a simpler solution: a way to keep the original dataset frustum positions / world frame fixed, so that training on a subset of cameras does not change the global pose convention in the first place.