feat: Add depth to Frame

[mrousavy]

What

TL;DR: Adds Frame.depth, a nullable CMSampleBuffer that holds depth data.

If enableDepthData is true (a prop passed to <Camera>), the Camera will be configured to set up a depth data output pipeline in addition to the video data output pipeline.
Ontop of that, a synchronizer output is created to sync depth data and video data buffers together for timestamps.

const device = Camera.getAvailableCameraDevices().find((d) => d.formats.some((f) => f.supportsDepthCapture))
const format = useCameraFormat(device, [{ depth: true }])

const fp = useFrameProcessor((frame) => {
  'worklet'
  console.log(frame.depth)
})

if (device == null) throw new Error(`Device does not support depth data!`)
return (
  <Camera
    device={device}
    format={format}
    enableDepthData={true}
    video={true}
    frameProcessor={fp}
  />
)

Changes

Tested on

Related issues

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
react-native-vision-camera	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Feb 14, 2025 3:08pm

[zigapk]

So glad to see this PR, just what I need. Would love to see this merged

[rui-batch]

merge this shit pls.

[mrousavy]

merge this shit pls.

Did you even bother to try this PR yourself to confirm everything works as expected?

[ruisg]

merge this shit pls.

Did you even bother to try this PR yourself to confirm everything works as expected?

I sincerely apologize for my previous comment. The frustration took over my keyboard.

Yes, I did test it locally and confirmed it was not working, I successfully fixed the dept locally as I needed it urgently. I’m more than happy to contribute so this can get merged.

[mrousavy]

This PR is not yet working as expected because it will stall the video pipeline. So it'll not be merged yet, until that's fixed. But I got lots of other stuff to do rn

[rologr35]

@mrousavy Hi. This is an amazing feature. Would be great if it also allows support for android. Is it possible to integrate ARCore with Raw Depth API or Depth API? Thank you.

[mrousavy]

@rologr35 CameraX doesn't support depth data streaming.
And any code I'd write in manual Camera2 bridging code would be a workaround, I don't want to release that stuff to production.

[WillTisdale]

@mrousavy
First, thank you for working on this depth data feature! I'm developing an app that requires LiDAR depth data for 3D measurements. While I wait for this PR to be merged, would it also be possible to expose depth data in the PhotoFile interface when taking photos with enableDepthData=true? This would be incredibly valuable for capturing high-quality depth maps without processing every frame.

[oliveryasuna]

@mrousavy We are very excited for this feature! I'm just curious if there's a timeline on its release?

[mrousavy]

No timeline.

[scm007]

What is remaining for this to work?

[yoann84]

Hi ! What's missing for that PR to be merged ?