Skip to content

Commit edc6b80

Browse files
hemalshahNVhemalshahNV
authored
Merge pull request #27 from NVIDIA-ISAAC-ROS/release-3.0
Isaac ROS 3.0.0
2 parents 466d062 + 3696e32 commit edc6b80

File tree

81 files changed

+7391
-309
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

81 files changed

+7391
-309
lines changed

‎README.md‎

Lines changed: 29 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -1,26 +1,30 @@
11
# Isaac ROS Image Segmentation
22

3-
Hardware-accelerated, deep learned semantic image segmentation
3+
NVIDIA-accelerated, deep learned semantic image segmentation
44

55
<div align="center"><img alt="sample input to image segmentation" src="https://media.githubusercontent.com/media/NVIDIA-ISAAC-ROS/.github/main/resources/isaac_ros_docs/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_image_segmentation_example.png/" width="320px"/>
66
<img alt="sample output from image segmentation" src="https://media.githubusercontent.com/media/NVIDIA-ISAAC-ROS/.github/main/resources/isaac_ros_docs/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_image_segmentation_example_seg.png/" width="320px"/></div>
77

88
## Overview
99

10-
[Isaac ROS Image Segmentation](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_image_segmentation) contains a ROS 2 package to produce semantic image segmentation.
11-
`isaac_ros_unet` provides a method for classification of an input image at the pixel level.
10+
[Isaac ROS Image Segmentation](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_image_segmentation) contains ROS packages for semantic image segmentation.
11+
12+
These packages provide methods for classification of an input image
13+
at the pixel level by running GPU-accelerated inference on a DNN model.
1214
Each pixel of the input image is predicted to belong to a set of defined classes.
13-
Classification is performed with GPU acceleration running DNN inference on a U-NET architecture model.
1415
The output prediction can be used by perception functions to understand where each
1516
class is spatially in a 2D image or fuse with a corresponding depth location in a 3D scene.
1617

1718
<div align="center"><a class="reference internal image-reference" href="https://media.githubusercontent.com/media/NVIDIA-ISAAC-ROS/.github/main/resources/isaac_ros_docs/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_image_segmentation_nodegraph.png/"><img alt="image" src="https://media.githubusercontent.com/media/NVIDIA-ISAAC-ROS/.github/main/resources/isaac_ros_docs/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_image_segmentation_nodegraph.png/" width="500px"/></a></div>
1819

19-
A trained model based on
20-
the [U-NET](https://en.wikipedia.org/wiki/U-Net) architecture is
21-
required to produce a segmentation mask. Input images may need to be
22-
cropped and resized to maintain the aspect ratio and match the input
23-
resolution of the U-NET DNN; image resolution may be reduced to improve
20+
| Package | Model Architecture | Description |
21+
|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------|--------------------------------------------------------------------------|
22+
| [Isaac ROS U-NET](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_unet/index.html#quickstart) | [U-NET](https://en.wikipedia.org/wiki/U-Net) | Convolutional network popular for biomedical imaging segmentation models |
23+
| [Isaac ROS Segformer](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_segformer/index.html#quickstart) | [Segformer](https://arxiv.org/abs/2105.15203) | Transformer-based network that works well for objects of varying scale |
24+
| [Isaac ROS Segment Anything](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_segment_anything/index.html#quickstart) | [Segment Anything](https://segment-anything.com/) | Segments any object in an image when given a prompt as to which one |
25+
26+
Input images may need to be cropped and resized to maintain the aspect ratio and match the input
27+
resolution expected by the DNN model; image resolution may be reduced to improve
2428
DNN inference performance, which typically scales directly with the
2529
number of pixels in the image.
2630

@@ -40,9 +44,11 @@ This package is powered by [NVIDIA Isaac Transport for ROS (NITROS)](https://dev
4044

4145
## Performance
4246

43-
| Sample Graph<br/><br/> | Input Size<br/><br/> | AGX Orin<br/><br/> | Orin NX<br/><br/> | Orin Nano 8GB<br/><br/> | x86_64 w/ RTX 4060 Ti<br/><br/> |
44-
|-------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|
45-
| [TensorRT Graph](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/scripts/isaac_ros_unet_graph.py)<br/><br/><br/>PeopleSemSegNet<br/><br/> | 544p<br/><br/><br/><br/> | [421 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_unet_graph-agx_orin.json)<br/><br/><br/>8.1 ms<br/><br/> | [238 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_unet_graph-orin_nx.json)<br/><br/><br/>9.6 ms<br/><br/> | [162 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_unet_graph-orin_nano.json)<br/><br/><br/>13 ms<br/><br/> | [704 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_unet_graph-nuc_4060ti.json)<br/><br/><br/>5.5 ms<br/><br/> |
47+
| Sample Graph<br/><br/> | Input Size<br/><br/> | AGX Orin<br/><br/> | Orin NX<br/><br/> | Orin Nano 8GB<br/><br/> | x86_64 w/ RTX 4060 Ti<br/><br/> | x86_64 w/ RTX 4090<br/><br/> |
48+
|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------|
49+
| [SAM Image Segmentation Graph](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/benchmarks/isaac_ros_segment_anything_benchmark/scripts/isaac_ros_segment_anything_graph.py)<br/><br/><br/>Full SAM<br/><br/> | 720p<br/><br/><br/><br/> | [2.22 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_sam_graph-agx_orin.json)<br/><br/><br/>470 ms @ 30Hz<br/><br/> | –<br/><br/><br/><br/> | –<br/><br/><br/><br/> | –<br/><br/><br/><br/> | [14.6 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_sam_graph-x86_4090.json)<br/><br/><br/>79 ms @ 30Hz<br/><br/> |
50+
| [SAM Image Segmentation Graph](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/benchmarks/isaac_ros_segment_anything_benchmark/scripts/isaac_ros_mobile_segment_anything_graph.py)<br/><br/><br/>Mobile SAM<br/><br/> | 720p<br/><br/><br/><br/> | [10.8 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_mobile_sam_graph-agx_orin.json)<br/><br/><br/>880 ms @ 30Hz<br/><br/> | [5.13 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_mobile_sam_graph-orin_nx.json)<br/><br/><br/>1500 ms @ 30Hz<br/><br/> | [2.22 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_mobile_sam_graph-orin_nano.json)<br/><br/><br/>360 ms @ 30Hz<br/><br/> | [27.0 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_mobile_sam_graph-nuc_4060ti.json)<br/><br/><br/>62 ms @ 30Hz<br/><br/> | [60.3 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_mobile_sam_graph-x86_4090.json)<br/><br/><br/>27 ms @ 30Hz<br/><br/> |
51+
| [TensorRT Graph](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/benchmarks/isaac_ros_unet_benchmark/scripts/isaac_ros_unet_graph.py)<br/><br/><br/>PeopleSemSegNet<br/><br/> | 544p<br/><br/><br/><br/> | [371 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_unet_graph-agx_orin.json)<br/><br/><br/>19 ms @ 30Hz<br/><br/> | [250 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_unet_graph-orin_nx.json)<br/><br/><br/>20 ms @ 30Hz<br/><br/> | [163 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_unet_graph-orin_nano.json)<br/><br/><br/>23 ms @ 30Hz<br/><br/> | [670 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_unet_graph-nuc_4060ti.json)<br/><br/><br/>11 ms @ 30Hz<br/><br/> | [688 fps](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_benchmark/blob/main/results/isaac_ros_unet_graph-x86_4090.json)<br/><br/><br/>9.3 ms @ 30Hz<br/><br/> |
4652

4753
---
4854

@@ -54,6 +60,16 @@ Please visit the [Isaac ROS Documentation](https://nvidia-isaac-ros.github.io/re
5460

5561
## Packages
5662

63+
* [`isaac_ros_segformer`](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_segformer/index.html)
64+
* [Quickstart](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_segformer/index.html#quickstart)
65+
* [Try More Examples](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_segformer/index.html#try-more-examples)
66+
* [Troubleshooting](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_segformer/index.html#troubleshooting)
67+
* [API](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_segformer/index.html#api)
68+
* [`isaac_ros_segment_anything`](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_segment_anything/index.html)
69+
* [Quickstart](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_segment_anything/index.html#quickstart)
70+
* [Try More Examples](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_segment_anything/index.html#try-more-examples)
71+
* [Troubleshooting](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_segment_anything/index.html#troubleshooting)
72+
* [API](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_segment_anything/index.html#api)
5773
* [`isaac_ros_unet`](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_unet/index.html)
5874
* [Quickstart](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_unet/index.html#quickstart)
5975
* [Try More Examples](https://nvidia-isaac-ros.github.io/repositories_and_packages/isaac_ros_image_segmentation/isaac_ros_unet/index.html#try-more-examples)
@@ -62,4 +78,4 @@ Please visit the [Isaac ROS Documentation](https://nvidia-isaac-ros.github.io/re
6278

6379
## Latest
6480

65-
Update 2023-10-18: Updated for Isaac ROS 2.0.0.
81+
Update 2024-05-30: Add SegFormer and Segment Anything packages

‎isaac_ros_gxf_extensions/gxf_isaac_ros_segment_anything/CMakeLists.txt‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
# SPDX-FileCopyrightText: NVIDIA CORPORATION & AFFILIATES
2+
# Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
3+
#
4+
# Licensed under the Apache License, Version 2.0 (the "License");
5+
# you may not use this file except in compliance with the License.
6+
# You may obtain a copy of the License at
7+
#
8+
# http://www.apache.org/licenses/LICENSE-2.0
9+
#
10+
# Unless required by applicable law or agreed to in writing, software
11+
# distributed under the License is distributed on an "AS IS" BASIS,
12+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
# See the License for the specific language governing permissions and
14+
# limitations under the License.
15+
#
16+
# SPDX-License-Identifier: Apache-2.0
17+
cmake_minimum_required(VERSION 3.22.1)
18+
project(gxf_isaac_ros_segment_anything LANGUAGES C CXX)
19+
20+
if(CMAKE_COMPILER_IS_GNUCXX OR CMAKE_CXX_COMPILER_ID MATCHES "Clang")
21+
add_compile_options(-fPIC -w)
22+
endif()
23+
24+
# Dependencies
25+
find_package(ament_cmake_auto REQUIRED)
26+
ament_auto_find_build_dependencies()
27+
find_package(CUDAToolkit)
28+
find_package(yaml-cpp)
29+
find_package(isaac_ros_nitros_detection2_d_array_type REQUIRED)
30+
enable_language(CUDA)
31+
# Create extension
32+
ament_auto_add_library(${PROJECT_NAME} SHARED
33+
gxf/segment_anything/segment_anything_ext.cpp
34+
gxf/segment_anything/segment_anything_postprocessor.cpp
35+
gxf/segment_anything/segment_anything_postprocessor.hpp
36+
gxf/segment_anything/segment_anything_postprocessor.cu.cpp
37+
gxf/segment_anything/segment_anything_postprocessor.cu.hpp
38+
gxf/segment_anything/segment_anything_prompt_processor.cpp
39+
gxf/segment_anything/segment_anything_prompt_processor.hpp
40+
gxf/segment_anything/segment_anything_msg_compositor.cpp
41+
gxf/segment_anything/segment_anything_msg_compositor.hpp
42+
)
43+
44+
# Mark as CUDA files with non-standard extensions
45+
set_source_files_properties(
46+
gxf/segment_anything/segment_anything_postprocessor.cu.cpp
47+
gxf/segment_anything/segment_anything_postprocessor.cu.hpp
48+
PROPERTIES LANGUAGE CUDA
49+
)
50+
target_link_libraries(${PROJECT_NAME}
51+
CUDA::cudart
52+
yaml-cpp
53+
)
54+
target_include_directories(${PROJECT_NAME} PRIVATE "${CMAKE_CURRENT_SOURCE_DIR}/gxf"
55+
${isaac_ros_nitros_detection2_d_array_type_INCLUDE_DIRS})
56+
57+
set_target_properties(${PROJECT_NAME} PROPERTIES
58+
BUILD_WITH_INSTALL_RPATH TRUE
59+
BUILD_RPATH_USE_ORIGIN TRUE
60+
INSTALL_RPATH_USE_LINK_PATH TRUE
61+
)
62+
63+
# Install the binary file
64+
install(TARGETS ${PROJECT_NAME} DESTINATION share/${PROJECT_NAME}/gxf/lib)
65+
66+
ament_auto_package(INSTALL_TO_SHARE)

‎isaac_ros_gxf_extensions/gxf_isaac_ros_segment_anything/gxf/segment_anything/segment_anything_ext.cpp‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
// SPDX-FileCopyrightText: NVIDIA CORPORATION & AFFILIATES
2+
// Copyright (c) 2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
3+
//
4+
// Licensed under the Apache License, Version 2.0 (the "License");
5+
// you may not use this file except in compliance with the License.
6+
// You may obtain a copy of the License at
7+
//
8+
// http://www.apache.org/licenses/LICENSE-2.0
9+
//
10+
// Unless required by applicable law or agreed to in writing, software
11+
// distributed under the License is distributed on an "AS IS" BASIS,
12+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
// See the License for the specific language governing permissions and
14+
// limitations under the License.
15+
//
16+
// SPDX-License-Identifier: Apache-2.0
17+
#include "segment_anything_postprocessor.hpp"
18+
#include "segment_anything_prompt_processor.hpp"
19+
#include "segment_anything_msg_compositor.hpp"
20+
#include "gxf/std/extension_factory_helper.hpp"
21+
22+
GXF_EXT_FACTORY_BEGIN()
23+
GXF_EXT_FACTORY_SET_INFO(0xa3ed574714ef4f11, 0xc127090d5b35a477,
24+
"SegmentAnythingExtension",
25+
"Isaac ROS Segmentation PostProcessor Extension", "NVIDIA", "0.0.1",
26+
"LICENSE");
27+
28+
GXF_EXT_FACTORY_ADD(0xe9681b9e1b864123, 0x8fc86530f45f9ab2,
29+
nvidia::isaac_ros::SegmentAnythingPostprocessor, nvidia::gxf::Codelet,
30+
"Generates a raw segmentation mask from a tensor");
31+
GXF_EXT_FACTORY_ADD(0xe8211b9e1b864ab1, 0x2de86530f45f9cd3,
32+
nvidia::isaac_ros::SegmentAnythingPromptProcessor, nvidia::gxf::Codelet,
33+
"Transforms the input bboxes/points to SAM format.");
34+
GXF_EXT_FACTORY_ADD(0xe12acb9e1b8642ba, 0x34c86170f32f9abc,
35+
nvidia::isaac_ros::SegmentAnythingMsgCompositor, nvidia::gxf::Codelet,
36+
"Composes a single msg with all the received tensors.");
37+
GXF_EXT_FACTORY_ADD_0(
38+
0xa321601525594206, 0xbc12d9f22a134452,
39+
std::vector<nvidia::isaac_ros::Detection2D>,
40+
"Array of decoded 2D object detections in an image");
41+
GXF_EXT_FACTORY_END()

0 commit comments

Comments
 (0)