|
1 | | -# Tutorial with Isaac Sim |
| 1 | +# Tutorial for DNN Image Segmentation with Isaac Sim |
| 2 | + |
| 3 | +<div align="center"><img alt="Coloured Segmentation Mask" src="../resources/Isaac_sim_peoplesemsegnet_shuffleseg_rqt.png" width="600px" title="U-Net Shuffleseg result in rqt_image_view"/></div> |
| 4 | + |
| 5 | +## Overview |
| 6 | + |
| 7 | +This tutorial walks you through a pipeline for [Image Segmentation](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_image_segmentation) of people using images from Isaac Sim. |
| 8 | + |
| 9 | +## Tutorial Walkthrough |
2 | 10 |
|
3 | 11 | 1. Complete the [Quickstart section](../README.md#quickstart) in the main README till step 9. |
4 | 12 | 2. Launch the Docker container using the `run_dev.sh` script: |
| 13 | + |
5 | 14 | ```bash |
6 | 15 | cd ~/workspaces/isaac_ros-dev/src/isaac_ros_common && \ |
7 | 16 | ./scripts/run_dev.sh |
8 | 17 | ``` |
| 18 | + |
9 | 19 | 3. Inside the container, build and source the workspace: |
| 20 | + |
10 | 21 | ```bash |
11 | 22 | cd /workspaces/isaac_ros-dev && \ |
12 | 23 | colcon build --symlink-install && \ |
13 | 24 | source install/setup.bash |
14 | 25 | ``` |
| 26 | + |
15 | 27 | 4. Install and launch Isaac Sim following the steps in the [Isaac ROS Isaac Sim Setup Guide](https://github.com/NVIDIA-ISAAC-ROS/isaac_ros_common/blob/main/docs/isaac-sim-sil-setup.md) |
16 | | -5. Open up the Isaac ROS Common USD scene (using the "content" window) located at: |
| 28 | +5. Open up the Isaac ROS Common USD scene (using the "content" window) located at: |
17 | 29 | `omniverse://localhost/NVIDIA/Assets/Isaac/2022.1/Isaac/Samples/ROS2/Scenario/carter_warehouse_apriltags_worker.usd`. |
18 | 30 |
|
19 | 31 | And wait for it to load completely. |
20 | 32 | > **Note:** To use a different server, replace `localhost` with `<your_nucleus_server>` |
21 | 33 | 6. Go to the stage tab and select `/World/Carter_ROS`, then in properties tab -> Transform -> Translate -> X change `-3.0` to `0.0`. |
22 | 34 | <div align="center"><img src="../resources/Isaac_sim_set_carter.png" width="400px"/></div> |
23 | | -</br> |
24 | | -7. Change the left camera topic name. Go to the stage tab and select `/World/Carter_ROS/ROS_Cameras/ros2_create_camera_left_rgb`, properties tab -> Compute Node -> Inputs -> topicName change `rgb_left` to `image`. |
| 35 | + |
| 36 | +7. Change the left camera topic name. Go to the stage tab and select `/World/Carter_ROS/ROS_Cameras/ros2_create_camera_left_rgb`, properties tab -> Compute Node -> Inputs -> topicName change `rgb_left` to `image`. |
25 | 37 | <div align="center"><img src="../resources/Isaac_sim_topic_rename.png" width="400px"/></div> |
26 | | -</br> |
27 | 38 | 8. Press **Play** to start publishing data from the Isaac Sim application. |
28 | 39 | <div align="center"><img src="../resources/Isaac_sim_image_segmentation.png" width="800px"/></div> |
29 | | -</br> |
| 40 | + |
30 | 41 | 9. Run the following launch files to start the inferencing: |
31 | | - |
32 | | -```bash |
33 | | - ros2 launch isaac_ros_unet isaac_ros_unet_triton.launch.py model_name:=peoplesemsegnet_shuffleseg model_repository_paths:=['/tmp/models'] input_binding_names:=['input_2:0'] output_binding_names:=['argmax_1'] network_output_type:='argmax' |
34 | | -``` |
35 | | -</br> |
| 42 | + |
| 43 | + ```bash |
| 44 | + ros2 launch isaac_ros_unet isaac_ros_unet_triton.launch.py model_name:=peoplesemsegnet_shuffleseg model_repository_paths:=['/tmp/models'] input_binding_names:=['input_2:0'] output_binding_names:=['argmax_1'] network_output_type:='argmax' |
| 45 | + ``` |
| 46 | + |
36 | 47 | 10. Visualize and validate the output of the package by launching `rqt_image_view` in another terminal: |
37 | 48 |
|
38 | | -```bash |
39 | | -cd ~/workspaces/isaac_ros-dev/src/isaac_ros_common && \ |
40 | | - ./scripts/run_dev.sh |
41 | | -``` |
42 | | -Then launch `rqt_image_view`: |
43 | | -```bash |
44 | | - ros2 run rqt_image_view rqt_image_view |
45 | | -``` |
46 | | -Then inside the `rqt_image_view` GUI, change the topic to `/unet/colored_segmentation_mask` to view a colorized segmentation mask. |
| 49 | + ```bash |
| 50 | + cd ~/workspaces/isaac_ros-dev/src/isaac_ros_common && \ |
| 51 | + ./scripts/run_dev.sh |
| 52 | + ``` |
| 53 | + |
| 54 | + Then launch `rqt_image_view`: |
| 55 | + |
| 56 | + ```bash |
| 57 | + ros2 run rqt_image_view rqt_image_view |
| 58 | + ``` |
47 | 59 |
|
48 | | -<div align="center"><img alt="Coloured Segmentation Mask" src="../resources/Isaac_sim_peoplesemsegnet_shuffleseg_rqt.png" width="350" title="U-Net Shuffleseg result in rqt_image_view"/></div> |
| 60 | + Then inside the `rqt_image_view` GUI, change the topic to `/unet/colored_segmentation_mask` to view a colorized segmentation mask. |
49 | 61 |
|
50 | | -**Note:** The raw segmentation is also published to `/unet/raw_segmentation_mask`. However, the raw pixels correspond to the class labels and so the output is unsuitable for human visual inspection. |
| 62 | + **Note:** The raw segmentation is also published to `/unet/raw_segmentation_mask`. However, the raw pixels correspond to the class labels and so the output is unsuitable for human visual inspection. |
0 commit comments