Inconsistent Topic Visibility and std::bad_alloc Crashes in Isaac ROS with iRobot Create 3 on Nvidia Jetson ORIN (rmw_cyclone_cpp)

[kmelti]

Hello there,

I am using an iRobot Create 3 connected via Ethernet to an Nvidia Jetson ORIN, running the Isaac-ROS NVIDIA Docker with the ROS 2 Humble distribution. I have installed the necessary icreate-robot packages, and the connection to the robot works correctly—I am able to send commands successfully.

However, when I launch the Isaac ROS Visual Slam package with the same domain ID as the robot (ROS_DOMAIN_ID=15), the following happens:

ROS_DOMAIN_ID 15
Launching irobot...
[INFO] [launch]: All log files can be found below /root/.ros/log/2024-10-18-14-23-09-413178-irobot-nav-46
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [component_container_mt-1]: process started with pid [60]
[component_container_mt-1] [INFO] [1729261390.376779482] [shared_nvblox_container]: Load Library: /isaac_ros/install/realsense_splitter/lib/librealsense_splitter_component.so
[component_container_mt-1] [INFO] [1729261390.399077701] [shared_nvblox_container]: Found class: rclcpp_components::NodeFactoryTemplate<nvblox::RealsenseSplitterNode>
[component_container_mt-1] [INFO] [1729261390.399214277] [shared_nvblox_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvblox::RealsenseSplitterNode>
[component_container_mt-1] [INFO] [1729261390.409318376] [camera.realsense_splitter_node]: Creating a RealsenseSplitterNode().
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/camera/realsense_splitter_node' in container 'shared_nvblox_container'
[component_container_mt-1] [INFO] [1729261390.439180045] [shared_nvblox_container]: Load Library: /isaac_ros/install/realsense2_camera/lib/librealsense2_camera.so
[component_container_mt-1] [INFO] [1729261390.688660088] [shared_nvblox_container]: Found class: rclcpp_components::NodeFactoryTemplate<realsense2_camera::RealSenseNodeFactory>
[component_container_mt-1] [INFO] [1729261390.688784441] [shared_nvblox_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<realsense2_camera::RealSenseNodeFactory>
[component_container_mt-1] [INFO] [1729261390.723771695] [camera.camera]: RealSense ROS v4.51.1
[component_container_mt-1] [INFO] [1729261390.723897391] [camera.camera]: Built with LibRealSense v2.56.1
[component_container_mt-1] [INFO] [1729261390.723924047] [camera.camera]: Running with LibRealSense v2.56.1
[component_container_mt-1] [INFO] [1729261390.725845622] [shared_nvblox_container]: Load Library: /isaac_ros/install/isaac_ros_visual_slam/lib/libvisual_slam_node.so
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/camera/camera' in container 'shared_nvblox_container'
[component_container_mt-1] [INFO] [1729261390.760082986] [shared_nvblox_container]: Found class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::visual_slam::VisualSlamNode>
[component_container_mt-1] [INFO] [1729261390.760212810] [shared_nvblox_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvidia::isaac_ros::visual_slam::VisualSlamNode>
[component_container_mt-1] [INFO] [1729261390.817062570] [visual_slam_node]: cuVSLAM version: 11.4
[component_container_mt-1] [INFO] [1729261391.286244764] [camera.camera]: Device with serial number 242322075051 was found.
[component_container_mt-1]
[component_container_mt-1] [INFO] [1729261391.286382812] [camera.camera]: Device with physical ID 2-1-2 was found.
[component_container_mt-1] [INFO] [1729261391.286404540] [camera.camera]: Device with name Intel RealSense D435I was found.
[component_container_mt-1] [INFO] [1729261391.287092383] [camera.camera]: Device with port number 2-1 was found.
[component_container_mt-1] [INFO] [1729261391.287135231] [camera.camera]: Device USB type: 3.2
[component_container_mt-1] [INFO] [1729261391.288714724] [camera.camera]: getParameters...
[component_container_mt-1] [INFO] [1729261391.289301126] [camera.camera]: JSON file is not provided
[component_container_mt-1] [INFO] [1729261391.289344742] [camera.camera]: Device Name: Intel RealSense D435I
[component_container_mt-1] [INFO] [1729261391.289364838] [camera.camera]: Device Serial No: 242322075051
[component_container_mt-1] [INFO] [1729261391.289380423] [camera.camera]: Device physical port: 2-1-2
[component_container_mt-1] [INFO] [1729261391.289393671] [camera.camera]: Device FW version: 5.13.0.50
[component_container_mt-1] [INFO] [1729261391.289406119] [camera.camera]: Device Product ID: 0x0B3A
[component_container_mt-1] [INFO] [1729261391.289417703] [camera.camera]: Sync Mode: Off
[component_container_mt-1] [WARN] [1729261395.221548045] [camera.camera]: re-enable the stream for the change to take effect.
[component_container_mt-1] [WARN] [1729261395.222460432] [camera.camera]: re-enable the stream for the change to take effect.
[component_container_mt-1] [WARN] [1729261395.224452663] [camera.camera]: re-enable the stream for the change to take effect.
[component_container_mt-1] [WARN] [1729261396.026933569] [camera.camera]: re-enable the stream for the change to take effect.
[component_container_mt-1] [WARN] [1729261396.027252386] [camera.camera]: re-enable the stream for the change to take effect.
[component_container_mt-1] [WARN] [1729261396.028845991] [camera.camera]: re-enable the stream for the change to take effect.
[component_container_mt-1] [INFO] [1729261396.051922677] [camera.camera]: Stopping Sensor: Stereo Module
[component_container_mt-1] [INFO] [1729261396.337416724] [camera.camera]: Starting Sensor: Stereo Module
[component_container_mt-1] [INFO] [1729261396.454462013] [camera.camera]: Open profile: stream_type: Infra(1), Format: Y8, Width: 640, Height: 480, FPS: 60
[component_container_mt-1] [INFO] [1729261396.454606877] [camera.camera]: Open profile: stream_type: Infra(2), Format: Y8, Width: 640, Height: 480, FPS: 60
[component_container_mt-1] [INFO] [1729261396.454640701] [camera.camera]: Open profile: stream_type: Depth(0), Format: Z16, Width: 640, Height: 480, FPS: 60
[component_container_mt-1] [INFO] [1729261396.455086271] [camera.camera]: Stopping Sensor: RGB Camera
[component_container_mt-1]  18/10 14:23:16,458 WARNING [281471294261552] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1] [INFO] [1729261396.462526264] [camera.camera]: Starting Sensor: RGB Camera
[component_container_mt-1] [INFO] [1729261396.492937054] [camera.camera]: Open profile: stream_type: Color(0), Format: RGB8, Width: 640, Height: 480, FPS: 15
[component_container_mt-1] [INFO] [1729261396.495037957] [camera.camera]: RealSense Node Is Up!
[component_container_mt-1] [WARN] [1729261396.530686461] [camera.camera]:
[component_container_mt-1]  18/10 14:23:16,574 WARNING [281471294261552] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  18/10 14:23:16,789 WARNING [281471294261552] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  18/10 14:23:16,799 WARNING [281471294261552] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  18/10 14:23:16,810 WARNING [281471294261552] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  18/10 14:23:16,821 WARNING [281471294261552] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[component_container_mt-1]  18/10 14:23:16,945 ERROR [281469993863472] (uvc-streamer.cpp:105) uvc streamer watchdog triggered on endpoint: 131
[component_container_mt-1]  18/10 14:23:18,493 ERROR [281469968685360] (uvc-streamer.cpp:105) uvc streamer watchdog triggered on endpoint: 132
[component_container_mt-1] [INFO] [1729261405.740218472] [visual_slam_node]: Time taken by CUVSLAM_WarmUpGPU(): 14.923056
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/visual_slam_node' in container 'shared_nvblox_container'
[component_container_mt-1] [INFO] [1729261405.763154004] [visual_slam_node]: left_pose_right reading from CameraInfo
[component_container_mt-1] [INFO] [1729261405.763272181] [visual_slam_node]: Baseline is : 0.050081
[component_container_mt-1] [INFO] [1729261405.763345589] [visual_slam_node]: Use use_gpu: true
[component_container_mt-1] [INFO] [1729261405.763369461] [visual_slam_node]: Enable IMU Fusion: false
[component_container_mt-1] [INFO] [1729261405.764972346] [shared_nvblox_container]: Load Library: /isaac_ros/install/odometry_flattener/lib/libodometry_flattener_component.so
[component_container_mt-1] [INFO] [1729261405.777143043] [shared_nvblox_container]: Found class: rclcpp_components::NodeFactoryTemplate<nvblox::OdometryFlattenerNode>
[component_container_mt-1] [INFO] [1729261405.777276611] [shared_nvblox_container]: Instantiate class: rclcpp_components::NodeFactoryTemplate<nvblox::OdometryFlattenerNode>
[component_container_mt-1] [INFO] [1729261405.791782580] [odometry_flattener_node]: Creating a OdometryFlattenerNode().
[INFO] [launch_ros.actions.load_composable_nodes]: Loaded node '/odometry_flattener_node' in container 'shared_nvblox_container'
[component_container_mt-1] [INFO] [1729261405.833976512] [visual_slam_node]: Time taken by CUVSLAM_CreateTracker(): 0.070590
[component_container_mt-1] [WARN] [1729261409.023345713] [visual_slam_node]: Delta between current and previous frame [100.175104] is above threshold [33.330000]

On the host system, after exporting ROS_DOMAIN_ID=15, I can only see the topics from the iRobot Create 3:
ros2 topic list

/irobot/battery_state
/irobot/cliff_intensity
/irobot/cmd_audio
/irobot/cmd_lightring
/irobot/cmd_vel
/irobot/cmd_vel_stamped
/irobot/dock_status
/irobot/hazard_detection
/irobot/imu
/irobot/interface_buttons
/irobot/ir_intensity
/irobot/ir_opcode
/irobot/kidnap_status
/irobot/mobility_monitor/transition_event
/irobot/mouse
/irobot/odom
/irobot/robot_state/transition_event
/irobot/slip_status
/irobot/static_transform/transition_event
/irobot/stop_status
/irobot/tf
/irobot/tf_static
/irobot/wheel_status
/irobot/wheel_ticks
/irobot/wheel_vels
/parameter_events
/rosout
/tf

However, when I launch the Isaac ROS Visual Slam package with the same domain ID as the robot (ROS_DOMAIN_ID=15), I encounter various issues:

In some cases, the topics from the isaac_ros package are visible, but no data is published.

In other cases, the topics from the robot disappear entirely.

Additionally, I sometimes have the topics from the robot, but other ROS 2 packages crash with the following error:

terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
/bin/bash: line 4:    46 Aborted

Additionally, running this command on the Jetson:

ros2 launch create3_republisher create3_republisher_launch.py republisher_ns:=/irobot_republish robot_ns:=/irobot
also crashes with the same error as above. I am using rmw_cyclone_cpp as the DDS.

I am using rmw_cyclone_cpp as the DDS.

Question:

1. Did I miss something in my configuration that could be causing the inconsistent topic visibility or the std::bad_alloc errors?
2. Are there specific settings or adjustments I need to make for DDS (rmw_cyclone_cpp) to ensure reliable communication across nodes with the same ROS_DOMAIN_ID?

[martincerven]

Hey @kmelti, by ethernet you mean USB cable?

I also have Jetson & Create3 and I wanted to test these Isaac ROS algorithms...

• Do you have Isaac in containers?
• Do you have cyclone set up everywhere?
• Do you have synced time? - with chrony for example

I will try them later, but It might be one of two:

• time issue: the navigating packages need fresh data, even if you are off by few seconds your queues could fill and crash?
• permissions/flags/RMW profile setup inside containers?

Are you using custom RMW profile? How are you syncing time?