Overview#
The Camera module processes depth images into policy observations. Its interfaces use C++ and ROS 2; a Python camera SDK is not yet provided.
C++ entry point#
DepthProvider(rclcpp::Node& node, Ort::Env& env,
std::size_t expected_output_size);
bool process_latest(std::vector<float>& output);
void reset();DepthProvider is noncopyable. Configure input cropping, history sampling, the encoder model, and output dimensions together according to the policy requirements.
ROS 2 entry point#
depth_node publishes std_msgs/msg/Float32MultiArray and registers the clear_depth_history Trigger service. The repository's parkour camera configuration is:
| Setting | Value in the source file |
|---|---|
depth_topic |
/camera/camera/depth/image_rect_raw |
depth_obs_topic |
/depth_obs |
depth_process_hz |
50.0 |
use_depth_encoder |
true |
depth_encoder_model_name |
encoder.onnx |
depth_encoder_output_dim |
128 |
depth_process_hz=50.0 configures the processing frequency. This pipeline is for RPO parkour; RP1 integration requires a matching perception policy and configuration.
Checks before integration#
Confirm that the devices/topics in the camera launch file, depth image dimensions, cropping parameters, distance processing, history taps, and encoder inputs/outputs match the policy observations. The repository provides launch files for D435i; other cameras require their own configuration and validation.
If the perception input is too short, the inference interface issues a warning and handles the input. During troubleshooting, check the actual output dimensions against the policy's observation requirements.