Himani Multimodal Perception & Understanding System (HMPUS) is the perception "brain" being built incrementally for a humanoid robot, starting from raw camera input and growing toward full multimodal scene understanding.
The system is designed bottom-up as a set of independent, single-responsibility ROS2 nodes rather than one monolithic AI pipeline:
Camera Node — streams frames from an ESP32-CAM over Wi-Fi into ROS2 as sensor_msgs/Image, with no knowledge of what's in the frame.
Perception Node — runs YOLOv8 (GPU-accelerated) on each frame to detect objects, publishing structured vision_msgs/Detection2DArray messages.
Tracking Node — a custom-built IoU-based multi-object tracker (implemented from scratch, not just a library) that assigns persistent track_ids across frames, with full lifecycle management (NEW → TRACKING → LOST → REACQUIRED/REMOVED) so the robot understands the same object over time, not just isolated per-frame detections.