One of the things I want my robot to do is to bring me beer when I'm retired and sitting on the couch. This involves the following actions:

1. Drive to the fridge
2. Open the fridge
3. Locate a bottle of beer
4. Fetch the bottle
5. Close the fridge
6. Drive to me
7. Hand me the bottle

My wild thumper robot can already drive autonomously to any location in the apartment. Also thanks to MoveIt I can position the manipulator to any given x/y/z-position that is physically accessible. The current task is to detect, locate and grab a bottle. The detection is done with a YOLO Real-Time Object Detection neural network, the 3d location is then determined from the distance given by the 3d camera and the grabbing is done with MoveIt. For this the following ROS configuration is used:

The realization with the Realsense 3d-camera is done in the following steps:

1. Use a neural network (NN) to detect a bottle in the RGB-image of the 3d camera. The NN runs on a stationary computer because the robot does not have a NN accelerator. The result is a box with row, column, width and height.
2. Calculate the middle pixel coordinate and get its distance from the depth image. For this the depth image needs to be aligned to the RGB-image. The realsense node can do that automatically.
3. With the known orientation of the 3d camera use projectPixelTo3dRay() to transform these values in goal coordinates x/y/z, relative to the robot.
4. Position the gripper of the manipulator 5cm before the goal
5. Open the gripper
6. Position the middle of the gripper on the goal
7. Close the gripper
8. Move the manipulator to a final position.

The resulting MoveIt script is in the Git repository and the video shows what it looks like: