The development of a successful robotic assistive feeding system depends on its ability to acquire bites of food accurately, time the bites appropriately, and transfer the bites easily in both individual and group dining settings. However, the task of automating bite acquisition is challenging due to the vast variety of foods, utensils, and human strategies involved, as well as the need for robust manipulation of a deformable and hard-to-model target.

Bite timing, especially in social dining settings, requires a delicate balance of multiple forms of communication, such as gaze, facial expressions, gestures, and speech, as well as action and sometimes coercion. Bite transfer is a unique form of robot-human handover that involves the use of the mouth.

Source

Gesture and speech controlled robot arms are being developed to assist in feeding individuals with upper-extremity mobility limitations.

These robot arms use sensors and cameras to detect and interpret hand gestures or voice commands, allowing the user to control the robot arm in a more natural way. Gesture controlled robot arms can pick up food, bring it to the user's mouth, and adjust the position of the food based on the user's gestures.

Speech controlled robot arms can interpret voice commands such as "open mouth" or "move food closer". These robot arms have the potential to improve the independence and quality of life for individuals with upper-extremity mobility limitations and can also be used in healthcare and elderly care settings to improve the efficiency and safety of feeding assistance.

In this project, we will be focusing on Face tracking, Gesture recognition, Robot-Assisted feeding and Object Detection in cluttered environments.

Sustainable Development Goals:

This project aims at assisting countries in reaching the 11th sustainable development goal by introducing technologies that make human societies more inclusive.

Hardware

Building the robot arm

You will be needing the LEGO Mindstorms 51515 kit. We followed this video by PhiloYT to create the arm. You can find the building instructions here.

Instead of attaching the Mindstorms Hub, attach the Raspberry Pi and the BuildHat. At the end it should look like this:

Specifications

1)OpenCV Spatial AI Kit OAK-D:

OAK-D is a device to inference Monocular Neural Inference fused with stereo depth and Stereo Neural Inference.

DepthAI is the embedded spatial AI platform built around Myriad X - a complete ecosystem of custom hardware, firmware, software, and AI training. It combines neural inference, depth vision, and feature tracking into an easy-to-use, works-in-30-seconds solution.OAK-D is a device to inference Monocular Neural Inference fused with stereo depth and Stereo Neural Inference.

DepthAI offloads AI, depth vision and more - processed direct from built-in cameras - freeing your host to process application-specific data.

DepthAI gives you power of AI, depth, and tracking in a single device with a simple, easy-to-use API, written in Python and C++.

Best of all, it is modular (System on Module) and built on MIT-licensed open source hardware, affording adding these Spatial AI/CV super powers to real commercial products.

The Geometry behind Calculation of Depth and Disparity from Stereo Images on OAK-D:

By tracking the displacement of points between the alternate images, the distance of those points from the camera can be determined. Different disparities between the points in the two images of a stereo pair are the result of parallax. When a point in the scene is projected onto the image planes of two horizontally displaced cameras, the closer the point is to the camera baseline, the more difference is observed in its relative location...