Close
0%
0%

Open Personal Assistant Robotic Platform

Voice-controlled assistant robot to keep you company

Similar projects worth following
The goal is to create an open-sourced modular robotic platform that is voice-controlled and teleoperated. It has the full capabilities of a normal voice assistant through the open-sourced Mycroft AI, but will be able to follow you around and see it's surroundings using machine vision. I also gave the assistant a friendly face to make it easier to connect to.

The ultimate goal is that this can be used as a help in elderly care. We are a lot of people that faced the problem of not being able to take care of our elderly relatives in a safe way during covid-19. The robot assistant could help with reminders of taking medicine, a tray can be added to help carry things, give some company by talking with the robot and make it play your favorite music or podcasts. You can also use it for teleoperation so you can talk to your relative and check in on them.

Goal

The goal with the project is to be able to get the robot to complete small tasks at home. Either through teleoperation, partly automated or fully automated. With the goal of automating tasks more and more over time. Also to have a friendly and easy to use interface to make it easy to use.

It would be especially useful for elderly or disabled people who need help but are at risk of being infected with COVID-19, influenza or any other illness. The robot can reduce human contact and relieve small tasks that can be difficult if, for example it is difficult to bend down or you sit in a wheelchair, some example tasks could be to help fill water and food bowls for pets, through object detection help to find things at home, or maybe watering the flowers.

Design

The design is kept quite simple to begin with to focus more on the software side. I will try to use cheap and easy to find components. Right now i have a simple base built on an existing robot platform with 20x20 aluminium extrusions attached to it as a frame for mounting components. On the "head" there is a 7" touchscreen for displaying an animated face and with the possibility to use as an input device. 

The goal is to have a good base built for the robot first with the possibility for modular attachments.

Software

I'm using Tensorflow for Object detection and facial recognition. A python script to read sensor data from the arduino and to stream the live video to a server and use Socket.io to send real time commands to the robot so it's able to be remote controlled and start different directives, for example the only one i have now is the "Follow"-mode that use the object detection to follow the object of choice.

Mycroft AI is used as the voice assistent to give commands, play music etc.

Here you can see an early test of the follow mode. Further improvements have been made after this, mostly on making the object recognition faster and i will post more videos of that soon.

  • 1 × Raspberry Pi 4
  • 1 × Arduino Uno
  • 2 × 30W Tweeter Speaker
  • 1 × Raspberry Pi 7" Touchscreen
  • 1 × Raspberry Pi Camera

View all 11 components

  • Speech commands

    Erik Knutsson06/29/2020 at 20:26 0 comments

    Been working on the custom app for Mycroft AI to be able to control the robot by voice. I just got some basic commands down so far, but it's pretty cool to see it in action.

    To give the robot a little more of a personal feel i also started training a new Wakeword/Name for the robot. So i named it David, seemed like a good name since two of my favorite robots from movies is named David :)


  • Adding sound

    Erik Knutsson06/25/2020 at 21:27 0 comments

    To fully integrate the new voice assistent i've been working on adding speakers to the robot, so used some speakers i had laying around from an old bluetooth speaker project and printed a holder for them. It's not the prettiest. But as you can see on my wiring in the robot, i try to get it functional at first and make it pretty later :) 

  • Voice assistent capabilities

    Erik Knutsson06/07/2020 at 20:51 0 comments

    The weekend was spent on testing out different voice assistants. The setup i had before was to simply have my own text-to-speech where i just checked for certain keywords. Not the best, but it worked. But i wanted to add something like Alexa so i instantly could get way much more functionality without having to program everything on my own. 

    So i started out with Alexa, was not that hard to implement a test version from their examples so i could play podcasts and radio and so on and it worked great. It was fast and accurate. However i really wanted Spotify support so i could play my music and podcasts. But it turns out Amazon have disabled these types of applications if you are not developing a public application. 

    So i moved on to Mycroft AI that i found had a Spotify Skill, this was a little bit more tricky to get working at first, it was a easy install but the output audio would not play, after some work i finally got it working, it's slower than Alexa, but i read you can do some optimizations and i do really like that it is Open Source, it was really accurate and it seems easy enough to develop my own Skill to do a custom skill to control the robot. So the direction i will be going in will be with Mycroft AI.

    Next step is to create the custom skill and to install a speaker, since im just using a Bluetooth speaker right now which is a bit unnecessary, but i have some speakers from an old build of my own bluetooth speaker that i will probably re-use.

    You probably get the idea of how a voice assistant work, but here is a demo video anyway, since some content is always nice :) Something is weird with the sound in the demo, because it jumps into the middle of the first sentence. But i think that is some weird problem with my speaker since i don't get that with headphones connected.

  • Congratulations it's a box!

    Erik Knutsson05/28/2020 at 20:25 0 comments

    So while it's not the most important part of the project it's always fun to do some designs. But it's also really easy to just end up with a box with wheels. It's easy to build and everything fits great together. But... it is a bit boring, i will see if i can come up with something better than this and of course suggestions are welcome!

  • Some CAD work

    Erik Knutsson05/28/2020 at 10:14 0 comments

    Been doing some work in cad to try and get some more parts 3D-printed. Been doing these bottom plates that is attached to the frame for holding components. Had to do a total of 4 plates to be able to fit them in the printer.

    In place it will look something like this. Could probably do a better connection between the plates to keep it nice and flat. So perhaps that will change later.

    While i'm waiting on some components and more filament to arrive in the mail i will be doing some more CAD work to hopefully get a complete design and overview of the robot.

  • Remote interface

    Erik Knutsson05/25/2020 at 20:50 0 comments

    Put some more work into the interface for the teleoperation part of the robot. From here i can see the video feed with object detection, steering and some controls for making the robot autonomously find and follow objects. It's actually quite nice to make it play music and make it follow you arround the apartment :)

  • Printing new parts

    Erik Knutsson05/21/2020 at 20:42 0 comments

    Alot of parts are currently being printed and tested. Here im printing new motormounts.

  • Follow Me

    Erik Knutsson05/20/2020 at 08:42 0 comments

    Here is an older test from a month ago in "Follow Me" mode. It's terribly slow, but since this i have been working more on getting rid of the delays and made the object detection faster. I will post a new video as fast as it is put togheter with the new design.

  • Status update

    Erik Knutsson05/20/2020 at 06:43 0 comments

    At the moment im printing new parts to create a new chassi for the robot. Most of the components will be in a base with just two extrusions goin up as a body with the ”Head” mounted to it. Images of the progress will be added soon.

View all 9 project logs

Enjoy this project?

Share

Discussions

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates