Collaborative Robot Arm with Artificial Intelligence, and Used as Assistant in: 1) Voice Command System & 2) Sorting Objects System

Public Chat
Similar projects worth following
Robot arm that pick-and-place an object like a box. Here I'm using the "MATRIX Creator" and Raspberry Pi boards, to control a robotic arm by mean of Artificial Intelligence (AI) with voice commands and the "Snips" assistant.

Hi everyone! Below I show you summaries of my Cobot in its three versions:

A video summarizing the journey of this project in the first version. Main goal, Features, Solutions, Content, Hardware and software prerequisites,

First code version of the robot arm

x-zip-compressed - 2.66 kB - 06/27/2021 at 23:09


Circuit diagram of the robot arm, version 1

x-zip-compressed - 295.79 kB - 06/28/2021 at 01:11


All the STL files necessary to create this project

x-zip-compressed - 11.19 kB - 06/27/2021 at 23:13


All the datashhets files used: 1) MATRIX_Creator.pdf, and 2) MG995_Tower-Pro.pdf

x-zip-compressed - 12.30 MB - 06/28/2021 at 00:48


  • 1 × Raspberry Pi 3 Model B+
  • 1 × MATRIX Labs MATRIX Creator
  • 1 × Alexa Echo Dot (3rd Gen)
  • 1 × 6DOF Robot Arm
  • 1 × Micro-USB Cable

View all 10 components

  • Step 9. Test and Conclusion

    Guillermo Perez Guillen06/28/2021 at 02:33 0 comments


    Before we start to do tests, we need to do next actions:

    On your Raspberry Pi type node assistant.js

    On your computer's terminal type sam status and sam watch

    Congrats! now you can test your robotic arm


    The prototype worked very well, and is very useful to develop more complex actions with the robotic arm. however, I also had some difficulties such as:

    • The board worked very well with 5 servos and when I used 6 servos the system was collapsed and the robotic arm made strange movements. Also in the servomotor configuration I changed to the value of: min_pulse_ms: 0.2 and the robotic arm worked much better.
    • As I explained at the beginning of this tutorial, I had problems configuring the audio board, but this was successfully solved and now the MATRIX Creator board is very efficient even when it receives keywords at half a meter.

    It's easy to say that in the future this prototype can serve as a model in every home, and may be, we will have a robotic arm at our tables to help us in multiple tasks.

  • Step 8. Printing and Assembling the Panel

    Guillermo Perez Guillen06/28/2021 at 02:27 0 comments

    I have designed a 10 x 10 cm panel for three functions:

    • The first is to hold the cables of the servos and that these don't get disconnected.
    • The second is to hold the connector of all physical grounds of the servos.
    • The third is to hold the connector of all Vcc connections of the servos.

    Below you can see a picture and download the STL file on my GitHub repository here:

  • Step 7. Wiring the Robot Arm

    Guillermo Perez Guillen06/28/2021 at 02:17 0 comments

    Below you can see the electrical diagram with the connections of this project, it's recommended to do it patiently so as not to make mistakes because there are many cables to be connected.

    I suggest you not wire the servo 5, since in the tests I carried out the system collapsed when I connected the six servos. Then I decided to disconnect the servomotor that was less used.

  • Step 6. Creating assistant.js

    Guillermo Perez Guillen06/28/2021 at 02:15 0 comments

    This step will go show you how to setup a MATRIX Lite project with Snips.

    First, MQTT must be installed and used to listen in on events from Snips.

    npm install mqtt --save

    assistant.js will be used to listen and respond to events from your Snips assistant. This file combines all the code from everloop.js & relay.js to control the everloop and relay switch through voice.

    var mqtt = require('mqtt');
    var client  = mqtt.connect('mqtt://localhost', { port: 1883 });
    var relay = require('./relay.js');
    var everloop = require('./everloop.js');
    var snipsUserName = 'guillengap';
    var wakeword = 'hermes/hotword/default/detected';
    var sessionEnd = 'hermes/dialogueManager/sessionEnded';
    var lightState = 'hermes/intent/'+snipsUserName+':roboticArmState';
    //ON CONNECT//
    client.on('connect', function() {
       console.log('Connected to Snips MQTT server\n');
    //ON MESSAGE//
    client.on('message', function(topic,message) {
       var message = JSON.parse(message);
       switch(topic) {
           // * On Wakeword
           case wakeword:
               console.log('Wakeword Detected');
           // * Robotic Arm Change
           case lightState:
               // Turn robotic arm Here/There
                   if (message.slots[0].rawValue === 'here'){
                       console.log('Robotic Arm Here');
                   else if(message.slots[0].rawValue === 'there'){
                       console.log('Robotic Arm There');
               // Expect error if `here` or `there` is not heard
                   console.log('Did not receive an Here/There state')
           // * On Conversation End
           case sessionEnd:
               console.log('Session Ended\n');

     everloop.js incorporates the code required to control the LEDs on the MATRIX Creator.

    const matrix = require('@matrix-io/matrix-lite');
    var matrix_device_leds = 0;// Holds amount of LEDs on MATRIX device
    var methods = {};// Declaration of method controls at the end
    var waitingToggle = false;
    var counter = 0;
       // Turns off all LEDs
       if (waitingToggle == false) {
               // Set individual LED value
       // Creates pulsing LED effect
       else if (waitingToggle == true) {
               // Set individual LED value
                   b: (Math.round((Math.sin(counter) + 1) * 100) + 10),// Math used to make pulsing effect
       counter = counter + 0.2;
       // Store the Everloop image in MATRIX configuration
    methods.startWaiting = function() {
       waitingToggle = true;
    methods.stopWaiting = function() {
       waitingToggle = false;
    module.exports = methods;// Export methods in order to make them avaialble to other files 

    relay.js includes the code required to turn the robotic arm here and there. This relay will control power to the robotic arm.

    const matrix = require('@matrix-io/matrix-lite');
    var methods = {};// Declaration of method controls at the end
    matrix.gpio.setFunction(0, 'PWM');
    matrix.gpio.setFunction(1, 'PWM');
    matrix.gpio.setFunction(2, 'PWM');
    matrix.gpio.setFunction(3, 'PWM');
    matrix.gpio.setFunction(4, 'PWM');
    matrix.gpio.setFunction(5, 'PWM');
    Read more »

  • Step 5. Deploy your Assistant

    Guillermo Perez Guillen06/28/2021 at 02:07 0 comments

    You can now deploy your assistant! On the bottom right of the page will be a Deploy Assistant button.

    Use the Sam CLI Tool to deploy the assistant to your Raspberry Pi. Below is an example of the command to use.

  • Step 4. Adding an Intent

    Guillermo Perez Guillen06/28/2021 at 02:02 0 comments

    Intents are what Snips uses to handle user requests. For this Assistant, we'll start by creating an intent for turning Here/There the MATRIX Creator's App.

    Create a new intent roboticArmState for your RoboticArm app

    Once you've defined your intent, you need to extract whether the user wants their robotic arm here or there. This is where slots come in.

    Create a new slot called power and a custom slot type with the same name.

    Make sure your power slot type has here & there as values.

    Create example sentences containing the words here and there. Highlight these words in your example sentences to assign them to your recently created power slot.

    Be sure to save!

  • Step 3. Creating a Snips Assistant and App

    Guillermo Perez Guillen06/28/2021 at 01:59 0 comments

    Sign into your account.

    Create an assistant called Robotic Arm

    Create an App named RoboticArm

  • Step 2. Connect Sam to Your Raspberry Pi

    Guillermo Perez Guillen06/28/2021 at 01:54 0 comments

    On the command prompt of you Raspberry Pi board type ifconfig to know the IP address of this device

    From your computer's terminal, you can connect to the Raspberry Pi board

    Sign in through the Sam CLI Tool

  • Step 1. Let's Get Started

    Guillermo Perez Guillen06/28/2021 at 01:37 0 comments

    Hardware Prerequisites

    Before getting started, you can check with more details what hardware I used:

    Software Prerequisites

    You can check what required software on my personal computer I used:

    • SD Card Formatter: To easily format our Micro SD Card
    • balenaEtcher: To easily flash our compressed Raspbian Stretch image
    • MATRIX kit image here which includes Stretch + MATRIX packages (2.18 GB)
    • Node.js: Dependancy for Sam CLI Tool (Windows 64 bits)
    • Git Bash: requirement asked by Node.js
    • Snips' Sam CLI Tool: Creates & manages Snip assistants on your Raspberry Pi

    Setting Up the Hardware

    Once everything is well installed on your Raspberry Pi board, You can check the versions of: npm, node and nodejs.

    In my case, I had two serious problems with the configuration of my hardware and these are the solutions:

    FIRST SOLUTION: to activate the pcm.speaker, I had to add a rate of 16000 in the assound.conf file.

    pcm.speaker {
    type plug
    slave {
    pcm "hw:0,0"
    rate 16000

    SECOND SOLUTION: My Raspberry Pi board has better audio with the "portaudio" board, so I had to activate it in the snips.toml file

    # frame = 256
    # bind = "default@mqtt"
    mike = "MATRIXIO-SOUND: - (hw:2,0)"
    # disable_playback = false
    # disable_capture = false

View all 9 project logs

  • 1

    Follow below steps to repeat the experience of this project

    • Step 1. Let's Get Started
    • Step 2. Connect Sam to Your Raspberry Pi
    • Step 3. Creating a Snips Assistant and App
    • Step 4. Adding an Intent
    • Step 5. Deploy your Assistant
    • Step 6. Creating assistant.js
    • Step 7. Wiring the Robotic Arm
    • Step 8. Printing and Assembling the Panel
    • Step 9. Test and Conclusion

    NOTE: All these steps are detailed in the Logs of this project

View all instructions

Enjoy this project?



Guillermo Perez Guillen wrote 07/19/2021 at 04:14 point

This project also participates in the "Supplyframe DesignLab: 2021 Hackaday Prize" contest, in the category of "Challenge 2: Refresh Work-From-Home Life".

  Are you sure? yes | no

Guillermo Perez Guillen wrote 06/28/2021 at 03:00 point

This project participates in the "Supplyframe DesignLab: 2021 Hackaday Prize" contest, in the category of "Challenge 4: Redefine Robots" and the updates will be published before the deadline.

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates