Close
0%
0%

Anna The Multibot!

Multifunctional robot to help bring the dream of a personal robot into reality.

Similar projects worth following
Personal robots have been a long time dream. With the power of modern computers, machine vision and WIFI robots, it’s time to work on a project that cannot just perform one automated task, but be expandable to perform many task.

Goals:

- An easy way to call different automated functions that anyone off the street could use.

- A voice response from the robot.

- Multiple useful automated functions on an expandable platform.

Accomplishments so far:

-Teleoperated through a WIFI network

- Vision based object avoidance

- Easy to use plain test user interface

- Face detection and tracking

- Voice response from the robot

- Story telling application

Details:

Anna is a simple robot (built mostly from stuff I found in my basement) with a Tenvis jpt3815w WIFI camera, a DIY WIFI motor-controller board powered by a PIC16F874A Micro controller from MicroChip Inc., a WIFI Module, two h-bridge motor drivers and a port to add an arm so Anna will be able to manipulate objects in her world in the future. I chose the Tenvis camera because it is low cost and has a built-in pan and tilt.

Anna is controlled by a desktop or laptop through a WIFI network. Anna's program is written with Visual Studios Community in Visual Basic. This allows for an easy way of programming and testing ideas, plus adds a lot of processing power. The vision system will be a combination of EMGU OpenCV along with some original code.

By designing an easy-to-use user interface that anyone can operate to call different functions and a voice response from the robot to make it easy to know the robot is executing your command, plus an expandable platform that can grow and perform an ever expanding list of useful tasks, we will be taking steps to realizing the dream of a personal robot that anyone can use.

EMGU License information available here:

http://www.gnu.org/licenses/gpl-3.0.txt

Microsoft Visual Studio Community License information available here:

https://msdn.microsoft.com/en-us/subscriptions/cc150618.aspx

main board.zip

Gerber file of Anna's WIFI motor control board.

x-zip-compressed - 16.36 kB - 08/06/2016 at 14:35

Download

Adobe Portable Document Format - 2.96 MB - 07/03/2016 at 21:34

Preview
Download

Adobe Portable Document Format - 145.06 kB - 06/30/2016 at 00:00

Preview
Download

  • 1 × TENVIS JPT3815W Wireless IP Camera

  • Adding storytelling to the robot. A great educational tool for small children!

    Dennis08/04/2016 at 01:39 0 comments

    With a working voice on the robot, it is a good time to add a practical automatic function such as storytelling. With a voice-coordinated head motion and a display to display the words being spoken, a storytelling robot could be a great educational tool to teach reading skills to small children. I could also see other educational functions to teach math, science and history. Lots of robot kits are designed to teach engineering, but why not have a robot to help teach other subjects as well.

    I added a new form to display the story. The form contains a “rich” textbox and 3 buttons for pause, resume and cancel. The display will display one sentence at a time as the robot tells the story so a child can read along.
    The below images shows the display, the code to handle the buttons and what the display looks like while the robot is telling the story. In the future, the display will be on the robot:

    To the “command class” I added the following code to decode the command to tell a story:
    If command_input.Contains("tell") And command_input.Contains("story") Then
    Speech.speak_text("ok...")
    story()
    End If
    Also under “command" class I added the following sub to load the story and call the read sub.

    Private Sub story()
    Anna.Command_text.Text = Nothing
    Anna.load_story()
    Anna.read_text()
    End Sub
    In the main class I added the following subs to load the story, tell the story and handle buttons to pause, resume and cancel the story and also automate some head motion to keep the viewers’ attention:
    Public Sub load_story()
    Try
    ' Open the file using a stream reader.
    Using sr As New StreamReader("C:\Anna\Anna_Robot_Project\Anna\story\jack and the beanstalk.txt")
    Dim line As String
    ' Read the stream to a string and write the string to the console.
    line = sr.ReadToEnd()
    s = line
    End Using
    Catch

    End Try
    'TextBox1.Text = s
    End Sub
    Public Sub read_text()
    Dim first_index As Integer
    Dim second_index As Integer
    Dim length As Integer
    Dim segment As String = Nothing
    Dim myForm2 As New Display()
    first_index = -1

    display_cancel = False
    Do
    second_index = (s.IndexOf(".", second_index + 1))
    length = second_index - (first_index)
    If length < 1 Then Exit Do
    segment = s.Substring(first_index + 1, length)
    myForm2.Show()
    myForm2.RichTextBox1.Text = segment
    first_index = second_index
    speech1(segment)
    If second_index > s.Length - 1 Then Exit Do
    Dim a As String = segment.Length
    If display_cancel = True Then Exit Do
    Loop
    myForm2.Close()
    End Sub
    Private Sub speech1(st As String)
    synthesizer.SelectVoiceByHints(VoiceGender.Female, VoiceAge.Adult)
    speaker.Rate = 0
    speaker.Volume = 100
    AddHandler speaker.SpeakCompleted, AddressOf SpeachComplete
    speaker.SpeakAsync(st)
    speech_head_motion()
    While speach_done = False
    Application.DoEvents()
    If display_cancel = True Then Exit While
    End While
    speach_done = False
    End Sub
    Private Sub SpeachComplete(sender As Object, e As SpeakCompletedEventArgs)
    speach_done = True
    End Sub
    Public Sub speach_pause()
    speaker.Pause()
    Speech.speak_text("Story is paused")
    End Sub
    Public Sub speach_resume()
    Speech.speak_text("Story is resuming")
    Threading.Thread.Sleep(500)
    speaker.Resume()
    End Sub
    Public Sub speach_stop()
    display_cancel = True
    speaker.SpeakAsyncCancelAll()
    Speech.speak_text("Story is terminated")
    End Sub
    Private Sub speech_head_motion()
    head_up(800)
    head_speech_count = head_speech_count + 3
    If head_speech_count > 10 Then head_speech_count = 0
    If head_speech_count > 8 Then head_forward()
    If head_speech_count > 8 Then head_up(500)
    If head_speech_count > 5 And head_speech_count < 8 Then head_left(500)
    If head_speech_count < 5 Then head_right(500)
    End Sub

  • Anna the robot calling OpenCV functions face detect/tracking and canny frame from plain text commands with voice response.

    Dennis07/29/2016 at 23:57 0 comments

    Here’s two automated functions added to Anna. Face detect and canny frame. Both are called by plain text commands and both have a voice response. Below the video is a breakdown of the steps.

    Now with a way to call different automated processes from the last log, it’s time to add some processes. I thought I would start off with some OpenCV functions. Luckily with a wrapper like EMGU, adding OpenCV is fairly easy. To automate the process, we need a process to call, a timer, points to check to see if the process is being requested and a way to make the point active.
    I added a new class to contain the OpenCV functions called "OpenCV_functions". Within the class, I added routines for face detection and canny frame (more to be added later).
    Two Boolean points are also added to the main page. One is called “face” and one is called “canny_frame”.
    Public canny_frame As Boolean = False
    Public face As Boolean = False
    Both are Public to allow them to be set/cleared from other classes and they are both initialized as false.
    In the “command” class under the “text_decode” sub I added the following to lines of code:
    If command_input.Contains("show") Then show()
    If command_input.Contains("cancel") Then cancel()
    Also in the command class I added the following subs:


    Public Sub show()
    If command_input.Contains("canny") And command_input.Contains("frame") Then
    Machine_vision.canny_frame = True
    Dim Thread2 As New Thread(Sub() Speech.speak_text("Canny frame is now active, by your command"))
    Machine_vision.Command_text.Text = Nothing
    Thread2.Start()
    End If
    If command_input.Contains("face") And command_input.Contains("detect") Then
    Machine_vision.face = True
    Dim Thread2 As New Thread(Sub() Speech.speak_text("face detect is now active, by your command"))
    Machine_vision.Command_text.Text = Nothing
    Thread2.Start()
    End If
    End Sub
    Public Sub cancel()
    If command_input.Contains("canny") And command_input.Contains("frame") Then
    Machine_vision.canny_frame = False
    Dim Thread2 As New Thread(Sub() Speech.speak_text("Canny frame is now inactive, by your command"))
    Machine_vision.Command_text.Text = Nothing
    Thread2.Start()
    End If
    If command_input.Contains("face") And command_input.Contains("detect") Then
    Machine_vision.face = False
    Dim Thread2 As New Thread(Sub() Speech.speak_text("face detect is now inactive, by your command"))
    Machine_vision.Command_text.Text = Nothing
    Thread2.Start()
    End If
    End Sub


    Last I added a timer to the main page:
    Private Sub Timer1_Tick(sender As Object, e As EventArgs) Handles Timer1.Tick
    load_jpg()
    PictureBox1.Image = picture
    If canny_frame = True Then Opencv_functions.canny()
    If face = True Then Opencv_functions.face_detect()
    End Sub
    The load() loads the picture from the camera to proccess and if the Boolean point is set the timer will call it. The Boolean points are set and cleared in the command class based on text commands.
    Below is the OpenCV functions:


    Public Sub face_detect()
    faces = Nothing
    face_count = 1
    Try

    Dim ColordImage As Image(Of Bgr, [Byte]) = New Image(Of Bgr, Byte)(picture)
    'Load the object detector
    Dim faceDetector As New CascadeClassifier("haarcascade_frontalface_default.xml")
    'Convert the image to Grayscale
    Dim grayImage As Image(Of Gray, [Byte]) = ColordImage.Convert(Of Gray, [Byte])()
    For Each face As Rectangle In faceDetector.DetectMultiScale(
    grayImage,
    1.1,
    10,
    New Size(25, 20),
    Size.Empty)
    CvInvoke.Rectangle(ColordImage, face, New MCvScalar(0, 0, 255))
    Dim p As PointF = New PointF
    p = face.Location
    Dim size As String = face.Size.ToString
    faces = faces + ((" face" & face_count & vbLf + (" X: " _
    + (p.X.ToString + (" Y: " + p.Y.ToString))) + " " + size + ";"))
    face_count = face_count + 1
    Next
    Dim MyImage As Image = ColordImage.ToBitmap()
    Machine_vision.PictureBox1.Image = MyImage
    Catch
    End Try
    End Sub


    Public Sub canny()
    Try
    Dim cannyFrame As New Mat()
    Dim cannyFrame2 As New Mat()
    Dim ColordImage As Image(Of Bgr, [Byte]) = New Image(Of Bgr, Byte)(picture)
    Dim grayImage...

    Read more »

  • Command the robot to do something with a voice response!

    Dennis07/22/2016 at 00:23 0 comments

    To have a robot with a lot of different functions, it is necessary to come up with a way to call the different functions. Ideally, any one off the street could be able to tell the robot to perform a task. On that thought, plain text commands seem the most obvious way to accomplish this, and a voice response also seems like a good idea.
    To add very complex capabilities with lots of functions to any project, I like to add a new class to house the functions and subs. It makes the program a bit easier to follow and keeps it looking neater.
    On that note, I added two new classes to the Visual Studios program “command and speech.”
    I added a “command” text box to receive the commands. As text is typed in the text box, the command class is called as each character is typed to determine if the command matches any of the command keywords with the below sub.


    Private Sub Command_TextChanged(sender As Object, e As EventArgs) Handles Command_text.TextChanged, Command_text.TextChanged
    command_input = Command_text.Text
    Command.text_decode()

    End Sub


    The Speech Class at the moment just contains one sun that speaks a string of text that is send when the sub is called. Below is the speech class.


    Imports System.Speech.Synthesis
    Public Class Speech
    Private speaker As New SpeechSynthesizer()

    Public Sub speak_text(text As String)
    Dim speech
    speech = CreateObject("bot.voice")
    speech.speak(text)
    End Sub
    End Class


    The Command class determines if any command key word is typed and if one or a combination of keywords is typed it responds with a voice response and calls the operation. The command class at the moment calls the robots motor and head commands. It also opens a second thread for the voice so both subs run at the same time.

    Imports System.Threading

    Public Class Command
    Dim Speech As New Speech
    Public Sub text_decode()
    If command_input.Contains("look") Then headcommand()
    If command_input.Contains("head") Then headcommand()
    If command_input.Contains("move") Then movecommand()
    If command_input.Contains("turn") Then movecommand()
    If command_input.Contains("backup") Then movecommand()
    If command_input.Contains("back up") Then movecommand()
    End Sub
    Public Sub headcommand()
    If command_input.Contains("up") Then
    Machine_vision.head_up(1500)
    Dim Thread2 As New Thread(Sub() Speech.speak_text("looking up by your command"))
    Machine_vision.Command_text.Text = Nothing
    Thread2.Start()
    End If
    If command_input.Contains("down") Then
    Machine_vision.head_down(1500)
    Dim Thread2 As New Thread(Sub() Speech.speak_text("looking down by your command"))
    Machine_vision.Command_text.Text = Nothing
    Thread2.Start()
    End If
    If command_input.Contains("forward") Then
    Machine_vision.head_forward()
    Dim Thread2 As New Thread(Sub() Speech.speak_text("looking forward by your command"))
    Machine_vision.Command_text.Text = Nothing
    Thread2.Start()
    End If
    If command_input.Contains("left") Then
    Machine_vision.head_left(1500)
    Dim Thread2 As New Thread(Sub() Speech.speak_text("looking left by your command"))
    Machine_vision.Command_text.Text = Nothing
    Thread2.Start()
    End If
    If command_input.Contains("right") Then
    Machine_vision.head_right(1500)
    Dim Thread2 As New Thread(Sub() Speech.speak_text("looking right by your command"))
    Machine_vision.Command_text.Text = Nothing
    Thread2.Start()
    End If
    If command_input.Contains("stop") Then
    Machine_vision.drive_stop()
    Dim Thread2 As New Thread(Sub() Speech.speak_text("stop by your command"))
    Machine_vision.Command_text.Text = Nothing
    Thread2.Start()
    End If
    End Sub
    Public Sub movecommand()
    If command_input.Contains("forward") Then
    Machine_vision.drive_forward(1500)
    Dim Thread2 As New Thread(Sub() Speech.speak_text("moveing forward by your command"))
    Machine_vision.Command_text.Text = Nothing
    Thread2.Start()
    End If
    If command_input.Contains("turn left") Then
    Machine_vision.drive_left_f(500)
    Dim Thread2 As New Thread(Sub() Speech.speak_text("turning left by your command"))
    Machine_vision.Command_text.Text = Nothing
    Thread2.Start()
    End If
    If...

    Read more »

  • Testing out the object avoidance vision system

    Dennis07/19/2016 at 02:41 0 comments

    A quick trip around the kitchen testing out the object avoidance vision system. The robot is turning based on left side or right side threats. Fairly simple: If right side threat then left turn routine. If left side threat then right turn routine.

  • Object proximity

    Dennis07/09/2016 at 01:16 0 comments

    The program at this point has both a canny image and the original colored bitmap the canny image was derived from. With a canny edge image, we can now start getting information on where the objects are in the image.
    The technique I used is to start at the bottom of the canny image and start checking to see if the pixels are black or white. A black pixel means no edge and a white pixel means an edge is detected at that location. If a black pixel is detected, the program moves up the image to the next pixel and continues to check pixels until it has either detected a white pixel or it has reached the top of the image.
    If a white pixel is detected, the program stores its data in an array which includes the white pixel’s x and y location from the bottom of the image. As the program is checking the pixels, it is also calculating the average hue and brightness from the colored bitmap of the areas that are detected not to be edges. This information is also stored in the array and will give us additional useful data for later. It is not necessary to map every line of pixels, so I’m only mapping every 10th line. This cuts down on processor time.
    This allows detecting of horizontal edges. More important than horizontal edges is vertical edges. A horizontal edge could be just a line on the floor but a vertical line could be a wall. After the above section of code has ran, two more sections run to detect vertical edges.
    The canny image is divided into a left and a right side. From a line in the center of the image, a section of code checks the left side of the image for edges from the center of the image to the far left side. The same is also performed on the right side of the image. The information is also stored in arrays for further analysis.
    There is a short section of code to redraw the lines mapped. On the image below, red lines are the mapped lines from the bottom on the image, and blue lines are the vertical lines from the center of the image to the left and right side of the image.

    Below is the sub I used to detect the lines. The arrays are defined in a module as PUBLIC to allow different sections of the program to use the data. I added a clase for vision to separate the code into sections.

    Public Sub edge_range()
    Dim color As Color
    Dim hue As Integer = Nothing
    Dim brightness As Integer
    Dim x As Integer
    Dim y As Integer
    Dim color_hue As Color
    Dim color_lum As Integer = Nothing
    Dim image1 As Bitmap = Machine_vision.edges()
    Dim image_color As Bitmap = Machine_vision.picture
    Dim i As Integer
    Dim object_location As Integer
    Dim a As Integer = 0
    Dim line_count As Integer = 0
    Dim longest_x As Integer = 0
    Dim longest_y As Integer = 0
    x = 5
    y = 479
    For a = 0 To 62
    Do Until y = 0
    color = image1.GetPixel(x, y)
    color_hue = image_color.GetPixel(x, y)
    brightness = color.GetBrightness
    If brightness = 1 Then
    object_location = y
    Exit Do
    Else
    y = y - 1
    If hue = Nothing Then hue = color_hue.GetHue Else hue = (color_hue.GetHue + hue) / 2
    If color_lum = Nothing Then color_lum = color_hue.GetBrightness * 10 Else color_lum = ((color_hue.GetBrightness * 10) + color_lum) / 2
    End If
    Loop
    range(a, 0, 0) = 480 - object_location
    If a + 1 <> 63 Then range(a + 1, 0, 0) = Nothing
    range(a, 1, 0) = x
    range(a, 2, 0) = hue
    range(a, 3, 0) = color_lum
    If a = 0 Then
    range(a, 4, 0) = 1 : line_count = 1
    range(a, 5, 0) = 0
    range(a, 6, 0) = 0
    ElseIf Math.Abs(range(a, 0, 0) - range(a - 1, 0, 0)) < 10 Then
    range(a, 4, 0) = line_count
    Else
    line_count = line_count + 1
    range(a, 4, 0) = line_count
    End If

    If 480 - object_location > longest_y Then longest_y = 480 - object_location : longest_x = range(a, 1, 0)
    y = 479
    x = x + 10

    Next

    x = 5
    For a = 1 To 62
    For i = 480 - range(a, 0, 0) To 479
    Machine_vision.picture.SetPixel(range(a, 1, 0), i, Color.FromArgb(255, 0, 0))

    Next
    Next

    '***********************look for wall edges **************************
    Dim line_count1 As Integer = 0
    Dim line_count2 As Integer = 0
    y = 475
    a = 0
    Do Until y < 10
    x = 320
    color = image1.GetPixel(x, y)
    color_hue...

    Read more »

  • Finding edges

    Dennis07/05/2016 at 18:24 0 comments

    With the image from the camera now in the program, we need to look for object edges. This is one of the easiest ways to locate threats to the robot’s movements. An easy way to do this is to take the bitmap image that we have from the camera and invoke the canny function from the OpenCV library to implement the Canny Edge Detector. Below is the function I use to accomplish this. It returns a black and white image. The background is black and the edges show up white.

    Public Function edges() As Bitmap
    Dim cannyFrame As New Mat()
    Dim cannyFrame2 As New Mat()
    Dim ColordImage As Image(Of Bgr, [Byte]) = New Image(Of Bgr, Byte)(picture)
    Dim grayImage As Image(Of Gray, [Byte]) = ColordImage.Convert(Of Gray, [Byte])()
    Dim pyrDown As New UMat()
    CvInvoke.Canny(grayImage, cannyFrame, 100, 60)
    Dim img As Image(Of Bgr, [Byte]) = cannyFrame.ToImage(Of Bgr, [Byte])()
    Dim MyImage As Image = img.ToBitmap()
    Return img.ToBitmap()
    End Function

    Here's the image with the bitmap:

    Here's the image after invoking the Canny Edge Detector:

  • Get the camera's image into the program.

    Dennis07/03/2016 at 23:19 0 comments

    First things first. In order to experiment with a vision system, you need a camera and you need to get the camera’s picture from the camera into a program. I configured the Tenvis camera to my local WIFI router. Follow the Tenvis easy setup in files.
    I added a tutorial to demonstrate how to get the camera picture from the camera into the program. After running the tutorial, you should be able to display the camera’s picture in the form like the picture below. This is also a good first step to building a telepresence robot!

  • Anna inside and out

    Dennis06/30/2016 at 03:02 0 comments

    Here’s an overview of Anna inside and out. All in all a fairly simple robotics system. I originally built it to experiment with everything robotics.

    And here's the inside.

    Here’s a video of Anna under remote control from a Laptop through a WIFI network with a video feed from the robot’s camera and testing control of the robot’s camera pan and tilt, along with control of the robot’s two drive motors.

View all 8 project logs

  • 1
    Step 1

    How to get a camera image into a program:

    Open Microsoft Visual Studios. Under file click “New”. Then click “Project”, then click Windows Form Application” Title it “retrieve cam”.
    Under Form1.vb[Design]
    Add a textbox and name it “Head_ip” and add the Camera’s IP address under text of the textbox. Add a label above it and set the text to “camera IP”. Add a picture box and set its size to 640,480. Add a timer and set it to enable.

    You’ll also need to add the 2 imports to the top of the class form.

    Imports System.IO

    Imports System.Net

    Every time the timer cycles the program will retrieve a snapshot from the camera and display it in the picturebox.

    Below is how the Form1.vb[Design] should look:

    Here's the complete code:

    Imports System.IO
    Imports System.Net

    Public Class Form1
    Private Sub Timer1_Tick(sender As Object, e As EventArgs) Handles Timer1.Tick
    load_jpg()
    End Sub
    Public Sub load_jpg()
    Try
    Dim sourceURL As String = "http://" & Head_ip.Text & ":7777/snapshot.cgi"
    Dim buffer As Byte() = New Byte(99999) {}
    Dim read As Integer, total As Integer = 0
    ' create HTTP request
    Dim req As HttpWebRequest = DirectCast(WebRequest.Create(sourceURL), HttpWebRequest)
    req.Credentials = New NetworkCredential("username", "password")
    ' get response
    Dim resp As WebResponse = req.GetResponse()
    ' get response stream
    Dim stream As Stream = resp.GetResponseStream()
    ' read data from stream
    While (InlineAssignHelper(read, stream.Read(buffer, total, 1000))) <> 0
    total += read
    End While
    ' get bitmap
    Dim bmp As Bitmap
    bmp = DirectCast(Bitmap.FromStream(New MemoryStream(buffer, 0, total)), Bitmap)
    PictureBox1.Image = bmp
    Catch
    End Try
    End Sub
    Private Function InlineAssignHelper(Of T)(ByRef target As T, ByVal value As T) As T
    target = value
    Return value
    End Function
    End Class

    When you run it you should get something like this:

View all instructions

Enjoy this project?

Share

Discussions

Orlando Hoilett wrote 07/04/2016 at 04:04 point

Sounds like a great idea! Quick question, are any of the software used for face recognition by products like Amazon Echo even remotely open source? Or are they usually in-house algorithms?

  Are you sure? yes | no

Dennis wrote 07/04/2016 at 18:37 point

Hi Orlando,
 And thanks. I’m not
sure what Amazon Echo is using but OpenCV has libraries for Face recognition
and is commonly used. OpenCV can also be used with Android.

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates