An autonomous robot with 3D vision based on a 3D webcam and a Raspberry Pi.
Already have an account? Log in.
To make the experience fit your profile, pick a username and tell us what interests you.
To control the robot I use my L1VM with some libraries written in C. The main control program is written in my language Brackets.
The motor driver is connected to the serial port of the Raspberry Pi.
With OpenCV the pictures of the webcam are scanned for identical parts in both pictures. An object in front is placed in a different location in the pictures. The control program uses this difference to calculate the objects distance. The goal is to avoid driving into obstacles.
I got help by my brother: he designed and printed the 3D webcam mounting for my robot!
Now I am writing a new control program in my Brackets language. The 3D points measurement is now interpolated up to 400 mm distance. This works together with the IR range sensor to detect obstacles in front of my robot. I have to do some test drives to check my program!
Currently I am writing a new control program for the robot. I am using my new language bra(et (bracket) to write the program. The program runs on my new virtual machine L1VM. L1VM is a lot faster then Nano VM, that is the reason why I write a new control program. It has the OpenCV object detection library already in the code. I still need to write a basic program and did no real drive tests so far.
The L1VM virtual machine is on GitHub: L1VM - fast virtual machine
Programs running on L1VM are 6-7 times faster as on the Nano VM! The L1VM binary is really small: it's only about 28 KB in the X86_64 Linux version. The VM has 60 opcodes and two interrupts (with up to 255 possible operations). The VM can run modules for SDL graphics, math operations, file I/O and more.
The modules are very basic. I continue development on them if I need something new.
The bra(et language makes it possible to use inline assembly code. This is sometimes needed to do some coding which is not possible in bra(et yet.
The full source code for L1VM is available on GitHub. Any feedback is welcome.
I finally got speech recognition with Jasper working. I can say commands like "go", "left" or "stop" and the robot will drive or turn in that direction. Or with the Jasper modules I can start a search in Wikipedia and the IMDB Movie database.
I began to write a little chat module for Jasper. So someone could ask the robot something and gets an answer.
I use the webcam microphone for speech input. I had to set the recording level to very low.
You have to turn it down until Jasper shows: "no disturbance detected" on silence. Then you have no "false" detected words on silence.
The vision system now generates a disparity map of the two webcam pictures (3D picture).
The robot adds near parts of this 3D picture to my 3D points and can see "more" objects.
If a threshold of objects in a set distance is reached the robot detects it as a obstacle. But how high should this threshold be set? I wrote some functions which do learn that. I did set the start thresholds very low, so the robot drives backwards to avoid objects. There is a "did drive backwards counter" to note this. If this counter reaches a set value the robot will increase the objects threshold. And the counter is set back to zero.
After some time "learning" the robot should drive forwards and avoid driving into objects. If the robot is stuck somewhere the objects threshold is decreased. The accelerometer is used to measure if an object was touched.
I did changes to my control program to log the accelerometer data before a move and while moving the robot. This way I can see the difference between the values of standing still and moving.
Later I will add code to check if the robot does move if it sends the commands to the motor driver.
So the robot can recognize if it is stuck somewhere and can't move forward.
I did install OpenCV on a fresh Raspbian. My programs need the "extra modules", so I had to compile them too. I installed my VM too. Now the control program runs again.
I now have to install Jasper for voice recognition. This time I made a backup of the SD card. So if anything goes wrong then I am on the safe side.
I added A heatsink onto the big chip on the Raspberry Pi 3. It got hot while compiling OpenCV. Now it is better with the heatsink installed. My control programs can reach up to 80% CPU time and I need this cooling then.
While testing some GPIO library stuff for my VM the SD card filesystem did crash!
It was corrupted beyond all repair. I did use photorec to rescue some important files. Now I have to reinstall everything from scratch. That is not funny :(.
Unfortunately my last backup image was done long time ago. Lesson learned: do more backup images of the SD card!
I can not work on this project until I am ready again, sorry for this!
I wrote an I2C library for Nano. With a simple test program I can now read out the raw data of a MPU 6050 accelerometer. Now I have to add some code to my main control program of the robot.
Would you please provide more information on using the 2 web cams for 3d vision? Are you using a library from OpenCV?
jay-t
BTom
Petar Crnjak
Sinclair Gurny
Yes, I use the OpenCV library. It is a algorithm to get keypoints in both pictures. The function to get the distances is my own code. I did use a "calibration" method to calculate the distances from 50 mm up to 250 mm. Everything farer away is filtered out. Maybe I do a calibration for it later.