08/04/2019 at 20:03 •
The goal of this series of articles is to create a compact indoor robot that can navigate using stereo vision. As a platform for it, we’ll use a small Dagu RP5 platform on tracks that we have. Here’s how it looks like next to the StereoPi.
Detailed TL;DR for fisheye cameras calibration can be found here in our blog.
- 06/21/2019 at 13:07 • 0 comments
06/20/2019 at 13:58 •
We have 3 more guides now:
1. The Art Of Stereoscopic Photo, part 1 (basics)
2. The Art Of Stereoscopic Photo, part 2 (assembling a camera)
3. Hacking Skybox on Oculus Go for StereoPi live streaming (just a hack)
04/09/2019 at 08:33 •
Today we’re pleased to share with you a series of Python examples for OpenCV development. This code works with either the StereoPi or the Raspberry Pi Development Board, as both support using two cameras simultaneously. Our ready-to-use code (and also Raspbian image) will help you every step of the way, from the first image capture to the Depth Map created via real-time video capture.
We would like to emphasize that all of these examples are for those new to OpenCV and are not intended for production use. If you are an advanced OpenCV user and have worked with the Raspberry Pi before, you’ll know it’s better to use C/C++ (instead of Python) and to utilize the GPU for better performance. At the end of this article we’ll provide some notes regarding the various bottlenecks we experienced using Python.
Here is our hardware setup:
We used the StereoPi board with Raspberry Pi Compute Module 3+. Also two Raspberry Pi cameras V1 connected (based on ov5647 sensor).
- Raspbian Stretch (kernel 4.14.98-v7+)
- Python 3.5.3
- OpenCV 3.4.4 (pre-compiled, ‘pip’ from Python Wheels)
- Picamera 1.13
- StereoVision lib 1.0.3 (https://github.com/erget/StereoVision)
The software installation process is beyond the scope of this article but we have prepared a Raspbian image with all software installed. Here is a link to our GitHub stereopi-tutorial repository.
All scripts support key stroke processing, and you can press ‘Q’ key to stop them. If you use Ctrl+C to stop the script, it may break the Python interaction with the cameras. In this case, you will need to reboot StereoPi.
Step 1: Image Capture
We use 1_test.py script for this purpose. Open the console and go to our examples folder:
After starting the script you can see a preview window with the stereoscopic video. Pressing ‘Q’ will stop the process and save the last captured image. This image will be used in the next scripts for Depth Map parameters tuning.
This script allows you to check if your hardware is operational and helps you obtain your first stereoscopic picture.
The following video shows how the first script works:
Step 2: Collecting Images For calibration
In an ideal world, a perfect depth map needs to use two identical cameras with their optical, vertical and horizontal axis all in parallel. In the real world, however, cameras are different and it’s impossible to align them perfectly. Thus, a software calibration method is used. Using two cameras you take multiple photos of an object. In our case, we used a printed chessboard. A special algorithm will then analyze these photos and find parameters for correction. This script begins this process by capturing a series of chessboard photos for calibration. Before each photo, the script starts a five (5) second countdown. Five seconds is generally enough time to reposition the chessboard. Make sure it can be seen by both cameras and ensure it’s stable to avoid “blurred” photos. The default number of photos captured per series is 30.
At the end, we have 30 stereoscopic photos, saved in /scenes folder.
Step 3: Image Separation
The third script 3_pairs_cut.py separates the captured photos into “left” and “right” images and saves them in /pairs folder. These separations could be done on-the-fly, without saving, but this step is helpful for your next experiments. You can save image-pairs from different capture series. Use your own code to work with this images, or use another stereoscopic camera’s images by putting them in this folder.
This script will show you every stereo pair before it’s separated (and waiting for key press). This lets you find bad photos and remove them before the next script.
Our code also includes the images we used for calibration. You may save them as an example before taking your own stereoscopic photos.
Step 4: Calibration
The next script 4_calibration.py loads all pairs saved on the previous step and calculates correction matrices. The script first tries to find a chessboard on the photo and, if it can’t, then it ignores the current pair and continues forward. So if you have some bad photos in your series, it won’t break the script. After all the calculations have been done, the program will rectify the last image and show you resulting “fixed” images. At this step, you can understand the quality of the calibration. In our case, calculations took about a minute and a half.
Calibration script doing his job:
Step 5: Depth Map Tuning
The next script, 5_dm_tune.py, loads image saved in script 1, and the calibration results from step 4. After that it presents you with a depth map and an interface for fine-tuning. If you want to use our parameters, just press the “Load settings” button. Before tuning the parameters, we recommend taking a photo with 3 objects at different distances from camera. For example one close-up at 30–40 cm, another at average distance (2–3 meters) and another “far” away. It will help you to find the right settings, where the closest objects will be red, and those far away will be dark-blue.
Here is how this looks like:
Step 6: Real Time Depth Map Using Video
Last script 6_dm_video.py builds depth map in a real time, using results from previous steps.
We hope our scripts will help you in your experiments!
Notes for advanced users:
- Once started, the first script will display the average time between captured frames and, once stopped, it will show the average FPS. This can be used as a simple tool to determine the camera and capture parameters at which Python can capture imagery without dropping frames. Using this script, we’ve found that setting the camera to 1280x480 at 20 FPS provided stable results, with no lost frames.
- You may have noticed that we capture images from camera at 1280x480 resolution and then scale it down to 640x240. You may ask, “why don’t we just capture 640x240, and avoid the Python processing to scale them?” The reason is that PiCamera has some bugs related to capturing images at low resolutions, which results in the image appearing broken (photo mixed with green squares). So, to avoid this, we use a method of acquiring a large image and then scaling it down via the GPU. In this way, there is no additional load to our ARM CPU!
- Why do we use BGRA capture, but not BGR? As we mentioned before, we resize the images using the GPU, and a native GPU format is BGRA (or RGBA, as you wish). If you use BGR you will find two issues; the first is a lower FPS (about 10–20% in our case) and the second is a PiCamera warning «PiCameraAlfaStripping: using alpha-stripping to convert to non-alpha format; you may find equivalent alpha format faster”. Googling this error led us to PiCamera documentation where we found the BGRA method.
- Where is PiRGBArray in your code? This is a native class of PiCamera for working with cameras. In our experiments, using a hand-made Numpy array instead of PiRGBArray gives us about 50% more FPS. We don’t think PiRGBArray is a poor choice and it may be we made some mistakes in implementation. We’ll look into this further at a later time.
- What is the CPU load while building the Depth Map with video?
Let me answer this with a screenshot:
We can see that only 1 kernel from 4 of the Pi’s CPUs has some load — even while rendering the GUI, 2 images and a depth map. This means that there is potential for performance here, and can try to use it by OpenCV optimization (OpenMP) using C instead of Python.
- What is the maximal Depth Map FPS we can have with these settings? We were able to achieve ~17 FPS, while the camera captures 20 FPS. FPS appears to be most sensitive for a number of parameters. MinDisparity and NumOfDisparities are obvious as these define the number of “steps” for running a window while comparing two images. Another parameter appears to be preFilterCap.
- What is the CPU temperature?
Using a Compute Module 3+ Lite, we’re seeing a temperature of 63.4’C
- How can we use GPU?
It can be used for undistortion and rectification in a real time. There are some similar approaches, for example using WebGL, or Python-based Pi3d, and also Processing for Raspberry Pi. There is also an interesting project called py-videocore by Koichi Nakamura. In our communications Koichi told me to use his kernel and StereoBM sources for CUDA to accelerate the depth map. So, there are lot of optimization options.
Thank you for reading!
Here is our source code with the Raspbian image links:
03/04/2019 at 11:31 •
As you know, all our Starter and Deluxe kits will include a microSD card with a ready-to-use Raspbian image so you can repeat all of our livestream experiments right out of the box. We’ve been busy polishing existing features and adding new ones to this image. In this update, we’ll share with you some new features and say a few words about our experiments with Skype and 3D video.
Latest StereoPi Livestream Playground (SLP) Image
- Image size reduced from 5 GB to 860 MB
- Video livestream to browser (2D and 3D)
- Livestream to Android over USB cable (Android accessory support)
- Bash console over web admin panel
- File editor over admin panel
- Access to video records over web admin panel
- RTSP livestream support
- MPEG-TS livestream support
- Linux partition now takes 2 GB instead of 4 GB
- FAT32 partition now created automatically on first boot
- RPi 3B+ and CM3+ support (updated kernel)
- Most settings are now in the
You can download image file from one of these three mirrors:
Full descriptions of all features will be added to the SLP section of our wiki in the coming days.
Skype and 3D Video
One of our new features is the ability to livestream MPEG-TS. We used this feature to livestream video from StereoPi to OBS (Open Broadcaster Software) with this OBS-VirtualCam plugininstalled. OBS creates a virtual webcam accessible to Skype. Here’s a proof-of-concept demo, recorded by Sergey:
And here is a screen capture of my iPhone and our first 3D Skype call:
As the iOS screen recorder does not record sounds, I added music to it.
To get all these things to work, we used two tricks. First, we used mic from Logitech webcam connected to the same computer to provide the audio to Skype, since the OBS-VirtualCam cannot emulate a sound device, so sound from StereoPi’s microphone isn’t available to Skype.
Second, we avoid a one-second delay between audio (due to an internal OBS video buffer) by first streaming the video from StereoPi to gstreamer on Windows and then pointing OBS to use the gstreamer window as its video source, which resulted in a delay of only about 100 milliseconds.
This test shows it is possible to use stereoscopic livestream with a lot of common software, like Skype and other video-related programs, all without modification since they already work with a traditional camera.
If you want to discuss more features, please join this thread on the Raspberry Pi forum.
02/07/2019 at 11:38 •
If you have classic Raspberry Pi with the camera, you can repeat all our video livestream experiments. Livestream to YouTube, Android and Oculus Go. Also you can repeat our behind-the-scene experiments with video livestream to WIndows desktop, Mac or any RTMP server.
Today we want to share with you our Raspbian image. We call it SLP (StereoPi Livestream Playground). It supports single-camera mode and also two-camera mode for StereoPi.
You can find image, Android application and brief manual in our Wiki
01/30/2019 at 20:04 •
We are pleased to announce the launch of the StereoPi campaign! :-)
12/25/2018 at 10:13 •
As we mentioned in our previous update, 3 weeks ago we started first step for preparing production at chosen factory. Now we are glad to inform you that first step is successfully complete!
During these 3 weeks these things happened:
---------- more ----------
- During first week factory started PCB manufacturing and begun components bought.
- During second week components was mounted on equipment, which will be used for batch production. At this step all components were mounted except some connectors. We received some photos at this stage:
- On the second week all components needed for tests also arrived at factory (Raspberry Pi Compute Module 3 Lite and 2 cameras).
- For all 20 pieces factory performed all tests, and all 20 pieces have passed them successfully!
If you’re curious, for the tests our team created MicroSD Raspbian image with auto-started self-tests, and with every StereoPi factory do the next things:
- Insert MicroSD with test software
- Insert Raspberry Pi Compute Module 3 Lite
- Connect two cameras
- Connect Ethernet cable from StereoPi to router
- Connect HDMI monitor
- Connect USB device
- Turn power on
Script runs on start checks all things like camera modules, USB, Ethernet. If every subtest was successful script shows “green” result. So we obtained “green” for all 20 pieces.
- All these 20 pieces have been packed and sent to us. We plan to receive it on the next week. Here’s the photo of final sample and batch packed in a box:
The last thing we plan to do is to make aggressive tests in our lab. If aggressive tests will not show any things needed to be patched, then it means we have approved hardware design and ready to press “start” button for production right after successfully crowdfunding. We do our best now to start crowdfunding campaign in the nearest few weeks!
12/25/2018 at 10:08 •
A friend of mine hosts a VR club and asked me if it’s possible to make a 3rd person view in a real life. Thus, we decided to conduct another experiment using our StereoPi (a stereoscopic camera with Raspberry Pi inside).---------- more ----------
My friend showed me some screenshots to clarify what it should look like:
After several days to compile mechanical parts and write some code for video livestream to Oculus Go, our team created a prototype of this 3rd person camera view.
I went to a friend’s office party on Friday and offered for his colleagues to take part in the first tests of this system. The results were really impressive! Here are some interesting moments:
What’s “under the hood”?
We used StereoPi v 0.7 with Raspberry Pi Compute Module 3 Lite. For cameras, we chose Waveshare 160 degree cameras.
We 3D printed a simple case and created a laser-cut camera support plate:
To attach the camera to the person’s back, we created this construction from plastic tubes and colored it with liquid rubber:
On the StereoPi side, we created a simple application to capture videos in stereoscopic mode with raspivid. This application also supports autodiscovery functions, to automatically find and connect to applications (currently ones on Android and Oculus Go). To provide for easy adjustment of settings, we created a simple admin panel, available over WiFi.
To stream video from the Oculus Go to the computer for observers, we used scrcpy-win64 and reconnected Oculus from a MicroUSB wire to a UDP connection. This allowed us to see the livestream on an external screen:
For the Oculus Go we used our Android application. It is not yet fully adjusted for Oculus, but it was enough for the first tests. This app uses the network to automatically find StereoPi, request access to its video, and begin livestreaming it to the user.
4. Some specific settings
To minimize latency, we set the camera to 42 FPS (maximum available on Raspberry Pi in stereoscopic mode without overclocking) with 1280x720 resolution. Bitrate was set to 3 Mbit. With these settings, latency was around 100 ms.
As I mentioned previously, we used two wide angle cameras. In this case, we cropped the left and right images to keep the aspect ratio that people are comfortable with. When using cameras on drones, we usually compress the images horizontally to maintain the original FOV; however, this time we planned to show it to people untrained in FPV flying.
There was no stereoscopic calibration or on-the-go rectification of stereoscopic video. We just tuned the cameras’ axes to be parallel as precise as possible and livestreamed it “as is”. We added this software to our to-do list for future additions.
The tests worked out very well. All of the testers left in a good mood and with new experiences. The first several seconds, it was best to support the users, to prevent them from falling until their perceptions adapted to the new reality
As for the hardware part — StereoPi met our expectations as a quick prototyping tool in this case as well. This, therefore, continues to prove its usefulness in such areas.
If you would like to know more about the developments in StereoPi production, and to take part in our upcoming crowdfunding campaign, you can subscribe to updates on our pre-launch page here: https://www.crowdsupply.com/virt2real/stereopi
12/25/2018 at 09:55 •
If you use ROS when creating robots, then you probably know that it supports utilization of stereo cameras. For example, you can create a depth map of the visible field of view, or make a point cloud. I began to wonder how easy it would be to use our StereoPi, a stereo camera with Raspberry Pi inside, in ROS. Earlier, I’d tested and confirmed that a depth map is easily built using OpenCV; but I had never tried ROS - and so, I decided to conduct this new experiment, and document my process of looking for the solution.---------- more ----------
1. Does ROS for Raspberry Pi exist?
First, I decided to find out if it’s even possible to create a ROS for Raspberry Pi. The first thing that came up on a Google search was a list of instructions for installing various versions of ROS onto Raspberry Pi. This was great - I already had something to go off of! I well remembered how long it took to assemble OpenCV for Raspberry (about 8 hours), so I decided to look for ready-made versions of MicroSD cards to save me some time.
2. Are there any ready-made MicroSD cards with ROS for Raspberry?
Apparently, this issue has also already been solved by several teams of engineers. If you don’t count the one-time creations by enthusiasts, there were 2 types that were consistently renewed with new versions of OS and ROS.
The first type was an ROS installed onto the native OS Raspbian, from the team ROSbots.
Here’s a regularly updated link.
The second was images by Uniquity Robotics on Ubuntu.
And so, the second question was also quickly solved. It was time to dive deeper.
3. What’s the setup for working with a Raspberry Pi camera on ROS?
I decided to check which stereo cameras had ready-made drivers for ROS via this page: http://wiki.ros.org/Sensors
Here, I found 2 subsections:
2.3 3D Sensors (range finders & RGB-D cameras)
It turned out that the first subsection listed not only stereo cameras, but also TOF sensors and scanning lidars - basically, everything that can immediately provide 3D information. The second was the one with the bulk of the stereo cameras. An attempt to look for drivers for several stereo cameras didn’t bring me any more joy, as it hinted at a gruelling amount of code.
Alright, I decided. Let’s take a step back. How does just one Raspberry Pi camera work in ROS?
Here, I was greeted by 3 pleasant surprises:
- Apparently, there exists a special node for ROS called a
raspicam_node, specialized for work with Raspberry Pi.
- The sources of this node are uploaded to github, and the code is regularly maintained and well documented: https://github.com/UbiquityRobotics/raspicam_node
- The creator of the node, Rohan Agrawal (@Rohbotics) works for a company which actively maintains one of the ready-made images for Raspberry Pi
I looked over the github repository
rasoicam_nodeand checked the issues section. There, I discovered an open issue called “stereo mode”, almost 7 months of age, without any answers or comments. In this, more or less, the rest of the story unfolded.
4. Hardcore or not?
To avoid asking the authors any childish questions, I decided to check the sources and see what an addition of stereo mode entails. I was mostly interested in this C++ subsection: https://github.com/UbiquityRobotics/raspicam_node/tree/kinetic/src
It turned out that the driver was coded at the MMAL level. I then remembered that the implementation code for stereoscopic mode was open-source, and readily available (you can find the implementation history here on Raspberry Pi forum); the task of coding a full stereoscopic driver for ROS was doable, but sizable. Furthermore, I looked at other stereoscopic cameras’ driver descriptions and found out that the driver needed not only to publish the left and right images, but also to implement separate calibration parameters for each camera and do a lot of other things. This stretched the experiments out to one or two months. After considering this, I decided not to put all my eggs in one basket: I would split up my efforts, asking the author about support for stereo, and meanwhile trying to find a simpler, but functional solution on my own.
5. Conversations with the author
In a github thread about stereo mode I asked the author a question, mentioning that stereo has been supported by Raspberry Pi since way back in 2014, as well as suggesting to send him a development board if it was needed for any experiments. Remember, at this point I still doubted that in this distribution kit the stereo would work as it did in its native Raspbian.
To my surprise, Rohan answered quickly, writing that their distribution uses a Raspbian kernel, and so everything should work fine. He asked me to test this on one of their builds.
A Raspbian kernel! Now I don’t have to sell my soul to capture a stereo image!
I downloaded and opened their latest version via a link from Rohan, and launched a simple python script for capturing stereoscopic images. It worked!
After this response, Rohan wrote that he’ll check the driver’s code for stereoscopic mode support, and asked several questions. For example, our stereo mode outputs one connected image, but we needed 2 halves - a left and a right. Another question was about the calibration of parameters for each camera.
I responded that for the initial stages, we could just download the images from the cameras individually. Of course, this would leave them unsynchronized in terms of image capture time and color/white balance settings, but for a first step it would serve the purpose just fine.
Rohan released a patch which permitted the user to select, in ROS, which of the cameras to pull images from. I tested this - the camera selection worked, which was already an excellent result.
6. Unexpected help
Suddenly, a comment from a user named Wezzoid appeared in the thread. He described his experiences in creating a project based on a stereoscopic setup with Pi Compute 3 and a Raspberry Pi devboard. His four-legged walking robot was able to track the location of an object in space, change the position of cameras, and stay a set distance away from him. Here's the Hackaday project itself by @Wes Freeman
He shared the code with which he was able to capture an image, cut it into two halves using python tools, and publish it as separate nodes of the left and right cameras. Python wasn’t the fastest in these situations, so he used a low resolution of 320x240, as well as a good lifehack. If we captured the stereo image side-by-side (one camera on the left on the image, and one on the right), python was forced to cut each of the 240 rows in half. However, if the image was stitched together in top-bottom format (the left camera on the top half of the image, and the right on the bottom), then python would cut it in half in just one operation - which Wezzoid had successfully accomplished.
Plus, he published the python code he used to execute this process on Pastebin.
7. Launching the publication of the left and right camera nodes
Upon first launch, the code proclaimed that it wasn’t able to access the YML files with the camera parameters. I was using the Raspberry cameras V2, and remembered that on github, in addition to raspicam_node, there were files with lists of the results of calibration for various camera models: https://github.com/UbiquityRobotics/raspicam_node/tree/kinetic/camera_info . I downloaded one of them, made two copies, and saved them as left.ymland right.yml after adding to them the camera resolutions from the author’s code. Here’s the code for the left camera as an example.
For the right camera, the camera name is changed to right, and the file is renamed right.yml; other than that, the file is identical.
Since I didn’t plan on creating a complex project, I didn’t recreate the author’s lengthy paths and subfolders, and instead simply placed the files in the home folder next to the python script. The code successfully started up, outputting status updates into the console.
All that remained was to check what the right and left cameras ended up publishing. To view this, I launched
rqt_image_view. The selections /left/image_raw and /right/image_raw appeared in the drop-down menu; when I chose them, I was shown the separate feeds from the left and right cameras.
Fantastic, this thing worked! Now it was time for the most interesting part.
8. Looking at the depth map
For the depth map, I didn’t attempt to create my own approach, and instead followed the basic ROS manual for setting up stereo parameters.
From that, I was able to figure out that it would be easiest to publish both nodes in a specific namespace, and not at the root as Wezzoid had done. After some tweaks, lines from the old code such as
left_img_pub = rospy.Publisher(‘left/image_raw’, Image, queue_size=1)
began to look more like this:
left_img_pub = rospy.Publisher(‘stereo/left/image_raw’, Image, queue_size=1)
We then launch the stereo processing node, stereo_image_proc:
ROS_NAMESPACE=stereo rosrun stereo_image_proc stereo_ige_proc
And of course we want to see the result, so we launch the viewer:
rosrun image_view stereo_view stereo:=/stereo image:=image_rect_color
Finally, to configure parameters of the depth map, we launch the configuration utility:
rosrun rqt_reconfigure rqt_reconfigure
In the end, we see the image embedded at the beginning of this article. Here’s a zoomed in screenshot:
I published all of the files for this program on github: https://github.com/realizator/StereoPi-ROS-depth-map-test
9. Plans for the future
After I published my results, Rohan wrote “Very cool! Looks like I am going to have to pick up a StereoPi”. I then mailed him the board. Hopefully, this will make it easier for him to tweak and debug this new acquisition into a full-fledged stereo driver for ROS and Raspberry.
It’s possible to create a depth map from a stereo image using ROS on StereoPi with Raspberry Pi Compute Module 3 inside, and in fact in several ways. The path we selected for quick testing isn’t the best in terms of performance, but can be used for basic application purposes. The beauty lies in its simplicity and ability to immediately begin experiments.
Oh, and fun fact: after I had already published my results, I noticed that Wezzoid, who had suggested the utilized solution, had actually been the author of the initial question about the publication of two stereo images. He asked it, and he himself resolved it!
- Apparently, there exists a special node for ROS called a