StereoPi has been successfully crowdfunded! https://www.crowdsupply.com/virt2real/stereopi Key features: - Supports Raspberry Pi Compute Module 1 and 3 / 3 Lite - Raspbian support out of the box - Support two cameras - Small size - Open source
StereoPi is designed to be friendly tool for experiments and quick prototyping with all kind of video-related projects. It will definitely helps you to enjoy with: - Making 3D photos or record stereoscopic video - Experiment with 3D video livestream to 3D helmets like Oculus Go or Internet - Build computer vision systems and work with OpenCV - Make a robots with ROS onboard - Prototyping 360 degree photo and video solutions - Creating AR/VR project - Livestream from your drone or robot in stereo mode or from two independent cameras
The goal of this series of articles is to create a compact indoor robot that can navigate using stereo vision. As a platform for it, we’ll use a small Dagu RP5 platform on tracks that we have. Here’s how it looks like next to the StereoPi.
Today we’re pleased to share with you a series of Python examples for OpenCV development. This code works with either the StereoPi or the Raspberry Pi Development Board, as both support using two cameras simultaneously. Our ready-to-use code (and also Raspbian image) will help you every step of the way, from the first image capture to the Depth Map created via real-time video capture.
We would like to emphasize that all of these examples are for those new to OpenCV and are not intended for production use. If you are an advanced OpenCV user and have worked with the Raspberry Pi before, you’ll know it’s better to use C/C++ (instead of Python) and to utilize the GPU for better performance. At the end of this article we’ll provide some notes regarding the various bottlenecks we experienced using Python.
Here is our hardware setup:
We used the StereoPi board with Raspberry Pi Compute Module 3+. Also two Raspberry Pi cameras V1 connected (based on ov5647 sensor).
Raspbian Stretch (kernel 4.14.98-v7+)
OpenCV 3.4.4 (pre-compiled, ‘pip’ from Python Wheels)
The software installation process is beyond the scope of this article but we have prepared a Raspbian image with all software installed. Here is a link to our GitHub stereopi-tutorial repository.
All scripts support key stroke processing, and you can press ‘Q’ key to stop them. If you use Ctrl+C to stop the script, it may break the Python interaction with the cameras. In this case, you will need to reboot StereoPi.
Step 1: Image Capture
We use 1_test.py script for this purpose.
Open the console and go to our examples folder:
After starting the script you can see a preview window with the stereoscopic video. Pressing ‘Q’ will stop the process and save the last captured image. This image will be used in the next scripts for Depth Map parameters tuning.
This script allows you to check if your hardware is operational and helps you obtain your first stereoscopic picture.
The following video shows how the first script works:
Step 2: Collecting Images For calibration
In an ideal world, a perfect depth map needs to use two identical cameras with their optical, vertical and horizontal axis all in parallel. In the real world, however, cameras are different and it’s impossible to align them perfectly. Thus, a software calibration method is used. Using two cameras you take multiple photos of an object. In our case, we used a printed chessboard. A special algorithm will then analyze these photos and find parameters for correction. This script begins this process by capturing a series of chessboard photos for calibration. Before each photo, the script starts a five (5) second countdown. Five seconds is generally enough time to reposition the chessboard. Make sure it can be seen by both cameras and ensure it’s stable to avoid “blurred” photos. The default number of photos captured per series is 30.
At the end, we have 30 stereoscopic photos, saved in /scenes folder.
Step 3: Image Separation
The third script 3_pairs_cut.py separates the captured photos into “left” and “right” images and saves them in /pairs folder. These separations could be done on-the-fly, without saving, but this step is helpful for your next experiments. You can save image-pairs from different capture series. Use your own code to work with this images, or use another stereoscopic camera’s images by putting them in this folder.
This script will show you every stereo pair before it’s separated (and waiting for key press). This lets you find bad photos and remove them before the next...
As you know, all our Starter and Deluxe kits will include a microSD card with a ready-to-use Raspbian image so you can repeat all of our livestream experiments right out of the box. We’ve been busy polishing existing features and adding new ones to this image. In this update, we’ll share with you some new features and say a few words about our experiments with Skype and 3D video.
Latest StereoPi Livestream Playground (SLP) Image
Image size reduced from 5 GB to 860 MB
Video livestream to browser (2D and 3D)
Livestream to Android over USB cable (Android accessory support)
Bash console over web admin panel
File editor over admin panel
Access to video records over web admin panel
RTSP livestream support
MPEG-TS livestream support
Linux partition now takes 2 GB instead of 4 GB
FAT32 partition now created automatically on first boot
RPi 3B+ and CM3+ support (updated kernel)
Most settings are now in the /boot/stereopi.config file
You can download image file from one of these three mirrors:
And here is a screen capture of my iPhone and our first 3D Skype call:
As the iOS screen recorder does not record sounds, I added music to it.
To get all these things to work, we used two tricks. First, we used mic from Logitech webcam connected to the same computer to provide the audio to Skype, since the OBS-VirtualCam cannot emulate a sound device, so sound from StereoPi’s microphone isn’t available to Skype.
Second, we avoid a one-second delay between audio (due to an internal OBS video buffer) by first streaming the video from StereoPi to gstreamer on Windows and then pointing OBS to use the gstreamer window as its video source, which resulted in a delay of only about 100 milliseconds.
This test shows it is possible to use stereoscopic livestream with a lot of common software, like Skype and other video-related programs, all without modification since they already work with a traditional camera.
If you have classic Raspberry Pi with the camera, you can repeat all our video livestream experiments. Livestream to YouTube, Android and Oculus Go. Also you can repeat our behind-the-scene experiments with video livestream to WIndows desktop, Mac or any RTMP server.
A friend of mine hosts a VR club and asked me if it’s possible to make a 3rd person view in a real life. Thus, we decided to conduct another experiment using our StereoPi (a stereoscopic camera with Raspberry Pi inside).
If you use ROS when creating robots, then you probably know that it supports utilization of stereo cameras. For example, you can create a depth map of the visible field of view, or make a point cloud. I began to wonder how easy it would be to use our StereoPi, a stereo camera with Raspberry Pi inside, in ROS. Earlier, I’d tested and confirmed that a depth map is easily built using OpenCV; but I had never tried ROS - and so, I decided to conduct this new experiment, and document my process of looking for the solution.