VrTracker uses at least two infrared camera to track an infrared dot on the user headset, and computes its position in a 3D space to allow the user to move around in a virtual environment.
Oculus Rift, VR One, Samsung Gear, Sony Playstation VR… All those Virtual Reality are great, but they are missing something ! They do not have a position tracking system ! This means you can look up down right left, but can’t stand up and move around in your room. Too bad because the best experiences happens when you can actually move around by yourself in your virtual environment.
How do I know ? Well I have been working with many of those headsets as a consultant for an augmented reality company, and I have developed a 3D position tracking system compatible with all those devices ! And let me tell you, I have tested this tracking system with Samsung Gears, and the result was AWESOME !!! I was able to move around in a 5 meters by 8 meters room and therefore in my virtual environment.
What is so great about position tracking ? The range of applications is much wider ! You can for example create an experience in a museum where users could visit a pyramid using VR headsets in an Egyptian exposition. Or visit an appartement or a building even before it has been built ! You could also create games with multiple users… The only limit is your imagination !
Now you might be wondering how this all works ! You can check out this short video for a quick introduction :
Here is very simply how the different components interacts with each others :
To sum up, the system is composed of at least 2 cameras (CMUcam5) that can track infrared light, an Arduino board with aNrf24L01 wireless transmitter to send the camera’s data over the air. A Raspberry Pi which acts as a gateway to calculate the 3D position from every 2D positions received from the cameras using OpenCV library. And finally an infrared emitter which is going to be tracked.
As the project is growing and I would like to release a Unity Asset so that people could start playing around with VR Tracker, I need to develop a software architecture that is stable and could easily expand to more features. Therefore, let's start with some UML to design the software :
If you are reading this, you might have understood by now that we are tracking an infrared light on the user's headset.
In the first version, the IR leds are simply powered by a small battery and always on. But now, what if we added a little Wifi (ESP8266) chip as there is on the camera to control them ?
What for ?
Well if the system is used with multiplayer or if there is some interferences due to sunlight, the wireless infrared tag could be "pinged" to ensure we are tracking the good tag. It would allow to track different users and even "objects".
Here is a big update for VR Tracker ! Now the Arduino-NRF24L01+ on every camera has been replaced with a WIFI chip, the ESP8266 (an ESP12E on NodeMCU board).
Why this choice ?
Well using Wifi allows to remove the gateway, you can access to your camera directly. Also the code can be updated wirelessly, which is very nice ! Moreover,, for future developments, it will allow a better flexibility.
Oh and BTW, the chip cost less than 4$ !
What is going to change ?
First we remove the Gateway. Therefore all calculations that used to happen in the Gateway must happen the PC / Smartphone. As we are going to use Unity to develop applications and games, we will have to include OpenCV in unity (which can be a bit tricky) and translate the code from python to C#.
Want a look at it ? Here you go :
How to plug it ? Here you go !
The code is coming soon, don't hesitate to ask for it ;)
They are two main components in the system : the cameras and the gateway. The cameras are based CMUcam5 from Charmed Labs which is a position tracking camera which can output 2D coordinates.
Why this camera ?
I have tested almost every alternatives, and the first prototypes were even based on Raspberry Pi camera, and the 2D position tracking was made using OpenCV image library on the Raspberry. I have also been working with the wiimote sensors and many standard HD webcam with FreeTrack. None of them did the job by a long shot !
Let's get started, this camera is nice, but we have to add some stuff to make it great. Here is what we are going to do :
In another post, we will see how to make it wireless, this one is already long enough !
Get to know your camera
An important part is to understand how your camera works and how it see the outside world and how it talks about it. I am not going to write a lot about it as the guys from CharmedLabs did a great job at explaining all about their product !
Play around with the camera, connect it to your computer and try to get an image.
Now you are supposed to know enough about the camera for the following steps. It is especially important to make it wireless.
Remove the IR blocking filter
As you have seen on the video above, the camera tracks objects based on their colors. After a little testing, I found out that the color tracking is not good enough for our application, therefore we have to improve the tracking. Here is what we will do : instead of tracking color, we are going to track an infrared light. Infrared light is a light with a wavelength around 940nm, that can’t be seen by humans. Infrared is commonly used in remotes for examples.
Almost any camera can see infrared even though our eyes can’t. Unfortunately, in most cases we don’t want to see infrared with a camera, so the lens contains an infrared blocking filter. The CmuCam 5 has such a filter in its lens. We will have toremove that filter !
Different methods can be used to remove IR filters, such as eating the lens until the glue melts. In my case I took a screwdriver, broke the lens and remove the small pieces of glass left.
Here is how to do it step by step :
Unscrew the lens from the camera
Identify the IR filter. It is on the sensor side of the lens. It has a red-ish reflection like so :
Break the filter with a small screwdriver. Don’t go to far, you would broke the others lenses. Remove the small pieces of glasses left.
You’re done !
Add an IR pass filter
Now your camera can see usual light and IR light. As we are only going to track IR, we could add a filter that removes usual light, called “infrared pass filter“. With this filter, the camera will see infrared and only infrared ! The tracking will be much better !
I am sure you are now thinking “Damn, where am I going to find such a filter ?”, well it used to be easy 15 years ago, today it’s going to be a little more complicated… Here is what you have to find :
Yes ! A goddamn floppy disk !!!
Because when you open a floppy disk, you see that round plastic thing on the next picture ? Well it can act an IR pass filter ! You just have to cut a small round in this disc with a pair of scissor and put it between the camera sensor and the lens, or at the end of lens.
Update the firmware
The firmware inside the camera is made to detect colors, not white (IR) on black. So we have to update that firmware. Thanks to IR-LOCK team, a special firmware has already been made. Those guys use a Pixy cam too on their drones to make a target tracking system using infrared, for automated landing and much more ! You can also order a Pixy Cam right from their website with the right lenses and filter and a bunch of infrared LEDs If you prefer for 99$. I wouldn’t personally recommend it if you are low on budget, but if you want to gain some time, it could be a solution.
Upload the following files Case CAD (you have to do it twice, once for the bottom and once for the top)
Finish the order (login, shipping…) and should have 5 cases delivered to your home for less than 20$ !
Once received, you just have to find some long screws and assemble everything !
Note : If you are planning to order on SeeedStudio, you might want to order the PCB for the wireless transceiver at the same time. So checkout the next article on this subject.
In the nextstep, we will make an electronic circuit to make the camera wireless. I’ll give you all the files and instructions to build it yourself ! It is in another post as this is pretty long…
Let's sum up what we have done so far :
we ordered a pair of color tracking cameras
we modified them into Infrared tracking cameras
we updated their firmware to track Infrared and tested them
we built a small case to give them a nice look
There is one last modification we have to bring to the cameras, and that modification is a game changer : we are going to make them wireless !
Of course we won't send the video, but the coordinates of the points the camera is tracking. In this article we are going to connect an ESP8266 Wifi shield to the CmuCam to send coordinates over Wifi.
Here is what we are going to do, step by step. All you need to know for this part is how to program an Arduino (I have seen 8 years old doing it so...I'll give you a link just in case).
Here we'll go through all you need to know to make it out of this tutorial alive.
First if you don't know what is Arduino, please read about it here : Arduino.
We are not going to use a regular Arduino board but an ESP8266, which has been made compatible with Arduino. To make it easier, we will start with a NodeMCU dev board which includes an ESP12E (which includes an ESP8266 ^^) and a USB programmer :
You will need one per camera, but luckily you can find them starting at 3.28$, shipping included on Aliexpress.
Connect NodeMCU to CmuCam
Here is how to wire the CmuCam to your brand new ESP8266 :
Program the ESP8266
We won't go through "How do we install Arduino IDE", if you are still reading this then I'm sure you'll be smart enough to know how to use Google ;)
For this part first go here to learn how to use Arduino with an ESP8266 and configure your Arduino IDE.
You should install the following library for the camera (please use my version otherwise it won't work with the ESP8266 : Pixy ESP8266 library (more documentation on the library here).
Make sure to download (from the Arduino library manager) all the libraires required for Wifi connection.
And here is the code you have been waiting for. Here is what is does :
Turn the ESP8266 into an Access Point so you can connect to it and enter your Wifi informations
Try to auto update from a server
Start communication with the camera, and broadcast datas over Websocket (it acts as a websocket server)
Please note that it is a very early version for tests purposes. Major fixes and improvements are yet to come.
Behind the curtains : calculating 3D position by image resectionning
The system uses OpenCV to calculate the 3D position form multiples 2D positions.
First a calibration sequence is required. The calibration consists of recording some 2D positions with the cameras and associate them to real 3D position. I’ll make a quick video about that. You have to recalibrate every time you move the cameras of course.
With the calibration and some OpenCV functions, we are then able the deduce the 3D position of each camera. With that position and we will be able to get a projection matrix and then do some triangulation !
Yes, to sum up we are going to do some regular triangulation !
But first we have to create a Unity environnement and import OpenCV. To do so we will use EmguCV, which is a cross platform OpenCV wrapper, compatible with PC, Mac OS, Iphone, Android... all we need for our applications.
Then we need to communicate with the cameras using websockets.
Finally we have to create a calibration sequence using OpenCV and add the tracking function. All of this had been done in the previous version using a Gateway, but I now have to translate all the code from python to C# and clean it of course.