They are two main components in the system : the cameras and the gateway.
The cameras are based CMUcam5 from Charmed Labs which is a position tracking camera which can output 2D coordinates.

I have tested almost every alternatives, and the first prototypes were even based on Raspberry Pi camera, and the 2D position tracking was made using OpenCV image library on the Raspberry. I have also been working with the wiimote sensors and many standard HD webcam with FreeTrack. None of them did the job by a long shot !

You can order this camera on Amazon for 69$.

• Multi point tracking (great for multiple users)
• 640*480 @ 50FPS
• 1280*800 @ 30FPS
• Lens field-of-view: 75 degrees horizontal, 47 degrees vertical
• Voltage : 5 to 12V
• Power consumption : 140mA
• Open Source (hardware and firmware)
• Removable optics

Let's get started, this camera is nice, but we have to add some stuff to make it great. Here is what we are going to do :

Get to know your camera

An important part is to understand how your camera works and how it see the outside world and how it talks about it.
I am not going to write a lot about it as the guys from CharmedLabs did a great job at explaining all about their product !

Here is the Main Website

Technical specifications

Technical documents

And her is the Quick Start page to setup your camera for its first use.

Play around with the camera, connect it to your computer and try to get an image.

Now you are supposed to know enough about the camera for the following steps. It is especially important to make it wireless.

Remove the IR blocking filter

As you have seen on the video above, the camera tracks objects based on their colors. After a little testing, I found out that the color tracking is not good enough for our application, therefore we have to improve the tracking. Here is what we will do : instead of tracking color, we are going to track an infrared light. Infrared light is a light with a wavelength around 940nm, that can’t be seen by humans. Infrared is commonly used in remotes for examples.

Almost any camera can see infrared even though our eyes can’t. Unfortunately, in most cases we don’t want to see infrared with a camera, so the lens contains an infrared blocking filter. The CmuCam 5 has such a filter in its lens. We will have toremove that filter !

Different methods can be used to remove IR filters, such as eating the lens until the glue melts. In my case I took a screwdriver, broke the lens and remove the small pieces of glass left.

Here is how to do it step by step :

1. Unscrew the lens from the camera
2. Identify the IR filter. It is on the sensor side of the lens. It has a red-ish reflection like so :
3. Break the filter with a small screwdriver. Don’t go to far, you would broke the others lenses. Remove the small pieces of glasses left.
4. You’re done !

Add an IR pass filter

Now your camera can see usual light and IR light. As we are only going to track IR, we could add a filter that removes usual light, called “infrared pass filter“. With this filter, the camera will see infrared and only infrared ! The tracking will be much better !

I am sure you are now thinking “Damn, where am I going to find such a filter ?”, well it used to be easy 15 years ago, today it’s going to be a little more complicated… Here is what you have to find :

Because when you open a floppy disk, you see that round plastic thing on the next picture ? Well it can act an IR pass filter ! You just have to cut a small round in this disc with a pair of scissor and put it between the camera sensor and the lens, or at the end of lens.

Update the firmware

The firmware inside the camera is made to detect colors, not white (IR) on black. So we have to update that firmware. Thanks to IR-LOCK team, a special firmware has already been made. Those guys use a Pixy cam too on their drones to make a target tracking system using infrared, for automated landing and much more ! You can also order a Pixy Cam right from their website with the right lenses and filter and a bunch of infrared LEDs If you prefer for 99$. I wouldn’t personally recommend it if you are low on budget, but if you want to gain some time, it could be a solution.

To download, install and configure the new firmware, please follow instructions 2 – 6 on IR-LOCK Getting Started page.

Now you can start playing around with infrared tracking

Build a case

This step is not necessary, but I am going to give you all the steps and files to make a nice case for your camera and wireless module (we’ll talk about that a bit later).

The case is very easy and low cost to make. It is not the most beautiful case ever, but it will be practical and won’t require any tools (except a screwdriver).

The case is made of two pieces of plastic, laser cut, assembled with a few screws.

1. Go on SeeedStudio Laser cutting service
2. Upload the following files Case CAD (you have to do it twice, once for the bottom and once for the top)
3. Finish the order (login, shipping…) and should have 5 cases delivered to your home for less than 20$ !

Once received, you just have to find some long screws and assemble everything !

Note : If you are planning to order on SeeedStudio, you might want to order the PCB for the wireless transceiver at the same time. So checkout the next article on this subject.

In the next step, we will make an electronic circuit to make the camera wireless. I’ll give you all the files and instructions to build it yourself ! It is in another post as this is pretty long…

Let's sum up what we have done so far :

• we ordered a pair of color tracking cameras
• we modified them into Infrared tracking cameras
• we updated their firmware to track Infrared and tested them
• we built a small case to give them a nice look

There is one last modification we have to bring to the cameras, and that modification is a game changer : we are going to make them wireless !

Of course we won't send the video, but the coordinates of the points the camera is tracking. In this article we are going to connect an ESP8266 Wifi shield to the CmuCam to send coordinates over Wifi.

Here is what we are going to do, step by step. All you need to know for this part is how to program an Arduino (I have seen 8 years old doing it so...I'll give you a link just in case).

The basics

Here we'll go through all you need to know to make it out of this tutorial alive.

First if you don't know what is Arduino, please read about it here : Arduino.

We are not going to use a regular Arduino board but an ESP8266, which has been made compatible with Arduino. To make it easier, we will start with a NodeMCU dev board which includes an ESP12E (which includes an ESP8266 ^^) and a USB programmer :

You will need one per camera, but luckily you can find them starting at 3.28$, shipping included on Aliexpress.

Connect NodeMCU to CmuCam

Here is how to wire the CmuCam to your brand new ESP8266 :

Program the ESP8266

We won't go through "How do we install Arduino IDE", if you are still reading this then I'm sure you'll be smart enough to know how to use Google ;)

For this part first go here to learn how to use Arduino with an ESP8266 and configure your Arduino IDE.

You should install the following library for the camera (please use my version otherwise it won't work with the ESP8266 : Pixy ESP8266 library (more documentation on the library here).

Make sure to download (from the Arduino library manager) all the libraires required for Wifi connection.

And here is the code you have been waiting for. Here is what is does :

1. Turn the ESP8266 into an Access Point so you can connect to it and enter your Wifi informations
2. Try to auto update from a server
3. Start communication with the camera, and broadcast datas over Websocket (it acts as a websocket server)

Please note that it is a very early version for tests purposes. Major fixes and improvements are yet to come.

https://codebender.cc/sketch:207000

Behind the curtains : calculating 3D position by image resectionning

The system uses OpenCV to calculate the 3D position form multiples 2D positions.

First a calibration sequence is required. The calibration consists of recording some 2D positions with the cameras and associate them to real 3D position. I’ll make a quick video about that. You have to recalibrate every time you move the cameras of course.

With the calibration and some OpenCV functions, we are then able the deduce the 3D position of each camera. With that position and we will be able to get a projection matrix and then do some triangulation !

Yes, to sum up we are going to do some regular triangulation !

But first we have to create a Unity environnement and import OpenCV. To do so we will use EmguCV, which is a cross platform OpenCV wrapper, compatible with PC, Mac OS, Iphone, Android... all we need for our applications.

Then we need to communicate with the cameras using websockets.

Finally we have to create a calibration sequence using OpenCV and add the tracking function. All of this had been done in the previous version using a Gateway, but I now have to translate all the code from python to C# and clean it of course.

To be continued...