Overview

My investigations into the camera start with frame rate. Motion capture benefits from high frame rates because it makes it easier to identify markers from frame to frame. I'd be happy with 60fps, but I know from reading the camera docs that it is capable of more, in fact the docs state that 90fps is achievable at a resolution of 1280x720. The full camera resolution is 3280x2464 which is really nice, but unfortunately the max is 15fps in that mode - if you can even get the Pi to handle that much video data.

So is 1280x720 enough? The resolution required will depend on the size of the markers and the range they are seen from the camera. This might be a good time to talk about the capture volume, a space where the recording process takes place. I like the idea of being able to capture in relatively small rooms, say 4x4 meters. This makes it possible to use a home office or another room in your house to do the recording.

Exposure time is also very important to consider. This determines how long the shutter of the camera is open, and therefore how blurred a marker will appear depending on its speed. As a marker moves faster it will begin to look more stretched. This is not necessarily a crippling problem, but having the shortest possible exposure will definitely assist in the marker analysis phase.

Active markers

This all points to having as much light coming from the markers as possible. The more light, the shorter the required exposure. I'm going to be using regular old 5mm LEDs as the markers (This won't be the final solution, but it is an interesting experiment).

I didn't have any, so I went out and bought a bunch of different LEDs. It also might be an idea to use different coloured LEDs to help with identifying which marker is which.

I powered up a few LEDs with coin cells. I can see attaching many of these to the actor could be a pain, especially managing batteries as they go flat.

Testing

I started with the 1280x720 mode running at 90fps, and the Pi was very happy with that, it had no problem recording and storing the video data at that speed. This means I've already achieved better than the 60fps goal I had in mind. But can it do better? Turns out, yes, you can push the camera all the way to 120fps! However, the higher you go the more the RPi struggles to process and shift the video data around. I'm not sure exactly where the bottleneck is but it looks related to the transfer from the GPU to the CPU. I don't think bandwidth is the issue, but simply the RPi CPU can't cope with the amount of data. This results in frames occasionally being dropped. There is a way to relieve this pressure, and that is to slightly reduce the resolution. There will always be unusable area in the image and why not cut that completely from the pipeline instead of throwing it out later. The RPi camera lets you specify a resolution and region that you want to capture. The nice thing about this is that those pixels will never make it through to the CPU, meaning that you will have less overall data. I found 1000x700 was a nice compromise, but until final lenses are in place this will be up for debate.

Here is a video of me waving one of the LED markers as fast as I can while capturing at 120fps. I want to simulate the fastest possible human motion, so that I get an idea of the worst case exposure issues.

You can see that the 120fps is quite effective, but if you pause the video you will notice that the LED has been stretched quite significantly.

What happens if the exposure is lowered by 75%?

The trail is now 75% shorter, but the rest of the image is much darker. Ironically this just helps out further when it comes to marker processing, since anything other than the markers is useless data.

Here is another test with two of the coin cell powered LEDs attached to my hands.

This is very promising, the markers show up really nicely and there is very little blurring due to the low exposure time. If you look at a single frame that has had a simple threshold applied (image below) it becomes clear that locating the markers in the image is very straightforward. But can the RPi process these images in real time at 120fps? (Spoiler: It can)

Conclusion

120fps is an absolutely fantastic result, already this is a system that will beat off the shelf webcams. I'm thrilled with the performance of the Pi camera. It definitely doesn't compare with modern mobile cameras that can do 1080p at 240fps, but there isn't much better for those in the maker space that don't have special access to 10000 per order camera components.