First, please enjoy the Holokart demo video so you can see this in action! It's a real life version of Mario Kart, where you get into a physical (as opposed to virtual) go-kart, and put on a Microsoft Hololens augmented reality visor. As you race around the track, you'll see holographic power-ups that you can collect by driving through them... and thanks to some electronics I've added to the go-karts, those virtual power-ups can actually boost the speed of your real-life go-kart, or slow the speed of your competition (depending on which power-up you get).
Since the hackaday crowd is an intellectually curious group, here are some behind-the-scenes details on the technology here.
One thing I've learned the Microsoft Hololens does insanely well is inside-out SLAM (simultaneous localization and mapping). If you're not familiar, this is a concept from autonomous robots where there are a variety of sensors and as the device moves around it creates a map of its surroundings and keeps track of where it is in those surroundings. The Hololens uses visible light cameras, an interial measurement unit, and also a structured grid of infrared light (along with an IR camera) and combines the data coming back from all of those sensors to produce an astoundingly accurate map of the environment despite the possibility of some error in one of those sensor streams. In order to maximize the Hololens' ability to track its location at "high speed" (10-15 miles per hour), we placed posterboards around the track with black rectangular stickers on them. The sharp corners of those stickers are the visual "features" that visual SLAM algorithms use to track movement. In the final production system, those posterboards will be replaced by cardboard standups of cars, racers, audience members, checkered flags... things that fit the racing theme and have a purpose (track decoration) beyond just providing sharp features for the Hololens to track. Those cardboard cutouts would be 3 as dimensional as possible, so that the grid of IR light and IR camera has good three-dimensional features to detect as well. I am confident that those improvements, along with switching to the Hololens 2 and improving lighting conditions at the track, will significantly reduce or eliminate the "lag" that you see in the video, where the virtual characters lag behind the racers by a few frames.
The coolest part of this project, in my opinion, is the interaction between the real-life go-karts and the virtual Hololens game. For example, if you use a speed-boosting power-up (the catchup and mustard "jetpack" you see in the demo video), your go-kart will speed up for a set amount of time. Likewise, if you drop an obstacle (the bowling pins you see in the demo) and an opponent runs into it, their go-kart will stop for a moment, allowing you to get ahead of them in the race. This is accomplished through remote control. Modern electric go-karts have remote controls that the venues use for example to stop all racers in the event of an accident, or to slow down or speed up karts depending on the ability of the drivers. The go-kart system that I implemented this with uses a 433 Mhz OOK system that was straightforward to decode. I implemented a 433 Mhz transceiver that was attached to the game server using USB serial, that allows the game server to selectively slow down or speed up karts in order to make the power-ups actually control the speed of the karts.
The remaining issue was how can the user activate the power-up? I added an "activate power-up" button to the go-karts. This button is connected to an arduino with a wifi shield, so it can tell the game server when a user has pressed the "activate" button. Each arduino was programmed so that it knows which kart it's in, and the hololens units know which kart they are associated with because of a "game start" procedure that involves pointing the Hololens at an identifying QR code on the back of the kart's electronics enclosure.
Lastly, I should address a common question. If you google "real life mario kart" you'll find some projects by other people. Here are those projects, along with what I wanted to do differently.
1. Virtual Reality Mario Kart - This totally beautiful project by Masters of Shape (amazing mixed media arts & tech group) uses virtual reality goggles, so you can't see your real-world surroundings at all. Personally I get nauseous easily in virtual reality goggles, so I wanted to do something similar with augmented reality. Also for tracking the kart, they use what's called outside-in tracking using IR beacons on the kart and an expensive grid of cameras on the ceiling, which was a 6-figure line item the last time I got a quote on the same system. The Hololens uses inside-out tracking, so venues would just have to pay for the headsets and not a separate tracking system. Still, let it be clear that I have a lot of respect for what Masters of Shape has done.
2. BattleKart - This creative project from a European company uses a grid of projectors on the ceiling and the same IR beacon system that Masters of Shape uses. Since this group uses projectors, users don't need to wear headsets. That makes things faster for the venue (putting headsets on people takes time), although personally the effect of projecting things on the ground isn't as exciting as the 3D holographic effect you get with the Hololens. Also, the outside-in tracking is a 6-figure line item on the budget, that you can remove if you use the Hololens.
3. HadoKart - In this project from Japan, users wear Hololens devices but race around in rather slow and uncomfortably small go-karts. See this video from an amusement trade show to see what I mean. They have promotional videos you'll find on YouTube where the video appears to be artificially sped up, or the holograms look like they were added in post-production. I really like the concepts for some of the games this group has come up with, but I wanted to do this with faster and more comfortable go-karts.
4. Mushroom Rally and MariKart - I hesitate to include these in this list, but people keep sending me links to these groups. The idea here is that you put on costumes of video game characters (like Super Mario) and then ride in go-karts. There's no AR or VR... no power-ups... no game really other than racing go-karts while wearing costumes. I think that sounds fun, and I'd like to try it... but I think adding holograms and speed-boosting power-ups was a worthwhile pursuit.