Immersive VR on Rpi to control a Hero with sensors

This project will give you full gesture control of any virtual Hero you like with a simple set of inexpensive devices using sensors on RPi

Similar projects worth following
Welcome to "Be Your Hero" project!

I hope you are ready to enter the next generation of Virtual Reality immersion!

This project will give you full gesture control of any virtual Hero you like with a simple set of inexpensive devices using sensors. All the data collected is wirelessly sent to a computer and will display your favorite hero on a normal screen or an a DIY HD virtual reality headset.

I spent a lot of time on this project to provide the most cost efficient solution. The resulting embedded device is really inexpensive, surprisingly reliable and comes in a very small package. In this tutorial you'll have all the information you need to develop the sensors, the Bluetooth communication, to build the VR headset, import your Hero from Blender and develop your own 3D immersive game!

So, if like me you spent your childhood dreaming about becoming a Jedi or a Super Sayan, follow the guide! Your first light saber training and Kamehameha is on its way :-)

How Does It Work?

So, how are we are going to build real time detection and interaction between the movements of our body and the movements of our Hero's?

This project uses a combination of bracelets with 6-DOF sensors and Wii Nunchuck that you can easily attach to your body. Those sensors can detect "6 Degrees Of Freedoms" with a 3 axes accelerometer and a 3 axes gyroscope.

The central system of each of the embedded devices is a Arduino FIO powered by a Lipo battery. The communication is provided by a NRF24 Bluetooth board which has the ability to modulate the transmission channels. Thanks to this device, we have a large number of modules simultaneously communicating with a central with the reliable Bluetooth stack.

The Bluetooth receptor is wired to an Arduino Micro. Within this particularly impressive board there is an atmega32u4 (same as Arduino Leonardo) embedded which gives it the capacity to emulate HID devices and COM ports at the same time and on the same USB line. Through this, we are able to emulate a keyboard, a mouse, a joystick at the same time and still send data over the serial terminal.

As it's probably the most commonly used computer by makers, I used a Raspberry Pi to analyse the incoming data and output the interface over HDMI. But you can also use any other computer OS as the code run on Python, as every computer emulates Python and reads HID and serial USB. I have only tested it on Linux for now.

The Python code provides 3 possibles outputs :

  • Full screen, to play on a normal computer or TV screen
  • Basic stereoscopic screen, to use with cardboard
  • Tunnel effect stereoscopic, to use on a VR headset with aspheric lenses.

It also provides several ways to interact :

  • The Be Your Hero USB HID Sensors
  • Keyboard & Mouse

You don't need to build the sensors & Arduino boards to test the code !!! The Python code has been tested on several linux machine including Raspberry Pi. You can use Mouse and Keyboard for control.

So this is our main hardware architecture... lets talk about software now!

Several years ago, a great contributor of the Raspberry Pi community built "Pi3D", a great light 3D Python library running on the small CPU/GPU of our favorite computer. He provided us with a full documentationand examples to start building games or platforms. To build the software I combined a Pi3D and VrZero to facilitate the Virtual Reality development. In addition I used Blender, a free professional software to build 3D objects.

By discovering the world of 3D Blender objects on the internet, I was amazed by the number of accessible drawings you can find. Basically, any of your favorite avatars, if they are slightly famous, have probably been designed a 100 times. So you just have to spend some time on the internet to find what you want ;). I found most of my designs on this website.

  • 1

    When I read a tutorial, most of the time, I am interested in the general concept of the project, or, in a very specific part of it. That's why I chose to split every main technical part of the project into 7 points. So you can quickly access the specific information that you need.

      Bluetooth -> USB/HID :
      Build a USB HID Bluetooth dongle with a multi transmitter Star typology. The USB HID device can simultaneously act as a Keyboard, a Mouse, a Joystick and still send data over Serial port.

      6 DOF -> Bluetooth :
      Build a 3, 6 or 9 DOF sensor with a Bluetooth interface. The device is 4*2*2cm large, it is powered on Lipo rechargeable battery, it has a DIP switch to manually give a unique address to nodes and it fits into a tiny 3D print bracelet box so it is easy to wear.

      Nunchuck Joystick -> Bluetooth : 
      Build a Wii Nunchuck/Bluetooth interface on a battery to transfer the joystick and button to the central using Bluetooth

      Wii Remote Camera -> Bluetooth :
      Hack a Wii Camera Remote to track the movement of your body in a room

      Create your Hero and a 3D python Map :
      This part teaches you all the basics for Blender to import your hero to your Python Map generated by Pi3D

      Test your Hero and create a game :
      Your Hero is now ready to be fully tested. You can test it to move all the parts of the body you have implemented. You can also try out a little "home made game"!

      Implement the Stereoscopic Hardware and Software :
      Now we can implement the "Stereoscopic" and "Tunnel Effect" so we'll be able to test our program on a basic cardboard or on a DIY High Quality headset.

View all instructions

Enjoy this project?



Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates