Immersive VR on Rpi to control a Hero with sensors

This project will give you full gesture control of any virtual Hero you like with a simple set of inexpensive devices using sensors on RPi

Similar projects worth following
Welcome to "Be Your Hero" project!

I hope you are ready to enter the next generation of Virtual Reality immersion!

This project will give you full gesture control of any virtual Hero you like with a simple set of inexpensive devices using sensors. All the data collected is wirelessly sent to a computer and will display your favorite hero on a normal screen or an a DIY HD virtual reality headset.

I spent a lot of time on this project to provide the most cost efficient solution. The resulting embedded device is really inexpensive, surprisingly reliable and comes in a very small package. In this tutorial you'll have all the information you need to develop the sensors, the Bluetooth communication, to build the VR headset, import your Hero from Blender and develop your own 3D immersive game!

So, if like me you spent your childhood dreaming about becoming a Jedi or a Super Sayan, follow the guide! Your first light saber training and Kamehameha is on its way :-)

How Does It Work?

So, how are we are going to build real time detection and interaction between the movements of our body and the movements of our Hero's?

This project uses a combination of bracelets with 6-DOF sensors and Wii Nunchuck that you can easily attach to your body. Those sensors can detect "6 Degrees Of Freedoms" with a 3 axes accelerometer and a 3 axes gyroscope.

The central system of each of the embedded devices is a Arduino FIO powered by a Lipo battery. The communication is provided by a NRF24 Bluetooth board which has the ability to modulate the transmission channels. Thanks to this device, we have a large number of modules simultaneously communicating with a central with the reliable Bluetooth stack.

The Bluetooth receptor is wired to an Arduino Micro. Within this particularly impressive board there is an atmega32u4 (same as Arduino Leonardo) embedded which gives it the capacity to emulate HID devices and COM ports at the same time and on the same USB line. Through this, we are able to emulate a keyboard, a mouse, a joystick at the same time and still send data over the serial terminal.

As it's probably the most commonly used computer by makers, I used a Raspberry Pi to analyse the incoming data and output the interface over HDMI. But you can also use any other computer OS as the code run on Python, as every computer emulates Python and reads HID and serial USB. I have only tested it on Linux for now.

The Python code provides 3 possibles outputs :

  • Full screen, to play on a normal computer or TV screen
  • Basic stereoscopic screen, to use with cardboard
  • Tunnel effect stereoscopic, to use on a VR headset with aspheric lenses.

It also provides several ways to interact :

  • The Be Your Hero USB HID Sensors
  • Keyboard & Mouse

You don't need to build the sensors & Arduino boards to test the code !!! The Python code has been tested on several linux machine including Raspberry Pi. You can use Mouse and Keyboard for control.

So this is our main hardware architecture... lets talk about software now!

Several years ago, a great contributor of the Raspberry Pi community built "Pi3D", a great light 3D Python library running on the small CPU/GPU of our favorite computer. He provided us with a full documentationand examples to start building games or platforms. To build the software I combined a Pi3D and VrZero to facilitate the Virtual Reality development. In addition I used Blender, a free professional software to build 3D objects.

By discovering the world of 3D Blender objects on the internet, I was amazed by the number of accessible drawings you can find. Basically, any of your favorite avatars, if they are slightly famous, have probably been designed a 100 times. So you just have to spend some time on the internet to find what you want ;). I found most of my designs on this website.

  • 6 × Arduino mini
  • 1 × Raspberry Pi 3
  • 6 × 9 DOF sensors
  • 1 × Wii Nunchuck
  • 1 × TFT 7" HDMI screen

View all 7 components

  • 6 DOF -> Bluetooth

    jean.perardel04/25/2018 at 16:45 0 comments

    DOF (Degree Of Freedom) Sensors

    When I started this project, I hesitated a lot on what DOF sensors I should buy. It was obvious to me that I would need a 9 degrees of freedom to avoid the infinite rotating effect that 6 DOF would create.


    But then I discovered this code with a demo running on Free_IMU. It was perfectly still! Even after many random moves, the Yaw axe seems to be only a few degrees different while the Pitch and Roll stays perfectly level (see the plane picture above if you don't understand the three axes) which is perfectly fine for me for now... In addition, Eulers angles can be calculated with a "home reference position". Very useful for calibrating our sensors!


    The schematic of the device is above. Here are the components to build one :

    • AirBoardArduino FIO or DIY solution. FIO Arduino bootloader runs with an 8MHz crystal (instead of 16MHz on the UNO) which allows it to be used in lower voltage. So if you have an Arduino project running on a battery, it is a great strategy to directly start with a 8MHz board (Or you can use the internal 8Mhz oscillator). Of course if your project needs the full performance of the UNO, this may create problems, but most of them don't...
    • MPU 6050, 6DOF sensor which communicates with the Arduino over I2C
    • NRF24L01+ Bluetooth chip which communicates with SPI. This board became very famous because of its great manufacturer NORDIC (which is a reference in BT devices) but also because of its amazingly cheap price : less than 1$! If we also consider the fact that it allows multi-channel communication, you have yourself a perfect toy to play with!
    • Dip switch to select devices
    • Lipo Battery
    • Headers and wires to charge the battery and push the program.

    For debugging this project, I used several Arduino Airboard from a Kickstarter project. This amazing little board embeds a FIO, a Battery, a power switch, LEDs... Plus, it accepts remote programming over a dedicated BLE chip and a USB dongle. I recommend using those for debugging embedded battery projects, it will save you a lot of time!


    Processing test software (Also in Github here) I already talked a bit about the library for the 6DOF sensor. If you want to give it a try, download this code and wire the sensor to an Arduino board. Once processing, start the IMU interface and you can start playing. The "h" key will recalibrate the sensor.

    The Euler angles problems & solutions

    Playing with 3D object rotations is a hard work. Using Arduino libraries and Pi3D helped me a lot to control the movement for one object. But I am still debugging a bit when the movement of one sensor depend on the movement of another. For example, hands movements depend on arms movements. So the angles of the hands have to be "HANDS_ANGLES - ARMS_ANGLES = REAL_HANDS_ANGLES". As Euler values aren't linear, "-ARMS_ANGLES" isn't the opposite of "ARMS_ANGLES"...

    To implement this part, I worked on two solutions :

    • SOLUTION 1 : The nodes send only the 4 quaternions on Bluetooth. The central receive and calculate Euler angles. When the quaternions of the hands is received, the Euler is calculated depending on the arms quaternions as "home reference". This solution works, but the central needs more time to calculate.
    • SOLUTION 2 : My first choice was to calculate Euler angles inside the nodes and send them over Bluetooth (which is nice because I still have one empty byte on the buffer for the button). In Pi3D I tried to implement the Hands depending on the arm, but it's still not working perfectly right now. If you are interested in this topic, you can take a look on the conversation I had with paddyg here.


    As you can see, those 6-DOF sensors are quite precise. But without a compass, they need...

    Read more »

  • Bluetooth -> USB/HID

    jean.perardel04/25/2018 at 16:08 0 comments

    Have you ever used HID devices?

    Of course! You are probably using several right now!

    HID stands for "Human Interaction Devices", it's the USB protocol used for Keyboards and Mouses and it's probably the easiest way to interact with a computer as it's recognized by any OS system.

    Some Arduino boards like Leonardo or Micro aren't using the original ATmega328p AVR chip as UNO. They implement a Atmega32u4. The name doesn't seems very different... but this controller will give you the possibility to communicate with the USB port of a computer through HID. When you play with an Arduino UNO, you need to add a chip to interconnect the USB (like FTDI or 32u2) as HID is incompatible (except V-USB for AVR library, but the implementation is slower).


    This HID capability will provide us with a great way to interface between Bluetooth and Raspberry Pi because we can re-use some of the Mouse or Keyboard libraries for an easy interaction and also use a Serial terminal communication for more complex data transfers.


    So, the hardware is pretty simple and can be re-used in many different projects:

    • A Leonardo or Micro Arduino board. I used Micro to save space, but the Leonardo has a 3.3V power supply, which is useful for powering the Bluetooth. If you use a Micro, I recommend you to add an external 3.3V VCC. 
    • NRF24L01+ on the SPI port (Pin 3/10/11/12/13, follow the schematic under). 
    • A button. To avoid headache, this safety button can block the HID. Sometimes a wrong loop can make your board send thousands of keyboard and mouse commands!!! Its only function is to say "send nothing if the button is pressed!" 
    • I used a tiny breadboard, some male/female headers and some wires to plug in everything together (See the picture above). I will try to optimize it and make a 3D printed box later when I will have finished other more important things :p



    All the software is available on this Github. For the Bluetooth HID, I had to manually add the libraries mouse/keyboard/Joystick (I use Arduino 1.6.9). This doesn't really make sense, as they are supposed to be already implemented...


    You can try removing the first three includes, then select Arduino Micro or Leonardo board and compile (Be careful, it's not going to work with UNO!). If it says keyboard, mouse or joystick not recognised, you have to download the library yourself "write arduino keyboard.h in google". Then you move them inside the Arduino installation folder (java/library in mac) and restart Arduino software!

    The libraries you need depend on what you want. At first I was using a joystick lib, but it seemed to slow the Python code down. So I made another function to send data through Serial as well.

    The mouse HID is very useful if you want to follow the movement of the head. So we keep the possibility of controlling our Python program with a normal mouse or with the Micro device.

View all 2 project logs

  • 1

    When I read a tutorial, most of the time, I am interested in the general concept of the project, or, in a very specific part of it. That's why I chose to split every main technical part of the project into 7 points. So you can quickly access the specific information that you need.

      Bluetooth -> USB/HID :
      Build a USB HID Bluetooth dongle with a multi transmitter Star typology. The USB HID device can simultaneously act as a Keyboard, a Mouse, a Joystick and still send data over Serial port.

      6 DOF -> Bluetooth :
      Build a 3, 6 or 9 DOF sensor with a Bluetooth interface. The device is 4*2*2cm large, it is powered on Lipo rechargeable battery, it has a DIP switch to manually give a unique address to nodes and it fits into a tiny 3D print bracelet box so it is easy to wear.

      Nunchuck Joystick -> Bluetooth : 
      Build a Wii Nunchuck/Bluetooth interface on a battery to transfer the joystick and button to the central using Bluetooth

      Wii Remote Camera -> Bluetooth :
      Hack a Wii Camera Remote to track the movement of your body in a room

      Create your Hero and a 3D python Map :
      This part teaches you all the basics for Blender to import your hero to your Python Map generated by Pi3D

      Test your Hero and create a game :
      Your Hero is now ready to be fully tested. You can test it to move all the parts of the body you have implemented. You can also try out a little "home made game"!

      Implement the Stereoscopic Hardware and Software :
      Now we can implement the "Stereoscopic" and "Tunnel Effect" so we'll be able to test our program on a basic cardboard or on a DIY High Quality headset.

View all instructions

Enjoy this project?



Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates