A Raspberry Pi and Arduino based voice enabled smart room system that controls lighting and music and responses to questions.

Similar projects worth following
The goal of this project is to not need any lights switches anymore, have a voice controlled digital assistant without the need of picking up the smartphone and to learn about every involved technology.

The system uses a WiFi enabled Raspberry Pi 2 as base station, extended by a USB linked Arduino Nano, which is responsible for gathering sensor data and executing tasks that are better done by a microcontroller. USB communication is based on the Firmata protocol.

The whole project is under active development and almost every key part of it is already working for itelf, at least in an early stage. When everything comes together, the system will be able to provide the following functionality:

  • Light Control
    • switch all lights either
      • fully automatic (presence detection, position tracking)
      • semi automatic (ceiling lights by presence, small lights only by voice command or Android App)
      • manual (all lights only by voice commands or Android App)
    • light switching is implemented using 433MHz outlets for the small lamps and a relay for the 12V ceiling lights.
  • Digital Assistant / Voice Control
    • reliable voice command understanding for
      • light control
      • question answering (Wolfram Alpha API)
      • Spotify music playback
      • maybe more to come
    • implemented using pocketsphinx for keyword spotting, Google Speech API for recognition and Ivona Speech API for final TTS. Command parsing is home made.
  • Presence Detection / Position Tracking
    • The system always knows if one or more individuals are currently inside the room and also tracks their current position.
      • This allows for context sensitive voice commands or even full auto mode, e.g. the common
        • "turn on the couch light" can be shortened to
        • "light on", because it is known that you are currently sitting on the couch.
      • In full auto mode the couch light would be switched on automatically as you sit down and off again as you move somewhere else.
    • implemented using a PIR sensor for initially detecting that someone enters the room and from then on...
    • ... using the live footage of a ceiling mounted 160° wide angle Raspberry Pi camera which is analyzed by an OpenCV app regarding the current position of all individuals and whether they are leaving the room through the door.
    • This way the ceiling light can be instantly switched on when entering and instantly switched off when leaving, without delay and without the need of waving at a sensor once in a while.

  • 1 × Raspberry Pi 2 for main logic, camera, microphone sensors
  • 1 × 433MHz Transmitter for controlling the wireless outlets
  • 3 × 433MHz Outlet for switching the small lights
  • 1 × 5V Relay for switching the 12V ceiling lights
  • 1 × 160° wide angle Raspberry Pi Camera ceiling mounted, for presence sensing and position tracking

View all 9 components

  • Plugin-API demonstration

    Bennet Krause03/22/2016 at 16:58 0 comments

    Here is a little example plugin that uses the CloudFileManager, Properties, RaspiCam, Logging and MQTT functionality provided by the plugin API.

    When you publish a string representing the desired file name into the topic home/room/picam/upload, this plugin takes a photo with dimensions and quality as specified in the and upload it to dropbox. If "current_time" is sent , the current date and time will be used as file name.

    public class PiCamUpload extends BasePlugin {
     public void onStart() {
      // get CloudFileManager implementation from plugins (dropbox in this case)
      CloudFileManager cfm = getSystemManager().getCloudFileManager();
      Objects.requireNonNull(cfm, "A CloudFileManager implementation is needed for this plugin to work.");
      // get properties from this plugins's .properties file
      int width = Integer.parseInt(getProperties().getProperty("width"));
      int height = Integer.parseInt(getProperties().getProperty("height"));
      int quality = Integer.parseInt(getProperties().getProperty("quality"));
      // subscribe to a fitting topic
      MQTT.get().subscribe("home/room/picam/upload", (MqttMessage m) -> {
      // convert binary payload to ascii string
      String filename = new String(m.getPayload(), StandardCharsets.US_ASCII);
      if (filename.equals("current_time"))
        filename = new SimpleDateFormat("yyyy-MM-dd_HH:mm:ss").format(new Date());
      File img = new File("./" + filename + ".jpg");
      try {
      } catch (IOException e) {
        getLog().error("Exception", e);
      RaspiCamera.take(img, width, height, quality);
      try {
        cfm.uploadFile(img, null);
      } catch (FileNotFoundException e) {
        getLog().error("Exception", e);

  • App/ XML-Config/ Plugin System/ MQTT [Update#2]

    Bennet Krause03/22/2016 at 13:04 0 comments

    Since quite some time the project has a Android app to control the lights, turn on the alarm system (just the PIR) and to view all important logs. Later I will add the possibility to live stream (or to slide show :P) from the RasPi-Cam.

    The app has a ForegroundService with a permanent notification that allows quick access even from the lock screen and the actual app can be killed in the task manager.

    Communication with the RasPi is done via MQTT with a broker on my vServer. Log entries generated on the Pi are currently sent to the App using Google Cloud Messaging; maybe I will switch that part to MQTT later on.

    The Rasperry Pi Java software now has a plugin system that allows to outsource all user specific logic into runnable, .jar-packed plugins. Additionally their are non-runnable plugins that implement interfaces. So the user can decide on specific implementations e.g using Dropbox as cloud file service or to use a local text-to-speech engine instead of a cloud based solution.

    The system supports hot deployment of runnable plugins.

    Thanks to MQTT and the new XML configuration it is easy to have multiple RasPis all running the SmartRoom software. The config allows to do simple wiring without the need to write any code. E.g. controlling 433mhz outlets and relays or publishing the current value of a PIR-sensor to the broker server is as easy as that:

    <?xml version="1.0" encoding="UTF-8"?>
    <smartroom broker_uri="ssl://">
    	<state topic="home/room/ceilinglight/state" type="boolean">
    	    <!-- enables the relay if a message on that topic is "1" -->
    	    <pinappender pin="GPIO 3"/> 
    	<state topic="home/room/desklight/state" type="boolean">
    	    <!-- controls a 433mhz outlet -->
    	    <outletappender pin="GPIO 0" system_code="9" unit_code="8"/> 
    	<state topic="home/room/pir/state" type="boolean">
    	    <!-- publishes a message with "1" or "0" on that topic depending on the current pin state -->
    	    <pinprepender pin="GPIO 4" mode="pull up"/> 

  • Presence Sensing / Position Tracking [Update#1]

    Bennet Krause12/26/2015 at 12:41 0 comments

    This is my current test setup for developing the OpenCV based presence sensing and room position tracking system component. It consists of my RasPi 2 and a 160° wide angle camera module from china (100% compatible with the original camera module) mounted on the power transformer of my ceiling lights (which will later on be disrupted by the Arduino controlled relay).

    After quite some struggling with the compilation, I finally managed to compile the brand new OpenCV 3.1.0 (including the Java bindings) on my Pi.

    Fortunatly, after loading the preinstalled (on Raspbian Jessie) V4L2 kernel driver, which creates a /dev/videoX, OpenCV can use the camera module out of the box, using it like a standard USB webcam. Therefore it only has a resolution of 640x480, but that is by far enough for motion tracking needs, a higher resolution would overload the RasPi anyways.

    That way, I get about 2 to 5 position updates per second (unoptimized), with a single core at full load, leaving enough resources for the other software modules.

    The detection of the current position of the occupant is done by taking the thresholded difference between two sequential images, blurring the result to remove noise, get a binary result image and filtering out too small or too close-by blobs and finally calculating the centroid of the result blob.

    Support for multiple occupants will be added later on, more important is the interpretation of the constantly updated position. I will need to define rectangular zones for all interesting room areas (e.g. couch, bed, desk).

    For presence sensing it is planned that a PIR will initially turn the camera on if a movement is detected. From then on the software constantly tracks the occupants position and (this is to be added) listens for a movement in outside direction through the door and thereby registers that the room is unoccupied again (turn off camera and lights).

    Visualized, the position tracking looks as follows:

View all 3 project logs

Enjoy this project?



Zeeshan Khan S wrote 01/08/2016 at 06:38 point

Can you guide me to the source code of the person tracking algorithm please?

  Are you sure? yes | no

Bennet Krause wrote 01/08/2016 at 19:01 point

Currently all my source code is privately hosted on Bitbucket, but I will soon put some projects on GitHub once I cleaned it and finished the javadoc.

  Are you sure? yes | no

Bennet Krause wrote 01/13/2016 at 12:51 point

I've added the GitHub link on the left.

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates