Close
0%
0%

Web Smart (Phone) Screen Blink Bot

Secure web page on smartphone safely drives cheap bot with flashes of light using internal sensors and web services.

Similar projects worth following
"...Let me sum up... No, there is too much" see the detail below. Basically this is about how to avoid all problems with our prior bots and safely use your expensive smartphone, with free online providers to host secure web pages that can gain access to the phones sensors, to control a very low cost robot. The phone is then also able to connect to extensive web services for additional processing.

What if I told you it was possible to SAFELY use ANY SmartPhone as a robot brain?

On any smartphone with a browser, load a freely hosted github.com webpage with Javascript that blinks areas on the screen to send motor commands to an e.g. Arduino. The page can also access web services API's for voice, object recognition, teleoperation, etc... via WiFI or Cellular data.

- Safe: No electrical connections to the phone, so you can't possibly damage it.
- Any SmartPhone: Not dependant on any hardware support; doesn't need USB, audio jack, BLE, anything.
- Secure: No app needed, so no security concerns, no development software required, nothing to learn but Javascript.
100% reliable, no configuration, easy debugging because everything is visible.
- Sensors: You still get access to all the sensors on the phone; camera, accelerometer, compass, GPS, and to the web via WiFI or cellular connection.
- Internet: You can remote control it, or have it log data, pics, etc... It can access online services for object reco, voice APIs, etc... but no for $ service provider is required; it's free.
- Stunningly Cheap: Total additional cost, beyond a tiny circuit or a cheap mcu (Arduino, Pi Pico, etc...), dc motors or servos, battery, whatever else you use for the robot, is pennies. All open source, no required services, no BS.

History: ESP's web servers provide great UI's on SmartPhones

The San Diego Makers Guild has provided literally hundreds of low cost robots to kids at events. These are based on the NodeMCU (ESP32 or 8266) devices, with a couple continuous rotation servos and a battery pack, programmed with a web server accessible via direct WiFi connections, which allowed them to be controlled by a web page on a cell phone or PC. There are many advantages:

- $5 chip, a pair of $5 continuous rotation servos, and a $2 battery pack. All the display, controls, user interface, etc... are on the SmartPhone
- Very clean, responsive, and fun remote control interface, without any app to install; just uses the browser on the phone. This video explains it pretty well. It's from a time when we were using the bots in a combat league. 
- You can serve any static web page, so not only the basic remote control interface, but also your own custom version, settings, and even a simple web based IDE to edit the files on the ESP; it becomes a mobile webdev platform!

The use of the ESP chips to host web pages for smartphones is a great trick but... it's a bit of a trap, in that it's limiting. 

Historical problems: No web services, no access to sensors:

It doesn't allow one to use the phone to contact the internet in general, and worse than that, you can't put the phone ON the robot /and/ use the sensors in the phone from a web page unless the web page is https with a valid certificate. The browsers security won't allow it. 

Which means you either have to write an app, or install a cert on the ESP (nightmare) or connect the ESP to your house WiFI (which is VERY insecure and just a general bad idea) or... you have to just live without the camera, mic, accelerometers, GPS, compass, etc... all the wonderful sensors that the phone can give us. Which is a huge loss, because they can tell the robot what is around it, that it bumped into something, what direction its moving, where it is, etc.. Not to mention sending pictures to google for classification of objects or sending sound to recognize verbal commands, etc...

If not WiFi...
Instead, let's connect the smartphone to the robot some other way so we can still just use a web page, avoid an app, and run on the net. But what ways can we connect? 

- Bluetooth Low Energy is horribly unreliable. Getting every model of phone to pair correctly is a nightmare, and... sending data from a web page to a BLE device isn't supported anyway. And you need a microcontroller that supports it, so it really limits your choices. 
- We could use Bluetooth...

Read more »

Two Photodiodes Detectors.stl

3D Model for bracket to hold the 2 photodiodes.

Standard Tesselated Geometry - 298.52 kB - 10/16/2022 at 14:45

Download

Web2Wire.zip

The test HTML, Javascript and CSS files.

x-zip-compressed - 1.61 kB - 10/16/2022 at 14:33

Download

TestWeb2Wire.ino

Test Nano program to read the sensors.

ino - 1.46 kB - 10/16/2022 at 14:32

Download

TestWeb2Wire.fzz

Fritzing showing connections.

fzz - 32.42 kB - 10/16/2022 at 14:31

Download

  • 1 × Arduino Nano The main microcontroller. Although a Nano is being used, any microcontroller should be able to be used.
  • 1 × Nano IO Expansion Shield This shield makes it easy to connect sensors/actuators to the Nano. Here's a link to one on Amazon: https://www.amazon.com/gp/product/B087ZRMG2B.
  • 2 × Photodiode module This detects the light levels on the screen. Although the original project used photo resistors, photodiodes are faster. You need one for Clock and another one for Data. Here's a link to ones on Amazon: https://www.amazon.com/gp/product/B0B389WHM6. In addition, a 3D printed holder is required to hold the photo diodes in place, the STL model is in the Files section.
  • 6 × Dupont female to female jumper wires Wires to connect photodiodes to Nano shield. See the Fritzing file for connection details.

  • Simple Simulation of Simple Circuit

    James Newton02/09/2023 at 06:00 0 comments

    A very very simple simulation of the very simplest version of the light controlled motor circuit. One of these on the left and one on the right. The photoresistor, with tape over it to keep it from activating the touch sensors in the screen, is placed facing the block displayed on the cell phone screen. When the block is white, the motor moves. When it's black, the motor stops. 

    You can play with it here:

    https://www.tinkercad.com/things/aQwsvZ6Vvcn-spectacular-fulffy/editel?tenant=circuits

    Limitations:
    1. Can't go backward. (A more complex circuit might do that)
    2. No speed regulation. (But the phone can use the compass / IMU to stay on course)
    3. Very limited power (resistive drop through transistor, meh)

    Note that the batteries shown here are just a sample. AA's or a USB battery are probably a less costly option. 

    KEY POINT: The cell phone controls the motor from a web page (or whatever) which has access to the phone sensors, the internet, javascript, etc... The web page can be served from any free host, like github. The javascript on that page can send sensor data to services, get back complex analysis, directions, etc... You can do voice, image, teleoperation, GPS, etc... With an old smartphone and a few dollars worth of electronics. 

  • More Simple Friction Drive idea

    James Newton02/08/2023 at 05:30 0 comments

    Clarifying the prior post on this idea. The key is that the motor shaft is pressed own against the outside of the wheel by the weight of the cell, motor, frame, etc... The wheel is something with a rubber outside, like a fat rubber band over cardboard or an o-ring between laser cut acrylic disks. The wheel shaft fits in a vertical slot not a hole, so the shaft slides up until the top of the wheel hits the motor shaft. Motor is driven by a circuit that senses light from blocks displayed on the cell phone screen. 

  • Easier with Analog?

    James Newton02/07/2023 at 05:41 0 comments

    A friend sent me a link to a very minimal version of this same idea. An analog DC motor driver looking at the light level on the screen of a smartphone and using that to drive a robot. Around. Honestly, I'm hard pressed to see why that isn't better? 

    https://www.voltpaperscissors.com/diy-smartphone-robot

    The only advantage of using servos (which require a controller) is that the speed regulation is better.

    I'm not crazy about the fact that he is using an app / photos / etc... instead of just using a web page. Also, his remote control depends on his server, but that could easily be changed. I think a local server is much better to start with especially as a gateway to the joy of simple e.g. node.js web servers or ESPs or whatever. Many security issues with an external server and apps. 

    Of course, the analog route doesn't really support controlling other devices, but... shrug... each new device is just a square on the screen and a simple circuit. 

    Also, speed regulation is really a non-issue if the program running in the phone can "learn" and watch it's accelerometers, and compass. 

  • Simple Friction Drive idea

    James Newton02/07/2023 at 05:37 0 comments

    (edit: picture didn't show up last time)

    The idea here is that the motor shaft presses against the outside of the wheel. The wheel is something with a rubber outside, like a fat rubber band over cardboard or an o-ring between laser cut acrylic disks. The shaft actually goes all the way through to the other side (only showing half here). The red block has a slot not a hole, so the shaft slides up until the top of the wheel hits the motor shaft. The motor is connected to the red block, or to a sheet or PCB connected to that block. A PCB behind the motor also holds the light sensor, pointed at the phone screen. 

    https://www.tinkercad.com/things/2JTX6x4Dj0k-dazzling-lappi/edit

  • Update to Setup and Software

    Aram Perez10/16/2022 at 14:44 1 comment

    Hi Folks,

    I've added some components and files to this project that I used to speed up the whole Screen Blink to Microcontroller communications. There might be some remnants from all my testing that are no longer required.

    I changed the "protocol" from I2C to what I'm calling TwoWire. I'm sending 8 bits at a time, MSB first (makes it easier to see what data was send on a scope or logic analyzer), with a small delay between bytes.

    Any comments/suggestions/corrections/money will be appreciated ;-)

    /Aram

  • Swap logic levels for circuit with pullup.

    James Newton09/08/2022 at 18:48 0 comments

    Assuming the use of a photo sensitive resistor between the analog pin and ground, and an internal or external pullup resistor to the power rail, a white screen will cause a low voltage, and a black screen will allow the voltage to rise. So, to make this the default (as it's the simplest circuit possible, consisting of only the sensors plugged into the Arduino header) I've swapped the black and white blocks on the test web page, but I also made it easy to switch back with a couple of "defines" at the start:

      var logic1 = black
      var logic0 = white

    https://github.com/JamesNewton/JamesNewton.github.io/commit/f36116a8cec1beee6d23ae3cc618ed002eca941c

    Next steps (please help if you can?): Update the Arduino code below to decode the I2C start, data, stop and check for reliability of communications. The time required for the serial output to dump all the data into the plotter far exceeds the timing of the actual binks, so it's good to set the analog levels, turn that print off, and only output the digital signal and decoding. 

  • A start on the Arduino code

    James Newton08/27/2022 at 21:43 0 comments

    This code reads the A2D pins from the sensors, tracks the middle of the signal range, chops that into a high or low signal, and displays all that for the Serial Plotter so you can debug the setup.

    The sensors in this case are nothing more than photoresistors using the internal pull up in the Arduino. (Yes, the analog pins in the Arduino CAN have a pull up enabled). The total hardware cost outside the Arduino are a pair of light dependant resistors. (!)

    Here is a sample: The red is the data, with the green showing the cutoff point / AGC (Automatic Gain Control). You can see the slight increase on agc when the signal goes low, and the decrease when it goes high. That provides some hysteresis to avoid "bounce" or multiple edges. The blue is the clock, which also has an AGC, but it isn't shown here. Note the difference in signal levels caused by different positioning of the sensors to the screen. The yellow and purple are the resulting SCL and SDA signals. I've not yet decoded those to retrieve the data. The data is coming from this page:
    https://jamesnewton.github.io/webbot.html
    and the source for that page is
    https://github.com/JamesNewton/JamesNewton.github.io/blob/master/webbot.html
    (not the free secure web page hosting from github and the ability to access the phone camera inside the browser)

    Here is the code, such as it is. Much work remains to be completed, and I will NOT have time for it. I really hope others will pickup and run with this. 

    /*
      ReadAnalogVoltage
    
      Reads an analog input on pin 0, converts it to voltage, and prints the result to the Serial Monitor.
      Graphical representation is available using Serial Plotter (Tools > Serial Plotter menu).
      Attach the center pin of a potentiometer to pin A0, and the outside pins to +5V and ground.
    
      This example code is in the public domain.
    
      https://www.arduino.cc/en/Tutorial/BuiltInExamples/ReadAnalogVoltage
    */
    
    #define SDA_CHANNEL A0
    #define SCL_CHANNEL A1
    #define FILTER 256
    
      float agc0, agc1, hi1, lo1;
      float hysteresis;
      bool scl, oscl, sda, osda;
      bool pause;
      
    
    // the setup routine runs once when you press reset:
    void setup() {
      // initialize serial communication at 9600 bits per second:
      Serial.begin(115200);
      pinMode(A0, INPUT_PULLUP);
      pinMode(A1, INPUT_PULLUP);
      //good starting values for midpoints, highs and lows.
      agc0 = 2.0;
      agc1 = 2.0;
      hi1 = 3.8;
      lo1 = 1.1;
      hysteresis = -0.1;
      scl=sda=oscl=osda=0;
      pause = 1;
    }
    
    // the loop routine runs over and over again forever:
    void loop() {
      int sensorValue0 = analogRead(SCL_CHANNEL); //clock on A1
      float voltage0 = sensorValue0 * (5.0 / 1023.0);
      int sensorValue1 = analogRead(SDA_CHANNEL); //data on A0
      float voltage1 = sensorValue1 * (5.0 / 1023.0);
      scl = (voltage0 > agc0)?1:0;
      if (scl != oscl) { //clock line changed
        agc0 += scl?hysteresis:-hysteresis; //noise suppression
        oscl = scl; //update
      }
      agc0 = ((agc0*FILTER) - agc0 + voltage0)/FILTER;
      //sda = (voltage1 > agc1)?1:0;
      if (voltage1 > agc1) { //data high
        hi1 = ((hi1*FILTER) - hi1 + voltage1)/FILTER;
        sda = 1;
      } else { //data low
        lo1 = ((lo1*FILTER) - lo1 + voltage1)/FILTER;
        sda = 0;
      }
      if (sda != osda) { //data line changed
        //agc1 += sda?hysteresis:-hysteresis; //noise suppression
        hi1 += sda?hysteresis*2:-hysteresis*2; //noise suppression
        osda = sda; //update
      }
      //agc1 = ((agc1*FILTER) - agc1 + voltage1)/FILTER;
      agc1 = (hi1 + lo1)/2;
    
      if (Serial.available() > 0) {
        // read the incoming byte:
        Serial.read(); //clear it but toss it
        pause = pause?0:1;
      }
      if (!pause) {
        Serial.print("sclv:"); 
        Serial.print(voltage0);
        Serial.print(", sdav:"); 
        Serial.print(voltage1);
        Serial.print(", agc1:"); 
        Serial.print(agc1);
        Serial.print(", scl:"); 
        Serial.print(scl);
        Serial.print(", sda:"); 
        Serial.print(sda);
        Serial.print("\n"); 
      }
    }

  • Optical Flow

    James Newton05/18/2022 at 22:33 0 comments

    I happened upon some brilliant code on github for doing optical flow in the browser. Open this URL in your smartphone, then go to the "Zone" demo (the Pong one is fun but doesn't give the detail to show what is happening). Allow it to access your camera, then instead of moving your hand, as instructed, just tip the phone slowly left and right, as if it were being moved on a robot. Note that the yellow dot does a good job of tracking rotation. 

    https://anvaka.github.io/oflow/

    If you move too fast, it loses track, but that can be adjusted, and for a slower moving (careful) version of the bot, it might be just fine, at least when combined with the compass and other sensors in the phone. 

    The code is open source (note the Fork me...) 

    Just another example of the amazing robotic control things that can be done in a smartphone browser without installing anything. 

  • Less complex (1 component) "circuit"

    James Newton05/16/2022 at 16:02 0 comments

    I purchased a set of these sensors for $10+ on Amazon:

    https://www.amazon.com/XLX-Photoresistor-Optoresistor-Assortment-Light-dependen/dp/B01M3TO7RA/ref=sr_1_9

    The GL5528 seems to work best, just plugged in between pin 7 and ground. Then I set pin 7 as an input with pull up and was able to sense light and dark, just passing my hand over the sensor. 

    TODO:

    - Add another channel for the clock

    - Write some Arduino code to show the state of those pins on a graph and try different transmission speeds to see if the signal gets through at "fast enough" rates.

    - Try the wire library to see if I can receive I2C data

    My time is stretched to the breaking point right now so if someone wants to help out by duplicating this and doing those steps and sharing the code / results I would be VERY appreciative. 

  • Auto power up on SmartPhone insert

    James Newton04/12/2022 at 17:26 0 comments

    Add a microswitch at the bottom of the slot where the SmartPhone is inserted so it automatically powers up the bot when the phone is inserted. 

    The script on the web page can listen to it's accelerometer sensors and send commands for short forward and back bursts to the motors in opposite directions; e.g. try to twist back and forth. If the sensors don't see that pattern, then the phone isn't in the robot or the bot isn't sensing it correctly. E.g. the phone will know it's in "not inserted" mode and keep sending that pattern looking for motion. 

    On power up, the bot will expect to see that pattern and will use it to set levels on the A2D. This will compensate for different screen brightness, lighting conditions, alignment, etc... Once it configures, it will start moving, and the phone will feel that "I'm in! Switch to ready to go mode"

    If during normal operation, a motor command doesn't generate the expected motion, it can initiate a test "twist" and if that fails, switch back to "I'm out!" mode. 

View all 10 project logs

Enjoy this project?

Share

Discussions

James Newton wrote 01/26/2023 at 16:37 point

A friend sent me a link to a very minimal version of this same idea. An analog DC motor driver looking at the light level on the screen of a smartphone and using that to drive a robot. Around. Honestly, I'm hard pressed to see why that isn't better? 
https://www.voltpaperscissors.com/diy-smartphone-robot
The only advantage of using servos (which require a controller) is that the speed regulation is better.

I'm not crazy about the fact that he is using an app / photos / etc... instead of just using a web page. Also, his remote control depends on his server, but that could easily be changed. I think a local server is much better to start with especially as a gateway to the joy of simple e.g. node.js web servers or ESPs or whatever. Many security issues with an external server and apps. 

  Are you sure? yes | no

James Newton wrote 01/26/2023 at 23:56 point

Of course, the analog route doesn't really support controlling other devices, but... shrug... each new device is just a square on the screen and a simple circuit. 

Also, speed regulation is really a non-issue if the program running in the phone can "learn" and watch it's accelerometers, and compass. 

  Are you sure? yes | no

Aram Perez wrote 10/16/2022 at 14:47 point

I've added some files and components to the projects.

  Are you sure? yes | no

Aram Perez wrote 09/12/2022 at 01:00 point

James, I've modified your Web code to use CSS instead of images and hopefully made it faster. I tried to create a new project on stackblitz but I didn't have the time to figure out how. Send me your email so I can send you the zipped files.

  Are you sure? yes | no

James Newton wrote 09/13/2022 at 04:57 point

Sorry, just now seeing this. JamesNewton@MassMind.org

  Are you sure? yes | no

Aram Perez wrote 09/02/2022 at 19:48 point

Hi James,

Here are two options you might consider:

1) If you're set on using full rotation servos, here's a board to which you can connect servos and sensors:  https://www.diymore.cc/collections/esp8266/products/esp8285-esp-m2-wireless-wifi-development-board-micro-usb-ch340-microcontroller-module-dm-strong-for-arduino-esp-m3-nodemcu-lua. It has an ESP8266 along with 2M flash. All the GPIO's have GND, Vin and signal pins. Vin can be 5 - 10 V.

2) If you want to consider using small DC motors, search for "nodemcu motor shield" on Aliexpress. The motor shield also breaks out the GPIO pins with GND, Vin and signal (however, 4 GPIOs are used for driving the motor controller). This is the combo I use for my STEAMbot Robot Kit and would give you 4M flash.

/Aram

  Are you sure? yes | no

James Newton wrote 09/02/2022 at 21:08 point

Thanks. That's a cute board. I'm moving /away/ from the ESP's though. Since I don't need wifi on the robot, or sensors other than the light sensors I'll use an Arduino, and my own servo adapter board. I'll eventually make a little PCB to hold the light sensors in place with cables back to the Arduino so they can be held to the phone. 

  Are you sure? yes | no

Aram Perez wrote 09/04/2022 at 01:03 point

Here’s a shield for a Nano: HiLetgo 5pcs Nano I/O Expansion Sensor Shield for Arduino UNO R1 Nano 3.0 Duemilanove 2009 https://a.co/fQXgGvb. 

  Are you sure? yes | no

James Newton wrote 09/08/2022 at 19:47 point

Again, thanks for listing a nice board for the nano, provides connection to a servo which is of value. It will make it a bit harder to connect the sensors, but it can still be done. e.g. the female headers are easier for just sticking component wires into. 
All in all, this is nice, but what we NEED is development on the decoding of the signals and a standard Arduino is fine for that. And easy to use. And we can add the servos with the existing adapter board or by just taking off the holder on a servo header and plugging in the wires individually. If you want to help, please see the latest log entry and the call for help at the end. 

  Are you sure? yes | no

Aram Perez wrote 09/08/2022 at 20:21 point

Ok, I’ll take a look at the log. However, I have to take care of some business concerning my 88 year old mom in Puerto Rico so it will be a couple of weeks before I can actually do anything. 

  Are you sure? yes | no

Ken Yap wrote 05/19/2022 at 01:15 point

Haha, very cool hack. I'll have to start thinking of other places where a phone can provide data contactless.

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates