Web Smart (Phone) Screen Blink Bot

Description

"...Let me sum up... No, there is too much" see the detail below. Basically this is about how to avoid all problems with our prior bots and safely use your expensive smartphone, with free online providers to host secure web pages that can gain access to the phones sensors, to control a very low cost robot. The phone is then also able to connect to extensive web services for additional processing.

Details

ESP's web servers provide great UI's

The San Diego Makers Guild has provided literally hundreds of low cost robots to kids at events. These are based on the NodeMCU (ESP32 or 8266) devices, with a couple continuous rotation servos and a battery pack, programmed with a web server accessible via direct WiFi connections, which allowed them to be controlled by a web page on a cell phone or PC. There are many advantages:

- $5 chip, a pair of $5 continuous rotation servos, and a $2 battery pack. All the display, controls, user interface, etc... are on the SmartPhone
- Very clean, responsive, and fun remote control interface, without any app to install; just uses the browser on the phone. This video explains it pretty well. It's from a time when we were using the bots in a combat league.
- You can serve any static web page, so not only the basic remote control interface, but also your own custom version, settings, and even a simple web based IDE to edit the files on the ESP; it becomes a mobile webdev platform!

The use of the ESP chips to host web pages for smartphones is a great trick but... it's a bit of a trap, in that it's limiting.

No web services, no access to sensors:

It doesn't allow one to use the phone to contact the internet in general, and worse than that, you can't put the phone ON the robot /and/ use the sensors in the phone from a web page unless the web page is https with a valid certificate. The browsers security won't allow it.

Which means you either have to write an app, or install a cert on the ESP (nightmare) or connect the ESP to your house WiFI (which is VERY insecure and just a general bad idea) or... you have to just live without the camera, mic, accelerometers, GPS, compass, etc... all the wonderful sensors that the phone can give us. Which is a huge loss, because they can tell the robot what is around it, that it bumped into something, what direction its moving, where it is, etc.. Not to mention sending pictures to google for classification of objects or sending sound to recognize verbal commands, etc...

If not WiFi...
Instead, let's connect the smartphone to the robot some other way so we can still just use a web page, avoid an app, and run on the net. But what ways can we connect?

- Bluetooth Low Energy is horrible unreliable. Getting every model of phone to pair correctly is a nightmare, and... sending data from a web page to a BLE device isn't supported anyway. And you need a device that supports it, so it really limits your controller choices.
- We could use Bluetooth audio... but again, not reliable, and quite hard to decode on the robot. And you need a controller that manages it.
- USB OTG is poorly supported; many phones just don't do it. Also, it's an electrical connection, and if a kid (or someone like me) wires it wrong, dads expensive smartphone can go up in smoke. Maybe. And... you can't send data to USB serial from most browsers.
- We could use the headphone jack, but again, not all phones have it. And it's a (minor) electrical risk.

But, you know what all phones DO have? And can't be blocked? And doesn't require permissions or anything to access?

A screen.

Der Blinkenlights
We can build a circuit that senses light. And a web page can blink out messages at VERY high frame rates these days. So I've written a page that uses light blocks to send data and some people have designed and built electronics for it:

https://hackaday.io/project/4196-2wire-i2c-from-blocks-of-light-on-an-lcd-screen

http://techref.massmind.org/techref/io/serial/i2c/withGIFs.html

By doing that, we can
1. Write a web page (e.g. on github.com which provides a good cert, and so can access your phones sensors)
2. Blink out messages to a /really/ low end processor (e.g. Arduino UNO) to control a pair of continuous rotation servos, etc...
3. In the Javascript on the page, we can use the phones sensors to control the robot locally, send data to cloud...

Project Logs

Collapse

Optical Flow

James Newton • 05/18/2022 at 22:33 • 0 comments

I happened upon some brilliant code on github for doing optical flow in the browser. Open this URL in your smartphone, then go to the "Zone" demo (the Pong one is fun but doesn't give the detail to show what is happening). Allow it to access your camera, then instead of moving your hand, as instructed, just tip the phone slowly left and right, as if it were being moved on a robot. Note that the yellow dot does a good job of tracking rotation.

https://anvaka.github.io/oflow/

If you move too fast, it loses track, but that can be adjusted, and for a slower moving (careful) version of the bot, it might be just fine, at least when combined with the compass and other sensors in the phone.

The code is open source (note the Fork me...)

Just another example of the amazing robotic control things that can be done in a smartphone browser without installing anything.

Less complex (1 component) "circuit"

James Newton • 05/16/2022 at 16:02 • 0 comments

I purchased a set of these sensors for $10+ on Amazon:

https://www.amazon.com/XLX-Photoresistor-Optoresistor-Assortment-Light-dependen/dp/B01M3TO7RA/ref=sr_1_9

The GL5528 seems to work best, just plugged in between pin 7 and ground. Then I set pin 7 as an input with pull up and was able to sense light and dark, just passing my hand over the sensor.

TODO:

- Add another channel for the clock

- Write some Arduino code to show the state of those pins on a graph and try different transmission speeds to see if the signal gets through at "fast enough" rates.

- Try the wire library to see if I can receive I2C data

My time is stretched to the breaking point right now so if someone wants to help out by duplicating this and doing those steps and sharing the code / results I would be VERY appreciative.

Auto power up on SmartPhone insert

James Newton • 04/12/2022 at 17:26 • 0 comments

Add a microswitch at the bottom of the slot where the SmartPhone is inserted so it automatically powers up the bot when the phone is inserted.

The script on the web page can listen to it's accelerometer sensors and send commands for short forward and back bursts to the motors in opposite directions; e.g. try to twist back and forth. If the sensors don't see that pattern, then the phone isn't in the robot or the bot isn't sensing it correctly. E.g. the phone will know it's in "not inserted" mode and keep sending that pattern looking for motion.

On power up, the bot will expect to see that pattern and will use it to set levels on the A2D. This will compensate for different screen brightness, lighting conditions, alignment, etc... Once it configures, it will start moving, and the phone will feel that "I'm in! Switch to ready to go mode"

If during normal operation, a motor command doesn't generate the expected motion, it can initiate a test "twist" and if that fails, switch back to "I'm out!" mode.

Discussions

Ken Yap wrote 05/19/2022 at 01:15

Haha, very cool hack. I'll have to start thinking of other places where a phone can provide data contactless.

Are you sure? yes | no

Web Smart (Phone) Screen Blink Bot

Description

Details