ESP's web servers provide great UI's

The San Diego Makers Guild has provided literally hundreds of low cost robots to kids at events. These are based on the NodeMCU (ESP32 or 8266) devices, with a couple continuous rotation servos and a battery pack, programmed with a web server accessible via direct WiFi connections, which allowed them to be controlled by a web page on a cell phone or PC. There are many advantages:

- $5 chip, a pair of $5 continuous rotation servos, and a $2 battery pack. All the display, controls, user interface, etc... are on the SmartPhone
- Very clean, responsive, and fun remote control interface, without any app to install; just uses the browser on the phone. This video explains it pretty well. It's from a time when we were using the bots in a combat league. 
- You can serve any static web page, so not only the basic remote control interface, but also your own custom version, settings, and even a simple web based IDE to edit the files on the ESP; it becomes a mobile webdev platform!

The use of the ESP chips to host web pages for smartphones is a great trick but... it's a bit of a trap, in that it's limiting. 

No web services, no access to sensors:

It doesn't allow one to use the phone to contact the internet in general, and worse than that, you can't put the phone ON the robot /and/ use the sensors in the phone from a web page unless the web page is https with a valid certificate. The browsers security won't allow it. 

Which means you either have to write an app, or install a cert on the ESP (nightmare) or connect the ESP to your house WiFI (which is VERY insecure and just a general bad idea) or... you have to just live without the camera, mic, accelerometers, GPS, compass, etc... all the wonderful sensors that the phone can give us. Which is a huge loss, because they can tell the robot what is around it, that it bumped into something, what direction its moving, where it is, etc.. Not to mention sending pictures to google for classification of objects or sending sound to recognize verbal commands, etc...

If not WiFi...
Instead, let's connect the smartphone to the robot some other way so we can still just use a web page, avoid an app, and run on the net. But what ways can we connect? 

- Bluetooth Low Energy is horrible unreliable. Getting every model of phone to pair correctly is a nightmare, and... sending data from a web page to a BLE device isn't supported anyway. And you need a device that supports it, so it really limits your controller choices. 
- We could use Bluetooth audio... but again, not reliable, and quite hard to decode on the robot. And you need a controller that manages it.
- USB OTG is poorly supported; many phones just don't do it. Also, it's an electrical connection, and if a kid (or someone like me) wires it wrong, dads expensive smartphone can go up in smoke. Maybe. And... you can't send data to USB serial from most browsers.
- We could use the headphone jack, but again, not all phones have it. And it's a (minor) electrical risk. 

But, you know what all phones DO have? And can't be blocked? And doesn't require permissions or anything to access?

A screen. 

Der Blinkenlights
We can build a circuit that senses light. And a web page can blink out messages at VERY high frame rates these days. So I've written a page that uses light blocks to send data and some people have designed and built electronics for it:

By doing that, we can 
1. Write a web page (e.g. on which provides a good cert, and so can access your phones sensors) 
2. Blink out messages to a /really/ low end processor (e.g. Arduino UNO) to control a pair of continuous rotation servos, etc... 
3. In the Javascript on the page, we can use the phones sensors to control the robot locally, send data to cloud services for deeper analysis, and... 
4. There is a system called BOSH which can xfer data for us for FREE to any terminal on the net. Here's how that works:

BOSH supports server push notifications by simply not responding to a request from the client until the server has something to send to the client. The request simply hangs until data arrives (via another connection to the server from the sender) or until it times out. In either case, it is the clients responsibility to re-establish the request as quickly as possible and so continue listening for data from the server, or more accurately from a sender via the server.

Free Teleoperation
But where would a student with no money host a server? Well, it turns out that Google app engine gives users a few free hours /every month/ instead of just at the start. This has been tested, and works quite well:

This means that we can move data between any PC browser /anywhere on the net/ and the cell phone on the robot! It's true teleoperation. It works best for small amounts of data (so you don't end up getting a usage bill) but I've found you can do quite a lot with it.

It's also amazing how much you can do in a web page on a SmartPhone running in Chrome. Even processing live video is quite possible; just with Javascript.

So here is a VERY rough implementation of using vision to guide a robot.

It uses the camera and frame subtraction to show differences over time and summarize those changes as being in the left, right, or top center of the area. It ignores the lower center area. If it sees motion on the left, it turns right, and vice versa. If it sees motion ahead it stops. The point is that anything close by will be seen to move, relative to the robots motion. It sends "L", "R" or "F" for Left Right Forward via the images at the bottom. 

If you build the circuit, and program an Arduino to drive the servos as commanded, then you have a sighted robot. 

And beyond!
MUCH more advanced systems are possible. 

- Even that page could be improved, by sending a binary value based on how much motion is detected. Something like 4 bits for the left motor, and 4 for the right. In hex, 88 is stop, 00 is full reverse, FF is full forward. F0 spins right in place, 89 is a slight turn to the left and so on. And optical flow is much better for sensing motion:

- Bumper: We can use the Accelerometer to detect when we have bumped into something, or when we are tipping

- Direction: We can look at the compass to keep the robot rolling straight:

- Faces: We can do face tracking:

- Voices: We can listen for voice commands:

- GPS: We might even be able to use GPS to drive to waypoints in a large enough space; like a parking lot or warehouse. 

All of that for less than $20 worth of parts. 

What's needed:
- A PCB with the light sensors, headers for the 2 (or 3? 4?) servos, power input, maybe some LEDs... I'd like a laser line driver. Let's not make it a shield, but instead something that plugs into the edge of any number of little controllers, like the RC Servo adapter. In fact, let's use that for the servo header, and just do the light sensors and LEDs on this one?

- A laser cut or 3D printed body that:
- - Holds the PCB with the light sensors against the bottom front of the Smartphone screen.
- - Holds the Smartphone vertically.
- - Holds the continuous rotation servos (with their wheels attached)
- - Holds a tailwheel, or bead, or something to provide a stable base.