The more I think about it, this really seems like something that should be done. Route xwindows client requests over a serial console to an xwindows server app listening on USB OTG. Key points:
1. No need for network hardware or setup. E.g. work s with a RasPi Zero / MicroZed / other SBC that doesn't have video on board. Even if you have WiFi / CAT5, you have to configure that somehow, and this would make that easy or unnecessary. 

2. Based on existing work. E.g. your Linux desktop or xwindows apps, xserver code, etc... No need to re-invent the wheel to get a gui going. Please note: Yes xwindows was developed for network connections, but that doesn't mean it can't work over other links. 

3. Uses commonly available and cheap hardware to provide a very nice user interface. Android tablets are a dime a dozen. And if it's a temporary need, you can just pull out your phone.

Someone said "write an app for it" but I honestly don't have the time or the experience. I'd pay someone (far less than they were worth) to write it, just because I think it needs to exist. Anyone? Anyone? Bueller?

You should be able to do this by running SLIP or PPP over the serial link. Hopefully the bandwidth of the serial link is high enough. Note that it's not just a matter of "route xwindows client requests", you also have to send back the X replies. The exchange can include large chunks of graphical data. Also the protocol is asynchronous, not lock step or the performance will be terrible. Messing with the innards of X to add a different transport is non-trivial. I ported the X server to a non-Unix platform way way back. Much easier to use available shims.

Other means you might consider are RDP or VNC over the serial link.

If you 'console' is via USB-OTG networking, you are good to go and just need to figure out what is blocking you.  If you are talking about a serial console, you have some work to do but it is possible..  You'll probably need to change what device the robot end of your USB is emulating when connecting via USB.  You can emulate different flavors of usb devices.  usb-serial presents tty's, usb-ethernet presents networking devices, mtp presents that crappy camera interface.. etc.  This is assuming you aren't just connecting to something tied in via an FTDI chip of course.  In that case, serial is your limit and you aren't going to get X running on that (well you could I suppose, but your will be disappointed.  :).  More information needed on your 'server'.

no problem with the USB console. I can open a USB terminal app on the android and connect just fine. The issue is 1. Finding an app on the android which can present a GUI rather than a text console. 2. Sending lxde or other GUI desktop presentation from the Linux robot, xwindows works fine over the network, but doesnt transport over USB. So then setting up the network on a new robot is a pain,

Yeah, you haven't differentiated between if it is using usb-serial or usb-ethernet.  That is the deciding factor here.  Log in.  See what your current tty is assigned to.  If it is serial, see if your platform can change that can be network instead.  Some information about what you are plugging into would be useful as well.

The USB console is a /serial/ port. It is ttyPS0 via a CP2102 serial to USB converter. The device is a MicroZed board, basically Ubuntu 16.04. The Android device, via a USB OTG convert cable, is any one of a number of cell phones, tablets, etc... 

@James Newton Yeah, that's going to be a lot harder.  You'll be piping stuff that works > 1mbit minimum over a maximum 500kbit pipe.  It could work, but you would be better served plugging in a usb wifi device.  Instant hack may be to setup a PPP connection via the serial link and move the stack up one level, if you really wanted to pursue using the standard stuff.  Good luck.

Ah. Well then, I guess what I want isn't xwindows. I just want to add a "display" and "keyboard" to my "RPi" via an android tablet. 

Write an app for it.

in my other spare time.

X requires a TCP/IP connection. Have you tried tunnening lt over ssh?

If not you could tether your RPi to the phone using USB or WiFI tethering. You'd have to get the phone's IP address (it will be the DNS relay in the DHCP lease). Not sure if Android runs a firewall which will block clients on the tethering subnet from connecting to the X server. The drawback of tethering is that you won't be able to use the WiFi connection.

The other way is if both the phone and the RPi are on the same WiFi subnet, in which case you'd need to find the IP address of the phone. This can be seen in About Phone. If you have control over the DHCP server you could pin the address of the phone. Again there is the question of whether Android blocks incoming connections to the X server running on it.

Finally if you are able to root Android you could install a USB Ethernet driver. You might have to root the phone anyway if the firewall gets in the way.

X tunneling works fine over the network. The point of the exercise is to NOT use the network, and instead communicate via a serial to USB port to an Android device to use that as a GUI. Then you can use that to setup your Network (or whatever) easily... or not setup the network and just work directly. So... no video generating circuit on the unit (think RasPi Zero or compute) and no network, just USB. Now, use an android device as the screen.

IP networks are so ubiquitous these days, witness the rise of IP over WiFi, that bespoke protocols over serial and USB don't get much of a look in because it has to be developed for every new device, whereas once you implement IP for that device you get all the other stuff based on IP.

You might be able to find an Android framework for writing GUI apps to send commands and receive responses over terminal connections. Maybe something like this:

https://www.androwish.org/home/home