Close

X11 Application Migration pt. 1

A project log for Metaverse Lab

Experiments with Decentralized VR/AR Infrastructure, Neural Networks, and 3D Internet.

alusionalusion 06/06/2015 at 03:443 Comments

I needed more than just having an SSH tunnel from within Janus. The ability to have remote X applications migrated into VR gives a metaverse developer much more power to access and create content without having to leave the client. I strongly believe that when done right, this is to be the next great leap in computing. The office can exist anywhere -- it doesn't carry the limitations of physical hardware and can be accessed with a Virtual Reality headset which is a multiplier of how many displays you can interact with instead of the single monitor on current laptops / tablets. You have better privacy as well since people can't shoulder surf you when the screen is directly in front of your eyes. I think that you, the reader, will understand better once you see a functional computing environment in Virtual Reality.

First, I knew that accessing VNC over the web browser is nothing new and that I wanted to do things differently. I began to look at methods of transporting individual x applications, learning much about how X server and websockets work in the process. The X server code is more complicated than you might think, which is the bad for security.

I understand that simpler and more maintainable compositors are in the works (Wayland) but I just wanted to make a point that the way we interact within VR in the future would abstract greatly from where we are today using 2D interfaces and a keyboard/mouse. JanusVR already supports leap-motion and soon we can walk around in 3D environments with technology coming from Valve such as Lighthouse later in the fall. Tomorrow we will be interacting with computers with via voice, touch, sight, and movement. A good example I can think of is how we currently use Blender, a free and open source 3D modeling program. It's too difficult for someone to use Blender for simple operations. I'm looking forward to programs such as Tilt Brush that use Virtual Reality to create 3D content. Anyways, PoC || GTFO.

I began to look into WebSockify, a WebSocket to TCP proxy/bridge that allows a browser to connect to any application/server/service. WebSockets traffic is translated to normal socket traffic and traffic is forwarded between the client and target in both directions. Another useful piece to know is Xvfb: (X Virtual Frame Buffer) is an X server that can run on machines with no display hardware and no physical input devices. It emulates a dumb framebuffer using virtual memory. The virtual server does not require the computer it is running on to have a screen or any input device, only a network layer is necessary. An alternative to Xvfb is Xdummy: a script that uses LD_PRELOAD to run a stock X server with the "dummy" video driver. Next, I needed to find a way to process X programs through a web browser where I can view it inside Janus...

Xpra (X Persistent Remote Applications) is free and open source software that allows you to run X11 programs from a remote host and direct their display to your local machine, and display existing desktop sessions remotely. You can disconnect from these programs and reconnect from another machine without losing any state as well as having remote access to individual programs. Compared to VNC, Xpra is rootless and applications forwarded by Xpra appear on your desktop as normal windows managed by your window manager. Think of it as 'screen' for X11. It connects as a compositing window manager to a Xvfb display server that takes the window pixels and transports them into a network connection to the Xpra client.

TL;DR: Xpra is 'screen for X': allows you to run X programs, usually on a remote host and direct their display to your local machine. Xpra works by connecting to an Xvfb server as a compositing window manager.

I learned during my initial tests that the Ubuntu repo maintained an older version of Xpra so the first step will be to upgrade to the latest version. Winswitch is a frontend to Xpra and the latest version (0.15.0) is bundled with it. The instructions for downloading it are here, and for Ubuntu 14.04:

# instructions must be run as root
sudo su -

# Import the packagers key
curl http://winswitch.org/gpg.asc | apt-key add -

# Add repo and install winswitch (comes with latest version of xpra)
echo "deb http://winswitch.org trusty main" > /etc/apt/sources.list.d/winswitch.list;
apt-get install software-properties-common >& /dev/null;
add-apt-repository universe >& /dev/null;
apt-get update;
apt-get install winswitch
# Must have these also installed for server with HTML5 support
apt-get install xvfb websockify
* You can also download a bleeding edge package from here or stable builds from here.

Now, to test that it works we will try forwarding xterm into the browser window.

xpra start --bind-tcp=0.0.0.0:10000 --html=on --start-child=xterm
Point the browser / log in through Janus and browse to http://localhost:10000, the xterm should now appear in the browser window.

Login via 127.0.0.1 and port 10000. The xterm should appear on a WebSurface asset within Janus. Single application forwarding works! Xpra is successfully forwarding single X11 programs through to my web browser and into Janus. I can potentially distribute programs individually to people via different ports and display servers. Another idea might be to to use Docker containers for distributing applications with into VR. Just know that the applications don't need to be mounted on a wall and can be held in your hands like a tablet or displayed transparently like an AR-HUD interface. The WebSurface is an Asset that can be applied to any kind of 3D object. The desktop is no longer restricted to a single display or monitor. but can be dispersed in a virtual environment. It is sort of like the difference between a television and a projector, your applications are free to be projected onto any surface in virtual reality! This is the beginning of a 3D window manager built inside a Metaverse application.

For my next test, I used Xnest which can be used to run a virtual desktop of another computer within a window. On separate terminals my command looks like this:

# Initiate the Xpra server with nested X server
xpra start --bind-tcp=0.0.0.0:10000 --html=oc --start-child="Xnest :200 -ac -geometry 1280x800+24" :100

# Start awesome window manager on :200 from separate terminal
DISPLAY=:200 awesome&

# tail the log file
tail -f /home/user/.xpra/:100.log

Now you can point the websurface URL to http://localhost:10000.

Not only it works but regular X11 applications can easily be launched from within too!

Tabbed browsing is possible because now you are interacting with YOUR web browser. It has all of your bookmarks and plugins loaded. This is an extension of YOUR desktop environment into JanusVR. Making rooms in Janus is much easier when you have access to your desktop; there's less need to exit the client.

The next step I am working on is to free the applications from being clumped together on a single plane. Imagine having a file browser in your hands or the image viewer on the wall of your room. The desktop environment expands beyond your screen and into the world around you as part of the space. Individual X11 window applications can be represented into virtual 3D objects using this method. The combination with low latency SSH tunneling is going to initially enable the developers to do more from inside Janus. My goal goes much further beyond that however as I aim to decentralize, demonetize, and distribute functional computing environments within collaborative Metaverse spaces. When people hunger for more beyond simply gaming or consuming media on their VR devices, I'll be there to give them productive multi-user environments that will enable them to work in ways never before possible.
To be continued...

Discussions

alusion wrote 01/20/2016 at 21:09 point

Very cool, I'll be looking into this!

  Are you sure? yes | no

bmullan.mail@gmail.com wrote 01/21/2016 at 14:32 point

by the way... I have a script which will install, configure guacamole, mysql, nginx et al for you if you'd like to use it.  Obviously, it still doesn't solve how to get Guacamole's ssh to support X11 forwarding but since Guacamole does support websockets I guessing someone who "is" a programmer could get that to work also.

Just send me an email if you're interested.

brian

  Are you sure? yes | no

bmullan.mail@gmail.com wrote 01/20/2016 at 03:16 point

I just happened upon your post because of some recent work with xnest & other remote X desktop solutions.  Sorry to say I don't know what JanusVR is ... yet but reading your post I will be looking it up.     

I'd also implemented a Guacamole HTML5 solution for myself that used mysql, nginx, xrdp and LXC containers (Guacamole currently only supports vnc, ssh & rdp) but it's ssh doesn't support ssh X11 Forwarding or it would be a good solution.  

 However, Guacamole (www.guac-dev.org) does have an extendable plugin API which perhaps you might be able to use for this?   Anyway, your works sounds great from reading this.

You might want to look at the Guacamole Devlopers Guide -

http://guac-dev.org/doc/gug/developers-guide.html

Brian

  Are you sure? yes | no