Janus Fleet

Part 2 of Microservice Architectures for a Scalable and Robust Metaverse

The Janus multi-server enables multiplayer and is allows one client to share information with another client. The server software talks to every connected JanusVR client to know who else is in the same virtual space and where other avatars are. It is what glues the Metaverse together.

It's time to give the Janus-Server the love and attention it needs to be more robust when receiving unexpected amounts of traffic. The solution I have been investigating is Kubernetes; a system for managing containerized applications across a cluster of nodes.

The idea is that you abstract away all your hardware from developers and the 'cluster management tool' sorts it all out for you. Then all you need to do is give a container to the cluster, give it some info (keep it running permanently, scale up if X happens etc) and the cluster manager will make it happen.

Greek for "Pilot" or "Helmsman of a Ship"

Kubernetes builds upon a decade and a half of experience at Google running production workloads at scale, combined with best-of-breed ideas and practices from the community.
Reference to Google paper [Lesson's Learned while developing Google's Infrastructure]
Production ready as of 1.01: https://github.com/kubernetes/kubernetes/releases/tag/v1.0.1

Key Points:

Isolation: Predictable, idealized state in which dependencies are sealed in.
Portability: Reliable deployments. Develop here, run there.
Efficient: Optimized packing; idealized environments for better scaling.
Robust: Active monitoring, self healing. Easy and reliable continuous integration.

Introspection is vital. Kubernetes ships with [ https://github.com/google/cadvisor ] cadvisor for resourcce monitoring and has a system for aggregating logs to troubleshoot problems quickly. Inspect and debug your application with command execution, port forwarding, log collection, and resource monitoring via CLI and UI

Kubernetes handles scheduling onto nodes in a compute cluster and actively manages workloads to ensure that their state matches the users declared intentions. Using the concepts of "labels" and "pods", it groups the containers which make up an application into logical units for easy management and discovery.
https://kubernetes.io/docs/

Best practice for containerized applications is to run one process per container. The node.js server should be linked to the mysql database container. Decoupling applications into multiple containers makes it much easier to scale horizontally and reuse containers. Let's start with the node.js portion. Note that this current implementation is nearly working. Make sure to have Docker installed, then docker pull node in a terminal.

**This current Dockerfile is still broken**

FROM node
EXPOSE 8080
RUN git clone https://gitlab.com/alusion/avalon-server.git
RUN cd avalon-server && npm install && npm install forever -g && \
    npm install forever-monitor
WORKDIR avalon-server/
RUN chmod +x start_node.sh
ENTRYPOINT ["/avalon-server/start_node.sh"]

forever is a simple CLI tool for ensuring that a given script runs continuously.

config.js is configured with hardlinks currently, this will be changed in the future.

I have some questions for the linux container community:

Does it matter if the node app be run as user as opposed to root?
Why does the server seem to exit after it starts?
How does one generate secure certificates without the interactive prompt? (my guess is a bash script but who wants to config?)

The start_node.sh script looks as follows:

# Generate the certificates
sh generate_key
# Start the server
node server.js
# Not sure yet where to add this but the container keeps exiting.
# tail the logs
# tail -f server.log

If I run tail -f server.log, the container will stay open but the container will fail to build because it hangs there.

In order to get the mysql container, type into a terminal docker pull mysql

** This Dockerfile is still broken **

FROM mysql
ADD janusvr.sql /tmp/
RUN adduser --home /home/janus --disabled-password --gecos '' janus
RUN MYSQL_ROOT_PASSWORD="changeme"
RUN MYSQL_DATABASE="janusvr"
RUN MYSQL_USER="janusvr"
RUN MYSQL_PASSWORD="janusvr"
CMD mysql -u janusvr -p janusvr janusvr < /tmp/janusvr.sql
EXPOSE 3306

The latest MySQL image has environment variables passed in for configuring the database.

## ------ MySQL Container ------ ##
# mysql basics #
SHOW DATABASES;
USE janus;
SHOW TABLES;
SHOW FIELDS FROM users;
DESCRIBE users;
# Start a mysql server instance
docker run --name some-mysql -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql:tag
# Connect to MySQL from an application
docker run --name some-app --link some-mysql:mysql -d application-that-uses-mysql
# View MySQL Server log through Docker's container log
docker logs some-mysql

Help Needed

The janusvr.sql tables and the other files can be found here: https://gitlab.com/alusion/avalon-server

In order to test if the multiserver works, you can clone https://gitlab.com/alusion/paprika and just have to change line 18 in https://gitlab.com/alusion/paprika/blob/master/dcdream.html to point to the mutliserver as indicated by the Docker container:

<Room server="DOCKER_IP_ADDRESS" port="5567" ...

Docker for the Raspberry Pi: http://blog.hypriot.com/downloads/ might be good for testing...

Lead engineer of kubernetes describe differences between Mesos and Kubernetes + Fleet / Swarm.
http://stackoverflow.com/questions/27640633/docker-swarm-kubernetes-mesos-core-os-fleet

The Open Container Initiative is a lightweight, open governance structure, to be formed under the auspices of the Linux Foundation, for the express purpose of creating open industry standards around container formats and runtime.
http://www.opencontainers.org/

Another solution recommended by a brilliant hacker: http://opennebula.org/

To be continued...