We made Reality as a Service (RaaS) https://raas.perciplex.com, an open source platform for hosting physical OpenAI Gym environments.
You can use it by submitting a GitHub repo with python controller code to a web based front end. The python interface matches the standard OpenAI gym interface https://gym.openai.com/. A job queue is popped by robot instances and the results of this run are then published on the web interface. You can also watch them run live on a twitch stream https://www.twitch.tv/perciplex.
All code and hardware (which is intentionally minimal, cheap, and off the shelf) plans are available in the repo listed below.