Close

Ricoh Theta S 360 Streaming

A project log for Metaverse Lab

Experiments with Decentralized VR/AR Infrastructure, Neural Networks, and 3D Internet.

alusionalusion 01/28/2016 at 20:074 Comments

Thinking Outside the Bubble

Live-Streaming from a $350 360 camera into Virtual Reality

The Ricoh Theta S has achieved much praise within the VR community as being an easily affordable 360 camera that does all the things. The camera has 14MP resolution with spectacular looking images that are stitched from the camera to produce equirectangular format jpeg sizes: 5376x2688 / 2048x1024

Full HD (1,920 x 1,080 pixel) video time has been increased more than eight-fold to 25 minutes with the frame rate doubling to 30 fps since the last model. This camera has a live streaming and remote shutter options with 8gb of space available. Shutter speed can be set from 1/6400 sec. to 60 seconds allowing for excellent long exposures, excellent for light-painting. I plan to test the 360 light-painting concept with a friend who just acquired the Pixelstick:

Controlling the Camera

Linux command line can be used to control the camera HTTP API. Wireless LAN is used for communication between the client and RICOH THETA. RICOH THETA operates as an HTTP server when the wireless LAN is ON, and API can be used for GET requests and POST requests. You can read more about Open Spherical Camera API. Here's a few examples that use curl:

# Get Information
$ curl http://192.168.1.1:80/osc/info
{"manufacturer":"RICOH","model":"RICOH THETA S","serialNumber":"XXXXXXXX","firmwareVersigon":"01.11","supportUrl":"https://theta360.com/en/support/","endpoints":{"httpPort":80,"httpUpdatesPort":80},"gps":false,"gyro":false,"uptime":583,"api":["/osc/info","/osc/state","/osc/checkForUpdates","/osc/commands/execute","/osc/commands/status"]}
# Initiate Session
$ curl -X POST http://192.168.1.1:80/osc/commands/execute -d '{"name": "camera.startSession"}'
{"name":"camera.startSession","state":"done","results":{ "sessionId":"SID_0001","timeout":180}}
# Shoot a Still Image
$ curl -X POST http://192.168.1.1:80/osc/commands/execute -d '{"name": "camera.takePicture", "parameters": {"sessionId": "SID_0001"}}'
{"name":"camera.takePicture","state":"inProgress","id":"2","progress":{"completion":0.0}}
# Acquire Photographic State
$ curl -X POST http://192.168.1.1:80/osc/state
{"fingerprint":"FIG_0006","state":{"sessionId":"SID_0001","batteryLevel":1.0,"storageChanged":false,"_captureStatus":"idle","_recordedTime":0,"_recordableTime":0,"_latestFileUri":"100RICOH/R0010174.JPG","_batteryState":"charging"}}
# Get the File 
$ curl -X POST http://192.168.1.1:80/osc/commands/execute -d '{"name": "camera.getImage", "parameters": {"fileUri": "100RICOH/R0010174.JPG"}}' > image.jpg && open image.jpg
# Terminate Session
$ curl -X POST http://192.168.1.1:80/osc/commands/execute -d '{"name": "camera.closeSession", "parameters": {"sessionId": "SID_0001"}}'
{"name":"camera.closeSession","state":"done"}

There are also a number of scripts to get media from the camera. First you have to set the Ricoh Theta S to WiFi mode and connect to the access point. Here's a python example to test the API:

import urllib
import json
urllib.urlopen("http://192.168.1.1/osc/info").read()
data = json.dumps({"name":"camera.startSession"})
res = urllib.urlopen('http://192.168.1.1/osc/commands/execute', data)
sessionId = json.loads(res.read())["results"]["sessionId"]
data = json.dumps({"name":"camera.takePicture", "parameters": {"sessionId": sessionId}})
urllib.urlopen('http://192.168.1.1/osc/commands/execute', data)
fileUri = ""
while not fileUri:
    res = urllib.urlopen('http://192.168.1.1/osc/state', urllib.urlencode({}))
    fileUri = json.loads(res.read())["state"]["_latestFileUri"]
data = json.dumps({"name":"camera.getImage", "parameters": {"fileUri": fileUri}})
res = urllib.urlopen('http://192.168.1.1/osc/commands/execute', data)
with open("image.jpg", "wb") as file:
    file.write(res.read())
data = json.dumps({"name":"camera.closeSession", "parameters": {"sessionId": sessionId}})
urllib.urlopen('http://192.168.1.1/osc/commands/execute', data)

Opening the image can look something like this:

import subprocess
subprocess.Popen(['open', 'image.jpg'])

Here's a more advanced version that uses the better requests module: https://raw.githubusercontent.com/theta360developers/python-download-rossgoderer/master/ThetaS-Download.py

There's also a Node.js client that is awesome and makes taking a picture simple as:

% theta --capture
% theta --capture out.jpg
Available options:
  -h, --help                     show help
      --capture [FILENAME]       take a picture
      --list                     list pictures
      --handle [Object Handle]   specify picture by Object Handle
      --save [FILENAME]          save picture
      --delete [Object Handle]   delete a picture
      --info [Object Handle]     show picture info
      --battery                  check battery level
      --volume [NUMBER]          get/set audio volume (0~100)

By default, the Ricoh Theta will produce still images in the Equirectangular format that can then be wrapped around a sphere for viewing as I have shown how to do in my first project log: https://hackaday.io/project/5077/log/15961-360-photospheres

This type of projection is better for processing with the neural networks [Sublime] but it has proven not be the most optimized format for streaming. According to Facebook, moving from equirectangular layouts to a cube format in 360 video reduced file size by 25 percent against the original and encoding with a pyramid geometry reduces file size by 80 percent. These are significant improvements when dealing with scale. We'll come back to this later.

The other type of image that the Ricoh Theta generates is a Dual-Fisheye. The Ricoh Theta S generates this mp4 format video at sizes 1920x1080 / 1280x720.

My development platform of choice is the Raspberry Pi 2 running a fresh copy of Raspbian Jessie. One method of live-streaming that has worked for me in the past has been mjpeg-streamer. You can refer back to https://hackaday.io/project/5077-metaverse-lab/log/17307-avalon-pt-2 for notes on how to build AVALON on top of the Rpi2 and configure mjpg-streamer to initiate during start-up. The condensed version for installing looks like this: (http://jacobsalmela.com/raspberry-pi-webcam-using-mjpg-streamer-over-internet/)

lsusb
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install libjpeg8-dev imagemagick libv4l-dev
sudo ln -s /usr/include/linux/videodev2.h /usr/include/linux/videodev.h
sudo apt-get install subversion
cd ~
svn co https://svn.code.sf.net/p/mjpg-streamer/code/mjpg-streamer/ mjpg-streamer
cd mjpg-streamer
make mjpg_streamer input_file.so input_uvc.so output_http.so
sudo cp mjpg_streamer /usr/local/bin
sudo cp output_http.so input_file.so input_uvc.so /usr/local/lib/
sudo cp -R www /usr/local/www
sudo vi ~/.bashrc
     export LD_LIBRARY_PATH=/usr/local/lib/
source ~/.bashrc
/usr/local/bin/mjpg_streamer -i "/usr/local/lib/input_uvc.so -r 1280x720" -o "/usr/local/lib/output_http.so -w /usr/local/www -p 8090"
     #!/bin/bash
     /usr/local/bin/mjpg_streamer -i "/usr/local/lib/input_uvc.so -r 1280x720" -o "/usr/local/lib/output_http.so -w /var/www/stream -c username:password"
sudo chmod 755 /etc/init.d/livestream.sh
sudo update-rc.d livestream.sh defaults

This is not ready to be streamed yet because the video remains to be in Dual-Fisheye format. We can see that Ricoh Theta has a desktop program for Windows and Mac which will do stitching from Dual-Fisheye into equirectangular and add the video metadata but this is unavailable for streaming. Linux users running the program through Wine will be disappointed as the it fails to convert video or do real-time stitching for streaming into equirectangular format. The lens geometry for the THETA is based on equidistant projection. RICOH does not make detailed lens parameter information available. This is also known as lens distortion data. Developers often ask for this information to improve stitching. It is proprietary and not available as of December 2015. However, stitching is still possible without this information. GOROman released a shader pack for Unity game engine that UV maps each Fisheye view to a sphere pretty well.

A custom UV mapped is formed for each side then the two images are aligned together as close to being seamless as possible.

Inspired by this, VR developer FireFoxG helped to create a custom UV mapped object that can stitch the two Fisheye images as close as he can. These first attempts were pretty shabby:

I took a picture while holding grid paper to help visualize the alignment between the lenses. This has greatly improved the stitching:

The object file can be found here: Download as .obj

The process for streaming 360 video over the web involved setting up dynamic DNS and port forwarding on the router.

As mentioned earlier, it is found that the equirectangular format is not the best in terms of streaming. Cube maps have been shown to deliver significantly higher quality while being easier to compress. John Carmack was quick to point this early but we have recently seen a new tool from Facebook emerge called Transform, implemented as an ffmpeg video filter that transforms 360 video in equirectangular projection into a cubemap projection. Facebook's primary goal was to tackle the drawback of the standard equirectangular layout for 360 videos, which flattens the sphere around the viewer onto a 2D surface. This layout creates warped images and contains redundant information at the top and bottom of the image. For the size of Facebook, every optimization method will go a long way in scalability.

https://code.facebook.com/posts/1126354007399553/next-generation-video-encoding-techniques-for-360-video-and-vr/

When a 360 video is uploaded, we transform it from an equirectangular layout to the pyramid format for each of the 30 viewports, and we create five different resolutions for each stream, for a total of 150 different versions of the same video. We chose to pre-generate the videos and store them on the server rather than encode them in real time for each client request. While this requires more storage and can affect the latency when switching between viewports, it has a simple server-side setup and doesn’t require additional software

The other method of file format compression would be to use H 265 codec for encoding the video. It's extremely efficient for HD and 4K video, using a fraction of predecessor H.264 bandwidth. This provides the highest quality content even at low-bandwidth connections at sometimes half the cost of H.264. Sad to say there's a deal breaker to using this codec:

¯\_(ツ)_/¯

PS. RICOH, please release code for proper stitching or atleast a Linux version of your desktop program.

Discussions

James wrote 07/19/2017 at 10:35 point

Would RemapFilter do the fisheye to equirectangular job? https://trac.ffmpeg.org/wiki/RemapFilter

  Are you sure? yes | no

Michael wrote 08/16/2016 at 11:08 point

Great work! We solved the stitching problem with our #360live player. Feel free to test our solution - you just have to login our platform and take the encoder settings over to your encoder. Please give us feedback. If you want a free trial, please contact me.

  Are you sure? yes | no

James wrote 07/19/2017 at 10:52 point

Does your solution work with the Raspberry Pi?

  Are you sure? yes | no

naths0127 wrote 04/28/2016 at 12:20 point

Iv followed through you tutorial and I get this error /usr/local/bin/mjpg_streamer -i "/usr/local/lib/input_uvc.so -r 1280x720" -o "/usr/local/lib/output_http.so -w /var/www/stream -c username:password"



MJPG Streamer Version: svn rev: 3:172



 i: Using V4L2 device.: /dev/video0



 i: Desired Resolution: 1280 x 720



 i: Frames Per Second.: 5



 i: Format............: MJPEG



Adding control for Pan (relative)



UVCIOC_CTRL_ADD - Error: Inappropriate ioctl for device



Adding control for Tilt (relative)



UVCIOC_CTRL_ADD - Error: Inappropriate ioctl for device



Adding control for Pan Reset



UVCIOC_CTRL_ADD - Error: Inappropriate ioctl for device



Adding control for Tilt Reset



UVCIOC_CTRL_ADD - Error: Inappropriate ioctl for device



Adding control for Pan/tilt Reset



UVCIOC_CTRL_ADD - Error: Inappropriate ioctl for device



Adding control for Focus (absolute)



UVCIOC_CTRL_ADD - Error: Inappropriate ioctl for device



mapping control for Pan (relative)



UVCIOC_CTRL_MAP - Error: Inappropriate ioctl for device



mapping control for Tilt (relative)



UVCIOC_CTRL_MAP - Error: Inappropriate ioctl for device



mapping control for Pan Reset



UVCIOC_CTRL_MAP - Error: Inappropriate ioctl for device



mapping control for Tilt Reset



UVCIOC_CTRL_MAP - Error: Inappropriate ioctl for device



mapping control for Pan/tilt Reset



UVCIOC_CTRL_MAP - Error: Inappropriate ioctl for device



mapping control for Focus (absolute)



UVCIOC_CTRL_MAP - Error: Inappropriate ioctl for device



mapping control for LED1 Mode



UVCIOC_CTRL_MAP - Error: Inappropriate ioctl for device



mapping control for LED1 Frequency



UVCIOC_CTRL_MAP - Error: Inappropriate ioctl for device



mapping control for Disable video processing



UVCIOC_CTRL_MAP - Error: Inappropriate ioctl for device



mapping control for Raw bits per pixel



UVCIOC_CTRL_MAP - Error: Inappropriate ioctl for device



 o: www-folder-path...: /var/www/stream/



 o: HTTP TCP port.....: 8080



 o: username:password.: username:password



 o: commands..........: enabled

Any idea how to resolve this?

  Are you sure? yes | no