I have been exploring the possibility of creating a 360 camera for street motion capture for a while. It all started with an article I came across on the internet about Surround360 - a solution that was designed and released as open-source by the Facebook team a few years ago. Unfortunately, the solution turned out to be far from budget-friendly: by the most optimistic estimates, assembling the setup would cost around $30,000.

The Facebook design includes certain extravagances, which, if eliminated, could significantly reduce the cost of the construction. For instance, the use of 14 cameras in a circular arrangement is justified for capturing depth information and subsequent processing, but it is entirely excessive for capturing urban street scenes.

Using the ready-made GS3-U3-41c6c-c module ($1,500/unit) simplifies the assembly and configuration of the camera, but it really hits the wallet hard.

On the other hand, when aiming for cost reduction, it's crucial to maintain camera synchronization. Even a slight delay of a few milliseconds can impact the quality of the stitched panorama.

Achieving perfect synchronization is possible with Global Shutter cameras, but such cameras are much more expensive than the conventional Rolling Shutter ones, and their resolution is lower. Fortunately, Sony offers a solution for synchronizing N Rolling Shutter cameras in their sensors. It's available, for example, for IMX415 or IMX477 sensors, and it's precisely the latter sensor that will be used in this project.

The Raspberry Pi High-Quality Camera module is quite old and provides a RAW image output, unlike its counterparts from OmniVision, for example. RAW means that the receiving end of the image will need to analyze it and provide "feedback" to the sensor in the form of exposure settings, white balance, and autofocus. When it comes to autofocus on the RPI HQ camera, it's straightforward: the focus is mechanically adjusted once for the entire sensor's lifespan. However, exposure and white balance will still need to be configured. Solutions can be purely software-based or include hardware acceleration.

My first prototype of 360 camera was made of 6 raspberry pi zero cameras and single board which aggregated the received data.

However, quite soon I found that this design has a lot of flaws:

• each RPI board has h264 encoder limit of 1080p which cannot handle full 12MP resolution;
• without h264 encoding there is no way to save raw data on single or even different SD cards with good FPS rate;
• power consumption of 7 boards becomes a pain and hell of wires.

All that said, I desided to move to another SOC instead of RPI. Luckaly, RockChip recently presented the rk3588s SoC which has 3 mipi-csi2 connectors on a single board and NVME slot, which makes it almost ideal candidate for a control board, allowing to control up to 8 cameras with only two boards.

OrangePI sells 8Gb version of PI 5 board for only 90$ each, making it the cheapest board currently available on market with this type of characteristics. 

When using single-board computer cameras, sooner or later you encounter the MIPI CSI-2 standard. This standard has been around for a while and is widely used, for example, in mobile phones. The nuance, however, is that manufacturers independently choose the connectors for these cameras.

Even Raspberry Pi, in its boards, uses two different connectors: one with 15 pins and another with 22 pins. In theory, the 22-pin connector can support 4 MIPI lanes instead of two (which means transmitting twice the number of frames), but in practice, on the PI Zero, the additional two lanes are simply not wired.

The manufacturer of the OPI 5 thoughtfully released the pinout diagram of the connections used on the board, making it available to the public.

Let's quickly draft a diagram for the simplest adapter from one interface to another, order it from JLCPCB, simultaneously purchasing a dozen connectors from Aliexpress. We'll put everything together and realize that nothing is working.

After studying another dozen PDFs with watermarked "confidential!11" labels, we'll find our mistakes:

• RPI uses a logic level of 3.3V for I2C, while the OPI uses 1.8V.
• the same voltage level issue exists with pins 11 and 12 of the camera, which control the power.
• RK3588s board does not have any driver for IMX477.

That would be our next steps:

• create Rockchip-compatible driver for IMX477.
• create hardware adapter allowing to connect RPI HQ camera to OPI 5 board.