Programming Application Notes
These notes define a scenario illustrating the application strategy we will follow to enable the Compute Module 3 to interface the 24Mp Sony sensor of the PONF camera.
Software Sources
All the referred documents and links mentioned and attached to this document are part of the publicly available documentation on the Videocore IV and Broadcom BCM2837 SoC multi core processor, mostly coming from the Broadcom documentation site itself.
The mentioned sources and sources links are available on GitHub released under some GNU Open Source license, without violating any copyrighted information.
The PONF multi camera project is an Open Source one. Of course, due copyright, not every detail will be made publicly available.
Videocore IV Data Acquisition
The Raspberry P. I compute Module 3 is based on the Broadcom soc together other components this processor includes the main Cpu and the Videocore IV, the graphic processor.
What are the advantages of this kind of architecture? The main processor runs Linux, but this multitasking operating system can't run real-time. To process images data streaming from a camera image sensor through we need it instead; this is the role of the Videocore IV GPU.
On the quad-core Videocore IV graphic processor run the VTOS, a real-time operating system just to solve this problem. This is the Broadcom BCM2837 SoC scenario in few words. VTOS RTOS use part of the Linux RAM reserved to the video, usually 128 Mb when a Raspberry Pi camera is installed in the system:
- ARM1176JZF-S 700 MHz processor which acts as the "main" processor and typically runs Linux.
- Dual-core Videocore IV CPU @250MHz with SIMD Parallel Pixel Units (PPU) which runs scalar (integer and float) and vector (integer only) programs. Runs ThreadX OS, and generally coordinates all functional blocks such as video codecs, power management, video out
- Image Sensor Pipeline (ISP) providing lens shading, statistics and distortion correction
- QPU units which provide 24 GFLOPS compute performance for coordinate, vertex and pixel shaders. Whilst originally not documented, Broadcom released documentation and source code for the QPU in 2014
There is a software mechanism to load the VTOS when the Linux OS boot.
After boot Linux can communicate to the GPU operating system through a series of documented API.
How the Camera Sensor Works
The camera sensor operates through two types of communication lines. Control and setting commands are sent through the I2C serial protocol. In the meantime, after sending the I2C commands to set up the image sensor for image acquisition the camera application initializes the VTOS driver to receive real-time the pixel data row by row.
The fast image data stream is granted by the VTOS driver through two differential lines, using the pins of the GPIO supporting this feature.
All of this is possible because Videocore IV includes the Pl camera driver.
24Mp Image Sensor Approach
The PONF sensor has higher resolution but works with similar principles; differs in the number of differential channels used to stream the image to the Videocore IV. In our case we need four differential channels, corresponding to a total of eight GPIO pins. According to the Compute Module 3 and Broadcom Soc pinout specifications, the lines we need are available and the GPU is able to manage all the channels through the VTOS real-time OS.
References and Available Documentation
A series of public information, documentation and source code is available on the net providing detailed information to drive the design changes to move to the new 24Mp image sensor.
Videocore IV 3D Architecture Reference Guide: Provided by Broadcom includes an in-depth detailed description and punctual reference on how the Videocore works and how it relates to the rest of the Soc. This is a highly valuable resource as it provides both hardware and software reference for the quad-core hardware.
Videocore IV programmers manual
MIPI Specifications from the MIPI site
QPU Driver from raspberry pi ARM core
Most of the useful information and references come from the GitHub Herman Hermitage repository: https://github.com/hermanhermitage/videocoreiv
Another source of important information is the GitHub Raspberry PI Userland: Source code for ARM side libraries for interfacing to Raspberry PI GPU. These sources are a good starting platform to setup the Linux side interfacing applications to the new 24Mp sensor.
(https://github.com/raspberrypi/userland )
Changes and Upgrades
Due to the characteristics of the differential channels, the physical wires connections between the Broadcom BCM2837 SoC and the 24Mp image sensor should be of the same length. This is the most important setting to be taken into account when designing the PCB during the circuit routing. The nets wiring the differential GPIO pins should be exactly of the same length, according to the specifications mentioned above.
Software Driver
The creation of the specific image sensor driver following the hardware specifications is the key step in the new design approach. The new driver implementation is derived from the current one available in the Raspberry PI Linux Raspbian distribution, modified to support the wider data stream of the new sensor. The low-level driver will run in the VTOS environment.
Sensor Control Software
The 24Mp new sensor is controlled via a fast SPl serial protocol instead of I2C; a new series of Linux commands should be implemented via a library exposing a set of API. to control the sensor. The commands will setup the driver, manage the data buffers and the other control parameters. The new command set will have a similar behavior of the already well-known Picamera Linux utilities. As far as we know, according to the Videocore IV software and programming documentation the header files already support in a similar way the commands and parameters to set the image sensor: resolution, image size, frame rate etc.
The Videocore IV methods also interface the hardware features of the Broadcom BCM2837 GPU for some image post-processing features, sensibility settings, RGB values settings etc.
According to the hardware specifications of the Videocore IV exposes post-processing performances to provide a ready-to-use sensor image in a buffer organized in pixel rows. We can compare this approach to a sort of memory file handle that the Linux side can easily convert and store as RAW data file or compressed in JPEG format.
Note that the Videocore IV also supports mp4 image compression; in future, a in-depth investigation will see if the camera can include video acquisition.
Previous article
PONF Project: Road Testing the Compute Module 3 as Desktop