Group Actions
Engagement
  • Author Author: albertabeef
  • Date Created: Date Created
  • Views 3990 views
  • Likes 3 likes
  • Comments 1 comment
Related
Recommended

Ultra96-V2 Dual Camera Design for 2021.2

albertabeef

Introduction

This blog post provides details on how to build and execute the new 2021.2 design for the Ultra96-V2 development board with Dual-Camera Mezzanine.

image
 

Design Overview

The following block diagram illustrates the 2021.2 version of the hardware design.

image

The following images capture the resource utilization of the design.

imageimage

 

When compared to the previous version of the design 2020.1, the following changes have been made:

  • removed the OSD core, including the entire live DisplayPort (DP) sub-system, which was determined to not be required
  • modified the capture pipeline
    • added a VPSS core for color space conversion
    • added BGR format support to the frame buffer write core
  • added of the AXI interrupt logic for use with Vitis

When compared to the previous version of the design 2020.2 (https://avnet.me/u96v2-dualcam-2020.2), the following changes have been made:

  • update ap1302 drivers 
    • add controls for brightness, contrast, saturation, gamma, exposure, gain 
    • add controls for auto-focus, horizontal/vertical flip, 3d_path
  • update launch scripts

 

Supported Sensor Modules

The following sensor modules are supported by the linux-based 2020.2 design:

 

Although the Dual Camera Mezzanine comes with two (2) AR0144 IAS sensor modules, it also supports a single AR1335 sensor module.

 

Capture Pipeline Overview

The linux-based 2021.2 design supports a statically configurable capture pipeline for the following three configurations:

  • Dual AR0144 Sensors
  • Single AR0144 Sensor
  • Single AR1335 Sensor

 

The dual-camera mezzanine makes use of MIPI in order to connect the image cameras to the processing board.

 

The hardware design implemented in the programmable logic (PL) includes the following components:

  • MIPI CSI-2 RX receiver IP core
  • Image Pipeline : implemented with Color-Space-Conversion, and Scaler
  • Frame Buffer Write : the DMA engine implementing writes to external DDR

 

Dual AR0144 Camera Pipeline

The following image illustrates the capture pipeline for the Dual AR0144 configuration.

image

It is important to known that the AP1302 ISP receives the stereo images and generates a single side-by-side-image, as shown below:

image

 

 

Although the side-by-side image reflects the frontal view of the dual camera mezzanine, it is important to know that it contains:

  • image from left (L) camera (SENSOR2) on right side
  • image from right (R) camera (SENSOR1) on left side

 

This clarification is essential for any stereo processing that is attempted.

When in doubt, or to convince yourself, place your finger in front of one of the cameras and notice which side of the side-by-side image is blocked.

 

Single AR0144 Camera Pipeline

The following image illustrates the capture pipeline for the Single AR0144 configuration.

image

It is important to known that the single AR0144 configuration expects the sensor to be populated on the sensor1 header, as shown below:

image

 

Single AR1335 Camera Pipeline

The following image illustrates the capture pipeline for the Single AR1335 configuration.

image

It is important to known that the single AR1335 configuration expects the sensor to be populated on the sensor1 header, as shown below:

image

 

Rebuilding the Design

 

This section describes how to re-build the linux-based 2021.2 design.

 

This can be done on a linux machine, which has the Vitis 2021.2 and Petalinux 2021.2 tools correctly installed.

 

The following commands will clone the Avnet “bdf”, “hdl”, and “petalinux” repositories, all needed to re-build the linux-based design.

$ git clone https://github.com/Avnet/bdf

$ git clone –b 2021.2 https://github.com/Avnet/hdl

$ git clone –b 2021.2 https://github.com/Avnet/petalinux





 


To rebuild the linux-based design, use the following commands:




$ cd petalinux


$ ./scripts/make_u96v2_sbc_dualcam.sh




 


This will launch the build for the hardware design and petalinux project.  The build artifacts that are of interest are the following:


  petalinux/projects/u96v2_sbc_dualcam_2021_2/images/linux/BOOT.BIN


  petalinux/projects/u96v2_sbc_dualcam_2021_2/images/linux/boot.scr


  petalinux/projects/u96v2_sbc_dualcam_2021_2/images/linux/image.ub


  petalinux/projects/u96v2_sbc_dualcam_2021_2/images/linux/rootfs.tar.gz


 


By default, the device tree will be built for the dual AR0144 configuration.  If you want to use a different configuration, edit the following file:


   petalinux/projects/u96v2_sbc_dualcam_2021_2/project-spec/meta-on-semiconductor/recipes-bsp/device-tree/files/system.dtsi


Un-comment only one of the following lines corresponding to your desired confiuration, then re-build the petalinux project.


   #define CONFIG_STATUS_AR0144_DUAL


   //#define CONFIG_STATUS_AR0144_SINGLE


   //#define CONFIG_STATUS_AR1335_SINGLE






These binaries can be programmed to a dual partition micro SD card, as follows:


    

  • BOOT partition (FAT format)

      

    • BOOT.BIN
      • 

    • boot.scr
      • 

    • image.ub
      • 

    

    • 

  • ROOTFS partition (EXT4 format)

      

    • contents of rootfs.tar.gz, extracted as root
      • 

    

    • 



 


 


 


Executing the Design with the Dual AR0144 configuration


After booting the board with the SD card, ensure that the /etc/init.d/load_default_camera_config script is specifying the following configuration


  DAEMON=/usr/bin/load_camera_config


  DAEMON_OPTS="ar0144_dual"


 


If this is not the case, edit the configuration, then reboot the design.


 


The X-windows desktop needs to be stopped in order to run the passthrough example, which is achieved with the following command:




$ /etc/init.d/xserver-nodm stop




The dual camera passthrough can be launched with the following command:




$ source run_1920_1080




This script will perform the following steps:


    

  • configure capture pipeline configuration using media-ctl utility
    • 

  • configure monitor resolution using modetest utility
    • 

  • launch gstreamer pipeline using gst-launch-1.0 utility
    • 



 


The following test pattern will be visible for ~1 sec (after execution of modetest).


 








image

 


Once the gstreamer pipeline is launched, the following side-by-side image will be visible on the monitor.  Notice that the width is compressed by a factor of 2, which is only to make the dual images fit on the monitor.








image




 




The lens for each AR0144 can be manually rotated to adjust the focus, if required.


 


Side-by-Side configuration


As previously mentioned, the default side-by-side configuration defaults to the following:


    

  • image from left (L) camera (SENSOR2) on right side
    • 

  • image from right (R) camera (SENSOR1) on left side
    • 



This configuration can be queried with the following command:




$ v4l2-ctl --get-ctrl 3d_path -d /dev/video0
3d_path: 0




This configuration can be changed by writing "1" to the "3d_path" control, as follows:



$ v4l2-ctl --set-ctrl 3d_path=1 -d /dev/video0




 



This will change the side-by-side configuration to the following:




    

  • image from left (L) camera (SENSOR2) on left side
    • 

  • image from right (R) camera (SENSOR1) on right side
    • 





 


Executing the Design with the Single AR1335 configuration


After booting the board with the SD card, ensure that the /etc/init.d/load_default_camera_config script is specifying the following configuration


  DAEMON=/usr/bin/load_camera_config


  DAEMON_OPTS="ar1335_single"


 


If this is not the case, edit the configuration, then reboot the design.


 


The X-windows desktop needs to be stopped in order to run the passthrough example, which is achieved with the following command:




$ /etc/init.d/xserver-nodm stop




 


In order to display the full resolution of the AR1335 sensor (3840x2160), the QoS configuration needs to be optimized for the DisplayPort (DP) monitor.


Otherwise, the DP controller will starve for bandwidth to external memory, and the image will contain glitches.


This is accomplished with the following command:




$ optimize_qos_for_dp




 


The 4K camera passthrough can be launched with the following command:




$ source run_3840_2160




This script will perform the following steps:


    

  • configure capture pipeline configuration using media-ctl utility
    • 

  • configure monitor resolution using modetest utility
    • 

  • launch gstreamer pipeline using gst-launch-1.0 utility
    • 



 


The following test pattern will be visible for ~1 sec (after execution of modetest).


image


Once the gstreamer pipeline is launched, the following image will be visible on the monitor.


image




Auto-Focus


There is no need to manually turn the lens for the AR1335 sensor module, since it implements Auto-Focus functionality.


 


The CAVBA-000A datasheet indicates the auto-focus is implemented with the following components:


    

  • Lens Type : Largan 50013A7
    • 

  • VCM Type : Jahwa QJ4C
    • 

  • VCM Driver : DW9790A
    • 



 


image


In order to power the VCM device, the following jumper setting needs to be placed in the 1-2 position on the dual camera mezzanine.


 


image


 


The Voice Coil Motor (VCM) is under control of the on-board AP1302 ISP device.


Although the AP1302 firmware has Auto-Focus "off" by default, it can be enabled by the user.


 


 




This configuration can be queried with the following command:




$ v4l2-ctl --get-ctrl auto_focus-d /dev/video0
auto_focus: 0




This configuration can be changed by writing "1" to the "auto_focus" control, as follows:



$ v4l2-ctl --set-ctrl auto_focus=1 -d /dev/video0




 


 


Where to Buy the Hardware


 


The following links describe the hardware requirements needed to run this design.


 


Ultra96-V2


Buy Ultra96-V2


or Ultra96-V2 I-grade


Buy Ultra96-V2 I-grade


 


Ultra96-V2 4A Power Supply


Buy Ultra96-V2 4A Power Supply


 


Ultra96-V2 JTAG/UART Adapter


Buy Ultra96-V2 JTAG/UART Adapter


 


96Boards onsemi Dual Camera Mezzanine


Buy 96Boards onsemi Dual Camera Mezzanine


 


Going Further


 


A Vitis-AI 2.0 design has been built on top of this 2021.2 design, and is documented as a hackster.io project.


http://avnet.me/vitis-ai-2.0-dualcam


image 


This hackster.io project includes the following additional content:


    

  • instructions for building Vitis platform and Vitis-AI design
    • 

  • pre-built SD card image for Vitis-AI enabled design
    • 

  • python scripts for stereo processing (anaglyph, stereo neural inference)
    • 



 


 


Conclusion


 


I hope this tutorial will help you get started quickly with the Ultra96-V2 and Dual-Camera Mezzanine, and inspire you to create innovative stereo applications.


 


If there is any other related content that you would like to see, please share your thoughts in the comments below.


 


 


Revision History


 








Date
Description






2022/02/07
Initial Version.













				

		
















	
	

	

	

		
	






	

	

		



			




	
	










	
    
	

  • 
	
    


    
	
    
					
			
    
	
    
		
			
											
					
		
			
		
			
    

    

    
			
    Hi, I tried to implement this project, although i am able to generate all the files, I can't boot using SD Card. I followed all instructions to get necessary files generated for AR1335 Single design, but when i copy those files and boot from SD card, I get this error over serial port on Putty "create_overlay: Failed to create overlay(err=22)".
    

    I searched internet for same/similar errors but no resolution, I have Onsemi daul cam mezzannine and one AR1335 camera, my laptop has Vitis & Petalinux 2021.2 . I also tried to use the image shared by you in your blog, and face the same error.
    

    Any help would be appreciated.
    

    Thanks and regards
    

    Nitin
    
	
    

    

    
	
    
		
	
    

    

    
		 		 		
    
		
	
    

    

    •