element14 Community
element14 Community
    Register Log In
  • Site
  • Search
  • Log In Register
  • About Us
  • Community Hub
    Community Hub
    • What's New on element14
    • Feedback and Support
    • Benefits of Membership
    • Personal Blogs
    • Members Area
    • Achievement Levels
  • Learn
    Learn
    • Ask an Expert
    • eBooks
    • element14 presents
    • Learning Center
    • Tech Spotlight
    • STEM Academy
    • Webinars, Training and Events
    • Learning Groups
  • Technologies
    Technologies
    • 3D Printing
    • FPGA
    • Industrial Automation
    • Internet of Things
    • Power & Energy
    • Sensors
    • Technology Groups
  • Challenges & Projects
    Challenges & Projects
    • Design Challenges
    • element14 presents Projects
    • Project14
    • Arduino Projects
    • Raspberry Pi Projects
    • Project Groups
  • Products
    Products
    • Arduino
    • Avnet Boards Community
    • Dev Tools
    • Manufacturers
    • Multicomp Pro
    • Product Groups
    • Raspberry Pi
    • RoadTests & Reviews
  • Store
    Store
    • Visit Your Store
    • Choose another store...
      • Europe
      •  Austria (German)
      •  Belgium (Dutch, French)
      •  Bulgaria (Bulgarian)
      •  Czech Republic (Czech)
      •  Denmark (Danish)
      •  Estonia (Estonian)
      •  Finland (Finnish)
      •  France (French)
      •  Germany (German)
      •  Hungary (Hungarian)
      •  Ireland
      •  Israel
      •  Italy (Italian)
      •  Latvia (Latvian)
      •  
      •  Lithuania (Lithuanian)
      •  Netherlands (Dutch)
      •  Norway (Norwegian)
      •  Poland (Polish)
      •  Portugal (Portuguese)
      •  Romania (Romanian)
      •  Russia (Russian)
      •  Slovakia (Slovak)
      •  Slovenia (Slovenian)
      •  Spain (Spanish)
      •  Sweden (Swedish)
      •  Switzerland(German, French)
      •  Turkey (Turkish)
      •  United Kingdom
      • Asia Pacific
      •  Australia
      •  China
      •  Hong Kong
      •  India
      •  Korea (Korean)
      •  Malaysia
      •  New Zealand
      •  Philippines
      •  Singapore
      •  Taiwan
      •  Thailand (Thai)
      • Americas
      •  Brazil (Portuguese)
      •  Canada
      •  Mexico (Spanish)
      •  United States
      Can't find the country/region you're looking for? Visit our export site or find a local distributor.
  • Translate
  • Profile
  • Settings
FPGA
  • Technologies
  • More
FPGA
Blog Ultra96-V2 Dual Camera Design for 2021.2
  • Blog
  • Forum
  • Documents
  • Quiz
  • Events
  • Polls
  • Files
  • Members
  • Mentions
  • Sub-Groups
  • Tags
  • More
  • Cancel
  • New
Join FPGA to participate - click to join for free!
  • Share
  • More
  • Cancel
Group Actions
  • Group RSS
  • More
  • Cancel
Engagement
  • Author Author: albertabeef
  • Date Created: 7 Feb 2022 1:59 PM Date Created
  • Views 3987 views
  • Likes 3 likes
  • Comments 1 comment
  • ultra96v2
  • ultra96-v2
  • isp
  • 96 boards
  • ar0144
  • onsemi dual mipi camera
  • ap1302
  • dual camera
  • ias
  • ar1335
Related
Recommended

Ultra96-V2 Dual Camera Design for 2021.2

albertabeef
albertabeef
7 Feb 2022

Introduction

This blog post provides details on how to build and execute the new 2021.2 design for the Ultra96-V2 development board with Dual-Camera Mezzanine.

image
 

Design Overview

The following block diagram illustrates the 2021.2 version of the hardware design.

image

The following images capture the resource utilization of the design.

imageimage

 

When compared to the previous version of the design 2020.1, the following changes have been made:

  • removed the OSD core, including the entire live DisplayPort (DP) sub-system, which was determined to not be required
  • modified the capture pipeline
    • added a VPSS core for color space conversion
    • added BGR format support to the frame buffer write core
  • added of the AXI interrupt logic for use with Vitis

When compared to the previous version of the design 2020.2 (https://avnet.me/u96v2-dualcam-2020.2), the following changes have been made:

  • update ap1302 drivers 
    • add controls for brightness, contrast, saturation, gamma, exposure, gain 
    • add controls for auto-focus, horizontal/vertical flip, 3d_path
  • update launch scripts

 

Supported Sensor Modules

The following sensor modules are supported by the linux-based 2020.2 design:

  • AR0144 IAS Sensor Module
    • part number : CAV10-000A
    • datasheet : http://avnet.me/ias-ar0144-datasheet
    • features :
      • 1 Mpixel monochrome sensor
      • global shutter
  • AR1335 IAS Sensor Module
    • part number : CAVBA-000A
    • datasheet : http://avnet.me/ias-ar1335-datasheet
    • features :
      • 13 Mpixel (4K) color sensor
      • rolling shutter
      • auto-focus

 

Although the Dual Camera Mezzanine comes with two (2) AR0144 IAS sensor modules, it also supports a single AR1335 sensor module.

 

Capture Pipeline Overview

The linux-based 2021.2 design supports a statically configurable capture pipeline for the following three configurations:

  • Dual AR0144 Sensors
  • Single AR0144 Sensor
  • Single AR1335 Sensor

 

The dual-camera mezzanine makes use of MIPI in order to connect the image cameras to the processing board.

 

The hardware design implemented in the programmable logic (PL) includes the following components:

  • MIPI CSI-2 RX receiver IP core
  • Image Pipeline : implemented with Color-Space-Conversion, and Scaler
  • Frame Buffer Write : the DMA engine implementing writes to external DDR

 

Dual AR0144 Camera Pipeline

The following image illustrates the capture pipeline for the Dual AR0144 configuration.

image

It is important to known that the AP1302 ISP receives the stereo images and generates a single side-by-side-image, as shown below:

image

 

 

Although the side-by-side image reflects the frontal view of the dual camera mezzanine, it is important to know that it contains:

  • image from left (L) camera (SENSOR2) on right side
  • image from right (R) camera (SENSOR1) on left side

 

This clarification is essential for any stereo processing that is attempted.

When in doubt, or to convince yourself, place your finger in front of one of the cameras and notice which side of the side-by-side image is blocked.

 

Single AR0144 Camera Pipeline

The following image illustrates the capture pipeline for the Single AR0144 configuration.

image

It is important to known that the single AR0144 configuration expects the sensor to be populated on the sensor1 header, as shown below:

image

 

Single AR1335 Camera Pipeline

The following image illustrates the capture pipeline for the Single AR1335 configuration.

image

It is important to known that the single AR1335 configuration expects the sensor to be populated on the sensor1 header, as shown below:

image

 

Rebuilding the Design

 

This section describes how to re-build the linux-based 2021.2 design.

 

This can be done on a linux machine, which has the Vitis 2021.2 and Petalinux 2021.2 tools correctly installed.

 

The following commands will clone the Avnet “bdf”, “hdl”, and “petalinux” repositories, all needed to re-build the linux-based design.

$ git clone https://github.com/Avnet/bdf

$ git clone –b 2021.2 https://github.com/Avnet/hdl

$ git clone –b 2021.2 https://github.com/Avnet/petalinux


 

To rebuild the linux-based design, use the following commands:

$ cd petalinux

$ ./scripts/make_u96v2_sbc_dualcam.sh

 

This will launch the build for the hardware design and petalinux project.  The build artifacts that are of interest are the following:

  petalinux/projects/u96v2_sbc_dualcam_2021_2/images/linux/BOOT.BIN

  petalinux/projects/u96v2_sbc_dualcam_2021_2/images/linux/boot.scr

  petalinux/projects/u96v2_sbc_dualcam_2021_2/images/linux/image.ub

  petalinux/projects/u96v2_sbc_dualcam_2021_2/images/linux/rootfs.tar.gz

 

By default, the device tree will be built for the dual AR0144 configuration.  If you want to use a different configuration, edit the following file:

   petalinux/projects/u96v2_sbc_dualcam_2021_2/project-spec/meta-on-semiconductor/recipes-bsp/device-tree/files/system.dtsi

Un-comment only one of the following lines corresponding to your desired confiuration, then re-build the petalinux project.

   #define CONFIG_STATUS_AR0144_DUAL

   //#define CONFIG_STATUS_AR0144_SINGLE

   //#define CONFIG_STATUS_AR1335_SINGLE

These binaries can be programmed to a dual partition micro SD card, as follows:

  • BOOT partition (FAT format)
    • BOOT.BIN
    • boot.scr
    • image.ub
  • ROOTFS partition (EXT4 format)
    • contents of rootfs.tar.gz, extracted as root

 

 

 

Executing the Design with the Dual AR0144 configuration

After booting the board with the SD card, ensure that the /etc/init.d/load_default_camera_config script is specifying the following configuration

  DAEMON=/usr/bin/load_camera_config

  DAEMON_OPTS="ar0144_dual"

 

If this is not the case, edit the configuration, then reboot the design.

 

The X-windows desktop needs to be stopped in order to run the passthrough example, which is achieved with the following command:

$ /etc/init.d/xserver-nodm stop

The dual camera passthrough can be launched with the following command:

$ source run_1920_1080

This script will perform the following steps:

  • configure capture pipeline configuration using media-ctl utility
  • configure monitor resolution using modetest utility
  • launch gstreamer pipeline using gst-launch-1.0 utility

 

The following test pattern will be visible for ~1 sec (after execution of modetest).

 

image

 

Once the gstreamer pipeline is launched, the following side-by-side image will be visible on the monitor.  Notice that the width is compressed by a factor of 2, which is only to make the dual images fit on the monitor.

image

 

The lens for each AR0144 can be manually rotated to adjust the focus, if required.

 

Side-by-Side configuration

As previously mentioned, the default side-by-side configuration defaults to the following:

  • image from left (L) camera (SENSOR2) on right side
  • image from right (R) camera (SENSOR1) on left side

This configuration can be queried with the following command:

$ v4l2-ctl --get-ctrl 3d_path -d /dev/video0
3d_path: 0

This configuration can be changed by writing "1" to the "3d_path" control, as follows:

$ v4l2-ctl --set-ctrl 3d_path=1 -d /dev/video0

 

This will change the side-by-side configuration to the following:

  • image from left (L) camera (SENSOR2) on left side
  • image from right (R) camera (SENSOR1) on right side

 

Executing the Design with the Single AR1335 configuration

After booting the board with the SD card, ensure that the /etc/init.d/load_default_camera_config script is specifying the following configuration

  DAEMON=/usr/bin/load_camera_config

  DAEMON_OPTS="ar1335_single"

 

If this is not the case, edit the configuration, then reboot the design.

 

The X-windows desktop needs to be stopped in order to run the passthrough example, which is achieved with the following command:

$ /etc/init.d/xserver-nodm stop

 

In order to display the full resolution of the AR1335 sensor (3840x2160), the QoS configuration needs to be optimized for the DisplayPort (DP) monitor.

Otherwise, the DP controller will starve for bandwidth to external memory, and the image will contain glitches.

This is accomplished with the following command:

$ optimize_qos_for_dp

 

The 4K camera passthrough can be launched with the following command:

$ source run_3840_2160

This script will perform the following steps:

  • configure capture pipeline configuration using media-ctl utility
  • configure monitor resolution using modetest utility
  • launch gstreamer pipeline using gst-launch-1.0 utility

 

The following test pattern will be visible for ~1 sec (after execution of modetest).

image

Once the gstreamer pipeline is launched, the following image will be visible on the monitor.

image

Auto-Focus

There is no need to manually turn the lens for the AR1335 sensor module, since it implements Auto-Focus functionality.

 

The CAVBA-000A datasheet indicates the auto-focus is implemented with the following components:

  • Lens Type : Largan 50013A7
  • VCM Type : Jahwa QJ4C
  • VCM Driver : DW9790A

 

image

In order to power the VCM device, the following jumper setting needs to be placed in the 1-2 position on the dual camera mezzanine.

 

image

 

The Voice Coil Motor (VCM) is under control of the on-board AP1302 ISP device.

Although the AP1302 firmware has Auto-Focus "off" by default, it can be enabled by the user.

 

 

This configuration can be queried with the following command:

$ v4l2-ctl --get-ctrl auto_focus-d /dev/video0
auto_focus: 0

This configuration can be changed by writing "1" to the "auto_focus" control, as follows:

$ v4l2-ctl --set-ctrl auto_focus=1 -d /dev/video0

 

 

Where to Buy the Hardware

 

The following links describe the hardware requirements needed to run this design.

 

Ultra96-V2

Buy Ultra96-V2

or Ultra96-V2 I-grade

Buy Ultra96-V2 I-grade

 

Ultra96-V2 4A Power Supply

Buy Ultra96-V2 4A Power Supply

 

Ultra96-V2 JTAG/UART Adapter

Buy Ultra96-V2 JTAG/UART Adapter

 

96Boards onsemi Dual Camera Mezzanine

Buy 96Boards onsemi Dual Camera Mezzanine

 

Going Further

 

A Vitis-AI 2.0 design has been built on top of this 2021.2 design, and is documented as a hackster.io project.

http://avnet.me/vitis-ai-2.0-dualcam

image 

This hackster.io project includes the following additional content:

  • instructions for building Vitis platform and Vitis-AI design
  • pre-built SD card image for Vitis-AI enabled design
  • python scripts for stereo processing (anaglyph, stereo neural inference)

 

 

Conclusion

 

I hope this tutorial will help you get started quickly with the Ultra96-V2 and Dual-Camera Mezzanine, and inspire you to create innovative stereo applications.

 

If there is any other related content that you would like to see, please share your thoughts in the comments below.

 

 

Revision History

 

Date Description
2022/02/07 Initial Version.
  • Sign in to reply
  • nitsvnit
    nitsvnit over 2 years ago

    Hi, I tried to implement this project, although i am able to generate all the files, I can't boot using SD Card. I followed all instructions to get necessary files generated for AR1335 Single design, but when i copy those files and boot from SD card, I get this error over serial port on Putty "create_overlay: Failed to create overlay(err=22)".

    I searched internet for same/similar errors but no resolution, I have Onsemi daul cam mezzannine and one AR1335 camera, my laptop has Vitis & Petalinux 2021.2 . I also tried to use the image shared by you in your blog, and face the same error.

    Any help would be appreciated.

    Thanks and regards

    Nitin

    • Cancel
    • Vote Up 0 Vote Down
    • Sign in to reply
    • More
    • Cancel
element14 Community

element14 is the first online community specifically for engineers. Connect with your peers and get expert answers to your questions.

  • Members
  • Learn
  • Technologies
  • Challenges & Projects
  • Products
  • Store
  • About Us
  • Feedback & Support
  • FAQs
  • Terms of Use
  • Privacy Policy
  • Legal and Copyright Notices
  • Sitemap
  • Cookies

An Avnet Company © 2025 Premier Farnell Limited. All Rights Reserved.

Premier Farnell Ltd, registered in England and Wales (no 00876412), registered office: Farnell House, Forge Lane, Leeds LS12 2NE.

ICP 备案号 10220084.

Follow element14

  • X
  • Facebook
  • linkedin
  • YouTube