Introduction

This project is part 2 of a 5 part series of projects, where we will progressively create AI enabled platforms for the following Tria development boards:

• ZUBoard
• Ultra96-V2
• UltraZed-7EV

These projects can be rebuilt using the source code on github.com:

• github.com/AlbertaBeef/tria-vitis-platforms/tree/2023.2

The following series of Hackster projects describe how the above github repository was created and serves as documentation:

• Part 1 : Tria Vitis Platforms — Building the Foundational Designs
• Part 2 : Tria Vitis Platforms — Creating a Common Platform
• Part 3 : Tria Vitis Platforms — Adding support for Vitis-AI
• Part 4 : Tria Vitis Platforms — Adding support for Hailo-8
• Part 5 : Tria Vitis Platforms — Adding support for ROS2

The motivation of this series of projects is to enable users to create their own custom AI applications.

Introduction — Part II

In the previous project ( Part 1 ), we learned how to build the foundational designs for the various Tria Zynq-UltraScale+ hardware targets:

• ZUBoard : zb1cg_sbc_base, zb1cg_sbc_trifecta
• Ultra96-V2 : u96v2_sbc_base, u96v2_sbc_dualcam
• UltraZed-EV : uz7ev_evcc_base, uz7ev_evcc_nvme

We built an SD image for each design and learned how to boot and use each design. Dealing with multiple SD images, however, can be cumbersome.

This project describes how to combine multiple designs into a single platform. This will be accomplished with the “xmutil” platform management utility from AMD (formerly Xilinx).

XMUTIL Overview

AMD (formerly Xilinx) introduced XMUTIL with the Kria portfolio. The utility can be used to query the platform status, and manage multiple designs (called accelerated apps), as well as several other features.

Of interest to us in this project are the following commands:

• xmutil listapps
• xmutil loadapp {app}
• xmutil unloadapp

These commands call DFX-MGR under the hood, which is AMD’s implementation of the open source FPGA MANAGER service in linux.

The following table lists which dfx-mgr-client commands are called by the xmutil utility.

It allows the user to dynamically load hardware designs as full bitstreams or as partial reconfiguration, along with the device tree content that describes this hardware.

Creating the new platform

There are many ways to go about combining the various designs into a common platform.

I first achieved this with the 2022.2 version of tools for ZUBoard and Ultra96-V2:

• ZUBoard — 2022.2 : Combining designs into a common platform
• Ultra96-V2–2022.2 : Combining designs into a common platform

Although I have chosen a slightly different strategy for how to re-use the Avnet github repositories, the procedure is essentially the same for this new implementation using version 2023.2 of tools :

• Determining a common super-set for the PS configuration
• Creating build scripts for the Vivado designs
• Creating build scripts for the Vitis platforms
• Creating the common Petalinux project
• Purging the device tree from the common platform
• Creating firmware recipes for each design
• Creating the dynamic device tree for each design

We will keep the naming, covered in the previous project ( Part 1 ), for each hardware target, and name our platforms as {hwcore}_{hwtype}:

• ZUBoard : zub1cg_sbc
• Ultra96-V2 : u96v2_sbc
• UltraZed-7EV : uz7ev_evcc

The results of this project have been captured in the following github repository:

• https://github.com/AlbertaBeef/tria-vitis-platforms/tree/2023.2

In order to re-build the projects, please refer to the README.md in the previous github repository. This project explains how this repository content was created.

The following diagram shows the contents of this repository:

The “common” directory contains content common to all platforms. This content, mostly reused from the foundational designs we saw in the previous project ( Part 1 ), are links (git sub-modules) to the original Avnet github repositories.

The “{hwcore}” directory represents a directory for each hardware target: zub1cg, u96v2, and uz7ev.

Each of these directories contains sub-directories for each design.

• platforms : contains the build scripts for each design
• petalinux : contains the common Petalinux project for the common platform

Determining a common super-set for the PS configuration

I have not been able to find how to change the PS configuration dynamically, so the petalinux project requires being created with a super-set of the PS configuration.

For the ZUBoard, this includes:

• enabling a PS based I2C controller, required for the DualCam HSIO
• enabling the PCIe functionality, required for the M.2 (B+M-key) HSIO

For the UltraZed-EV, this includes:

• enabling the PCIe functionality, required for the M.2 (M-key) support

I have visually captured what these PS configuration super-sets look like for each platform:

The designs that have the PS configuration super-sets we need are the following:

• ZUBoard : zub1cg_sbc_trifecta
• Ultra96-V2 : zub1cg_sbc_dualcam
• UltraZed-EV : uz7ev_evcc_nvme

Therefore we will be reusing the Vivado project settings from those designs when creating the common Petalinux project.

But before that, we still need to create the scripts to generate all the designs we want to support.

Creating build scripts for the Vivado designs

For each design, we will create the following sub-directory structure, inspired from the AMD kria-vitis-platforms github repository:

All the designs built with the original Avnet github repositories were created in the following location:

~/Avnet_2023_2/hdl/projects/{hwcore}_{hwtype}_{design}_2023_2

For each design, we will open the Vivado project, and run the following TCL command in the TCL console:

write_bd_tcl -no_ip_version {hwcore}_{hwtype}_{design}_2023_2_write_bd_tcl.tcl

This will become the content for the config_bd.tcl script. I have modified these scripts to change the vendor information from “Avnet” to “Tria Technologies”, which is the new identity for our group.

The design’s I/O constraints, are copied to the pin.xdc constraints file.

The other files are modified versions of the kria-vitis-platforms template, modified for our purposes.

The main.tcl script, is augmented to include the Tria board definition files, which are located in the “bdf” sub-directory, under the “common” directory.

tria-vitis-platform/common/platforms/bdf

After building a Vivado project, its build artifacts will be found in the following location:

NOTE : For the ZUBoard hardware target, the “dualcam” design is actually equivalent to the “trifecta” design, augmented with PCIe functionnality. I preferred to keep the “dualcam” name for this one.

Creating build scripts for the Vitis platforms

With the Vivado projects created, we now want to create the Vitis wrappers for these projects. This will allow us to use Vitis to create additional designs, augmented with accelerators.

Once again, these build scripts are inspired from the AMD kria-vitis-platforms github repository:

Vitis platforms usually include a hardware (Vivado project) and software (bare metal, linux, or other …) component. Note that in our case the software is greyed out, since we will be creating a common Petalinux project, apart from the Vitis platforms. This is the strategy used by the AMD Kria platforms, which we are taking inspiration from.

The most important point to note is that the Vitis platform provides the following information regarding the available resources to the Vitis toolflow:

• clocks (and associated resets)
• interrupts
• AXI connectivity

This information can be found in the Vivado project (look for PFM attributes in the config_bd.tcl script), and allows Vitis to automatically insert accelerators into the Vivado project when creating new accelerated designs. We will be seeing this in a future project ( Part 3 ).

Once created, each Vitis platforms can be found in the following location:

Creating the common Petalinux project

The initial version of the Petalinux project for each hardware target was created from the BSPs generated in the previous project ( Part 1 ).

As an example, for the ZUBoard, the petalinux project was created as follows:

cd ~/tria-vitis-platforms/zub1cg

petalinux-create -t project -s ~/Avnet_2023_2/petalinux/projects/zub1cg_sbc_trifecta_2032_2.bsp -n petalinux

Once created, we want to change the name of our new petalinux project, from “zub1cg_sbc_{design}” to “zub1cg_sbc”, by modifying the following file:

tria-vitis-platforms/zub1cg/petalinux/project-spec/configs/config

...
#
# Firmware Version Configuration
#
CONFIG_SUBSYSTEM_HOSTNAME="zub1cg-sbc-2023-2"
CONFIG_SUBSYSTEM_PRODUCT="zub1cg_sbc_2023_2"
CONFIG_SUBSYSTEM_FW_VERSION="1.00"
...

We also want to configure our petalinux project to NOT generate any device tree content for the PL, by modifying the same file as follows:

...
CONFIG_SUBSYSTEM_REMOVE_PL_DTB=y
...

Finally, we configure the petalinux project to use the “dualcam” Vivado project:

cd ~/tria-vitis-platforms/zub1cg

petalinux-config --silentconfig --get-hw-description=../platforms/vivado/zub1cg_sbc_dualcam/project/zub1cg_sbc_dualcam.xsa

Note that this is not to import PL content, but rather for the PS configuration, or PS configuration super-set, that we identified earlier. For the ZUBoard, this is “trifecta”, which I have renamed to “dualcam” in this repository.

Attempting to build the petalinux project at this point will generate build errors related to non-existing device tree content references, such as the following references to the MIPI capture pipeline content:

Label or path CAPTURE_PIPELINE_mipi_csi2_rx_subsystem_0 not found
Label or path CAPTURE_PIPELINE_v_frmbuf_wr_0 not found
Label or path CAPTURE_PIPELINE_v_proc_ss_scaler_0 not found
Label or path CAPTURE_PIPELINE_v_proc_ss_csc_0 not found

The Avnet provided meta yocto layers define different machine configurations, which correspond to specific designs, which have specific PL device tree content.

I could have defined new machine configurations with no corresponding PL device tree content. Instead, I chose to re-organize these layers, and cherry-pick the content that I wanted to keep, modify, or omit.

Purging the device tree from the common platform

As already mentioned in the previous section, I chose to re-organize the Avnet meta layers, in order to purge the device tree content of all PL references for the common platform.

For the ZUBoard and Ultra96-V2, I re-organized the Avnet meta layers as follows:

I kept the “meta-avnet” layer, but only kept a sub-set of the “meta-on-semiconductor” layer. This was done with symbolic links to the original content in the “common” directory, and represented with pink arrows.

I omitted the “recipes-apps” and “recipes-bsp” recipes which contained the default application scripts, and device tree content for the dualcam design. This content will be defined in the firmware recipes instead.

I modified the “recipes-core” recipes which contained the following file, which defines which recipes to include in the build for the dualcam design:

recipes-core/images/petalinux-image-minimal.bbapend

DESCRIPTION = "Image definition for zuboard & ultra96v2 dual cameras boards"
LICENSE = "MIT"
DUALCAM_PACKAGES += "\
   ap1302 \
   libdrm \
   libdrm-tests \
"

IMAGE_INSTALL:append:u96v2-sbc-dualcam = " \
   ${DUALCAM_PACKAGES} \
"
IMAGE_INSTALL:append:zub1cg-sbc-dualcam = " \
   ${DUALCAM_PACKAGES} \
"

I kept the “recipes-kernel” and “recipes-modules” recipes, which contain the AP1302 driver and firmware, required for the DualCam functionality.

For the UltraZed-EV, I re-organized the Avnet meta layers as follows:

I kept all the recipes, except for the “recipes-bsp”, where I explicity removed the PL related device tree content for the UltraZed-EV common platform.

At this point, our petalinux projects successfully build with no PL design and no PL related device tree content.

Creating firmware recipes for each design

Now that we have our petalinux project set up as a “clean” slate, we can add our designs as firmware apps, which we will name:

• {vendor}_{hwcore}_{design}

Petalinux provides a command to create a yocto recipe for these firmware apps:

$ petalinux-create -t apps 
                  --template dfx_user_dts -n {firmware} 
                  --enable 
                  --srcuri"{path}/{firmware}.bit 
                           {path}/{firmware}.dtsi 
                           {path}/{firmware}.xclbin 
                           {path}/shell.json" 
                  --force

Using the above command, the following firmware recipes were created in the tria-vitis-platform github repository:

Instead of including a copy of the bitstream, however, a symbolic link was used to point to the bitstream in the Vivado project’s directory structure.

For the ZUBoard, the following firmware recipes were created:

• tria-zub1cg-base
• tria-zub1cg-dualcam

For the Ultra96-V2, the following firmware recipes were created:

• tria-u96v2-base
• tria-u96v2-dualcam

For the UltraZed-EV, the following firmware recipes were created:

• tria-uz7ev-base
• tria-uz7ev-nvme

Creating the dynamic device tree for each design

In the previous section, I was vague on where the device tree content (.dtsi) came from. This is actually the most complex piece to put in place, so merits its own section.

Here is how I create this dynamic device tree:

• Start with a standard template
• Add the PL generated device tree content
• Add the user generated device tree content

The standard template looks like the following:

/dts-v1/;
/plugin/;

#include <dt-bindings/gpio/gpio.h>
&fpga_full {
   #address-cells = <2>;
   #size-cells = <2>;
   firmware-name = "tria_zub1cg_base.bit.bin";
   resets = <&zynqmp_reset 116>, <&zynqmp_reset 117>, <&zynqmp_reset 118>, <&zynqmp_reset 119>;
};
&amba {
   afi0: afi0 {
      compatible = "xlnx,afi-fpga";
      config-afi = <0 0>, <1 0>, <2 0>, <3 0>, <4 0>, <5 0>, <6 0>, <7 0>, <8 0>, <9 0>, <10 0>, <11 0>, <12 0>, <13 0>, <14 0xa00>, <15 0x000>;
      resets = <&zynqmp_reset 116>, <&zynqmp_reset 117>, <&zynqmp_reset 118>, <&zynqmp_reset 119>;
      reset-names = "pl0", "pl1", "pl2", "pl3";
   };
};
&amba {
   /* Insert content from PL generated device tree content */
   /*    {original petalinux project}/components/plnx_workspace/device-tree/device-tree/pl.dtsi */
   /* Insert content from user generated device tree content */
   /*    {original petalinux project}/project-spect/meta-*/recipes-bsp/.../*.dtsi */
};
&zynqmp_dpsub {
   status = "okay";
};
&zynqmp_dp_snd_pcm0 {
   status = "okay";
};
&zynqmp_dp_snd_pcm1 {
   status = "okay";
};
&zynqmp_dp_snd_card0 {
   status = "okay";
};
&zynqmp_dp_snd_codec0 {
   status = "okay";
};

The first section “fpga_full” describes the firmware overlay, which consists of the entire PL (bitstream).

The second section “amba — afi0”, describes the configuration of the AXI interconnect between the PS and PL.

The “xlnx,afi-fpga” binding is documented here:

• https://github.com/Xilinx/linux-xlnx/blob/master/Documentation/devicetree/bindings/fpga/xlnx%2Cafi-fpga.txt

It configures the following 16 interfaces:

Array of values to be written.for FM0_RDCTRL(0) the valid values-fabric width   2: 32-bit,1 : 64-bit ,0: 128-bit enabled
for FM0_WRCTRL(1) the valid values-fabric width   2: 32-bit,1 : 64-bit ,0: 128-bit enabled
for FM1_RDCTRL(2) the valid values-fabric width   2: 32-bit,1 : 64-bit ,0: 128-bit enabled
for FM1_WRCTRL(3) the valid values-fabric width   2: 32-bit,1 : 64-bit ,0: 128-bit enabled
for FM2_RDCTRL(4) the valid values-fabric width   2: 32-bit,1 : 64-bit ,0: 128-bit enabled
for FM2_WRCTRL(5) the valid values-fabric width   2: 32-bit,1 : 64-bit ,0: 128-bit enabled
for FM3_RDCTRL(6) the valid values-fabric width   2: 32-bit,1 : 64-bit ,0: 128-bit enabled
for FM3_WRCTRL(7) the valid values-fabric width   2: 32-bit,1 : 64-bit ,0: 128-bit enabled
for FM4_RDCTRL(8) the valid values-fabric width   2: 32-bit,1 : 64-bit ,0: 128-bit enabled
for FM4_WRCTRL(9) the valid values-fabric width   2: 32-bit,1 : 64-bit ,0: 128-bit enabled
for FM5_RDCTRL(10) the valid values-fabric width   2: 32-bit,1 : 64-bit ,0: 128-bit enabled
for FM5_WRCTRL(11) the valid values-fabric width   2: 32-bit,1 : 64-bit ,0: 128-bit enabled
for FM6_RDCTRL(12) the valid values-fabric width   2: 32-bit,1 : 64-bit ,0: 128-bit enabled
for FM6_WRCTRL(13) the valid values-fabric width   2: 32-bit,1 : 64-bit ,0: 128-bit enabledfor AFI_FA(14)
   dw_ss1_sel	bits (11:10)
   dw_ss0_sel	bits (9:8)
      0x0: 32-bit AXI data width),
      0x1: 64-bit AXI data width,
      0x2: 128-bit AXI data
All other bits are 0 write ignored.for AFI_FA(15)  selects for ss2AXI data width valid values
   0x000: 32-bit AXI data width),
   0x100: 64-bit AXI data width,
   0x200: 128-bit AXI data

Getting the content for the dynamic device tree correct is usually a trial and error process for me, so be patient and try different iterations (as described in the next sections).

Verifying the firmware apps

In order to verify the firmware apps, program the SD image for the hardware target’s petalinux project, and boot the board.

Login as the “root” user as follows:

zub1cg-sbc-2023-2 login: root

root@zub1cg-sbc-2023-2:~#

If we made it this far, our petalinux project is working. We can now verify our firmware packages, which are referred to as “apps” by xmutil.

We start by querying which “apps” are present:

root@zub1cg-sbc-2023-2:~# xmutil listapps
        Accelerator  Accel_type                 Base  Base_type   #slots Active_slot
   tria-zub1cg-base    XRT_FLAT     tria-zub1cg-base   XRT_FLAT    (0+0)          -1
tria-zub1cg-dualcam    XRT_FLAT  tria-zub1cg-dualcam   XRT_FLAT    (0+0)          -1

Now that we have verified our common u96v2-sbc-2023.2 image, and the presence of our firmware apps, we can start verifying these apps.

Verifying these apps is an iterative process, due to the complexity of the dynamic device tree content.

Verifying the “base” app

We start with the simpler of the apps, the “base” app.

root@zub1cg-sbc-2023-2:~# xmutil loadapp tria-zub1cg-base
[   50.161905] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /fpga-full/firmware-name
[   50.172032] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /fpga-full/resets
[   50.182415] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /axi/display@fd4a0000/status
[   50.192853] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /axi/display@fd4a0000/zynqmp-dp-snd-pcm0/status
[   50.204950] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /axi/display@fd4a0000/zynqmp-dp-snd-pcm1/status
[   50.217038] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /axi/display@fd4a0000/zynqmp-dp-snd-card/status
[   50.229132] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /axi/display@fd4a0000/zynqmp-dp-snd-codec0/status
[   50.241421] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/afi0
[   50.250917] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/axi_gpio_0
[   50.260931] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/axi_gpio_1
[   50.270939] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/axi_gpio_2
[   50.280947] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/axi_iic_0
[   50.290875] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/axi_iic_1
[   50.300799] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/axi_iic_2
[   50.310720] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/axi_intc_0
[   50.320729] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/axi_quad_spi_0
[   50.331083] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/axi_uartlite_0
[   50.341438] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/system_management_wiz_0
[   51.033639] OF: graph: no port node found in /axi/display@fd4a0000
[   51.051716] zocl-drm axi:zyxclmm_drm: error -ENXIO: IRQ index 32 not found
tria-zub1cg-base: loaded to slot 0

root@zub1cg-sbc-2023-2:~# xmutil desktop_enable

Note that the following WARNING occurs for the case of a working dynamic device tree, so can be ignored:

OF: overlay: WARNING: memory leak will occur if overlay removed, ...

We can also verify that we can unload the base app:

root@zub1cg-sbc-2023-2:~# xmutil desktop_disable

root@zub1cg-sbc-2023-2:~# xmutil unloadapp
[  326.686110] OF: ERROR: memory leak, expected refcount 1 instead of 97, of_node_get()/of_node_put() unbalanced - destroy cset entry: attach overlay node /axi/interrupt-controller@a0060000
remove from slot 0 returns: 0 (Ok)

Note that the following WARNING occurs for the case of a working dynamic device tree, so can be ignored:

OF: ERROR: memory leak, ...

Verifying the “dualcam” app

Next, we tackle the more complex app, the “dualcam” app.

root@zub1cg-sbc-2023-2:~# xmutil loadapp tria-zub1cg-dualcam
[   34.752952] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /fpga-full/firmware-name
[   34.763077] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /fpga-full/resets
[   34.773599] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /axi/display@fd4a0000/status
[   34.784042] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /axi/display@fd4a0000/zynqmp-dp-snd-pcm0/status
[   34.796137] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /axi/display@fd4a0000/zynqmp-dp-snd-pcm1/status
[   34.808226] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /axi/display@fd4a0000/zynqmp-dp-snd-card/status
[   34.820323] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /axi/display@fd4a0000/zynqmp-dp-snd-codec0/status
[   34.832608] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/afi0
[   34.842097] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/ap1302_osc
[   34.852106] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/misc_clk_0
[   34.862114] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/ias_out0
[   34.871946] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/CAPTURE_PIPELINE_mipi_csi2_rx_subsyst_0
[   34.884476] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/mipi_csi_portsCAPTURE_PIPELINE_mipi_csi2_rx_subsyst_0
[   34.898222] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/mipi_csi_port1CAPTURE_PIPELINE_mipi_csi2_rx_subsyst_0
[   34.911961] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/mipi_csirx_outCAPTURE_PIPELINE_mipi_csi2_rx_subsyst_0
[   34.925702] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/mipi_csi_port0CAPTURE_PIPELINE_mipi_csi2_rx_subsyst_0
[   34.939442] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/mipi_csi_inCAPTURE_PIPELINE_mipi_csi2_rx_subsyst_0
[   34.952928] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/CAPTURE_PIPELINE_v_frmbuf_wr_0
[   34.964675] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/CAPTURE_PIPELINE_v_proc_ss_csc_0
[   34.976595] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/csc_portsCAPTURE_PIPELINE_v_proc_ss_csc_0
[   34.989300] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/csc_port1CAPTURE_PIPELINE_v_proc_ss_csc_0
[   35.001999] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/csc_outCAPTURE_PIPELINE_v_proc_ss_csc_0
[   35.014527] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/csc_port0CAPTURE_PIPELINE_v_proc_ss_csc_0
[   35.027232] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/CAPTURE_PIPELINE_v_proc_ss_csc_0CAPTURE_PIPELINE_mipi_csi2_rx_subsyst_0
[   35.042535] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/CAPTURE_PIPELINE_v_proc_ss_scaler_0
[   35.054715] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/scaler_portsCAPTURE_PIPELINE_v_proc_ss_scaler_0
[   35.067941] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/scaler_port1CAPTURE_PIPELINE_v_proc_ss_scaler_0
[   35.081164] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/sca_outCAPTURE_PIPELINE_v_proc_ss_scaler_0
[   35.093955] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/scaler_port0CAPTURE_PIPELINE_v_proc_ss_scaler_0
[   35.107174] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/CAPTURE_PIPELINE_v_proc_ss_scaler_0CAPTURE_PIPELINE_v_proc_ss_csc_0
[   35.122130] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/GPIO_axi_gpio_0
[   35.132572] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/axi_iic_0
[   35.142497] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/axi_intc_0
[   35.152513] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/system_management_wiz_0
[   35.163655] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/vcap_portsCAPTURE_PIPELINE_v_proc_ss_scaler_0
[   35.176700] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/vcap_portCAPTURE_PIPELINE_v_proc_ss_scaler_0
[   35.189661] OF: overlay: WARNING: memory leak will occur if overlay removed, property: /__symbols__/CAPTURE_PIPELINE_v_frmbuf_wr_0CAPTURE_PIPELINE_v_proc_ss_scaler_0
[   35.828194] debugfs: Directory '0-003c' with parent 'regmap' already present!
[   35.907911] OF: graph: no port node found in /axi/display@fd4a0000
[   35.930502] zocl-drm axi:zyxclmm_drm: error -ENXIO: IRQ index 32 not found
tria-zub1cg-dualcam: loaded to slot 0

root@zub1cg-sbc-2023-2:~# xmutil desktop_enable

The last verification step is to check the presence of the /dev/media node.

root@zub1cg-sbc-2023-2:~# v4l2-ctl --list-devices
Xilinx Video Composite Device (platform:axi:vcap_CAPTURE_PIPEL):
        /dev/media1

vcap_CAPTURE_PIPELINE_v_proc_ss (platform:vcap_CAPTURE_PIPELINE_):
        /dev/video2
UVC Camera (046d:0825) (usb-xhci-hcd.1.auto-1):
        /dev/video0
        /dev/video1
        /dev/media0

root@zub1cg-sbc-2023-2:~# media-ctl -p -d /dev/media1
Media controller API version 6.1.30
Media device information
------------------------
driver          xilinx-video
model           Xilinx Video Composite Device
serial
bus info        platform:axi:vcap_CAPTURE_PIPEL
hw revision     0x0
driver version  6.1.30
Device topology
- entity 1: vcap_CAPTURE_PIPELINE_v_proc_ss (1 pad, 1 link)
            type Node subtype V4L flags 0
            device node name /dev/video2
        pad0: Sink
                <- "b0040000.v_proc_ss":1 [ENABLED]
- entity 5: b0000000.mipi_csi2_rx_subsystem (2 pads, 2 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev0
        pad0: Sink
                [fmt:UYVY8_1X16/1920x1080 field:none colorspace:srgb]
                <- "ap1302.0-003c":2 [ENABLED]
        pad1: Source
                [fmt:UYVY8_1X16/1920x1080 field:none colorspace:srgb]
                -> "b0020000.v_proc_ss":0 [ENABLED]
- entity 8: ap1302.0-003c (6 pads, 3 links)
            type V4L2 subdev subtype Sensor flags 0
            device node name /dev/v4l-subdev3
        pad0: Sink
                [fmt:SGRBG12_1X12/1280x800 field:none colorspace:srgb
                 crop.bounds:(0,0)/2560x800
                 crop:(0,0)/2560x800]
                <- "0-003c.ar0144.0":0 [ENABLED]
        pad1: Sink
                [fmt:SGRBG12_1X12/1280x800 field:none colorspace:srgb
                 crop.bounds:(0,0)/2560x800
                 crop:(0,0)/2560x800]
                <- "0-003c.ar0144.1":0 [ENABLED]
        pad2: Source
                [fmt:UYVY8_1X16/1280x800 field:none colorspace:srgb
                 crop.bounds:(0,0)/2560x800
                 crop:(0,0)/2560x800]
                -> "b0000000.mipi_csi2_rx_subsystem":0 [ENABLED]
        pad3: Source
                [fmt:UYVY8_1X16/1280x800 field:none colorspace:srgb
                 crop.bounds:(0,0)/2560x800
                 crop:(0,0)/2560x800]
        pad4: Source
                [fmt:UYVY8_1X16/1280x800 field:none colorspace:srgb
                 crop.bounds:(0,0)/2560x800
                 crop:(0,0)/2560x800]
        pad5: Source
                [fmt:UYVY8_1X16/1280x800 field:none colorspace:srgb
                 crop.bounds:(0,0)/2560x800
                 crop:(0,0)/2560x800]
- entity 15: 0-003c.ar0144.0 (1 pad, 1 link)
             type V4L2 subdev subtype Sensor flags 0
             device node name /dev/v4l-subdev1
        pad0: Source
                [fmt:SGRBG12_1X12/1280x800 field:none colorspace:srgb]
                -> "ap1302.0-003c":0 [ENABLED]
- entity 19: 0-003c.ar0144.1 (1 pad, 1 link)
             type V4L2 subdev subtype Sensor flags 0
             device node name /dev/v4l-subdev2
        pad0: Source
                [fmt:SGRBG12_1X12/1280x800 field:none colorspace:srgb]
                -> "ap1302.0-003c":1 [ENABLED]
- entity 23: b0020000.v_proc_ss (2 pads, 2 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev4
        pad0: Sink
                [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709]
                <- "b0000000.mipi_csi2_rx_subsystem":1 [ENABLED]
        pad1: Source
                [fmt:UYVY8_1X16/1280x720 field:none colorspace:rec709]
                -> "b0040000.v_proc_ss":0 [ENABLED]
- entity 26: b0040000.v_proc_ss (2 pads, 2 links)
             type V4L2 subdev subtype Unknown flags 0
             device node name /dev/v4l-subdev5
        pad0: Sink
                [fmt:UYVY8_1X16/1280x720 field:none colorspace:srgb]
                <- "b0020000.v_proc_ss":1 [ENABLED]
        pad1: Source
                [fmt:UYVY8_1X16/1920x1080 field:none colorspace:srgb]
                -> "vcap_CAPTURE_PIPELINE_v_proc_ss":0 [ENABLED]

root@zub1cg-sbc-2023-2:~# v4l2-ctl -D -d /dev/video2
Driver Info:
        Driver name      : xilinx-vipp
        Card type        : vcap_CAPTURE_PIPELINE_v_proc_ss
        Bus info         : platform:vcap_CAPTURE_PIPELINE_
        Driver version   : 6.1.30
        Capabilities     : 0x84201000
                Video Capture Multiplanar
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps      : 0x04201000
                Video Capture Multiplanar
                Streaming
                Extended Pix Format

Debugging the firmware apps

There is a good chance that the first attempt did not work, which will require an iterative approach to verification… especially for the device tree content…

Now the fun begins !

It is not required to re-program the SD card image at each iteration. Instead, we can re-build the tria-{hwcore}-{design} packages, and re-install them on our image with the dnf utility, as shown below:

$ petalinux-build -c tria-zub1cg-dualcam

$ ls build/tmp/deploy/rpm/zub1cg-sbc-base/tria-zub1cg-dualcam-1.0-r0.*.*.rpm
tria-zub1cg-dualcam-1.0-r0.2.zub1cg_sbc_base.rpm

This rpm package can be copied over to the embedded platform (via the SD card image, or via SSH), and installed as follows:

root@zub1cg-sbc-2023-2:~# dnf install rpm/tria-zub1cg-dualcam-1.0-r0.2.zub1cg_sbc_base.rpm
...
Dependencies resolved.
======================================================================================
Package                Architecture     Version     Repository      Size
======================================================================================
Upgrading:
 tria-zub1cg-dualcam   zub1cg_sbc_base  1.0-r0.2    @commandline    130 k  

Transaction Summary
======================================================================================
Upgrade  1 Package
Total size: 130 k
Is this ok [y/N]: y
Downloading Packages:
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                              1/1
  Upgrading        : tria-zub1cg-dualcam-1.0-r0.2.zub1cg_sbc_base                 1/2
  Cleanup          : tria-zub1cg-dualcam-1.0-r0.1.zub1cg_sbc_base                 2/2
  Verifying        : tria-zub1cg-dualcam-1.0-r0.2.zub1cg_sbc_base                 1/2
  Verifying        : tria-zub1cg-dualcam-1.0-r0.1.zub1cg_sbc_base                 2/2
Upgraded:
  tria-zub1cg-dualcam-1.0-r0.2.zub1cg_sbc_base
Complete!

Automatically booting the “base” app

The user can configure the image to automatically boot one of the firmware apps. Before attempting to automatically boot an app, make sure it loads successfully (like we did previously), and does not crash the system. Otherwise, your SD image will always automatically crash.

The /etc/dfx-mgrd/daemon.conf file indicates which firmware app (default_accel) to load at boot in the /etc/dfx-mgrd/default_firmware.

root@zub1cg-sbc-2023-2:~# cat /etc/dfx-mgrd/daemon.conf
{
"firmware_location": ["/lib/firmware/xilinx"],
"default_accel":"/etc/dfx-mgrd/default_firmware"
}

This file does not exist by default, but can be created as follows:

root@zub1cg-sbc-2023-2:~# cat /etc/dfx-mgrd/default_firmware
cat: /etc/dfx-mgrd/default_firmware: No such file or directory

root@zub1cg-sbc-2023-2:~# echo tria-zub1cg-base > /etc/dfx-mgrd/default_firmware
root@zub1cg-sbc-2023-2:~# cat /etc/dfx-mgrd/default_firmware
tria-zub1cg-base

The change will take effect at the next boot.

root@zub1cg-sbc-2023-2:~# reboot

After boot, we start by querying which “apps” are present:

root@zub1cg-sbc-2023-2:~# xmutil listapps
        Accelerator  Accel_type                 Base  Base_type   #slots Active_slot
   tria-zub1cg-base    XRT_FLAT     tria-zub1cg-base   XRT_FLAT    (0+0)          0,
tria-zub1cg-dualcam    XRT_FLAT  tria-zub1cg-dualcam   XRT_FLAT    (0+0)          -1

Notice that the tria-zub1cg-base app has been loaded. The same can be done for any firmware (design) on each platform.

Known Issues

The current version of this project has the following known issues:

• repeatedly unloading and loading apps will sometimes crash …

The workaround for this issue is to not unload and load apps … decide which configuration you need, and load it once :)

Conclusion

I hope this tutorial helped to understand the benefits of using “xmutil” and the dynamic configuration of the PL to create a single SD image for your custom AI applications on the Tria development boards.

If you would like to have the pre-built petalinux BSP or SDcard image for this common platform, please let me know in the comments below.

If you would like to see more AMD based Tria boards supported, let me know in the comments below.

Revision History

2024/11/04 — Preliminary Version