This is my second entry for the AMD-sponsored “The Eye on Intelligence” experimentation challenge. In my initial post, I provided a brief introduction to my smart fruit scale project and explored the potential of the Arty Z7 development board.
In this follow-up post, I explore PYNQ, an AMD framework designed to simplify the use of adaptive computing systems.
Getting started with fruit recognition with PYNQ for smart scales.
What is PYNQ
Applying for this design challenge was partly motivated by my desire to expand my understanding of PYNQ. Although I had experimented with PYNQ and the Kria KV260 Vision AI Starter Kit, I had not yet embarked on a personal project. PYNQ is an AMD open-source project with a large community that simplifies the development of adaptive computing systems. PYNQ offers a flexible and versatile platform for embedded systems development. It combines the power of a programmable logic (PL) device with a user-friendly Python API. This unique combination allows developers to easily interact with a wide range of peripherals and devices.PYNQ enables application development with:
- Parallel hardware execution: to accelerate computationally intensive tasks.
- High frame rate video processing: ideal for real-time video applications and analytics.
- Hardware-accelerated algorithms: optimization of performance-critical algorithms.
- Real-time signal processing: for applications that require immediate response to input data.
- High-bandwidth I/O: Facilitates efficient data transfer.
- Low-latency control: Ensures accurate and timely system responses.
The PYNQ framework extends the Linux kernel with specific APIs that provide PYNQ libraries and drivers.
These APIs allow interactive Python (iPython) applications running in a Jupyter notebook to interact with the programmable logic design.
Communication between the processor system (PS) and the programmable logic (PL) depends on the interface used. PYNQ implementations offer five primary PS/PL interfaces:
- Bitstream: Configures the PL for the desired application. The xdevcfg driver is used in PYNQ.
- GPIO: Provides simple input/output. The sysgpio driver supports this.
- Interrupts: Enables interrupt generation from the PL to the PS. The Userspace IO driver supports this.
- Master AXI Interfaces: Used for PS-to-PL data transfer when the PS initiates. The devmem driver is used.
- Slave AXI Interfaces: Used for PL-to-PS data transfer when the PL initiates. The xlnk driver enables these transfers.
PYNQ applications are developed using a Jupyter notebook via a web interface.
When creating PYNQ designs, a system optimization compiler can be used to generate the PL design and C libraries, which are then wrapped in Python.
PYNQ and Overlays
Overlays, or hardware libraries, are configurable FPGA designs that can be dynamically loaded to the PL to extend the capabilities of the PS. This allows for hardware acceleration of software applications and customization of the hardware platform.
PYNQ provides a Python interface for controlling overlays from the PS. This simplifies the process for software developers, as they can leverage pre-designed overlays without needing to understand FPGA design in detail.
A common use case is image processing. Overlays can be created for functions like edge detection, thresholding, and filtering. These overlays can then be loaded and controlled from Python, allowing for efficient and customizable image processing pipelines.
PYNQ overlays, or bitstreams, are dynamically loaded FPGA designs that extend the functionality of the Zynq SoC. By default, a base overlay is loaded at boot time. Additional overlays can be installed and loaded as needed.
Components of an Overlay
- Bitstream: The configuration file for the FPGA fabric.
- HWH File: Describes the available IP cores in the overlay.
- Python API: Exposes the IP cores as attributes for Python interaction.
Loading an Overlay
- Import the Overlay class: Use from pynq import Overlay.
- Instantiate the Overlay: Specify the bitstream file name. This automatically downloads the bitstream and parses the HWH file.
from pynq import Overlay overlay = Overlay("base.bit")
Use the BaseOverlay class: For the base overlay, use BaseOverlay to access pre-defined IP attributes.
Installing PYNQ on the Digilent Arty Z7
Prerequisites for the Arty Z7
- Arty Z7 board
- Computer with compatible browser (Supported Browsers)
- Ethernet cable
- Micro USB cable
- Micro-SD card with preloaded image, or blank card (Minimum 8GB recommended)
To install PYNQ on the Digilent Arty Z7, visit the official PYNQ boards page. Download the PYNQ image for the PYNQ-Z1; the PYNQ-Z1 and the Arty Z7 have nearly identical schematics, although the PYNQ-Z1 includes a PDM microphone, which the Arty Z7 lacks.
Go to https://www.pynq.io/boards.html
Or try these links:
PYNQ-Z1 | v3.0.1 | v2.7 v2.6 | PYNQ setup guide | Digilent Pynq-Z1 |
Flashing the PYNQ image
I used Balena Etcher, you don't need to unzip the image. https://etcher.balena.io/
Open Balena and select the downloaded zip file
Flashing
Then validating
We now have the card ready to be inserted into the card reader of the Arty Z7 development board.
The Arty Z7 supports three different boot modes: microSD, Quad SPI Flash, and JTAG. The boot mode is selected using the Mode jumper (JP4), which affects the state of the Zynq configuration pins after power-on. Before booting we must select the boot from the SD card by changing that jumper.
After a few seconds, the Yellow/Green LD12 / Done LED will light up to show that the Zynq device is operational.
After a minute you should see two Blue LD4 & LD5 LEDs and four Yellow/Green LD0-LD3 LEDs flash simultaneously. The Blue LD4-LD5 LEDs will then turn on and off while the Yellow/Green LD0-LD3 LEDs remain on. The system is now booted and ready for use.
Serial console USB UART Bridge (Serial Port)
The Arty Z7 includes an FTDI FT2232HQ USB-UART bridge (attached to connector J14) that lets you use PC applications to communicate with the board using standard COM port commands (or the tty interface in Linux). Drivers are automatically installed in Windows and newer versions of Linux. Serial port data is exchanged with the Zynq using a two-wire serial port (TXD/RXD). After the drivers are installed, I/O commands can be used from the PC directed to the COM port to produce serial data traffic on the Zynq pins. The port is tied to PS (MIO) pins and can be used in combination with the UART 0 controller.
Using Teraterm to acces the serial console: configure Serial POrt communication
tty interface
Boot console log
In the following capture you can see the entire log generated by the messages to the tty interface when starting the Ubuntu Linux distribution that supports PYNQ.
U-Boot 2022.01 (Apr 04 2022 - 07:53:54 +0000) CPU: Zynq 7z020 Silicon: v3.1 DRAM: ECC disabled 512 MiB Flash: 0 Bytes NAND: 0 MiB MMC: mmc@e0100000: 0 Loading Environment from FAT... *** Error - No Valid Environment Area found *** Warning - bad env area, using default environment In: serial@e0000000 Out: serial@e0000000 Err: serial@e0000000 Net: ZYNQ GEM: e000b000, mdio bus e000b000, phyaddr -1, interface rgmii-id SF: Detected w25q128jv with page size 256 Bytes, erase size 4 KiB, total 16 MiB eth0: ethernet@e000b000 Hit any key to stop autoboot: 0 switch to partitions #0, OK mmc0 is current device Scanning mmc 0:1... Found U-Boot script /boot.scr 2776 bytes read in 10 ms (270.5 KiB/s) ## Executing script at 03000000 Trying to load boot images from mmc0 6464304 bytes read in 365 ms (16.9 MiB/s) ## Loading kernel from FIT Image at 10000000 ... Using 'conf-1' configuration Verifying Hash Integrity ... OK Trying 'kernel-0' kernel subimage Description: Linux Kernel Type: Kernel Image Compression: uncompressed Data Start: 0x100000d4 Data Size: 6442720 Bytes = 6.1 MiB Architecture: ARM OS: Linux Load Address: 0x00080000 Entry Point: 0x00080000 Hash algo: sha1 Hash value: b056acb229d05b806f975396dacae4ad04002e0c Verifying Hash Integrity ... sha1+ OK ## Loading fdt from FIT Image at 10000000 ... Using 'conf-1' configuration Verifying Hash Integrity ... OK Trying 'fdt-0' fdt subimage Description: Flattened Device Tree blob Type: Flat Device Tree Compression: uncompressed Data Start: 0x106250a8 Data Size: 19767 Bytes = 19.3 KiB Architecture: ARM Hash algo: sha1 Hash value: 5779ba3e5b1fe87f7fdec42e7688329e3f583819 Verifying Hash Integrity ... sha1+ OK Booting using the fdt blob at 0x106250a8 Loading Kernel Image Loading Device Tree to 1ead4000, end 1eadbd36 ... OK Starting kernel ... Booting Linux on physical CPU 0x0 Linux version 5.15.19-xilinx-v2022.1 (oe-user@oe-host) (arm-xilinx-linux-gnueabi-gcc (GCC) 11.2.0, GNU ld (GNU Binutils) 2.37.20210721) #1 SMP PREEMPT Mon Apr 11 17:52:14 UTC 2022 CPU: ARMv7 Processor [413fc090] revision 0 (ARMv7), cr=18c5387d CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache OF: fdt: Machine model: xlnx,zynq-7000 Memory policy: Data cache writealloc cma: Reserved 128 MiB at 0x16800000 Zone ranges: Normal [mem 0x0000000000000000-0x000000001fffffff] HighMem empty Movable zone start for each node Early memory node ranges node 0: [mem 0x0000000000000000-0x000000001fffffff] Initmem setup node 0 [mem 0x0000000000000000-0x000000001fffffff] percpu: Embedded 12 pages/cpu s18828 r8192 d22132 u49152 Built 1 zonelists, mobility grouping on. Total pages: 129920 Kernel command line: root=/dev/mmcblk0p2 rw earlyprintk rootfstype=ext4 rootwait devtmpfs.mount=1 uio_pdrv_genirq.of_id="generic-uio" clk_ignore_unused Unknown kernel command line parameters "earlyprintk", will be passed to user space. Dentry cache hash table entries: 65536 (order: 6, 262144 bytes, linear) Inode-cache hash table entries: 32768 (order: 5, 131072 bytes, linear) mem auto-init: stack:off, heap alloc:off, heap free:off Memory: 373332K/524288K available (9216K kernel code, 324K rwdata, 2512K rodata, 1024K init, 296K bss, 19884K reserved, 131072K cma-reserved, 0K highmem) rcu: Preemptible hierarchical RCU implementation. rcu: RCU event tracing is enabled. rcu: RCU restricting CPUs from NR_CPUS=4 to nr_cpu_ids=2. Trampoline variant of Tasks RCU enabled. Tracing variant of Tasks RCU enabled. rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies. rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2 NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16 efuse mapped to (ptrval) slcr mapped to (ptrval) GIC physical location is 0xf8f01000 L2C: platform modifies aux control register: 0x72360000 -> 0x72760000 L2C: DT/platform modifies aux control register: 0x72360000 -> 0x72760000 L2C-310 erratum 769419 enabled L2C-310 enabling early BRESP for Cortex-A9 L2C-310 full line of zeros enabled for Cortex-A9 L2C-310 ID prefetch enabled, offset 1 lines L2C-310 dynamic clock gating enabled, standby mode enabled L2C-310 cache controller enabled, 8 ways, 512 kB L2C-310: CACHE_ID 0x410000c8, AUX_CTRL 0x76760001 random: get_random_bytes called from start_kernel+0x364/0x5f8 with crng_init=0 zynq_clock_init: clkc starts at (ptrval) Zynq clock init sched_clock: 64 bits at 162MHz, resolution 6ns, wraps every 4398046511101ns clocksource: arm_global_timer: mask: 0xffffffffffffffff max_cycles: 0x257a3bfb55, max_idle_ns: 440795207830 ns Switching to timer-based delay loop, resolution 6ns Console: colour dummy device 80x30 printk: console [tty0] enabled Calibrating delay loop (skipped), value calculated using timer frequency.. 325.00 BogoMIPS (lpj=1625000) pid_max: default: 32768 minimum: 301 Mount-cache hash table entries: 1024 (order: 0, 4096 bytes, linear) Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes, linear) CPU: Testing write buffer coherency: ok CPU0: Spectre v2: using BPIALL workaround CPU0: thread -1, cpu 0, socket 0, mpidr 80000000 Setting up static identity map for 0x100000 - 0x100060 rcu: Hierarchical SRCU implementation. smp: Bringing up secondary CPUs ... CPU1: thread -1, cpu 1, socket 0, mpidr 80000001 CPU1: Spectre v2: using BPIALL workaround smp: Brought up 1 node, 2 CPUs SMP: Total of 2 processors activated (650.00 BogoMIPS). CPU: All CPU(s) started in SVC mode. devtmpfs: initialized VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 4 clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns futex hash table entries: 512 (order: 3, 32768 bytes, linear) pinctrl core: initialized pinctrl subsystem NET: Registered PF_NETLINK/PF_ROUTE protocol family DMA: preallocated 256 KiB pool for atomic coherent allocations thermal_sys: Registered thermal governor 'step_wise' cpuidle: using governor menu amba f8801000.etb: Fixing up cyclic dependency with replicator amba f8803000.tpiu: Fixing up cyclic dependency with replicator amba f8804000.funnel: Fixing up cyclic dependency with replicator amba f889c000.ptm: Fixing up cyclic dependency with f8804000.funnel amba f889d000.ptm: Fixing up cyclic dependency with f8804000.funnel hw-breakpoint: found 5 (+1 reserved) breakpoint and 1 watchpoint registers. hw-breakpoint: maximum watchpoint size is 4 bytes. zynq-ocm f800c000.ocmc: ZYNQ OCM pool: 256 KiB @ 0x(ptrval) e0000000.serial: ttyPS0 at MMIO 0xe0000000 (irq = 32, base_baud = 6250000) is a xuartps printk: console [ttyPS0] enabled raid6: int32x8 gen() 117 MB/s raid6: int32x8 xor() 78 MB/s raid6: int32x4 gen() 125 MB/s raid6: int32x4 xor() 88 MB/s raid6: int32x2 gen() 200 MB/s raid6: int32x2 xor() 117 MB/s raid6: int32x1 gen() 180 MB/s raid6: int32x1 xor() 94 MB/s raid6: using algorithm int32x2 gen() 200 MB/s raid6: .... xor() 117 MB/s, rmw enabled raid6: using intx1 recovery algorithm vgaarb: loaded SCSI subsystem initialized usbcore: registered new interface driver usbfs usbcore: registered new interface driver hub usbcore: registered new device driver usb mc: Linux media interface: v0.10 videodev: Linux video capture interface: v2.00 pps_core: LinuxPPS API ver. 1 registered pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it> PTP clock support registered EDAC MC: Ver: 3.0.0 FPGA manager framework Advanced Linux Sound Architecture Driver Initialized. clocksource: Switched to clocksource arm_global_timer NET: Registered PF_INET protocol family IP idents hash table entries: 8192 (order: 4, 65536 bytes, linear) tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 6144 bytes, linear) TCP established hash table entries: 4096 (order: 2, 16384 bytes, linear) TCP bind hash table entries: 4096 (order: 3, 32768 bytes, linear) TCP: Hash tables configured (established 4096 bind 4096) UDP hash table entries: 256 (order: 1, 8192 bytes, linear) UDP-Lite hash table entries: 256 (order: 1, 8192 bytes, linear) NET: Registered PF_UNIX/PF_LOCAL protocol family RPC: Registered named UNIX socket transport module. RPC: Registered udp transport module. RPC: Registered tcp transport module. RPC: Registered tcp NFSv4.1 backchannel transport module. PCI: CLS 0 bytes, default 64 armv7-pmu f8891000.pmu: hw perfevents: no interrupt-affinity property, guessing. hw perfevents: enabled with armv7_cortex_a9 PMU driver, 7 counters available Initialise system trusted keyrings workingset: timestamp_bits=14 max_order=17 bucket_order=3 squashfs: version 4.0 (2009/01/31) Phillip Lougher jffs2: version 2.2. (NAND) (SUMMARY) © 2001-2006 Red Hat, Inc. xor: measuring software checksum speed arm4regs : 1045 MB/sec 8regs : 805 MB/sec 32regs : 836 MB/sec xor: using function: arm4regs (1045 MB/sec) Key type asymmetric registered Asymmetric key parser 'x509' registered io scheduler mq-deadline registered io scheduler kyber registered zynq-pinctrl 700.pinctrl: zynq pinctrl initialized dma-pl330 f8003000.dmac: Loaded driver for PL330 DMAC-241330 dma-pl330 f8003000.dmac: DBUFF-128x8bytes Num_Chans-8 Num_Peri-4 Num_Events-16 brd: module loaded loop: module loaded random: fast init done spi-nor spi0.0: w25q128jv (16384 Kbytes) spi0.0: error parsing ofpart partition /axi/spi@e000d000/flash@0/partition@0 (/axi/spi@e000d000/flash@0) tun: Universal TUN/TAP device driver, 1.6 CAN device driver interface macb e000b000.ethernet eth0: Cadence GEM rev 0x00020118 at 0xe000b000 irq 34 (00:18:3e:04:a1:50) e1000e: Intel(R) PRO/1000 Network Driver e1000e: Copyright(c) 1999 - 2015 Intel Corporation. ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver ehci-pci: EHCI PCI platform driver usbcore: registered new interface driver cdc_acm cdc_acm: USB Abstract Control Model driver for USB modems and ISDN adapters usbcore: registered new interface driver cdc_wdm usbcore: registered new interface driver usb-storage usbcore: registered new interface driver usbserial_generic usbserial: USB Serial support registered for generic usbcore: registered new interface driver usb_serial_simple usbserial: USB Serial support registered for carelink usbserial: USB Serial support registered for zio usbserial: USB Serial support registered for funsoft usbserial: USB Serial support registered for flashloader usbserial: USB Serial support registered for google usbserial: USB Serial support registered for libtransistor usbserial: USB Serial support registered for vivopay usbserial: USB Serial support registered for moto_modem usbserial: USB Serial support registered for motorola_tetra usbserial: USB Serial support registered for novatel_gps usbserial: USB Serial support registered for hp4x usbserial: USB Serial support registered for suunto usbserial: USB Serial support registered for siemens_mpi ULPI transceiver vendor/product ID 0x0451/0x1507 Found TI TUSB1210 ULPI transceiver. ULPI integrity check: passed. ci_hdrc ci_hdrc.0: EHCI Host Controller ci_hdrc ci_hdrc.0: new USB bus registered, assigned bus number 1 ci_hdrc ci_hdrc.0: USB 2.0 started, EHCI 1.00 hub 1-0:1.0: USB hub found hub 1-0:1.0: 1 port detected i2c_dev: i2c /dev entries driver cdns-wdt f8005000.watchdog: Xilinx Watchdog Timer with timeout 10s device-mapper: ioctl: 4.45.0-ioctl (2021-03-22) initialised: dm-devel@redhat.com EDAC MC: ECC not enabled Xilinx Zynq CpuIdle Driver started sdhci: Secure Digital Host Controller Interface driver sdhci: Copyright(c) Pierre Ossman sdhci-pltfm: SDHCI platform and OF driver helper ledtrig-cpu: registered to indicate activity on CPUs clocksource: ttc_clocksource: mask: 0xffff max_cycles: 0xffff, max_idle_ns: 551318127 ns timer #0 at (ptrval), irq=47 usbcore: registered new interface driver usbhid usbhid: USB HID core driver xlnk xlnk: Major 243 xlnk xlnk: xlnk driver loaded xlnk xlnk: xlnk_pdev is not null mmc0: SDHCI controller on e0100000.mmc [e0100000.mmc] using ADMA fpga_manager fpga0: Xilinx Zynq FPGA Manager registered IPVS: Registered protocols (TCP, UDP) IPVS: Connection hash table configured (size=4096, memory=32Kbytes) IPVS: ipvs loaded. IPVS: [rr] scheduler registered. Initializing XFRM netlink socket NET: Registered PF_INET6 protocol family Segment Routing with IPv6 In-situ OAM (IOAM) with IPv6 sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver mmc0: new high speed SDHC card at address aaaa NET: Registered PF_PACKET protocol family mmcblk0: mmc0:aaaa SE32G 29.7 GiB can: controller area network core NET: Registered PF_CAN protocol family mmcblk0: p1 p2 can: raw protocol can: broadcast manager protocol can: netlink gateway - max_hops=1 Registering SWP/SWPB emulation handler Loading compiled-in X.509 certificates Btrfs loaded, crc32c=crc32c-generic, zoned=no, fsverity=no of-fpga-region fpga-full: FPGA Region probed of_cfs_init of_cfs_init: OK cfg80211: Loading compiled-in X.509 certificates for regulatory database cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7' platform regulatory.0: Direct firmware load for regulatory.db failed with error -2 cfg80211: failed to load regulatory.db clk: Not disabling unused clocks ALSA device list: No soundcards found. EXT4-fs (mmcblk0p2): mounted filesystem with ordered data mode. Opts: (null). Quota mode: disabled. VFS: Mounted root (ext4 filesystem) on device 179:2. devtmpfs: mounted Freeing unused kernel image (initmem) memory: 1024K Run /sbin/init as init process systemd[1]: System time before build time, advancing clock. systemd[1]: Failed to find module 'autofs4' systemd[1]: systemd 249.11-0ubuntu3 running in system mode (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS -OPENSSL +ACL +BLKID +CURL +ELFUTILS -FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP -LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified) systemd[1]: Detected architecture arm. Welcome to PynqLinux, based on Ubuntu 22.04! systemd[1]: Hostname set to <pynq>. systemd[1]: Queued start job for default target Multi-User System. random: systemd: uninitialized urandom read (16 bytes read) systemd[1]: Created slice Slice /system/modprobe. [ OK ] Created slice Slice /system/modprobe. random: systemd: uninitialized urandom read (16 bytes read) systemd[1]: Created slice Slice /system/serial-getty. [ OK ] Created slice Slice /system/serial-getty. random: systemd: uninitialized urandom read (16 bytes read) systemd[1]: Created slice User and Session Slice. [ OK ] Created slice User and Session Slice. systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password …ts to Console Directory Watch. systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password R…uests to Wall Directory Watch. systemd[1]: Condition check resulted in Arbitrary Executable File Formats File System Automount Point being skipped. systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. systemd[1]: Reached target Remote File Systems. [ OK ] Reached target Remote File Systems. systemd[1]: Reached target Slice Units. [ OK ] Reached target Slice Units. systemd[1]: Reached target Local Verity Protected Volumes. [ OK ] Reached target Local Verity Protected Volumes. systemd[1]: Listening on Syslog Socket. [ OK ] Listening on Syslog Socket. systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. systemd[1]: Condition check resulted in Journal Audit Socket being skipped. systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. systemd[1]: Condition check resulted in Huge Pages File System being skipped. systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... systemd[1]: Condition check resulted in Kernel Trace File System being skipped. systemd[1]: Starting Journal Service... Starting Journal Service... systemd[1]: Starting Restore / save the current clock... Starting Restore / save the current clock... systemd[1]: Starting Set the console keyboard layout... Starting Set the console keyboard layout... systemd[1]: Condition check resulted in Create List of Static Device Nodes being skipped. systemd[1]: Starting Load Kernel Module configfs... Starting Load Kernel Module configfs... systemd[1]: Starting Load Kernel Module drm... Starting Load Kernel Module drm... systemd[1]: Starting Load Kernel Module fuse... Starting Load Kernel Module fuse... systemd[1]: Started Nameserver information manager. [ OK ] Started Nameserver information manager. systemd[1]: Reached target Preparation for Network. [ OK ] Reached target Preparation for Network. systemd[1]: Starting Load Kernel Modules... Starting Load Kernel Modules... systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... systemd[1]: Starting Coldplug All udev Devices... Starting Coldplug All udev Devices... systemd[1]: Started Journal Service. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Finished Restore / save the current clock. [ OK ] Finished Load Kernel Module configfs. [ OK ] Finished Set the console keyboard layout. [ OK ] Finished Load Kernel Module drm. [ OK ] Finished Load Kernel Module fuse. [ OK ] Finished Load Kernel Modules. [ OK ] Finished Remount Root and Kernel File Systems. Activating swap /var/swap... Mounting Kernel Configuration File System... Adding 524284k swap on /var/swap. Priority:-2 extents:1 across:524284k SS Starting Flush Journal to Persistent Storage... systemd-journald[110]: Received client request to flush runtime journal. Starting Load/Save Random Seed... systemd-journald[110]: File /var/log/journal/b4e19d1e11f24633a2f2e1fe2798cbe4/system.journal corrupted or uncleanly shut down, renaming and replacing. Starting Apply Kernel Variables... Starting Create System Users... [ OK ] Activated swap /var/swap. [ OK ] Mounted Kernel Configuration File System. [ OK ] Reached target Swaps. [ OK ] Finished Apply Kernel Variables. [ OK ] Finished Create System Users. Starting Create Static Device Nodes in /dev... [ OK ] Finished Coldplug All udev Devices. Starting Helper to synchronize boot up for ifupdown... [ OK ] Finished Flush Journal to Persistent Storage. [ OK ] Finished Create Static Device Nodes in /dev. [ OK ] Reached target Preparation for Local File Systems. [ OK ] Reached target Local File Systems. Starting Enable support fo…l executable binary formats... Starting Set console font and keymap... Starting Create Volatile Files and Directories... Starting Rule-based Manage…for Device Events and Files... [ OK ] Finished Enable support fo…nal executable binary formats. [ OK ] Finished Set console font and keymap. [ OK ] Finished Create Volatile Files and Directories. [ OK ] Started Entropy Daemon based on the HAVEGE algorithm. Starting Network Name Resolution... Starting Network Time Synchronization... Starting Record System Boot/Shutdown in UTMP... [ OK ] Started Rule-based Manager for Device Events and Files. [ OK ] Finished Record System Boot/Shutdown in UTMP. [ OK ] Found device /dev/ttyPS0. zocl-drm axi:zyxclmm_drm: IRQ index 0 not found [ OK ] Started Network Time Synchronization. [ OK ] Finished Helper to synchronize boot up for ifupdown. [ OK ] Finished Load/Save Random Seed. [ OK ] Started Network Name Resolution. [ OK ] Reached target Host and Network Name Lookups. [ OK ] Reached target System Time Set. [ OK ] Reached target Hardware activated USB gadget. [ OK ] Stopped Entropy Daemon based on the HAVEGE algorithm. [ OK ] Started Entropy Daemon based on the HAVEGE algorithm. [ OK ] Reached target System Initialization. [ OK ] Started resolvconf-pull-resolved.path. [ OK ] Started Trigger to poll fo…y enabled on GCP LTS non-pro). [ OK ] Started Daily apt download activities. [ OK ] Started Daily apt upgrade and clean activities. [ OK ] Started Daily dpkg database backup timer. [ OK ] Started Periodic ext4 Onli…ata Check for All Filesystems. [ OK ] Started Discard unused blocks once a week. [ OK ] Started Daily rotation of log files. [ OK ] Started Daily man-db regeneration. [ OK ] Started Message of the Day. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started Ubuntu Advantage Timer for running repeated jobs. [ OK ] Reached target Path Units. [ OK ] Reached target Timer Units. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Listening on UUID daemon activation socket. [ OK ] Reached target Socket Units. [ OK ] Reached target Basic System. Starting LSB: automatic crash report generation... Starting Executing boot.py from the boot partition... [ OK ] Started Regular background program processing daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Save initial kernel messages after boot. Starting Remove Stale Onli…t4 Metadata Check Snapshots... [ OK ] Started ifup for eth0. Starting Jupyter Notebook Server... Starting LSB: Load kernel …d to enable cpufreq scaling... Starting Dispatcher daemon for systemd-networkd... Starting Raise network interfaces... Starting Resize Filesystem on SD card... Starting resolvconf-pull-resolved.service... Starting System Logging Service... Starting User Login Management... [ OK ] Finished Resize Filesystem on SD card. [ OK ] Started System Logging Service. [ OK ] Finished resolvconf-pull-resolved.service. [ OK ] Started LSB: automatic crash report generation. [ OK ] Stopped Entropy Daemon based on the HAVEGE algorithm. [ OK ] Started Entropy Daemon based on the HAVEGE algorithm. [ OK ] Started LSB: Load kernel m…ded to enable cpufreq scaling. Starting LSB: set CPUFreq kernel parameters... [ OK ] Started LSB: set CPUFreq kernel parameters. [ OK ] Started Dispatcher daemon for systemd-networkd. [ OK ] Started User Login Management. [ OK ] Finished Remove Stale Onli…ext4 Metadata Check Snapshots. [ OK ] Stopped Entropy Daemon based on the HAVEGE algorithm. [ OK ] Started Entropy Daemon based on the HAVEGE algorithm. [ OK ] Stopped Entropy Daemon based on the HAVEGE algorithm. [ OK ] Started Entropy Daemon based on the HAVEGE algorithm. [ OK ] Stopped Entropy Daemon based on the HAVEGE algorithm. [ OK ] Started Entropy Daemon based on the HAVEGE algorithm. Starting resolvconf-pull-resolved.service... [ OK ] Finished resolvconf-pull-resolved.service. Starting resolvconf-pull-resolved.service... Stopping Network Time Synchronization... [ OK ] Stopped Network Time Synchronization. Starting Network Time Synchronization... [ OK ] Finished resolvconf-pull-resolved.service. [ OK ] Stopped Entropy Daemon based on the HAVEGE algorithm. [ OK ] Started Entropy Daemon based on the HAVEGE algorithm. [ OK ] Started Network Time Synchronization. [ OK ] Finished Raise network interfaces. [ OK ] Reached target Network. [ OK ] Reached target Network is Online. [ OK ] Started ISC DHCP IPv4 server. [ OK ] Started ISC DHCP IPv6 server. Starting Samba NMB Daemon... Starting OpenBSD Secure Shell server... Starting Permit User Sessions... [ OK ] Started Unattended Upgrades Shutdown. [ OK ] Finished Permit User Sessions. [ OK ] Started Serial Getty on ttyPS0. Starting Set console scheme... PYNQ Linux, based on Ubuntu 22.04 pynq ttyPS0 pynq login: xilinx (automatic login) Welcome to PYNQ Linux, based on Ubuntu 22.04 (GNU/Linux 5.15.19-xilinx-v2022.1 armv7l) Last login: Thu Sep 12 19:40:39 UTC 2024 on ttyPS0 xilinx@pynq:~$
Shutdown console log
And the log generated when performing an orderly shutdown.
Last login: Thu Sep 12 20:51:48 UTC 2024 on ttyPS0 xilinx@pynq:~$ sudo shutdown now [sudo] password for xilinx: Stopping Session c1 of User xilinx... [ OK ] Removed slice Slice /system/modprobe. [ OK ] Stopped target Multi-User System. [ OK ] Stopped target Login Prompts. [ OK ] Stopped target Host and Network Name Lookups. [ OK ] Stopped target Timer Units. [ OK ] Stopped Daily apt upgrade and clean activities. [ OK ] Stopped Daily apt download activities. [ OK ] Stopped Daily dpkg database backup timer. [ OK ] Stopped Periodic ext4 Onli…ata Check for All Filesystems. [ OK ] Stopped Discard unused blocks once a week. [ OK ] Stopped Daily rotation of log files. [ OK ] Stopped Daily man-db regeneration. [ OK ] Stopped Message of the Day. [ OK ] Stopped Daily Cleanup of Temporary Directories. [ OK ] Stopped Ubuntu Advantage Timer for running repeated jobs. [ OK ] Stopped target System Time Set. [ OK ] Stopped target Hardware activated USB gadget. Stopping LSB: automatic crash report generation... Stopping LSB: set CPUFreq kernel parameters... Stopping Regular background program processing daemon... Stopping Getty on tty1... Stopping Jupyter Notebook Server... Stopping Dispatcher daemon for systemd-networkd... Stopping System Logging Service... Stopping Serial Getty on ttyPS0... Stopping Samba SMB Daemon... Stopping OpenBSD Secure Shell server... Stopping Load/Save Random Seed... Stopping Unattended Upgrades Shutdown... [ OK ] Stopped Regular background program processing daemon. [ OK ] Stopped Dispatcher daemon for systemd-networkd. [ OK ] Stopped System Logging Service. [ OK ] Stopped Unattended Upgrades Shutdown. [ OK ] Stopped Serial Getty on ttyPS0. [ OK ] Stopped Getty on tty1. [ OK ] Stopped OpenBSD Secure Shell server. [ OK ] Stopped Samba SMB Daemon. [ OK ] Stopped Load/Save Random Seed. [ OK ] Stopped Session c1 of User xilinx. [ OK ] Removed slice Slice /system/getty. [ OK ] Removed slice Slice /system/serial-getty. Stopping Samba NMB Daemon... Stopping User Login Management... Stopping User Manager for UID 1000... [ OK ] Stopped User Login Management. [ OK ] Stopped Samba NMB Daemon. [ OK ] Stopped User Manager for UID 1000. [ OK ] Stopped LSB: automatic crash report generation. [ OK ] Stopped LSB: set CPUFreq kernel parameters. [ OK ] Stopped target Network is Online. Stopping LSB: Load kernel …d to enable cpufreq scaling... Stopping User Runtime Directory /run/user/1000... [ OK ] Unmounted /run/user/1000. [ OK ] Stopped LSB: Load kernel m…ded to enable cpufreq scaling. [ OK ] Stopped User Runtime Directory /run/user/1000. [ OK ] Removed slice User Slice of UID 1000. Stopping Permit User Sessions... [ OK ] Stopped Permit User Sessions. [ OK ] Stopped target Network. [ OK ] Stopped target Remote File Systems. Stopping ifup for eth0... Stopping Raise network interfaces... Stopping Network Name Resolution... [ OK ] Stopped Network Name Resolution. [ OK ] Stopped Raise network interfaces. [ OK ] Stopped ifup for eth0. [ OK ] Stopped target Preparation for Network. [ OK ] Stopped Jupyter Notebook Server. [ OK ] Stopped target Basic System. [ OK ] Stopped target Path Units. [ OK ] Stopped resolvconf-pull-resolved.path. [ OK ] Stopped Trigger to poll fo…y enabled on GCP LTS non-pro). [ OK ] Stopped target Slice Units. [ OK ] Removed slice User and Session Slice. [ OK ] Stopped target Socket Units. [ OK ] Closed Syslog Socket. [ OK ] Closed UUID daemon activation socket. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password …ts to Console Directory Watch. [ OK ] Stopped Forward Password R…uests to Wall Directory Watch. [ OK ] Stopped target Swaps. [ OK ] Stopped target Local Verity Protected Volumes. Deactivating swap /var/swap... Stopping Restore / save the current clock... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Load Kernel Modules. Stopping Network Time Synchronization... Stopping Record System Boot/Shutdown in UTMP... [ OK ] Stopped Network Time Synchronization. [ OK ] Deactivated swap /var/swap. [ OK ] Stopped Restore / save the current clock. [ OK ] Stopped Record System Boot/Shutdown in UTMP. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. Unmounting boot.mount... Unmounting /run/credentials/systemd-sysusers.service... [ OK ] Unmounted boot.mount. [ OK ] Unmounted /run/credentials/systemd-sysusers.service. [ OK ] Stopped target Preparation for Local File Systems. [ OK ] Reached target Unmount All Filesystems. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create System Users. [ OK ] Stopped Remount Root and Kernel File Systems. [ OK ] Reached target System Shutdown. [ OK ] Reached target Late Shutdown Services. [ OK ] Finished System Power Off. [ OK ] Reached target System Power Off. reboot: System halted
Checking IP address using my router.
Before I started changing the network configuration parameters, I quickly checked if my router had registered the new connection. To my pleasant surprise, it appeared on the network as PYNQ, no problem. You can also check the assigned IP from the command line, use whatever is fastest for you.
Pinging the board
The board responds correctly to ICMP echoes ping.
PS C:\Users\ealbertos> ping pynq
Pinging pynq [192.168.1.41] with 32 bytes of data:
Reply from 192.168.1.41: bytes=32 time<1ms TTL=64
Reply from 192.168.1.41: bytes=32 time<1ms TTL=64
Reply from 192.168.1.41: bytes=32 time<1ms TTL=64
Reply from 192.168.1.41: bytes=32 time<1ms TTL=64
Ping statistics for 192.168.1.41:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 0ms, Average = 0ms
Opening Jupyter
Once your board is setup, to connect to Jupyter Notebooks open a web browser from your PC or laptop and navigate to: http://pynq:9090
If your board is configured correctly, you will be presented with a login screen. The username is xilinx and the password is also xilinx.
Updating PYNQ files
If you are not sure you have the latest version of the files you can update them from the command line using the serial console.
sudo pip3 install --upgrade --upgrade-strategy only-if-needed pynq
xilinx@pynq:~$ cat REVISION
Release 2022_10_22 93ddd21
Board 2022_10_22 93ddd21 https://github.com/Xilinx/PYNQ
xilinx@pynq:~$
Accessing Files on The Board
Samba, a file sharing service, is running on the board. This allows you to access the Pynq home area as a network drive, to transfer files to and from the board.
Windows explorer
Welcome to PYNQ
Included in the PYNQ distribution are several Jupyter notebooks with tutorials, ready to be run on demand.
PYNQ Libraries
A significant benefit of PYNQ is its capability to support a wide array of peripherals. It accommodates common interfaces such as USB, Ethernet, and HDMI, as well as specialized ones like Pmods and Arduino devices. To utilize peripherals connected to the PL, one must load an overlay that includes the necessary hardware IP for peripheral control, ensuring smooth hardware-software integration.
The Python API of PYNQ eases the interaction with peripherals. Developers can control devices using the familiar Python syntax, broadening accessibility. Moreover, the API is extendable to include additional IP, offering customized solutions for various applications.
For advanced control, PYNQ provides low-level access to the overlay, featuring memory-mapped IO, memory allocation, and bitstream management, perfect for developers aiming to enhance performance or add custom features.
PYNQ also boasts the PynqMicroBlaze subsystem, facilitating the loading and creation of pre-compiled applications, thus expediting the development workflow.
Arty-Z7 Overlays
The following overlays are included by default in the PYNQ image for the PYNQ-Z1 and the Arty-Z7 board:
- Base Overlay: The base overlay for the PYNQ-Z1/Arty Z7 board enables PYNQ to utilize its peripherals immediately after booting, thanks to embedded hardware IP that manages these peripherals and connects them to the Zynq’s Processing System. Typical peripherals include GPIO devices (like LEDs, switches, and buttons), video, audio, and other custom interfaces, which can also serve as reference designs for creating custom overlays. Additionally, for general-purpose interfaces such as Pmod or Arduino headers, the base overlay may include a PYNQ MicroBlaze, allowing control of devices with various interfaces and protocols on the same port without modifying the programmable logic design.
Mic is not included on the Arty Z7
- Logictools Overlay: The LogicTools overlay includes programmable hardware blocks that interact with external digital logic circuits. Finite state machines, Boolean logic functions, and digital patterns can be generated using Python. Additionally, a programmable switch links the inputs and outputs of these hardware blocks to external I/O pins. And the LogicTools overlay includes a trace analyzer for capturing and analyzing data from the I/O interface, aiding in debugging and analysis.
The Arty-Z7 board has the following features:
- Zynq XC7Z020-1CLG400C
- 512MB DDR3
- 1G Ethernet
- USB 2.0
- MicroSD
- Uart
- 3.5mm mono audio output jack
- 2x HDMI (can be used as input or output)
- 4 push-buttons
- 2 slide switches
- 4 LEDs
- 2 RGB LEDs
- 2x Pmod ports
- 1x Arduino header
Exploring the video base overlay sample
The PYNQ Video subpackage provides a comprehensive set of drivers for managing video data on the PYNQ platform. It enables developers to capture video from the HDMI-In port, process it, and display it on the HDMI-Out port.
The underlying hardware consists of three key components: the HDMI-In block, the HDMI-Out block, and the Video DMA. The HDMI-In block captures video data, while the HDMI-Out block displays it. Both blocks support color space conversions and channel adjustments, allowing for flexible video processing.
The video library accommodates two distinct video front ends. These front-ends are tasked with transforming the device pin signals into a 24-bit-per-pixel, BGR formatted AXI stream, suitable for the subsequent pipeline stages. The Pynq-Z1 and the Arty Z7 boards feature a DVI-based front-end capable of supporting resolutions up to 1080p; however, the official support is limited to 720p due to the differential pins' speed ratings. This front-end is readily available as a drop-in IP within the PYNQ IP library.
HDMI-In
The Pixel Unpack and the Color Convert block allow conversion between different color spaces at runtime.
HDMI-out
The HDMI-Out is similar to HDMI-In. It has a Pixel Pack block (instead of the Unpack block for HDMI-In) and a Color Convert block.
The PYNQ video library offers three primary methods for processing video data:
- Python Processing on the PS: This approach is flexible and straightforward to implement, accommodating complex algorithms. However, it can be computationally intensive, especially for high-resolution tasks or those requiring real-time execution.
- Frame Handling: Input and output frames are represented as NumPy arrays, allowing for seamless integration with other Python libraries like OpenCV. Frames are read from the HDMI input and written to the HDMI output.
- Synchronization: The HDMI input will either return the most recent complete frame or wait until a frame is ready. The HDMI output will persist in displaying the last frame written until a new frame is supplied.
- IP Insertion in the Overlay: This method offers high-performance, hardware-accelerated processing. However, it requires modifying the overlay and recompiling the bitstream, which can be time-consuming.
- Pipeline Processing: Custom AXI-stream IP can be integrated into the video pipeline, either on the HDMI-In or HDMI-Out side.
- IP Placement: For optimal performance, consider placing the IP after the pixel_pack block in the HDMI-In section or before the pixel_unpack block in the HDMI-Out section. This allows for efficient color space conversion.
- IP Connection to an Available Interface: This modular approach allows for easy integration of existing IP cores. However, it may require additional interfaces or adapters.
- Batch Processing: An IP block can be added to an overlay and connected to an available interface. A DMA is typically used to stream data between the IP and the frame buffers in DRAM.
- DRAM Bottleneck: The DRAM can be a bottleneck for video processing, especially for high-resolution content. Consider optimizing memory access patterns and potentially using external memory for demanding applications.
- For the Arty-Z7 with a 512MB DDR3 with 16-bit bus, up to 1080p grayscale (8-bits per pixel) can be processed at approximately 60 FPS, but this is close to the system's total memory bandwidth.
The Apple Ipad detects the HDMI sink.
HDMI Sink and Source Demo: A Real-Time Look
Initially, I had reservations about sharing these unedited videos; however, their effectiveness in illustrating the examples has persuaded me. Although lengthy, they offer a clear demonstration of real-time image processing utilizing the HDMI sink and source on the Arty Z7 from PYNQ.
What You'll See:
- Two examples showcasing accelerated and non-accelerated image processing.
- A live session using the base overlay.
- No editing, just raw footage.
Setup Breakdown:
- HDMI Input: Apple iPad's camera app connected via USB to HDMI.
- HDMI Output: Arty Z7 to HDMI to USB video capture card.
- Display: Surface workstation with OBS screen capture.
- Software: Jupyter server (PYNQ 2.0), camera program, TeraTerm (serial console).
Screen Breakdown:
- Left: Chrome browser connected to Jupyter server.
- Right: Top: Camera program window. Bottom: TeraTerm serial console.
HDMI introduction
Face Detection Demo
USB Webcam demo
This notebook shows how to use a USB web camera attached to the board. An image is captured using fswebcam. The image can then be manipulated using the Python Image Library (Pillow).
The webcam used is the Ziquian 1080p Webcam and the driver for this webcam has already been installed on the board.
Recreating the base overlay from Windows
Clone https://github.com/Xilinx/PYNQ/tree/master/pynq
The build_ip.tcl file compiles any HLS IP used in the design. The base.tcl builds the overlay.
In Windows, the two Tcl files can be sourced in Vivado to rebuild the overlay. The Tcl files to rebuild the overlay can be sourced from the Vivado GUI, or from the Vivado Tcl Shell (command line).
From Vivado 2022.1
To rebuild from the Vivado GUI, open Vivado. In the Vivado Tcl command line window change to the correct directory, and source the Tcl files:
source build_ip.tcl
source base.tcl
Once completed the process you can review the overlay block diagram and start your customizations.
Overlay Block Diagram:
PYNQ resources
- https://github.com/Xilinx/PYNQ
- PYNQ | Python Productivity to AMD Adaptive Coompute platforms
- awai54st/PYNQ-Classification: Python on Zynq FPGA for Convolutional Neural Networks (github.com)
- Xilinx/BNN-PYNQ: Quantized Neural Networks (QNNs) on PYNQ (github.com)
- Xilinx/PYNQ-ComputerVision: Computer Vision Overlays on Pynq (github.com)
- Xilinx/PYNQ_Composable_Pipeline: PYNQ Composabe Overlays (github.com)
- briansune/PYNQ-2.7-MNIST: PYNQ-Based MNIST with Tensorflow Lite (github.com)
Summary
In this blog I have presented a quick introduction to the PYNQ framework, from its installation on the Arty-Z7 development board and a brief walk through some of the official examples and tutorials. I have also rebuilt the base overlay from Vivado by obtaining the original code from the github repository. I hope this is a blog that will help you get started quickly with the Arty-Z7 for this experimentation challenge.