Get Your Xilinx FPGA/Programmable SoC Questions Answered here

Thanks, that is an interesting approach and I will try that out.

I also learnt, that Digilent Arty-S7 board comes with a demo design that uses the DDR3 memory. It was recommended to start with the demo code and add my functionality instead of starting from scratch. I have not found the code yet, but it might be a 2nd option to get to a working code base.

For what it's worth, there is an Integrating Arm Cortex-M soft CPU IP into FPGAs virtual workshop that features Adam Taylor as an instructor and uses the Digilent Arty S7-50T to be held August 14th, 2019.

https://events.hackster.io/designstart

A couple of areas I'm interested in exploring with FPGA ( initially using the TUL PYNQ-Z2 board ) are:

1) HDMI video capture and processing where the PYNQ-Z2 is used for low cost HDMI capture / pass through for basic object/pedestrian detection/tracking which is then further used for indexing video recordings or automated tracking such as remote camera control. 
(It would also be interesting to look at accessing the HDMI I2C bus to extract/inject EDID information via the PYNQ framework to enable video signal diagnostic type applications.)

Some interesting online resources I've found in this area so far are:

• Adam Taylor - Use Python, Zynq and OpenCV to Implement Computer Vision
  https://www.hackster.io/adam-taylor/use-python-zynq-and-opencv-to-implement-computer-vision-361e1b
  which is based on the Arty Z7-20 so I am expecting similar performance on the PYNQ-Z2. THe 29.7fps reported looks good
  enough for experimentation purposes with an option to move up to the UltraScale+ boards.

• Adrian Rosebrook - Pedestrian Detection OpenCV
  https://www.pyimagesearch.com/2015/11/09/pedestrian-detection-opencv/
  using OpenCV's histogram of gradients model for pedestrian detection.

• Adam Geitgey - snagging parking spaces with mask r-cnn and python
  https://medium.com/@ageitgey/snagging-parking-spaces-with-mask-r-cnn-and-python-955f2231c400
  Computer vision with some machine learning applied.

Some inital questions perhaps arise around what are the quick wins in terms of hardware acceleration vs productivity in this area. A lot of the processing appears to still be done in software on the ARM processor as oppossed to being accelerated on the FPGA. Are there perhaps more appropriate computer vision libraries that would make better use of the FPGA hardware ?

2) Serial data capture and processing where the FPGA hardware or MicroBlaze subsystem are used to create a customizable UART to decode the incoming serial data and make it available via the PYNQ framework. Perhaps using the hardware to detect
certain conditions to filter the data stream off-loading some of the processing.

One application I had in mind was to be able to ingest a DMX512 lighting protocol on a RS-485 interface and have Python scripts react when certain data frames are sent out from a lighting console. Perhaps using the HDMI inputs and outputs to pass through live video and switch it with still images (or computer generated art / real time data from the web) controlled by the lighting console to create intelligent lighting displays using video projection or video walls.

For experimentation purposes one perhaps could even use the MicroBlaze subsystem to set up a virtual DMX512 transmitter and and virtual DMX512 receiver and access  both via the PYNQ framework all on the one board without any worries of timing issues.

However at a more general level, similar approaches perhaps could be used for more general serial data acquisition and status display on an attached HDMI screen as a protocol diagnostic tool.

Unfortunately there doesn't yet appear to be PYNQ driver available for the existing PMOD 485 interface
https://store.digilentinc.com/pmod-rs485-high-speed-isolated-communication/
and I've not found much in the way of associated on-line FPGA tutorials in this particular area, especially not with using the PYNQ framework.

- What trends do you see happening in the FPGA world in the upcoming years?

- Do you see FPGAs moving into the realm of PC's? GPUs, even though they are completely different, have gained a lot of attention in the general computation arena, while at the same time FPGAs haven't apparently gained much more popularity. What do think is the cause of this?

- Do you see any upcoming official opening of the bitstream formats? The formats appear to be slowly being reverse-engineered, whats is the rationale in keeping them closed? I would think that opening them would allow the creation of an ecosystem of 3rd party tools which probably would attract more users to the platform. Making software opensource could also lower development costs, etc. What are your thoughts on all this?

These are great questions.

I think the trend is pretty clear they are evolving (and have been for many years) in to platforms, which combine different compute engines e.g Programmable Logic and Processors cores. The Xilinx Versal devices take this to the next step. These heterogeneous devices enable the deployment of elements of the application being developed to the most optimal compute engine, what we will see along with rapid device development in this area is a significant increases in development SW which offers a combined platform which can be used to develop the application and then deploy elements as required in the different engines example include SDSoC, SDAccel for example.

I am not sure PC's will ever contain a programmable logic device on board the motherboard but I think programmable logic will be easily available via the cloud e.g. AWS and using modules like the Alveo card either locally or in the cloud. GPU's became popular for computation as PC do need them as well for graphics in a number of applications therefore it was a natural extension to be able to use them for computation as needed.

I do not and I am not sure why people get hung up on it in my opinion. I am not 100% sure why they are kept closed behind the need for IP protection and most importantly getting the bitstream wrong could potentially damage the device which will cause several issues. If it was me I would focus open tool development on areas such as simulation, verification, synthesis, formal verification etc I know there is work going on in this area. Regarding development costs, most vendors offer free versions which cover a significant range of components, which should address the most maker, hobbyist and many commercial applications. While the larger projects using tools which require licensing are often developed by organisations which can afford the costs. I have also found the vendors very willing to talk and support if your doing interesting projects.

The Pynq Z2 should give a similar performance if you use the OpenCV overlay as outlined in the blog of mine you mention. If you want to look at the Ultra96 which is the best board available for many projects I think in my opinion then you can get significant speeds up including >100 frames per second for optical flow algorithm implementation https://blog.hackster.io/microzed-chronicles-ultra96-and-pynq-da3b22cc982

Regarding the PYNQ driver for the Pmod485 it is possible to write C code for the micro-blaze in the Jupyter note book, so you could use this approach and leverage the Pmod485 driver from the Digilent PMod library. I covered it briefly in a blog here https://blog.hackster.io/pynq-edition-interfacing-with-pmods-arduino-and-raspberry-pi-61676f9f0fab

adamtaylorcengfiet  wrote:

 

 

I am not sure PC's will ever contain a programmable logic device on board the motherboard but I think programmable logic will be easily available via the cloud e.g. AWS and using modules like the Alveo card either locally or in the cloud. GPU's became popular for computation as PC do need them as well for graphics in a number of applications therefore it was a natural extension to be able to use them for computation as needed.

 

Actually, FPGAs have been placed on Motherboards for some time now,.

https://www.anandtech.com/show/11960/asus-announces-ws-c621e-sage-workstation-motherboard-dual-xeon-overclocking

Intel is now producing Xeon processors that include a FPGA in the same package as well as offering the Intel Acceleration Stack for Intel Xeon CPU with FPGAs.  However, as you mentioned, these would be more inline with the Web Giants and such due to the cost of the devices at this time. 

I like the comment regarding open source tools.  This is starting to be a thing now where folks are getting frustrated with the FPGA vendor tools and are reverse engineering the FPGA code to create their own open source tools to program the device. I suspect this will continue to expand.

I knew FPGA where on motherboards but as I understand it, it is not accessible by the user for programming instead it is smallish device which provides functions on the motherboard design. I read the question as being will FPGA be deployed on motherboards in a manner that they are accessible by the user to program and accelerate algorithms like you can with GPU's

It will be interesting to see what the open source community does, I suspect your right re reverse engineering.

neuromodulator  wrote:

 

- Do you see any upcoming official opening of the bitstream formats? The formats appear to be slowly being reverse-engineered, whats is the rationale in keeping them closed? I would think that opening them would allow the creation of an ecosystem of 3rd party tools which probably would attract more users to the platform. Making software opensource could also lower development costs, etc. What are your thoughts on all this?

I've been advocating for open bitstream formats for decades.  It's great that some are being reverse-engineered and I heartily congratulate the developers of IceStorm and related tools.

I don't see FPGA vendors opening and documenting their bitstream formats.  They have various "official" reasons, none of which I find credible.  I've written about this a lot over the years at element14, for example in my article Taming the Wild Bitstream.  Here's an excerpt:

The excuse most commonly heard from FPGA vendors is that they insist their customers want the bitstream format to be secret to prevent reverse-engineering customer designs. OK, that makes sense except for one thing: millions of μP/μC-based products are designed and sold each year using open CPU architectures, yet there is very little worry that someone is going to reverse-engineer those designs. IMO most FPGA designers would gladly have better tools in exchange for open bitstream formats.

 

Most FPGAs are part of hardware/software systems, and unless the FPGA performs a trivial function, trying to understand what it does and how it interacts with the software is pretty nasty. Indeed, it’s hard enough to understand someone else’s FPGA even if you have commented VHDL/Verilog source code :-) The best you can do is to make an exact copy, which you can do now with the bitstream itself. Yet people rarely do, because you’re much better off designing better products with better features than copying your competitor’s last-generation products and trying to modify them enough so that you’re not blatently violating copyright.

I also make the claim in my article that closed bitstreams have held back FPGAs since their inception and have prevented FPGAs from having the huge success of microprocessors and microcontrollers.  Where would microprocessors be today if Intel had kept its instruction set secret and insisted that everyone program in PL/M?

Is there an ADC in the Ultra 96 board and if so how do I access it using pynq?