FPGA for Smart Camera - using 2-3 cameras

It isn't possible to answer your question.

You need to give some more information about what you propose to do with the FPGA.

Indeed, I'm not sure that you even need an FPGA for this application (yet).

There are two ways to approach the hardware scoping problem - one is to prototype the application in a simulation environment which will allow you to specify the computing task and select the hardware that will do it.  The other is to select the hardware, based on a cost (and possibly power consumption)  target, and try to work out if it can be powerful enough to be useful.

Since modern FPGAs (the chips not boards) range in price from a few $ to a few k$ there are a great many possibilities.

For my 2c worth, a smart camera at this stage is a move in the wrong direction - you will get bogged down with packaging, heat and other environmental problems. Get the thing working in a nice big box in the middle of the car with the sensors connected by nice reliable cables carrying uncompressed data.

Once it works you can start thinking about how to re-package it.

MK

Hi Michael,

Thanks for the answer.

Here are more details about what I'm trying to achieve.

I'm looking to build a vision/perception system for self driving cars, similar with what Tesla has, but at much lower cost, to help as many people interested in learning how to approach open source (and DIY) SDC development.
I would like to mount 6-8 good cameras in my car, and although I could use USB (web) camera, I think I can build better and cheaper ones with off the shelf components.

So I need a way to convert the image captured by the sensor to my PC with GPU where I can process it and predict the next move of the car (for now I'm not looking at controlling the car), just based on image (IMU and GPS will be fused also).
Basically to achieve what you said "the sensors connected by nice reliable cables carrying uncompressed data", I need to figure out the cameras, cables and input interface for PC (USB3, Ethernet, PCI) or Nvidia TX2 (also MIPI CSI).

The image sensors I understand could be accessible over different interfaces, so an FPGA that allows me to connect 1-3 cameras at the same time, and output without latency over USB3, Ethernet or PCI would be very good, if it not very expensive, in between $150-$400. I looked at FPGAs but there are so many and I'm not even sure where to start.

The idea with FPGA is to also allow me to later add programs for image processing, specifically stereo and HDR, beside other simpler algorithms.

Would be great if will allow also larger programs like neural nets, but I doubt it would be under $500.

This FPGA (the chip) should allow me to build smart camera at a low cost, prototyping cost can be larger, but if I can (with help of experts in this area) build a smart and flexible stereo camera for less around $150 in 1000+ quantities, I think that would be a great solution for open source SDC/robotics community
It may not be $150, but the idea is to keep it low cost while allowing high quality sensors if needed, and more important to permit improvements through programming the FPGA or replacing it with a bigger one, same for image sensors.

This is an old project, would be nice to reuse the work done here, I assume this kind of design would be the end goal, although initially if I can just connect 8 cameras through USB3, PCI or Ethernet (through FPGA in all cases if possible) would be super:

https://danstrother.com/2011/01/24/fpga-stereo-vision-project/

https://danstrother.com/2011/06/10/fpga-stereo-vision-core-released/

The design is fully parameterized and highly scalable; some example implementations include:

• 320×240 @ 120 FPS in Spartan-3E 250
• 640×480 @ 30 FPS in Spartan-3E 500
• 800×480 @ 60 FPS in Spartan-6 LX25
• 800×480 @ 60 FPS in Cyclone IV EP4CE22
• 1920×1080 @ 30 FPS in Spartan-6 LX75

Thanks,
Marius

Hi Michael,

Thanks for the answer.

Here are more details about what I'm trying to achieve.

I'm looking to build a vision/perception system for self driving cars, similar with what Tesla has, but at much lower cost, to help as many people interested in learning how to approach open source (and DIY) SDC development.
I would like to mount 6-8 good cameras in my car, and although I could use USB (web) camera, I think I can build better and cheaper ones with off the shelf components.

So I need a way to convert the image captured by the sensor to my PC with GPU where I can process it and predict the next move of the car (for now I'm not looking at controlling the car), just based on image (IMU and GPS will be fused also).
Basically to achieve what you said "the sensors connected by nice reliable cables carrying uncompressed data", I need to figure out the cameras, cables and input interface for PC (USB3, Ethernet, PCI) or Nvidia TX2 (also MIPI CSI).

The image sensors I understand could be accessible over different interfaces, so an FPGA that allows me to connect 1-3 cameras at the same time, and output without latency over USB3, Ethernet or PCI would be very good, if it not very expensive, in between $150-$400. I looked at FPGAs but there are so many and I'm not even sure where to start.

The idea with FPGA is to also allow me to later add programs for image processing, specifically stereo and HDR, beside other simpler algorithms.

Would be great if will allow also larger programs like neural nets, but I doubt it would be under $500.

This FPGA (the chip) should allow me to build smart camera at a low cost, prototyping cost can be larger, but if I can (with help of experts in this area) build a smart and flexible stereo camera for less around $150 in 1000+ quantities, I think that would be a great solution for open source SDC/robotics community
It may not be $150, but the idea is to keep it low cost while allowing high quality sensors if needed, and more important to permit improvements through programming the FPGA or replacing it with a bigger one, same for image sensors.

This is an old project, would be nice to reuse the work done here, I assume this kind of design would be the end goal, although initially if I can just connect 8 cameras through USB3, PCI or Ethernet (through FPGA in all cases if possible) would be super:

https://danstrother.com/2011/01/24/fpga-stereo-vision-project/

https://danstrother.com/2011/06/10/fpga-stereo-vision-core-released/

The design is fully parameterized and highly scalable; some example implementations include:

• 320×240 @ 120 FPS in Spartan-3E 250
• 640×480 @ 30 FPS in Spartan-3E 500
• 800×480 @ 60 FPS in Spartan-6 LX25
• 800×480 @ 60 FPS in Cyclone IV EP4CE22
• 1920×1080 @ 30 FPS in Spartan-6 LX75

Thanks,
Marius

I don't think you can ever get this to fly without some serious funding.

Playing with modern FPGAs gets quite pricey - even Xilinx's entry level series 7 part (Artix - there aren't any Spartan 7s yet) comes only in BGA, needs multiple power supplies etc etc. So that puts home brew PCBs pretty much out of the question. You then get to development boards and modules -

these people make nice modules https://shop.trenz-electronic.de/en/

For dev boards try the usual suspects and Google.

MK

Thanks Michael!

I don't plan to build the PCB myself, but hire some expert, as long as the design is made open source.

Looks like someone just started to do part of what I need, in open source:

ZyncMV is an open source machine/computer vision platform using the Xilinx Zync FPGA+ ARM Cortex A9 SoC

https://github.com/adamjvr/ZyncMV

I'll look into integrating his project also.

If anyone in this FPGA comunity would like to help us build such an open source #SmartCamera please join our efforts at:

http://ossdc.org

Thank you,

Marius