SecuCam #1: Experimenting with the MKR Vidor 4000

This is the first blog post of my "SecuCam" project.

The goal of the project is build a basic Security Camera while also experimenting with hardware-accelerated video processing features like Motion Detection.

1. Arduino MKR Vidor 4000

So, after I proposed an idea, tariq.ahmadkindly sent me an Arduino MKR Vidor 4000 board. Thanks for that!

The Arduino MRK Vidor 4000 board is a pretty interesting Arduino board from the MKR Family.

It is based on a Atmel / Microchip SAMD21 controller, but what makes it special is the Altera / Intel Cyclone 10 FPGA.

Along the standard stuff, the board has a MIPI CSI Camera Input, a Micro-HDMI Ouput and a WiFi module.

1.1 Getting Started

Getting started with Arduino MKR Vidor 4000 is fairly simple.

We need an Arduino IDE, with Arduino SAMD Boards support installed:

At this point we should be able to run the Blink sketch on the board, the same way as on any other Arduino board.

Note:
if the upload to the MKR Vidor 4000 does not works try "double-clicking" the Reset button. It should bring the board into boot-loader mode.

1.1 Camera to HDMI Example

Blink is boring! So, I jumped right into searching how to use the Camera and HDMI features of the board

For this we need the following external components:

• Raspberry Pi Camera Module v1.3 with the OV5647 sensor
• HDMI Display
• MicroHDMI to HDMI cable (or adapter + HDMI cable)

We need to connect them like this:

On software side we need to install the VidorGraphics library:

Now we can have some fun by running the VidorEnableCam example:

#include "VidorGraphics.h"
#include "VidorCamera.h"


VidorCamera vcam;


void setup() {
  Serial.begin(9600);
  
  // wait for the serial monitor to open,
  // if you are powering the board from a USB charger remove the next line
  while (!Serial) {}


  if (!FPGA.begin()) {
    Serial.println("Initialization failed!");
    while (1) {}
  }


  // begin() enables the I2C communication and initializes the camera


  if (!vcam.begin()) {
    Serial.println("Camera begin failed");
    while (1) {}
  }


  delay(4000);
  Serial.println("Power On");


  // The camera should be on now, streaming to the HDMI output
}


void loop()
{
  // Wait for the user input to stop the camera


  String res = Serial.readStringUntil('\n');
  if (res.indexOf("STOP") > 0) {
    vcam.end();
  }
}

Once we upload this example to the Vidor 4000 and open a Serial Monitor, the board should start streaming the RaspiCam's image to the HDMI Display.

The VidorGraphics library comes with pre-built FPGA bitstream image, capable of handling MIPI CSI-2 Camera inputs, HDMI Display outputs and some miscellaneous functionality.

The library also offers a Vidor_GFX graphics API, similar to the one implemented in the Adafruit_GFX library. We will use this later.

2. Re-Compiling the VidorGraphics FPGA Bitstream?

The VidorGraphics library comes with a pre-built FPGA image which contains the implementation the GFX API and some miscellanies functionality.

My initial idea was to try to rebuild this FPGA image, and then try to implement some image processing on the Camera video stream.

The FPGA project for VidorBitstream library, along with some other ones, are available at:

https://github.com/vidor-libraries/VidorBitstream

MKR Vidor 4000 FPGA Development

Although Arduino made the FPGA project available, FPGA development on the MKR Vidor 4000 does not seems to be officially supported.

To compile the projects either the Intel Quartus II 18.1 Lite (free) or Standard (paid) versions are needed. The official image is build with the Standard (paid) version, and it is supposed to have significantly better performance over a version built with the Lite (free) version.

I attempted to build open the project with Quartus II 18.1 Lite, but unfortunately it fails because apparently some custom components are missing.

There are some patches provided to fix this, but it does not seems to work.

At this point I decided to give this up custom FPGA development on the MKR VIdor 4000, at least for now.

Anyways, I was still interested in what the Cyclone 10 FPGA is used for in the MKR Vidor 4000.

From a brief on look on the FPGA project and a the VidorGraphics library implementation, I think the FPGA project may be structured something like this:

• MIPI CSI RX subsystem is captured the camera image
• a HDMI TX subsystem drives the MicroHDMI output
• the GFX API seems to have its own Buffer (probably in the SDRAM)
• some Video Mixer component may be used to combine the Camera and GFX images

• there is a NIOS II Soft-Core Processor, which handles commands received from SAMD21 over UART
• the FPGA is programmed by the SAMD21 via JTAG
• there are some GPIO, UART, SPI modules for GPIO and communication with the WiFi module

Disclaimer:

the above diagram is more like an educated guess on what is implemented in the VidorGraphic's FPGA image. I did not fully checked what is actually in the project.

As I can't easily do FPGA development on the MKR Vidor 4000, the device will be used basically as a MIPI CSI to HDMI converter. Along with the RaspiCam V1.3 it forms a fancy HDMI Camera.

But, I came up with a new plan . The actual hardware accelerated video processing will be done on PYNQ Z2 using Pynq!

3. Customizing the HDMI Output

At this point the MKR Vidor 4000 was still was just a MIPI CSI to HDMI converter. So, I thought we need some features!

I decided on adding a Project14 "logo" and a Timestamp as overlays to the HDMI Output.

To draw on the HDMI output we can use the VidorGraphics library. Fortunately, a separate image buffer is used for VidorGraphics. The means we can draw or logos and timestamps as slow as we want, and the camera image still remains responsive.

Adding the "Logo" was easy. I just printed the "Project14" text with some hack to make the text bold. I was too lazy covert an actual Project 14 logo into a bitmap and try to actually display that on the output.

Adding the "Timestamp" caused some headaches, as I wanted to get the current time from the Internet and for this I needed to get the Wifi Module to work.

The Wifi Module can be used with the WiFiNINA library from Arduino, but with the latest versions the MKR Vidor 1000 is not officially supported. So, to get it work took some browsing on the Arduino forum, building a firmware and flashing it on the Wifi module.

Next, I used a Rest API form WorldClockApi.com to get the Current Time (UTC). The time than is displayed on the screen.

This is what I ended up with:

Note:

The source code is attached to this post. It is pretty hackish. Do not take it too seriously!

4. Fixing Camera Settings

The VidorGraphics library has support for the OV5647 Camera Sensor. This is the same sensor as used in the RaspiCam v1.3 camera modules.

The OV5647 has a maximum resolution of 5 MegaPixels. With the MKR Vidor 4000 works with the 640x480 resolution.

Now, in the VidorGraphics library the OV5647 is configured with somewhat non-optimal settings.

Namely, the just the top-left corner of the sensor is used, which especially on a wide angle camera leads to a weird view-port.

 
(Red is what is used, Blue is the optimal for 640x480 resolution, Green is what the camera is capable of)

We can improve this by using and area in the middle of the sensor. This can be done by setting the X and Y offset registers. On the top of that we can use 2x2 sub-sampling (this was already enabled), which leads to an area of 1280x960 pixels used in the middle of the sensor.

Here is what the Original and Fixed images look like:

To achieve this in the VidorCamera.h file complete cam_640x480 list with the following values:

  { 0x5003, 0x09 },
  { 0x5025, 0x11 },
  { 0x504B, 0x30 },
  { 0x3814, 0x31 },
  { 0x3815, 0x31 },
    
  { 0x3800, 0x00 },
  { 0x3801, 0x00 },
  { 0x3802, 0x00 },
  { 0x3803, 0x20 },
  { 0x3804, 0x0A },
  { 0x3805, 0x1F },
  { 0x3806, 0x07 },
  { 0x3807, 0x8F },


  { 0x3811, 0x08 },
  { 0x3813, 0x02 },


  { 0x5006, 0xF0 },
  { 0x5007, 0xC0 },
  { 0x5018, 0x31 },
  { 0x5019, 0x31 },
  { 0x5046, 0x09 },


  { 0x3800, 0x02 },
  { 0x3801, 0x98 },
  { 0x3802, 0x01 },
  { 0x3803, 0xF0 },
  
  { 0x380c, 0x07 },
  { 0x380d, 0xCC },

Note:

It would be even better if we could run the camera in a higher resolution, but this needs a new FPGA image and we may not have enough memory for this.

In the next blog post I will experiment with hardware accelerated video processing on a PYNQ Z2 board.

Hope You enjoyed this post!