The following post gives an overview of the open-source verification tools used as part of the Summer of FPGAs -- OrangeCrab Dev Bd - Review I did for element14 (link to road-test).
Tools
CoCoTB (verification)
CoCoTB is a set of Python modules that in conjunction with an HDL (Hardware Description Language) verification tool can be used to verify and test hardware design using Python.
cocotb is a COroutine based COsimulation TestBench environment for verifying VHDL and SystemVerilog RTL using Python.
The main characteristics to choose CoCoTB over other options are:
- Python
- you can code your tests in Python which I found is faster and easier than UVM (Universal Verification Methodology), SystemVerilog (SV), VHDL or similar verification languages (e, SystemC, ...)
- you can leverage thousands of available Python libraries, code, and examples - system models, you name it.
- Simulator independent
- It is not tied to a particular simulator, and as it is open source, so you can add another simulator if you ever have to.
CoCoTB has several optional modules. Among them, I use the cocotb-test. cocotb-test adds pytest capabilities and removes the need for Makefile(s). Enabling pytest added an extra feature of running simulations (tests) in parallel (pytest-xdist or pytest-parallel).
As a final note, I want to clarify that everything has its place. I think SV or VHDL are OK for some basic unit testing, and UVM is a complex set of SV modules that should be used to test complex systems. But for most of the part, a CoCoTB test suite should suffice. Unless there is a particular need, I personally would stay with CoCoTB.
Verilator (simulator)
Verilator is the fastest Verilog/SystemVerilog simulator, and is open-source and free. It is widely used and has a growing community.
Verilator converts a SV design into a C++ (or SystemC) model which can be wrapped with a C/C++ testbench. But in this case, the wrapper is done in Python with CoCoTB handling the TB <-> RTL interface.
GTKwave (waveform viewer)
GTKwave is the de facto HDL simulation waveform viewer in the open-source community. For good or for bad, I do not know any decent alternatives. If you can recommend a better one, just left a comment.
Installation
CoCoTB installation
sudo apt-get install make gcc g++ python3 python3-dev python3-pip pip install cocotb pip install cocotb-test
Verilator installationI install the tools usually under ~/tools/ directory
cd ~ mkdir tools cd tools
Prerequisites
# Prerequisites: sudo apt-get install git perl python3 make autoconf g++ flex bison ccache sudo apt-get install libgoogle-perftools-dev numactl perl-doc sudo apt-get install libfl2 # Ubuntu only (ignore if gives error) sudo apt-get install libfl-dev # Ubuntu only (ignore if gives error) sudo apt-get install zlibc zlib1g zlib1g-dev # Ubuntu only (ignore if gives error) git clone https://github.com/verilator/verilator
Patching Verilator for CoCoTB
To use the latest Verilator version in CoCoTB, the following patch is required: VPI: simulation time callbacks should be one-time #2778 at the moment
# vim instructions (any other editor should suffice)
vim verilator/include/verilated_vpi.cpp
:603 # Goto line 603
VerilatedVpiCbHolder& ho = *it;
VL_DEBUG_IF_PLI(VL_DBG_MSGF("- vpi: reason_callback reason=%d id=%" VL_PRI64 "d\n",
reason, ho.id()););
>>>>>
add next line =>
ho.invalidate();
<<<<<
(ho.cb_rtnp())(ho.cb_datap());
called = true;
if (was_last) break;From the link above:
diff --git a/include/verilated_vpi.cpp b/include/verilated_vpi.cpp
index c354d89f..64099b98 100644
--- a/include/verilated_vpi.cpp
+++ b/include/verilated_vpi.cpp
@@ -603,6 +603,7 @@ public:
VerilatedVpiCbHolder& ho = *it;
VL_DEBUG_IF_PLI(VL_DBG_MSGF("- vpi: reason_callback reason=%d id=%" VL_PRI64 "d\n",
reason, ho.id()););
+ ho.invalidate();
(ho.cb_rtnp())(ho.cb_datap());
called = true;
if (was_last) break;Building Verilator
unsetenv VERILATOR_ROOT # For csh; ignore error if on bash
unset VERILATOR_ROOT # For bash
cd ~/tools/verilator
git pull # Make sure git repository is up-to-date
git tag # See what versions exist
#git checkout master # Use development branch (e.g. recent bug fixes)
#git checkout stable # Use most recent stable release
#git checkout v{version} # Switch to specified release version
autoconf # Create ./configure script
./configure # Configure and create Makefile
make -j # Build Verilator itself
sudo make installGTKwave installation
sudo apt install gtkwave
Simple example
SV code
module dummy_multiplier
#(
parameter WL = 32,
parameter DONE_PIPE_LVL = 4
)
(
input wire clk,
input wire reset,
// Inputs
input wire start,
input wire [WL-1:0] multiplier,
input wire [WL-1:0] multiplicand,
// Outputs
output wire done,
output wire [2*WL-1:0] product
);
logic [DONE_PIPE_LVL-1:0] done_dummy_pipe;
logic [2*WL-1:0] product_dummy_pipe[DONE_PIPE_LVL];
always_ff @(posedge clk) begin : tx_block
if(reset) begin
done_dummy_pipe <= '0;
end else begin
product_dummy_pipe[0] <= multiplier * multiplicand;
for(int i=1; i<DONE_PIPE_LVL; ++i)
product_dummy_pipe[i] <= product_dummy_pipe[i-1];
done_dummy_pipe <= {done_dummy_pipe[DONE_PIPE_LVL-2:0], start};
end
end
assign done = done_dummy_pipe[DONE_PIPE_LVL-1];
assign product = product_dummy_pipe[DONE_PIPE_LVL-1];
endmoduleCoCoTB wrapper
import os
import logging
from pathlib import Path
import cocotb
from cocotb.clock import Clock
from cocotb.triggers import FallingEdge, RisingEdge, Timer
from cocotb_test import simulator
# -----------------------------------------------------------------------------
# CoCoTB Module
@cocotb.test()
async def dummy_multiplier_basic_test(dut):
"""
Basic multiplier test
"""
# Set logger
log = logging.getLogger("TB")
log.setLevel('INFO')
# Start test
log.info('Multiplier basic test')
# Setup TB
clock = Clock(dut.clk, 10, units='ns')
cocotb.fork(clock.start())
# Reset system
await FallingEdge(dut.clk)
log.info('Set reset')
dut.reset <= 1
await RisingEdge(dut.clk)
log.info('Release reset')
dut.reset <= 0
# Set inputs
log.info("Doing 5x7 = 35")
dut.multiplier <= 5
dut.multiplicand <= 7
dut.start <= 1
await RisingEdge(dut.clk)
dut.start <= 0
# Wait result
max_cnt = 10
while dut.done == 0 and max_cnt:
log.info("Waiting DUT ready")
max_cnt -= 1
await RisingEdge(dut.clk)
if(max_cnt != 0):
log.info(f'Data captured: {dut.product}')
assert dut.product == 35, f'5x7 -> 35, but got {dut.product.value}'
else:
log.error(f'max_cnt reached without done set')
assert False
# Sim done
await Timer(100, units='ns')
log.info('Test done!')
# -----------------------------------------------------------------------------
# Invoke test
if __name__ == '__main__':
# cocotb-test default is icarus, switching to verilator
sim = os.environ.get('SIM', 'verilator')
os.environ['SIM'] = sim
# ---------------------------------------
_workpath = str(Path(__file__).resolve().parent)
# ---------------------------------------
sim_build = str(Path(_workpath) / "sim_build" / 'dummy_multiplier_basic_test')
# ---------------------------------------
simulator.run(
toplevel='dummy_multiplier',
module=Path(__file__).stem,
verilog_sources=['../rtl/dummy_multiplier.sv'],
sim_build=sim_build,
)
Running the test
WAVES=1 python multiplier_simple_tb.pyConsole output
INFO cocotb: Running command: perl /usr/local/bin/verilator -cc --exe -Mdir /home/dramoz/dev/sv_blocks/tb/sim_build/dummy_multiplier_basic_test -DCOCOTB_SIM=1 --top-module dummy_multiplier --vpi --public-flat-rw --prefix Vtop -o dummy_multiplier -LDFLAGS -Wl,-rpath,/home/dramoz/.virtualenvs/cocotb/lib/python3.8/site-packages/cocotb/libs -L/home/dramoz/.virtualenvs/cocotb/lib/python3.8/site-packages/cocotb/libs -lcocotbvpi_verilator --trace-fst --trace-structs /home/dramoz/.virtualenvs/cocotb/lib/python3.8/site-packages/cocotb/share/lib/verilator/verilator.cpp /home/dramoz/dev/sv_blocks/rtl/dummy_multiplier.sv
INFO cocotb: Running command: make -C /home/dramoz/dev/sv_blocks/tb/sim_build/dummy_multiplier_basic_test -f Vtop.mk
INFO cocotb: make: Entering directory '/home/dramoz/dev/sv_blocks/tb/sim_build/dummy_multiplier_basic_test'
INFO cocotb: ccache g++ -std=c++11 -I. -MMD -I/usr/local/share/verilator/include -I/usr/local/share/verilator/include/vltstd -DVM_COVERAGE=0 -DVM_SC=0 -DVM_TRACE=1 -DVM_TRACE_FST=1 -faligned-new -fcf-protection=none -Wno-bool-operation -Wno-sign-compare -Wno-uninitialized -Wno-unused-but-set-variable -Wno-unused-parameter -Wno-unused-variable -Wno-shadow -std=c++14 -Os -c -o verilator.o /home/dramoz/.virtualenvs/cocotb/lib/python3.8/site-packages/cocotb/share/lib/verilator/verilator.cpp
INFO cocotb: ccache g++ -std=c++11 -I. -MMD -I/usr/local/share/verilator/include -I/usr/local/share/verilator/include/vltstd -DVM_COVERAGE=0 -DVM_SC=0 -DVM_TRACE=1 -DVM_TRACE_FST=1 -faligned-new -fcf-protection=none -Wno-bool-operation -Wno-sign-compare -Wno-uninitialized -Wno-unused-but-set-variable -Wno-unused-parameter -Wno-unused-variable -Wno-shadow -std=c++14 -Os -c -o verilated.o /usr/local/share/verilator/include/verilated.cpp
INFO cocotb: ccache g++ -std=c++11 -I. -MMD -I/usr/local/share/verilator/include -I/usr/local/share/verilator/include/vltstd -DVM_COVERAGE=0 -DVM_SC=0 -DVM_TRACE=1 -DVM_TRACE_FST=1 -faligned-new -fcf-protection=none -Wno-bool-operation -Wno-sign-compare -Wno-uninitialized -Wno-unused-but-set-variable -Wno-unused-parameter -Wno-unused-variable -Wno-shadow -std=c++14 -Os -c -o verilated_dpi.o /usr/local/share/verilator/include/verilated_dpi.cpp
INFO cocotb: ccache g++ -std=c++11 -I. -MMD -I/usr/local/share/verilator/include -I/usr/local/share/verilator/include/vltstd -DVM_COVERAGE=0 -DVM_SC=0 -DVM_TRACE=1 -DVM_TRACE_FST=1 -faligned-new -fcf-protection=none -Wno-bool-operation -Wno-sign-compare -Wno-uninitialized -Wno-unused-but-set-variable -Wno-unused-parameter -Wno-unused-variable -Wno-shadow -std=c++14 -Os -c -o verilated_vpi.o /usr/local/share/verilator/include/verilated_vpi.cpp
INFO cocotb: ccache g++ -std=c++11 -I. -MMD -I/usr/local/share/verilator/include -I/usr/local/share/verilator/include/vltstd -DVM_COVERAGE=0 -DVM_SC=0 -DVM_TRACE=1 -DVM_TRACE_FST=1 -faligned-new -fcf-protection=none -Wno-bool-operation -Wno-sign-compare -Wno-uninitialized -Wno-unused-but-set-variable -Wno-unused-parameter -Wno-unused-variable -Wno-shadow -std=c++14 -Os -c -o verilated_fst_c.o /usr/local/share/verilator/include/verilated_fst_c.cpp
INFO cocotb: /usr/bin/perl /usr/local/share/verilator/bin/verilator_includer -DVL_INCLUDE_OPT=include Vtop.cpp Vtop___024root__DepSet_heccd7ead__0.cpp Vtop__Dpi.cpp Vtop__Trace__0.cpp Vtop___024root__Slow.cpp Vtop___024root__DepSet_heccd7ead__0__Slow.cpp Vtop__Syms.cpp Vtop__Trace__0__Slow.cpp > Vtop__ALL.cpp
INFO cocotb: ccache g++ -std=c++11 -I. -MMD -I/usr/local/share/verilator/include -I/usr/local/share/verilator/include/vltstd -DVM_COVERAGE=0 -DVM_SC=0 -DVM_TRACE=1 -DVM_TRACE_FST=1 -faligned-new -fcf-protection=none -Wno-bool-operation -Wno-sign-compare -Wno-uninitialized -Wno-unused-but-set-variable -Wno-unused-parameter -Wno-unused-variable -Wno-shadow -std=c++14 -Os -c -o Vtop__ALL.o Vtop__ALL.cpp
INFO cocotb: echo "" > Vtop__ALL.verilator_deplist.tmp
INFO cocotb: Archive ar -rcs Vtop__ALL.a Vtop__ALL.o
INFO cocotb: g++ verilator.o verilated.o verilated_dpi.o verilated_vpi.o verilated_fst_c.o Vtop__ALL.a -Wl,-rpath,/home/dramoz/.virtualenvs/cocotb/lib/python3.8/site-packages/cocotb/libs -L/home/dramoz/.virtualenvs/cocotb/lib/python3.8/site-packages/cocotb/libs -lcocotbvpi_verilator -lz -o dummy_multiplier
INFO cocotb: rm Vtop__ALL.verilator_deplist.tmp
INFO cocotb: make: Leaving directory '/home/dramoz/dev/sv_blocks/tb/sim_build/dummy_multiplier_basic_test'
INFO cocotb: Running command: /home/dramoz/dev/sv_blocks/tb/sim_build/dummy_multiplier_basic_test/dummy_multiplier
INFO cocotb: -.--ns INFO cocotb.gpi ..mbed/gpi_embed.cpp:100 in set_program_name_in_venv Using Python virtual environment interpreter at /home/dramoz/.virtualenvs/cocotb/bin/python
INFO cocotb: -.--ns INFO cocotb.gpi ../gpi/GpiCommon.cpp:105 in gpi_print_registered_impl VPI registered
INFO cocotb: -.--ns INFO cocotb.gpi ..mbed/gpi_embed.cpp:240 in _embed_sim_init Python interpreter initialized and cocotb loaded!
INFO cocotb: 0.00ns INFO cocotb __init__.py:220 in _initialise_testbench_ Running on Verilator version 4.212 2021-09-01
INFO cocotb: 0.00ns INFO cocotb __init__.py:226 in _initialise_testbench_ Running tests with cocotb v1.5.2 from /home/dramoz/.virtualenvs/cocotb/lib/python3.8/site-packages/cocotb
INFO cocotb: 0.00ns INFO cocotb __init__.py:247 in _initialise_testbench_ Seeding Python random module with 1632919199
INFO cocotb: 0.00ns INFO cocotb.regression regression.py:127 in __init__ Found test multiplier_simple_tb.dummy_multiplier_basic_test
INFO cocotb: 0.00ns INFO cocotb.regression regression.py:468 in _start_test Running test 1/1: dummy_multiplier_basic_test
INFO cocotb: 0.00ns INFO ..tiplier_basic_test.0x7fe425b2b310 decorators.py:312 in _advance Starting test: "dummy_multiplier_basic_test"
INFO cocotb: Description:
INFO cocotb: Basic multiplier test
INFO cocotb:
INFO cocotb: 0.00ns INFO TB ..plier_simple_tb.py:39 in dummy_multiplier_basic_test Multiplier basic test
INFO cocotb: 5.00ns INFO TB ..plier_simple_tb.py:47 in dummy_multiplier_basic_test Set reset
INFO cocotb: 10.00ns INFO TB ..plier_simple_tb.py:50 in dummy_multiplier_basic_test Release reset
INFO cocotb: 10.00ns INFO TB ..plier_simple_tb.py:54 in dummy_multiplier_basic_test Doing 5x7 = 35
INFO cocotb: 20.00ns INFO TB ..plier_simple_tb.py:64 in dummy_multiplier_basic_test Waiting DUT ready
INFO cocotb: 30.00ns INFO TB ..plier_simple_tb.py:64 in dummy_multiplier_basic_test Waiting DUT ready
INFO cocotb: 40.00ns INFO TB ..plier_simple_tb.py:64 in dummy_multiplier_basic_test Waiting DUT ready
INFO cocotb: 50.00ns INFO TB ..plier_simple_tb.py:64 in dummy_multiplier_basic_test Waiting DUT ready
INFO cocotb: /home/dramoz/dev/sv_blocks/tb/multiplier_simple_tb.py:69: FutureWarning: `str(ModifiableObject)` is deprecated, and in future will return `ModifiableObject._path`. To get a string representation of the value, use `str(ModifiableObject.value)`.
INFO cocotb: log.info(f'Data captured: {dut.product}')
INFO cocotb: 60.00ns INFO TB ..plier_simple_tb.py:69 in dummy_multiplier_basic_test Data captured: 0000000000000000000000000000000000000000000000000000000000100011
INFO cocotb: 160.00ns INFO TB ..plier_simple_tb.py:77 in dummy_multiplier_basic_test Test done!
INFO cocotb: 160.00ns INFO cocotb.regression regression.py:364 in _score_test Test Passed: dummy_multiplier_basic_test
INFO cocotb: 160.00ns INFO cocotb.regression regression.py:487 in _log_test_summary Passed 1 tests (0 skipped)
INFO cocotb: 160.00ns INFO cocotb.regression regression.py:557 in _log_test_summary **********************************************************************************************************
INFO cocotb: ** TEST PASS/FAIL SIM TIME(NS) REAL TIME(S) RATIO(NS/S) **
INFO cocotb: **********************************************************************************************************
INFO cocotb: ** multiplier_simple_tb.dummy_multiplier_basic_test PASS 160.00 0.01 11913.81 **
INFO cocotb: **********************************************************************************************************
INFO cocotb:
INFO cocotb: 160.00ns INFO cocotb.regression regression.py:574 in _log_sim_summary *************************************************************************************
INFO cocotb: ** ERRORS : 0 **
INFO cocotb: *************************************************************************************
INFO cocotb: ** SIM TIME : 160.00 NS **
INFO cocotb: ** REAL TIME : 0.05 S **
INFO cocotb: ** SIM / REAL TIME : 3549.59 NS/S **
INFO cocotb: *************************************************************************************
INFO cocotb:
INFO cocotb: 160.00ns INFO cocotb.regression regression.py:259 in tear_down Shutting down...
INFO cocotb: - :0: Verilog $finish
INFO cocotb: - /home/dramoz/.virtualenvs/cocotb/lib/python3.8/site-packages/cocotb/share/lib/verilator/verilator.cpp:118: Verilog $finish
INFO cocotb: - /home/dramoz/.virtualenvs/cocotb/lib/python3.8/site-packages/cocotb/share/lib/verilator/verilator.cpp:118: Second verilog $finish, exiting
Results file: /home/dramoz/dev/sv_blocks/tb/sim_build/dummy_multiplier_basic_test/_r77l7fe_results.xml
View wave
gtkwave sim_build/dummy_multiplier_basic_test/dump.fst
