Integrate a Third-Party Accelerator

Latest update: 2026-08-21

ESP offers several design flows for designing accelerators, see for example the tutorials on how to design an accelerator in SystemC, in C/C++ and in Keras/PyTorch/ONNX. In many cases however, one may want to reuse an existing accelerator (i.e. a third-party accelerator), rather than designing a new one with ESP.

This tutorial explains the current third-party socket, how to add an accelerator to ESP, and how to design and test an SoC that contains it. ESP includes two substantial examples: the fixed-function NVIDIA Deep Learning Accelerator (NVDLA) and the programmable RISC-V Vortex GPGPU.

The Vortex integration and Intel/Altera DE10-Pro SX flow document unreleased development work. Coming soon

NVDLA

NVDLA is a deep learning accelerator from NVIDIA. It’s open source (nvdla.org), fixed function and highly configurable. NVDLA is composed of multiple engines needed to perform deep learning inference (e.g. convolution, activation, pooling, etc…). NVDLA has a configuration interface (APB) through which a host processor writes to a large set of memory-mapped registers. Once configured, NVDLA exchanges data with the memory hierarchy through a memory interface (AXI4). As soon as it completes the task, NVDLA notifies back the host processor by raising an interrupt signal. This is a very common invocation model for loosely-coupled fixed-function accelerators and it’s pretty much the same as for the ESP accelerators.

Although NVDLA is highly configurable, the NVDLA Compiler supports only a few configurations, named NVDLA full, NVDLA large and NVDLA small. To test and evaluate the integration of NVDLA in ESP, we use the NVDLA small, which has an 8-bit integer precision, 64 multiply-and-accumulate units, 128 KB of local memory and a 64-bit AXI4 interface.

NVDLA accelerator schematic

The NVDLA Compiler takes as inputs the network topology in prototxt format, a trained Caffe model, a calibration table needed for adjusting the network model (trained in full precision) to work with reduced precision, such as 8-bit integer precision in the case of NVDLA small. The NVDLA Compiler produces an NVDLA Loadable containing the layer-by-layer information to configure NVDLA. The NVDLA runtime leverages the user-mode driver to load the inputs and the NVDLA Loadable and to submit inference jobs to the kernel-mode driver, that is a Linux device driver.

NVDLA software stack

There are many pairs of Caffe trained model and prototxt network topology available online (Caffe Model Zoo by Berkeley), however, we found that the NVDLA compiler can parse successfully only a small number of them out of the box. Working with the 8-bit integer precision of NVDLA small requires the additional step of generating the calibration table, the instructions for that are available in the NVDLA sw GitHub repository. With TensorRT it’s possible to generate the calibration scales to compensate for the reduced precision.

For the purpose of this tutorial we will use a LeNet network trained for the MNIST dataset. We have experimented with a few other networks, including ResNet50, as described in our CARRV’20 paper. Here you can find the inputs for the NVDLA Compiler for the LeNet network:

A prebuilt version of the NVDLA compiler is available in the NVDLA sw repository at the path prebuilt/x86-ubuntu/nvdla_compiler. Move to the x86-ubuntu folder, place there the prototxt file, the Caffe model and the calibration table and then run the following:

./nvdla_compiler \
  --prototxt lenet_mnist.prototxt \
  --caffemodel lenet_mnist.caffemodel \
  --profile fast-math \
  --cprecision int8 \
  --configtarget nv_small \
  --calibtable lenet_mnist.json \
  --quantizationMode per-filter \
  --informat nchw \
  -o .


The output file is a NVDLA Loadable called fast-math.nvdla, which we rename lenet_mnist.nvdla. This is the loadable that we will use for the experiments in this tutorial, together with an input image from the MNIST dataset:

All the steps down to the generation of the NVDLA Loadable are done offline. Instead, the user application, the user-mode driver (UMD) and the kernel-mode driver (KMD) are executed at runtime, and therefore in this tutorial they will run on FPGA on the Ariane/CVA6 core in an ESP SoC.

Back to top

Vortex

Coming soon

Vortex is an open-source, programmable RISC-V GPGPU. ESP integrates the Vortex v2.2 hardware and software stack as GT_VORTEX. Unlike NVDLA, Vortex runs user-supplied kernels and exposes the Vortex runtime API to applications on ESP Linux.

The implementation is available on ESP’s official public vortex-2.2-integration development branch. It is newer than the 2026.1.0 release and remains marked Coming soon on this site until it is released.

The integration is intentionally split between two ESP accelerators:

  • accelerators/third-party/GT_VORTEX contains the ESP Vortex fork, the third-party wrapper, RTL source manifests, and the ESP runtime backend.
  • accelerators/rtl/gt_vortex_rtl provides the ESP Linux kernel driver and the application hooks used to build and stage Vortex software. This companion is a software-integration shim, not a second Vortex hardware tile.

The libvortex-esp.so backend talks to the gt_vortex_rtl device and accesses the reserved Vortex memory window through /dev/mem; the normal libvortex.so stub selects it with VORTEX_DRIVER=esp. This is a privileged, development oriented interface and should only be exposed to trusted software.

The ESP configuration GUI lets you choose the number of Vortex cores, warps and threads and enable the L2 and L3 caches. ESP derives the AXI transaction-ID width from that configuration and only offers combinations that fit the socket’s 10-bit ID limit. In this single-cluster integration, the optional L3 cache remains useful as the last-level cache between Vortex and its external AXI interface.

Hardware, runtime libraries and .vxbin kernels must use the same core, warp, thread, L2 and L3 configuration. make gt_vortex_rtl-app forwards the selected SoC settings automatically. A normal ESP Linux build stages the available runtime, host applications and kernels:

cd <esp>/socs/<soc-name>
make linux

On the resulting Ariane Linux system, list or run a staged regression with:

vortex-regression --list
vortex-regression basic
vortex-regression printf -n1

The checked-in bare-metal Vortex tests use prebuilt single-core byte-array images. The Linux regression flow is the preferred path for configurations with multiple Vortex cores because it builds configuration-matched .vxbin kernels.

DE10-Pro SX status: Vortex is available through the Intel/Altera flow, but sustained Vortex workloads launched from HPS Linux can currently trigger an HPS kernel panic. Treat this path as experimental; the AMD/Xilinx flow is the currently validated stable path for sustained Vortex workloads.

Back to top

Integrate a third-party accelerator

ESP hosts third-party accelerators through a 64-bit AXI4 master memory interface, a 32-bit APB slave configuration interface, and an interrupt. The socket maps memory traffic onto ESP’s coherent or non-coherent DMA paths. See Current socket interface and limits for the supported AXI behavior and remaining constraints.

Here are the step by step instructions to integrate a third-party accelerator in ESP. Use NV_NVDLA for a fixed-function example and GT_VORTEX for a programmable accelerator with a more involved runtime stack.

  • Accelerator folder. Create a folder with the name of the accelerator at the path <esp-root>/accelerators/third-party/<accelerator-name>. Then move into this new folder. The folder for NVDLA is <esp-root>/accelerators/third-party/NV_NVDLA; Vortex uses GT_VORTEX.

  • HW source files. Place the source code of the hardware implementation of the accelerator in a folder named ip. In the case of NVDLA, ip is a Git submodule pointing to our fork of the NVDLA hw repository.

  • List of HW source files. Create the files <accelerator-name>.vhdl, <accelerator-name>.pkgs (for VHDL packages), <accelerator-name>.verilog, <accelerator-name>.sverilog and populate them with the lists of all the RTL files in the ip folder that need to be compiled. In the case of NVDLA all the sources are in Verilog, so the VHDL source files, VHDL packages and SystemVerilog files are left empty. For SystemVerilog designs that require an include directory, follow the GT_VORTEX/vlog_incdir example.

  • SW source files. Place the source code of the software applications and drivers for invoking the accelerator in a folder named sw. In the case of NVDLA, sw is a Git submodule pointing to our fork of the NVDLA sw repository.

  • List of compiled SW files. Create the files <accelerator-name>.umd and <accelerator-name>.kmd and populate them with the lists of the device drivers, executable files and libraries generated after compiling the software in the sw folder. The kmd file is for the kernel-mode drivers, while the umd file is for user-space applications and libraries. The optional <accelerator-name>.bc manifest lists bare-metal artifacts copied to the selected processor’s bare-metal build directory.

    A .hosts file may document intended processors, but the current SoC GUI does not use it to filter accelerators. It filters by DMA width only. Verify software compatibility yourself when selecting a CPU; the current NVDLA and Vortex Linux stacks are intended for Ariane.

  • Supported DMA widths. Create the file <accelerator-name>.dma_widths and list the supported DMA NoC widths, one per line. With the current third-party accelerator socket this file should contain 64. ESP uses this file to decide whether the accelerator can be selected for the SoC’s configured DMA NoC width.

  • Accelerator wrapper. Design a wrapper for the accelerator compliant to the interface of ESP, using the NV_NVDLA_wrapper.v as an example. The wrapper doesn’t implement any logic, its purpose is to adapt the accelerator interface to the signal naming convention that ESP expects. The GT_VORTEX_wrapper.v example also shows a parameterized AXI ID width.

  • Accelerator description. Describe the accelerator in an XML file. Follow the XML for NVDLA as an example (NV_NVDLA.xml). These parameters match the content of the accelerator wrapper (NV_NVDLA_wrapper.v). The <accelerator-name>.dma_widths file described above must sit next to this XML file in the third-party accelerator folder.

    • name: should match the name of the accelerator folder
    • desc: arbitrary and concise description of the accelerator
    • device_id: unique hexadecimal device ID in the range 0x040 - 0x3FF. Make sure not to use the same ID for any other accelerator
    • axi_prefix: prefix added to each AXI port in the accelerator wrapper
    • interrupt: name of the interrupt output port
    • addr_width: bit-width of the AXI address fields
    • id_width: bit-width of the AXI transaction ID fields
    • user_width: bit-width of the AXI transaction USER fields
    • clock: one entry for each clock port that will be connected to the ESP accelerator tile clock
    • reset: one entry, including name and polarity, for each reset port that will be activated when the reset of the ESP accelerator tile is active

    id_width must not exceed ESP’s current 10-bit AXI ID width. Socket generation overrides this value for GT_VORTEX so that it matches the Vortex configuration selected in the SoC GUI.

Once again, as an example, you can see the relationship between the entries in NV_NVDLA.xml and the ports in NV_NVDLA_wrapper.v.

  • Makefile. Populate a simple Makefile with two required targets: sw and hw. hw is used for RTL code generation, when that applies, like in the case of NVDLA. sw should cross-compile both user-space applications and libraries (umd) and kernel-space drivers (kmd). The Makefile variables for the cross compilation are set by the main ESP Makefile and don’t need to be set in the third-party accelerator Makefile, these variables are KSRC, ARCH and CROSS_COMPILE. We recommend to follow the NVDLA example.

  • Vendor. You also have the opportunity to specify the vendor of the accelerator by creating a file called vendor containing only the name of the vendor.

  • Additional description. The only file to be edited outside of the newly created third-party accelerator folder is <esp-root>/tools/socgen/thirdparty.py. This file specifies the device name to be used in the compatible field during the generation of the device tree and whether the interrupt of the third-party accelerator is edge-sensitive (0) or level-sensitive (1). The compatible field is a string generated as <VENDOR>,<DEVICE ID>. This string must match the corresponding compatible field in the open firmware data structure struct of_device_id of the device driver (i.e. the kernel module).

After these steps, the accelerator is fully integrated and ESP sees it the same way it sees any other accelerator.

Back to top

Current socket interface and limits

The current third-party socket supports the AXI behavior needed by NVDLA and Vortex, including multi-beat incrementing memory transfers, variable AXI transfer sizes, independently arriving write-address and write-data channels, and byte-lane write strobes.

Partial WSTRB memory writes are preserved across ESP’s memory paths:

  • Non-coherent writes are split into naturally aligned byte, halfword, word or doubleword NoC transactions.
  • Coherent writes use read-modify-write so bytes disabled by WSTRB are preserved.
  • A write beat with all strobes clear is discarded locally.

The remaining integration constraints are:

  • The AXI data interface and third-party DMA NoC width are currently 64 bits.
  • AXI transaction IDs are limited to 10 bits.
  • The memory path implements incrementing burst addressing; fixed and wrapping burst modes are not interpreted by the proxy.
  • Partial write strobes are supported for memory traffic, but not for writes routed to a remote AHB peripheral.
  • The socket does not contain the ESP accelerator TLB. Software and wrapper logic must follow the address model expected by the third-party IP.
  • The automated wrapper flow currently targets an AXI4 memory master, APB configuration slave and interrupt. Other native buses such as Avalon, TileLink and Wishbone require a custom adapter.

Back to top

Design and test an SoC with a third-party accelerator

Move into the SoC working folder for the FPGA board of your choice.

AMD/Xilinx example: VCU118 
cd <esp-root>/socs/xilinx-vcu118-xcvu9p
Intel/Altera example: Terasic DE10-Pro SX 
cd <esp-root>/socs/terasic-de10-pro-sx

Use Quartus Prime Pro 19.4 and SoC EDS 19.1 for this validated flow. The board’s HPS preparation and payload-loading steps are covered in the single-core SoC guide.

Back to top

SoC configuration

Configure the SoC with the ESP mix&match floor-planning GUI. You can launch the GUI with:

make esp-xconfig


In addition to the usual processor tiles, memory tiles and ESP accelerator tiles, you can instantiate third-party accelerator tiles. The tile menu lists NV_NVDLA, GT_VORTEX, and any other compatible third-party accelerator that has been integrated.

The GUI does not currently filter this list by processor software support. Choose a compatible processor explicitly. The NVDLA and Vortex Linux stacks in ESP are intended for Ariane.

The memory interface of a third-party accelerator must currently have a 64-bit word width. ESP filters the third-party accelerator list against the SoC’s DMA NoC width by checking accelerators/third-party/<accelerator>/<accelerator>.dma_widths.

When selecting GT_VORTEX, also configure its core, warp and thread counts and optional L2/L3 caches. The GUI rejects combinations whose derived AXI ID width would exceed 10 bits. Keep these values consistent with all Vortex runtime and kernel builds.

The following screenshot shows the original multi-NVDLA example (remember to save with Generate SoC Config):

ESP GUI showing a multi-NVDLA SoC


After closing the GUI, check the Ariane device tree (socgen/esp/riscv.dts) for the NVDLA or Vortex devices instantiated by the GUI.

Recall the “Debug link configuration” instructions from the “How to: design a single-core SoC” guide. Everything described there applies whenever designing an SoC with ESP.

Back to top

Third-party source and software compilation

NVDLA

Both the NVDLA runtime application and device driver provided by NVIDIA only work with a single instance of NVDLA. In our fork of the NVDLA sw repository, we modified the NVDLA UMD and KMD to support systems with multiple instances of NVDLA executing in parallel. The maximum number of instances is currently set to 4, but you can increase it by changing the following constants:

// In kmd/firmware/dla_engine_internal.h
#define MAX_N_NVDLA 4

// In core/src/runtime/include/priv/Runtime.h
size_t getMaxDLADevices() { return 4; }


Build boot-loader and Linux image.

make linux


Compile/build the NVDLA hardware and software sources.

make NV_NVDLA


This step copies the runtime executable (nvdla_runtime) and library (libnvdla_runtime.so) as well as the device driver (opendla.ko) into the root file system that will be deployed on FPGA, namely the soft-build/ariane/sysroot folder.

Place the files lenet_mnist.nvdla and seven.pgm in soft-build/ariane/sysroot/root/NV_NVDLA.

Every time something changes in the sysroot folder, the Linux image needs to be re-generated.

make linux


Back to top

Vortex

The Vortex Linux integration uses the companion gt_vortex_rtl accelerator to build and stage its runtime. From the configured SoC working folder, run:

make linux

The Linux build invokes make gt_vortex_rtl-app, which builds libvortex-esp.so, the libvortex.so stub and regression host applications. It also stages any available .vxbin kernels under /applications/test/vortex_kernels. If the Vortex kernel toolchain is not available, the build can still stage the host side, but missing kernels must be built on a supported host with exactly the same Vortex hardware configuration.

To restage the Vortex applications without rebuilding the full Linux tree, use:

make gt_vortex_rtl-app

Do not mix kernels or runtime artifacts built with different core, warp, thread, L2 or L3 settings. ESP forwards the current SoC settings automatically through this target.

Back to top

FPGA prototyping

Recall the “FPGA prototyping” instructions from the “How to: design a single-core SoC” guide. Everything described there applies whenever designing an SoC with ESP.

AMD/Xilinx flow 
make vivado-syn
make fpga-program
make fpga-run-linux
Intel/Altera DE10-Pro SX flow

Build the HPS boot artifacts with SoC EDS 19.1, then synthesize and program the FPGA with Quartus Prime Pro 19.4. ESP Linux is loaded through the HPS Linux host:

make hps
make quartus-syn
make fpga-program
make fpga-run-linux INTEL_HPS_HOST=<hps-host>

Complete the board’s one-time SD-card and HPS utility setup before using these commands. Sustained Vortex workloads on this path remain experimental because they can currently trigger an HPS kernel panic.


Observe the boot through a serial communication program (e.g. minicom). After boot, you can optionally connect to the ESP Linux instance over SSH.

Run NVDLA

Toward the end of the boot the device drivers of all the accelerators get registered. You will see 4 NVDLA devices registered like in the picture below:

[   33.479107] Probe NVDLA config nvidia,nv_small
[   33.526396] [drm] Initialized nvdla 0.0.0 20171017 for 60400000.nv_nvdla on minor 0
[   33.695024] 0 . 12 . 5
[   33.703127] reset engine done
[   33.733930] Probe NVDLA config nvidia,nv_small
[   33.775771] [drm] Initialized nvdla 0.0.0 20171017 for 60500000.nv_nvdla on minor 1
[   33.944645] 0 . 12 . 5
[   33.952729] reset engine done
[   33.977915] Probe NVDLA config nvidia,nv_small
[   34.019091] [drm] Initialized nvdla 0.0.0 20171017 for 60600000.nv_nvdla on minor 2
[   34.184605] 0 . 12 . 5
[   34.192690] reset engine done
[   34.217802] Probe NVDLA config nvidia,nv_small
[   34.257816] [drm] Initialized nvdla 0.0.0 20171017 for 60700000.nv_nvdla on minor 3
[   34.424548] 0 . 12 . 5
[   34.432611] reset engine done


After logging in with the development image’s default credentials (user root, password openesp), move to the accelerator folder. Change or disable these credentials before connecting the board to an untrusted network.

  cd <accelerator-name> # e.g. cd NV_NVDLA


At this point you can execute the runtime application for testing the accelerator. Of course each accelerator may have a completely different application, so this step is specific to each accelerator. In this case we will proceed by running the runtime application for NVDLA.

In the NV_NVDLA folder, together with the application executable (nvdla_runtime) and the runtime library, you will find the LeNet NVDLA Loadable (lenet_mnist.nvdla) and the input image seven.pgm.

You can execute inference on one specific instance of NVDLA thanks to the --instance argument that we added to the NVDLA UMD.

  # Inference on instance 0
  ./nvdla_runtime --loadable lenet_mnist.nvdla --image seven.pgm --rawdump --instance 0

  # Inference on instance 1
  ./nvdla_runtime --loadable lenet_mnist.nvdla --image seven.pgm --rawdump --instance 1

  # Inference on instance 2
  ./nvdla_runtime --loadable lenet_mnist.nvdla --image seven.pgm --rawdump --instance 2

  # Inference on instance 3
  ./nvdla_runtime --loadable lenet_mnist.nvdla --image seven.pgm --rawdump --instance 3


Here is an example of the terminal output for running the LeNet inference on one of the NVDLA instances: sample terminal output.

output.dimg contains the classification results: MNIST has 10 classes which correspond to the digits from 0 to 9. A correct classification for the seven.pgm image would mean that the number in position 8 of 10 is the highest. Run the following to see the classification results:

$ cat output.dimg
0 2 0 0 0 0 0 124 0 0


You can also run multiple inference jobs in parallel.

  # Instances 0-1-2-3 in parallel
  ./nvdla_runtime --loadable lenet_mnist.nvdla --image seven.pgm --instance 0 & \
  ./nvdla_runtime --loadable lenet_mnist.nvdla --image seven.pgm --instance 1 & \
  ./nvdla_runtime --loadable lenet_mnist.nvdla --image seven.pgm --instance 2 & \
  ./nvdla_runtime --loadable lenet_mnist.nvdla --image seven.pgm --instance 3 &


By default both the NVDLA UMD and KMD print a lot of information. If the KMD prints do not appear on screen, you can see them at the end of the execution by running dmesg.

To silence all the debug and info prints by the KMD, you have to go back to the NVDLA compilation step.

  make NV_NVDLA-clean # do not skip this step
  VERBOSE=0 make NV_NVDLA


The runtime application reports the execution time, but all the terminal prints make it inaccurate. To collect executing time data you may want to remove some of the UMD prints as well.

Run Vortex regressions

The Vortex helper selects the ESP runtime backend and locates the matching staged host executable and .vxbin kernel:

vortex-regression --list
vortex-regression basic

The equivalent explicit invocation is:

export VORTEX_DRIVER=esp
/applications/test/gt_vortex_rtl_vortex_<test>.exe \
  -k /applications/test/vortex_kernels/<test>.vxbin

If a test fails unexpectedly, first confirm that the host application, runtime libraries, kernel and instantiated Vortex hardware all use the same core, warp, thread, L2 and L3 configuration.

Back to top