Reproduction case · OpenHW on ALINX · Episode 08

Storage hierarchy: 32 KiB BRAM vs DDR3, CPU vs DMA bandwidth and latency

Episode 08 stops measuring one device and puts the two memory levels this CVE2 can actually reach on one table: a 32 KiB on-chip scratch BRAM and 256 MiB of board DDR3. A CPU word read of the scratch reaches only 25.0 MB/s; the same 8 KiB copy through a hardware DMA reaches 199.8 MB/s, about 9x; DDR3 through the DMA reaches 1,422 MB/s read and 733 MB/s write, with a single-beat latency of 241 ns. Two independent board runs end in EP08 RESULT: PASS.

Verified on board · 2026-11-21 CVE2 / RV32IM · 50 MHz 32 KiB scratch BRAM · 40 / 50 DDR3 256 MiB · native UI 100 MHz Episode 08 / 12

The conclusion first: the SRAM nearest the CPU is not "fast". With no cache and no burst, a CVE2 moving 32-bit data with ordinary loads and stores sees only 22–25 MB/s. Hand the same copy to a hardware DMA and it jumps to 199.8 MB/s. DDR3 at 1,422 MB/s is another order entirely, but only the DMA can reach it; the CPU has no DDR window. These are not three engines in a fair race but a vertical cut through the real memory hierarchy of this SoC.

BoardALINX PI-AR035-F1-1Engineering board, not yet in the board selector; link will follow
ChipAMD Artix-7 · XC7A35TSpecs and sibling boards → ProjectOpenHW CVE2 / CV32E20Upstream repo and commit →
CPU BRAM read
25.0 MB/s · 8 cycles/word
CPU 8 KiB copy
22.2 MB/s · 9 cycles/word
DMA 8 KiB copy
199.8 MB/s · 1 word/cycle
DDR3 read / write
1,422 / 733 MB/s
DDR3 single beat
241 ns · 23–36 ui cycles
BRAM dependent load
2.5 cycles · 50 ns
Timing
WNS 0.853 / WHS 0.036 ns
Resources
LUT 9,581 · BRAM 40 / 50

1. Board result: seven paths, 19 UART lines

One firmware pass runs everything: four BRAM measurements, then DDR3 write, read-compare, CRC and ping. The block below is the raw uart_ep08_run1.txt; bytes, cycles, beats and crc_data are read back from RTL counters, not recomputed in a script.

EP08 storage hierarchy on ALINX PI-AR035-F1-1 EP08 VERSION ddr=0x45503038 bram=0x45503038 DDR_SIZE=268435456 CPU_HZ=50000000 UI_HZ=100000000 EP08 BRAM CPU_READ bytes=32768 cycles=65541 MBps=24 sum=0x281ffc00 EP08 BRAM LATENCY loads=32768 cycles=81924 cyc_per_load_x100=250 EP08 BRAM CPU_COPY bytes=8192 cycles=18437 MBps=22 bad=0 EP08 BRAM DMA_COPY bytes=8192 cycles=2050 MBps=199 bad=0 EP08 CALIB: PASS EP08 ROUND 1 BEGIN EP08 DDR WRITE bytes=67108864 status=0x000000f5 wr_cycles=9148314 wr_MBps=733 beats=0 err=0 EP08 DDR READ bytes=67108864 status=0x000000f5 rd_cycles=4718705 rd_MBps=1422 beats=4194304 err=0 EP08 DDR CRC bytes=67108864 status=0x000000f5 rd_cycles=4718612 rd_MBps=1422 crc_cycles=167772193 crc_MBps=39 crc_data=0x07e96c24 crc_exp=0x07e96c24 beats=4194304 err=0 EP08 DDR PING n=16 min_ui_cycles=23 max_ui_cycles=36 avg_x100=2412 avg_ns=241 last_addr=0x07800000 EP08 ROUND 1 PASS EP08 ROUND 2 BEGIN EP08 DDR WRITE bytes=67108864 status=0x000000f5 wr_cycles=9148282 wr_MBps=733 beats=0 err=0 EP08 DDR READ bytes=67108864 status=0x000000f5 rd_cycles=4718749 rd_MBps=1422 beats=4194304 err=0 EP08 DDR CRC bytes=67108864 status=0x000000f5 rd_cycles=4718625 rd_MBps=1422 crc_cycles=167772164 crc_MBps=39 crc_data=0x07e96c24 crc_exp=0x07e96c24 beats=4194304 err=0 EP08 ROUND 2 PASS EP08 RESULT: PASS
PathEngineVolumeCyclesBandwidthLatency / note
CPU read BRAMCVE2 load32 KiB / 8,192 words65,541 CPU25.0 MB/s8.0 cycles/word
CPU copy BRAMCVE2 load + store8 KiB / 2,048 words18,437 CPU22.2 MB/s9.0 cycles/word, bad=0
DMA copy BRAMbram_dma8 KiB / 2,048 words2,050 CPU199.8 MB/s1.0 word/cycle, bad=0
BRAM dependent loadCVE2 pointer chain32,768 loads81,924 CPU—2.5 cycles/load = 50 ns
DDR3 writeddr3_dma64 MiB9,148,314 UI733 MB/s45.8% of peak
DDR3 read-compareddr3_dma64 MiB / 4,194,304 beats4,718,705 UI1,422 MB/s88.9% of peak, err=0
DDR3 CRC-32ddr3_dma64 MiB167,772,193 UI39 MB/sbyte-wise second check
DDR3 single-beat pingddr3_dma16 beats, 8 MiB step24.12 UI average—min 23 / max 36 = 241 ns
The UART MBps field is integer-truncated: 24 / 22 / 199. The table recomputes 25.0 / 22.2 / 199.8 from the same cycle counts, so a reader does not divide 24 by 22 and get 1.09. The two independent runs match on every stable field (bytes, beats, CRC, err, PASS); only cycle counts and ping min/max jitter. Run 2 ping is min 23 / max 26 / avg 23.31.

2. Structure: two data paths, one CPU, no DDR window

CVE2RV32IM · 50 MHz
→
CSR decode0x2001 / 0x2002 / 0x2003
→
bram_dmaword copy · 50 MHz
↔
32 KiB scratchtrue dual port · 40 BRAM
CVE2CSR writes and status polls only
→
ddr3_dma128-bit · native UI
→
MIG 7 Series4:1 · ui_clk 100 MHz
→
DDR3MT41J128M16 · 256 MiB

Two facts matter. First, the CPU has only a BRAM window (0x2002_0000..0x2002_7FFF) and no DDR3 window. DDR3 addresses, data and CRC are all handled by ddr3_dma; the CPU writes CTRL and reads STATUS and counters. So "CPU memcpy vs DMA" can only be measured on BRAM, and "DDR3 bandwidth" can only be measured through the DMA. The page never merges the two into one curve.

Second, the scratch is a real dual-port BRAM, but only one port is exposed to the CPU. Port A serves CPU loads and stores, Port B serves DMA reads and writes. They never access at the same time: while dma_active is high, CPU scratch requests are gated. The cost is that the CPU cannot touch the scratch during a DMA; the benefit is that 32 KiB infer as 40 BRAM blocks instead of dissolving into LUTRAM.

3. Four BRAM numbers: 25 vs 22 vs 199, the difference is who moves the data

CPU read. 8,192 sequential word reads take 65,541 CPU cycles, or 8.0 cycles/word. CVE2 has no cache; each load is followed by address increment, accumulation, compare and branch. The load handshake itself is one cycle, the loop is not. 25.0 MB/s is exactly "8 cycles per word × 50 MHz × 4 B".

CPU copy. 2,048 words move from word 0 to word 2,048 in 18,437 cycles, 9.0 cycles/word, 22.2 MB/s. One cycle more than the read because a store is added. Both runs end bad=0, so source and destination do not alias.

DMA copy. The same 8 KiB range is copied by bram_dma in 2,050 CPU cycles, 1.0 word/cycle, 199.8 MB/s. It does not fetch instructions, compute addresses or pass data through CPU registers: source, destination and length go into CSRs, then the state machine issues one read and one write per cycle. The 9.0x gap is not a faster clock — both run at 50 MHz — it is the cost of instruction fetch and loop overhead per word.

Dependent load. A 1,024-node pointer ring runs 32,768 dependent loads in 81,924 cycles, 2.5 cycles/load, 50 ns. This measures the true round trip when the next address depends on the previous data, not streaming throughput. It gives the on-chip baseline for the DDR3 ping: one dependent access is 50 ns in BRAM and 233–241 ns in DDR3, about 4.7x apart.

The limits ship with the numbers. This scratch accepts full 32-bit words only; byte enables are not wired. While dma_active is high, CPU scratch accesses are gated, so the firmware only polls CSRs and never touches the scratch during a copy. Before reusing this as a general memory pool, add byte enables or split the ports, otherwise byte and half-word stores silently lose data.

4. Three DDR3 passes: 733 write, 1,422 read, 39 CRC

DDR3 reuses the Episode 07 MIG native UI and the 128-bit DMA, but fixes the test window at 64 MiB and runs three separate passes: write only, read-compare only, read plus CRC. The pattern still depends on the address (tag={4'hA, addr} expanded to 128 bits), so address decode errors, bank aliasing or stuck data lanes show up on readback; all 4,194,304 beats of 64 MiB compare with err=0.

CRC reads the range again, splits every 128-bit beat into 16 bytes and updates the reflected 0xEDB88320 polynomial byte by byte, with 0xFFFFFFFF as both initial and final XOR. The 64 MiB pass takes 167,772,193 UI cycles (about 1.68 s) and returns 0x07e96c24, equal to the expected value. It issues one read at a time and computes serially, hence 39 MB/s; its value is a second check independent of the pattern compare, not CRC throughput.

Single-beat ping. Sixteen reads step the address by 8 MiB to cross banks and rows; the RTL records the UI-cycle difference from an accepted app_en to app_rd_data_valid. Run 1 gives min 23 / max 36 / avg 24.12 UI cycles, or 241 ns; run 2 gives min 23 / max 26 / avg 23.31. The 36 is one long queueing event; the stable lower bound is 23 cycles, 230 ns.

ModeCTRLCoverageKey result
Write onlywr=1, crc=0, rd=064 MiB linear9,148,314 UI cycles / 733 MB/s
Read-comparerd=1, wr=0, crc=064 MiB linear4,718,705 UI cycles / 1,422 MB/s / 4,194,304 beats / err=0
Read + CRCrd=1, crc=164 MiB linear167,772,193 UI cycles / 39 MB/s / 0x07e96c24
Pingping toggle16 × single beat, 8 MiB stepmin 23 / max 36 / avg 24.12 UI cycles

5. How to read the numbers: hierarchy gap, peak, and what cannot be compared

ComparisonRatioHow to read it
DMA copy ÷ CPU copy (both 8 KiB)199.8 / 22.2 = 9.0xSame 50 MHz, same BRAM; the difference is whether data passes through instruction fetch and a loop. This is the cleanest comparison in this episode.
DDR3 read ÷ DMA BRAM copy1,422 / 199.8 = 7.1xDDR3 is 100 MHz 128-bit burst; BRAM DMA is 50 MHz 32-bit single word. Width and clock both differ.
DDR3 read ÷ CPU BRAM read1,422 / 25.0 = 56.9xOne side is a DMA 128-bit burst, the other a CPU word load. It shows the hierarchy gap, not that "DDR is faster than SRAM".
DDR3 write ÷ DMA BRAM copy733 / 199.8 = 3.7xThe write path still pays the BL8 rhythm of data one cycle before command.
DDR3 ping ÷ BRAM dependent load241 / 50 = 4.8xPing includes command transfer and cross-bank/row queueing; BRAM is an on-chip round trip.

DDR3 peak is still 16-bit × 800 MT/s = 1.6 GB/s. Read reaches 1,422 MB/s, about 88.9%; write 733 MB/s, about 45.8%; CRC 39 MB/s, about 2.4%. Read is fast because addresses and commands stream and 32 outstanding reads hide latency. Write is slower because this version keeps the factory self-test rhythm of data one cycle before command, so every BL8 occupies two UI cycles and the ceiling sits near half of peak. Issuing data and command together is the next direction, but it is not verified here.

Do not read this table as "SRAM slow, DDR fast". The real statement is: when the CPU moves data word by word in software, even on-chip SRAM gives only 22–25 MB/s; once a DMA takes over, on-chip SRAM reaches 199.8 MB/s; DDR3 reaches 1.4 GB/s because of width, burst and MIG scheduling. The hierarchy gap is the product of engine, width and protocol, not a victory of one memory medium.

6. Resources, timing and the final bit

ItemResultNote
LUT9,581 / 20,800 · 46.06%248 more than Episode 07 for scratch and bram_dma control
FF9,212 / 41,600 · 22.14%Two DMA state machines and ping timing
BRAM40 / 50 · 80.00%32 KiB scratch (32 blocks) + CVE2 RAM + FIFO
DSP11 / 90 · 12.22%Used inside MIG
IOB73 / 250 · 29.20%48 DDR3 signals plus board peripherals
WNS / WHS0.853 / 0.036 nsTNS / THS 0, 26,767 endpoints, 0 failing
DRC0 error / 0 warningFinal implementation report

The final bit is 2,192,137 B, SHA-256 f07d96c23ce78ee835b19ea932099f03df99e7d6d25449de5c0600c6de1a868c. BRAM rises from 32 blocks in Episode 07 to 40, exactly the price of the 32 KiB scratch; LUT grows by only 248 because the memory did not dissolve into LUTRAM.

Timing did not converge on the first try: the first final implementation had WHS −0.040 ns and 2 failing hold endpoints, and a single phys_opt_design -hold_fix inside impl_cdc_ep08.tcl was not enough. The flow switched to launch_runs impl_1, waited for the managed run, then applied two phys_opt_design -hold_fix passes and route_design, reaching 0.853 / +0.036 / 0 failing. The −0.040 bit was discarded; only the final version ships.

WHS is only 0.036 ns, so hold margin remains thin. With 0 failing it is publishable, but a different device, temperature corner, MIG configuration or scratch structure requires a fresh implementation; this timing result cannot be carried over.

7. Pitfalls hit along the way

8. Reproduction

# build machine ssh alinx@192.168.8.237 cd /media/alinx/nvme4t/fpga-love/pi_ar035_f1_1_cve2_ep08 # 1) synthesis + implementation + reports; final timing needs the managed run vivado -mode batch -source build_ep08.tcl vivado -mode batch -source impl_run_ep08.tcl # 2) firmware (the HEX is embedded in the bit, so a C change needs re-synthesis) cd sw && make && python3 bin2hex.py && cd .. # 3) start capture first, then program; trim at the last EP08 storage hierarchy line python3 cap_serial.py /dev/ttyUSB1 115200 /tmp/ep08.log & vivado -mode batch -source program_ep08.tcl

The second independent run means a full power cycle and reconfiguration, another program_ep08.tcl, and a fresh UART capture. Compare bytes, beats, CRC, err and the final PASS; cycle counts and ping min/max may jitter, stable fields must match. Restarting only the CPU without resetting the FPGA does not count as a second independent download.

Start the capture before programming. Reconfiguration briefly records the first two lines of the previous bit, so the log must be trimmed at the last EP08 storage hierarchy line; otherwise old output contaminates the new evidence.

9. Project files (38)

RTL, XDC, Tcl, firmware sources and prebuilt outputs, the final bit, three reports, two independent UART logs, build and programming logs, the MIG configuration and the DDR3 pin table are all below. Checksums are in SHA256SUMS.txt; the files/ trees on the Chinese and English pages are identical.

README.txtPackage notes: method, CSR map, results, boundaries metrics/ep08_metrics.jsonMachine-readable metrics: seven paths, timing, resources, SHA-256 bit/top_cve2_ep08.bitFinal bitstream, 2,192,137 B reports/ep08_timing_final.rptWNS 0.853 / WHS 0.036 / 0 failing reports/ep08_util_final.rptLUT 9,581 / FF 9,212 / BRAM 40 / DSP 11 reports/ep08_drc_final.rptDRC 0 error / 0 warning logs/uart_ep08_run1.txtFirst independent run, 19 UART lines logs/uart_ep08_run2.txtSecond independent run: stable fields identical to run1 logs/synth_ep08.logVivado synthesis log logs/impl_ep08.logVivado implementation log logs/impl_run_ep08.logManaged impl_1 plus two hold-fix passes logs/program_ep08_run1.logFirst programming log logs/program_ep08_run2.logSecond programming log logs/build_ep08.logCombined synthesis, implementation and managed-run log rtl/top_cve2_ep08.svTop: CVE2, MMCM, MIG, two DMAs, CSR, UART rtl/ddr3_dma.svnative UI write, compare, CRC-32 and ping engine rtl/dpram_scratch.sv32 KiB true-dual-port scratch, 40 BRAM blocks rtl/bram_dma.svWord-granularity BRAM copy engine xdc/pi_ar035_f1_1_cve2_ep08.xdcBoard pins and asynchronous clock groups xdc/ddr3_mig.xdcMIG-generated DDR3 pins and I/O standards tcl/build_ep08.tclcreate + synthesis + implementation + reports tcl/impl_cdc_ep08.tclEarly post-synthesis flow with clock groups tcl/impl_run_ep08.tclFinal managed impl_1 plus two hold-fix passes tcl/program_ep08.tclJTAG programming of bit and ELF sw/main.cSeven measurement paths, two rounds and reporting sw/start.SBSS clear, mtvec, 64-entry trap table sw/link.ldCode, data and stack layout in 64 KiB RAM sw/MakefileRISC-V GCC build and HEX generation sw/syscalls.cBare-metal newlib stubs sw/bin2hex.pyBIN → 32-bit HEX sw/firmware.elfDownloadable ELF, 67,008 B sw/firmware.binFlat binary, 9,480 B sw/firmware.hexBRAM init, 21,330 B / 2,370 words sw/firmware.disDisassembly, 98,804 B sw/firmware.mapLinker map SHA256SUMS.txtSHA-256 of all 37 artifacts mig/factory_mig.prjMIG project config (MT41J128M16XX-125) pins/ddr3_pins.csv48 DDR3 signals → package pin / IOSTANDARD

10. Related links

This episode verifies 32 KiB scratch BRAM and 256 MiB DDR3 read, write, copy, CRC and single-beat latency on one CVE2. Not verified: a CPU-visible DDR window, AXI4 interconnect, scatter/gather, ECC, DDR3 margin or temperature corners, long soak, multiple concurrent masters, byte or half-word scratch writes, and efficiency comparisons with other SoCs.

Subscribe to fpga.love

Daily Briefing · Weekly Digest · Monthly Deep Dive