INT8 Inference on FPGA: Fitting a CNN into DSP Slices

1. Quantization: The First Choice Between Precision and Efficiency

Computing FP32 weights and activations directly on an FPGA is expensive: the DSP48 multiply width is limited and on-chip memory bandwidth is tight. INT8 quantization cuts compute width to a quarter; combined with DSP packing (two INT8 multiplies into one DSP48), it multiplies MAC density. On strategy, min/max calibration is simple to implement but sensitive to long-tail distributions, while KL-divergence calibration (following the TensorRT idea) preserves the distribution body via information theory — accuracy loss can usually be kept within 1%.

2. Convolution Mapping: im2col vs Direct Convolution

Two mainstream data flows: im2col turns convolution into matrix multiply, good for reusing systolic arrays but adds data movement; direct convolution computes in sliding-window streaming fashion, using line buffers to cache row data and saving resources. In practice, small kernels (3×3) mostly use direct convolution; large strides or transposed convolutions call for trade-offs.

3. Data Reuse: Bandwidth Is the Real Bottleneck

Data between CNN layers far exceeds on-chip storage. Practice points: input/weight ping-pong buffering hides memory latency; on-chip accumulation of partial sums reduces write-backs. The design goal is to fit external DRAM bandwidth within the budget, not to blindly stack parallelism.

4. Measured Perspectives and Engineering Advice

• LeNet-class small networks: INT8 can hit millisecond-level latency on entry-level FPGAs — ideal for teaching and prototype validation;
• YOLO-class detection networks: combine depthwise separable convolutions and channel pruning; watch DSP/LUT usage vs frame-rate balance;
• Deployment order: software simulation to confirm quantization error → single-layer hardware bring-up → full-network integration, avoiding “all-at-once” rework.