 |
Layer/Channel Gate is a systolic array of size 10Ąż10 used to perform MLP layers. Feature maps flow vertically and weights are accessed independently by the PEs. The results are accumulated in the local buffers and finally accumulated in each column. After the accumulation is completed, results are sent to the ALU unit for activation, quantization, and other operations. The activation information of layers and channels is sent to the fine-tuning module.
Fine-tuning module consists of four parts. Before processing the layers, the parameters of each layer of the network and the fused model are loaded from DRAM to the cache. The performance calculation module obtains the activation ratio of each layer at runtime, loads the network parameters and the fusion model, and then calculates the indicators such as CTC and off-chip memory access of the fine-tuned model. Since the search space is further limited, the amount of computation here is small and does not affect the overall performance. The search results are fed into the arbiter, which selects the optimal model and updates the fused model cache.
To adapt to different cross-layer scheduling strategies, we introduce a reconfigurable processing module composed of fine-grained PE arrays, ALUs, and buffers. Each PE array consists of 8 PEs. The PEs in the PE array are cascaded to each other. Data flow can be transmitted and broadcast among the PE arrays and PEs for data reuse.
All the configurable units are divided into multilayers. The division is according to the proportion of the computational cost of different layers so that the throughput of different layers is balanced. With the appropriate division, the intermediate data can be consumed by the next layer in time. When the processing module starts working, the PE arrays read the features and weights from the SRAM and share them with different PEs.
PEs produce the partial sums and pass them to the Arithmetic Logic Units (ALUs). The partial sums are accumulated in the ALUs. After the accumulation of partial sums is completed, the ALUs will re-quantize the output data from 32-bit to 8-bit. Then the output will be cached in the output buffer and passed to the next layer.
|
IEEE/ICACT20230038 Slide.12
[Big Slide]
[YouTube]