DYNAMIC SPARSITY PATTERNS FOR ATTENTION HEADS

원문 (중국어)

A computing system including processing circuitry configured to, during a calibration stage, perform a sparsity pattern search on a plurality of attention heads included in one or more transformer layers to select a respective sparsity pattern associated with each of the attention heads. During an inferencing stage, processing circuitry receives an inferencing input. The processing circuitry pre-fills a context based at least in part on the inferencing input. Pre-filling the context includes computing sparse attention scores at each of the attention heads. Computing the sparse attention scores includes masking each of the attention heads using the respective sparsity pattern selected for that attention head during the calibration stage. The processing circuitry computes an inferencing output by performing inferencing starting from the sparse attention scores. The processing circuitry outputs the inferencing output.