EXPERT LOAD BALANCING IN TRANSFORMER MODELS
EXPERT LOAD BALANCING IN TRANSFORMER MODELS
In response to one or more conditions, a processing system determines whether transferring one or more experts to different processing units would improve load balancing at the processing system. The processing system determines an amount of variance between the utilization for each expert relative to the average utilization of all experts at their currently-assigned processing units. The processing system then measures the amount of variance under one or more different configurations of expert-processing unit assignments. If so, the processing system transfers one or more of the experts to different processing units.