A PARAMETER SERVER ORCHESTRATION PROCEDURE FOR ACCELERATING THE TRAINING OF LARGE LANGUAGE MODELS

원문 (중국어)

Disclosed is a parameter server orchestration architecture and procedure for accelerating the training of large language models that advantageously improves inter-machine network traffic thereby accelerating the training speed of LLMs. Our inventive procedure advantageously (i) minimizes the amount of inter-machine network traffic thereby accelerating the training speed of LLMs from a global perspective; (ii) optimizes the topology design for a given LLM training job so that less inter-machine traffic is produced; (iii) optimizes the number of workers used and their placement for a given LLM training job; (iv) optimizes number of parameter servers employed and their placement for a given LLM training job; (v) optimizes the workload distribution between different parameter servers such that inter-machine network traffic is reduced; and (vi) efficiently allocates network bandwidth and routing paths for inter-machine network traffic.