FAULT IDENTIFICATION AND RECOVERY FOR DISTRIBUTED TRAINING

원문 (중국어)

Example embodiments of the present disclosure relate to a method, a device and a non-transitory computer-readable medium for distributed training. The method comprises obtaining, during a distributed training task performed across a plurality of computing nodes, at least one heartbeat message from the plurality of computing nodes, each computing node including multiple GPU workers; detecting, based on the at least one heartbeat message, an abnormal status of the distributed training task; commanding the plurality of computing nodes to run at least one self-check diagnostics test; identifying, based on results of the at least one self-check diagnostics test, at least one faulty node from the plurality of computing nodes; and replacing the at least one faulty node with an equivalent number of heathy computing nodes that have passed the at least one self-check diagnostics test.