Systems and Methods for Robotic System Control Using Generative Models

原文（中国語）

In one embodiment, a method includes receiving a natural language instruction from an interface device, receiving sensor data of a physical environment from a sensing system, converting the natural language instruction into instruction tokens and the sensor data into sensor tokens. The method further includes executing an autoregressive generative model on the instruction tokens and sensor tokens to output control tokens. The method further includes generating control commands based on the control tokens and transmitting the control commands to a robotic system. The control commands cause the robotic system to perform an action affecting the physical environment.