Systems and Methods for Robotic System Control Using Generative Models
개요
발명자
Yang Liu; Carlos Florensa; Jeremy Welborn; Nikhil Mishra; Anusha Nagabandi; Hassan Farooq; Varun Vijay; Juyue Chen; Daniel Adelberg; Andrew Sohn; Yan Duan; Pieter Abbeel; Xi Chen
IPC 분류
CPC 분류
In one embodiment, a method includes receiving a natural language instruction from an interface device, receiving sensor data of a physical environment from a sensing system, converting the natural language instruction into instruction tokens and the sensor data into sensor tokens. The method further includes executing an autoregressive generative model on the instruction tokens and sensor tokens to output control tokens. The method further includes generating control commands based on the control tokens and transmitting the control commands to a robotic system. The control commands cause the robotic system to perform an action affecting the physical environment.
원문 (중국어)
In one embodiment, a method includes receiving a natural language instruction from an interface device, receiving sensor data of a physical environment from a sensing system, converting the natural language instruction into instruction tokens and the sensor data into sensor tokens. The method further includes executing an autoregressive generative model on the instruction tokens and sensor tokens to output control tokens. The method further includes generating control commands based on the control tokens and transmitting the control commands to a robotic system. The control commands cause the robotic system to perform an action affecting the physical environment.