Systems and Methods for Robotic System Control Using Generative Models
Dossier Overview
Applicant
Covariant, Inc.
Inventor
Yang Liu; Carlos Florensa; Jeremy Welborn; Nikhil Mishra; Anusha Nagabandi; Hassan Farooq; Varun Vijay; Juyue Chen; Daniel Adelberg; Andrew Sohn; Yan Duan; Pieter Abbeel; Xi Chen
IPC Classification
CPC Classification
In one embodiment, a method includes receiving a natural language instruction from an interface device, receiving sensor data of a physical environment from a sensing system, converting the natural language instruction into instruction tokens and the sensor data into sensor tokens. The method further includes executing an autoregressive generative model on the instruction tokens and sensor tokens to output control tokens. The method further includes generating control commands based on the control tokens and transmitting the control commands to a robotic system. The control commands cause the robotic system to perform an action affecting the physical environment.
Original (Chinese)
In one embodiment, a method includes receiving a natural language instruction from an interface device, receiving sensor data of a physical environment from a sensing system, converting the natural language instruction into instruction tokens and the sensor data into sensor tokens. The method further includes executing an autoregressive generative model on the instruction tokens and sensor tokens to output control tokens. The method further includes generating control commands based on the control tokens and transmitting the control commands to a robotic system. The control commands cause the robotic system to perform an action affecting the physical environment.
External Resources