Robots let ChatGPT touch the real world thanks to Microsoft

Last week, Microsoft researchers unveiled an experimental framework for controlling robots and drones using the language capabilities of ChatGPT, a popular AI language model created by OpenAI. Using natural language commands, ChatGPT can write special code to control the robot’s movements. A human then reviews the results and adjusts as necessary until the task is successfully completed.

The study arrived in a paper titled “”ChatGPT for Robotics: Design Principles and Model Abilities” was written by Sai Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor of Microsoft Autonomous Systems and Robotics Group.

In a demo video, Microsoft shows the robot using a robotic arm to place blocks on the Microsoft logo, flying a drone to inspect shelf contents, and using the robot to find objects. . With vision function.

To make ChatGPT work with robotics, researchers taught ChatGPT a custom robotics API. Given a command such as “pick up the ball,” ChatGPT can generate robot control code, much like writing a poem or completing an essay. After a human inspects and edits the code for correctness and safety, a human operator can perform the task and evaluate its performance.

Thus, ChatGPT speeds up robot control programming, but it is not an autonomous system. “Using ChatGPT for robotics emphasizes that it is not a fully automated process, but rather a tool that augments human capabilities.”

Although most of the feedback (about the success or failure of an action) to ChatGPT appears to come from humans in text form, researchers claim to have also successfully fed ChatGPT itself with visual data. increase. In one instance, the researchers instructed her ChatGPT to command the robot to catch a basketball with feedback from the camera. Track implicit world models that go beyond text-based probabilities. “

Although the results look rudimentary so far, they represent an early attempt to apply one of the hottest technologies today, large-scale language models, to robot control. According to Microsoft, the ChatGPT interface could open up robotics to a wider audience in the future.

“The goal of this research is to see if ChatGPT can think beyond text and reason about the physical world to aid in robotics tasks,” reads a Microsoft Research blog post. “We want to make it easier for people to interact with robots without having to learn complex programming languages ​​or the details of robotic systems.”