Open X-Embodiment dataset, RT-X model a leap for AI robots

October 5, 2023

Google’s DeepMind worked with 33 different academic labs to create an AI training dataset based on 22 different robot types.

Robots are really good at doing one specific thing. If you want it to do something even slightly different, the robot needs to be trained from scratch. The ultimate goal for robotics is to have a robot that is good at a general range of actions with the ability to learn new skills by itself.

To train an AI model you need a large dataset of data related to the purpose of the model. Language models like GPT-4 are trained on vast amounts of written data. Image generators like DALL-E 3 are trained on large amounts of images.

With X-Embodiment, DeepMind has created a dataset of robotic actions based on 22 different types of robots. It then used that dataset to train new models based on its RT-1 and RT-2 robotic models.

The data for X-Embodiment was derived from “22 robot embodiments, demonstrating more than 500 skills and 150,000 tasks across more than 1 million episodes,” according to DeepMind’s post.

The earlier test results of the RT-1 and RT-2 models were already impressive but DeepMind found that the RT-X versions performed significantly better due to the general nature of the new dataset.

Testing involved comparing a robot controlled by a model trained for a specific task with that same robot controlled by the RT-1-X model. RT-1-X performed on average 50% better than the models designed specifically for tasks like opening a door or routing a cable.

RT-2, Google’s vision-language-action (VLA) robotic model, allows robots to learn from web, verbal, and visual data and then act without being trained. When engineers trained RT-2-X with the X-Embodiment dataset they found that RT-2-X was three times as successful as RT-2 for emergent skills.

RT-2-X demonstrates an understanding of spatial relationships between objects. Source: DeepMind

In other words, the robot was learning new skills it didn’t have before, based on abilities that other robots had contributed to the dataset. Skills transfer between different types of robots could be a game-changer for rapid robotics development.

These results are cause for optimism that we’ll soon see robots with more general skills as well as the ability to learn new ones without being specifically trained for them.

DeepMind says that this research could be applied to the self-improvement property of RoboCat, its self-improving AI agent for robotics.

The prospect of having a robot that continues to improve and learn new skills would be a huge advantage in fields like manufacturing, agriculture, or healthcare. Those new skills could equally be applied in the defense industry which is perhaps a less appealing, if inevitable prospect.

Join The Future


Clear, concise, comprehensive. Get a grip on AI developments with DailyAI

Eugene van der Watt

Eugene comes from an electronic engineering background and loves all things tech. When he takes a break from consuming AI news you'll find him at the snooker table.


Stay Ahead with DailyAI

Sign up for our weekly newsletter and receive exclusive access to DailyAI's Latest eBook: 'Mastering AI Tools: Your 2024 Guide to Enhanced Productivity'.

*By subscribing to our newsletter you accept our Privacy Policy and our Terms and Conditions