Researchers build synthetic dog dataset in GTA-V to create 3D models

March 3, 2024

dog ai

Researchers from the University of Surrey have developed a method to transform photographs of dogs into detailed 3D models. 

The training material? Not real dogs, but rather computer-generated images from the virtual world of the hit game Grand Theft Auto V (GTA V). 

Moira Shooter, a postgraduate research student involved in the study, shared of the study, “Our model was trained on CGI dogs – but we were able to use it to make 3D skeletal models from photographs of real animals. That could let conservationists spot injured wildlife, or help artists create more realistic animals in the metaverse.”

To date, methods for teaching AI about 3D structures involve using real photos alongside data about the objects’ actual 3D positions, often obtained through motion capture technology. 

However, when applying these techniques to dogs or other animals, there are often too many movements to track, and getting dogs to behave for long enough is tough. 

To build their dog dataset, researchers altered GTA V’s code to replace its human characters with dog avatars through a process known as “modding.” 

Examples from the synthetic dog dataset generated using GTA-V. Source: University of Surrey.

Researchers produced 118 videos capturing these virtual dogs in various actions – sitting, walking, barking, and running – across different environmental conditions.

This culminated in the creation of ‘DigiDogs,’ a rich database containing 27,900 frames of dog movement, captured in a way real-world data collection hadn’t allowed.

With the dataset in hand, the next steps used Meta’s DINOv2 AI model for its strong generalization skills, fine-tuning it with DigiDogs to accurately predict 3D poses from single-view RGB images.

Researchers demonstrated that using the DigiDogs dataset for training resulted in more accurate and lifelike 3D dog poses than those trained on real-world datasets, thanks to the variety in dog appearances and actions captured.

The model’s enhanced performance was confirmed through thorough qualitative and quantitative evaluations.

AI dogs
Models trained on the synthetic dataset DigiDogs showed improved accuracy versus those trained only with the real-world dataset RGBD-Dogs. You can see this here if you look carefully at the predicted mappings (in pink) versus the ground truth (in green), which are improved on the right-hand side. Source: University of Surrey.

While this study represented a big step forward in 3D animal modeling, the team acknowledges there’s more work to be done, especially in improving how the model predicts the depth aspect of the images (the z-coordinate). 

Shooter described the potential impact of their work, saying, “3D poses contain so much more information than 2D photographs. From ecology to animation—this neat solution has so many possible uses.”

The paper won the Best Paper prize at the IEEE/CVF Winter Conference on Applications of Computer Vision.

It opens the door to better model performance in areas like wildlife conservation and 3D object rendering for VR. 

Join The Future


Clear, concise, comprehensive. Get a grip on AI developments with DailyAI

Sam Jeans

Sam is a science and technology writer who has worked in various AI startups. When he’s not writing, he can be found reading medical journals or digging through boxes of vinyl records.


Stay Ahead with DailyAI

Sign up for our weekly newsletter and receive exclusive access to DailyAI's Latest eBook: 'Mastering AI Tools: Your 2024 Guide to Enhanced Productivity'.

*By subscribing to our newsletter you accept our Privacy Policy and our Terms and Conditions