Scientists from EPFL have developed a deep-learning-based motion-capture software that uses several camera angles to imitate the actions of a fly in three dimensions. By using this information, scientists are hoping to meet their goal of designing a fly-like robot.
Flies aren’t very appealing to humans. We more than often have some unfortunate experiences with them as they buzz about, feast on garbage and make us think of how unhygienic some places can be. However, robots may provide some form of redemption. It appears that flies have specific features and abilities that can inform a new design for robotic systems.
“Unlike most vertebrates, flies can climb nearly any terrain,” says Professor Pavan Ramdya, whose lab at EPFL’s Brain Mind Institute led the study. “They can stick to walls and ceilings because they have adhesive pads and claws on the tips of their legs. This allows them to basically go anywhere. That's interesting also because if you can rest on any surface, you can manage your energy expenditure by waiting for the right moment to act.”
The different poses of a fly are captured in 3D by several cameras. The images are collected and processed with DeepFly3D. (Image Credit: P. Ramdya, EPFL.)
This vision of extracting the principles that control fly behavior contributed to the design of robots that pushed the development of DeepFly3D, a motion-capture system for the fly Drosophila melanogaster, a common model organism used throughout biology. 
During one of Ramdya’s experiments, a fly walked on top of a small floating ball, while seven cameras recorded every action it took. The top facet of the fly was glued onto a still stage, making the fly stay in position while walking on the ball. This causes the fly to “believe” that it’s moving freely.
The images are then processed by DeepFly3D. “This is a fine example of where an interdisciplinary collaboration was necessary and transformative,” says Ramdya. “By leveraging computer science and neuroscience, we’ve tackled a long-standing challenge.”
What’s unique about DeepFly3D is that it’s capable of figuring out the 3D pose of the fly or any other animal. This means that it can predict and create behavioral measurements at extraordinary resolutions for many different biological applications. DeepFly3D doesn’t require manual calibration, and if it makes any errors in its calculations of the fly’s pose, it will use images to find and correct them. It also comes bundled with active learning, which allows it to improve performance on its own.
The software paves the way to efficiently and precisely model the actions, poses, and joint angles of a fly in 3D. This may encourage a way to automatically model 3D poses in various living organisms as well.
“The fly, as a model organism, balances tractability and complexity very well,” says Ramdya. “If we learn how it does what it does, we can have an important impact on robotics and medicine and, perhaps most importantly, we can gain these insights in a relatively short period of time.”
This video shows a recording of the fly’s behavior using optogenetics and deep learning.
Have a story tip? Message me at: cabe(at)element14(dot)com