Image from the 3D-POP dataset.
Sensory ecology - 3D Tracking of movement and behavior.
Scroll ↓
My research on sensory ecology focuses on studying the mechanisms behind complex behavior patterns of animals by measuring their movement and posture. I use compute vision & AI to automatically measure the subtle differences in behavioral patterns of individuals in a group by tracking their interactions in the three dimensions. 3D tracking and individual identification offers a remarkable window to measure complex attributes such as gaze or visual field of view of an animal. Since 2018, I have developed several datasets, tracking methods and infrastructures with birds as my primary system. My work has extended 3D posture tracking from indoor to outdoor environments and now being extended to primates to study social learning and tool use. I am also deeply interested in sensory ecology research conducted with use of Virtual Reality (VR) systems for animals.
In 2023, We introduced SMART-BARN, one of the largest facility for tracking animals (insects, mammals and birds) in indoor environments. We combined motion capture cameras, video cameras, and audio microphones for complete 3D tracking of movement and sound. We conducted three concrete case studies to demonstrate capability of the system in supporting a wide range of biological questions. The research facility is now adapted as Imaging Hangar by the Center of the Advanced Studies of Collective Behavior (CASCB) at the University of Konstanz. The study is published in Science Advances and a hands on overview can be found at New Scientist or Uni Konstanz. The facility is currently supporting a wide range of experiments in collective behavior, robotics and swarm intelligence.
Tracking animals in 3D with high accuracy
I used SMART-BARN to create the first large-scale datasets for markerless 3D posture tracking of birds. I developed a novel technique to record the 3D postural movements (head and body orientation) of birds using motion capture markers. Furthermore, I used this technique to create millions of annotations necessary to train a machine learning model for markerless tracking. 3D-POP is one of the first datasets consisting sub-millimeter accurate 3D posture for an entire group of birds for several hours. We show that the model works with pigeons without markers. Read more details from here.
Introducing markerless 3D posture tracking for a flock of birds
My collaborators used 3D-POP to extend the work on 3D postures to track posture of birds in the wild. We show that data created in indoor environments can be used effectively for tracking birds in the wild with minimal additional annotations. This approach is further extended by myself and collaborators to other systems such as Great tits, Siberian jays to study social learning.
Going from the lab to the wild
What did we learn from this technology?
My research on 3D posture has lead to novel understanding of visual mechanism of pigeons. The SMART-BARN contributed to research of predator-pray dynamic and collective information processing in pigeons. Pigeons are model systems for behavioral studies and therefore the measurement techniques and vision models developed for tracking pigeons are universally useful for scientists across the globe. The concept of SMART-BARN is replicated in Japan and the 3D gaze tracking inspired development of 3D-SOCS a miniature setup to track gaze of small birds like Great tits.
A side quest into the world of Virtual Reality (VR) with animals
I have published one of the first review papers on applications of Virtual Reality with animals for the computer science community. This work review traces the history of visual stimulation based experiments designed to study a range of topics including animal vision, cognition, bio-mechanics and navigation.