Image credit: Christian Pfeiffer

Visual attention prediction improves performance of autonomous drone racing agents


Humans race drones faster than neural networks trained for end-to-end autonomous flight. This may be related to the ability of human pilots to select task-relevant visual information effectively. This work investigates whether neural networks capable of imitating human eye gaze behavior and attention can improve neural networks’ performance for the challenging task of vision-based autonomous drone racing. We hypothesize that gaze-based attention prediction can be an efficient mechanism for visual information selection and decision making in a simulator-based drone racing task. We test this hypothesis using eye gaze and flight trajectory data from 18 human drone pilots to train a visual attention prediction model. We then use this visual attention prediction model to train an end-to-end controller for vision-based autonomous drone racing using imitation learning. We compare the drone racing performance of the attention-prediction controller to those using raw image inputs and image-based abstractions (i.e., feature tracks). Comparing success rates for completing a challenging race track by autonomous flight, our results show that the attention-prediction based controller (88% success rate) outperforms the RGB-image (61% success rate) and feature-tracks (55% success rate) controller baselines. Furthermore, visual attention-prediction and feature-track based models showed better generalization performance than image-based models when evaluated on hold-out reference trajectories. Our results demonstrate that human visual attention prediction improves the performance of autonomous vision-based drone racing agents and provides an essential step towards vision-based, fast, and agile autonomous flight that eventually can reach and even exceed human performances.

Aerial Robotics Autonomous Systems Deep Learning Drone Racing Eye Tracking Imitation Learning
Dr. Christian Pfeiffer
Consulting for Professors at ETH Zurich

I am passionate about supporting university professors in their personal development and on the topic of leadership. With a background in psychology and cognitive neuroscience, and 10+ years experience as academic scholar, my consulting combines theoretical depth and practical relevance to help leaders excell in their professional career development. Expertise: Psychology, Cognitive Neuroscience, Computer Science, Robotics. Interests: Leadership, Personal Development