Geometric Deep Reinforcement Learning for Quadrotor UAV
Open Access DepositedThis dissertation presents geometric deep reinforcement learning frameworks for the low-level control of quadrotor unmanned aerial vehicles. The control of highly agile quadrotors is challenging due to their inherently underactuated and heavily coupled translational and yawing dynamics. Furthermore, traditional single-agent monolithic policies lack structural awareness of the dynamics, struggling to simultaneously manage conflicting control objectives and requiring massive datasets to learn physical symmetries through trial and error. To address these limitations, this dissertation develops modular and equivariant reinforcement learning architectures based on the geometric decomposition of quadrotor dynamics and introduces a dynamics-level domain adaptation framework for adaptive sim-to-real transfer. First, a multi-agent modular reinforcement learning architecture is constructed by decomposing the quadrotor dynamics into a translational subsystem and a decoupled yaw subsystem. By assigning specialized independent learning modules to handle each distinct dynamic characteristic, the proposed framework mitigates the severe optimization conflicts inherent in traditional single-agent monolithic policies. This decoupled approach not only demonstrates superior tracking performance and robustness, particularly during aggressive yaw maneuvers, but also significantly improves sample efficiency. Second, an equivariant reinforcement learning framework is introduced to fundamentally resolve the sample inefficiency of standard neural network policies. Standard multi-layer perceptrons are symmetry-agnostic and redundantly learn optimal control strategies across physically equivalent state-action pairs. Through the formalization of the continuous rotational symmetry of the translational dynamics and the discrete reflectional symmetry of the yaw dynamics, these geometric properties are directly encoded into equivariant multi-layer perceptrons. This structural embedding guarantees that a policy learned in one state automatically generalizes across all symmetrically equivalent configurations, drastically accelerating learning convergence. Third, a dynamics-level domain adaptation framework is proposed to achieve safe and data-efficient sim-to-real transfer. While traditional domain randomization provides a baseline level of robustness, it inherently struggles to capture complex unmodeled aerodynamics and actuator latencies, particularly for highly unstable systems such as a flying inverted pendulum. Furthermore, standard domain adaptation methods fundamentally couple system identification and control by directly updating the policy with real data. This requires the simultaneous identification of unmodeled dynamics and optimization of the control policy, often leading to sample inefficiency and unstable training convergence. In contrast, the proposed approach explicitly decouples these objectives. By learning the dynamics mismatch from real flight data, a data-driven delta action model is trained to construct an augmented simulator that captures the real-world physical gap without requiring explicit system identification of these residual discrepancies. The control policy is then safely re-trained entirely within this stable, augmented simulation environment. By correcting the dynamic model prior to policy optimization, this approach prevents risky real-world reinforcement learning exploration while maximizing data efficiency and training stability. Finally, the proposed frameworks are verified through comprehensive numerical simulations and real-world flight experiments. The integration of geometric decomposition, equivariant networks, and dynamics-level adaptation overcomes the traditional barriers of model-free reinforcement learning. The resulting frameworks establish a highly effective and scalable paradigm for the reinforcement learning control of quadrotors and other complex robotic systems.
- All rights reserved
Notice to Authors
If you are the author of this work and you have any questions about the information on this page, please use the Contact form to get in touch with us.