Safe and Trustworthy Deep Reinforcement Learning for Aircraft Separation Assurance
Open AccessEnsuring the safety separation among aircraft has become a key challenge with the rapid growth of global air traffic. Deep Reinforcement Learning (DRL) has been recently applied to aircraft separation assurance systems, ensuring aircraft safety in complex and dense traffic. These DRL models can learn compact representations of complex aircraft states and make tactical decisions. However, DRL models are trained as black boxes with complex structures, leaving the system with no safety guarantees and the model output without explanations. Specifically, existing DRL solutions fail to answer the following three critical questions: (1) How to provide immediate safety enhancement for DRL models without additional training in unseen environments. (2) How to validate the safety of DRL models given their black-box nature and complex structures. (3) How to explain the behaviors of DRL agents and help users understand the decision-making in real-time. These questions create crucial obstacles to providing safe and trustworthy decision-making in DRL models for safety-critical applications. In order to build safe and trustworthy DRL models for aircraft separation assurance systems, three novel frameworks are proposed in this dissertation to answer these questions.To solve the first problem that DRL models fail to maintain safe separation in unseen environments without further training, an immediate safety enhancement module for DRL models is proposed. The proposed module can support safe decision-making based on Monte-Carlo Dropout and execution-time data augmentation under state and model uncertainties in unseen environments. The module is named "DropOut and Data Augmentation safety module" (for short DODA).To tackle the second problem that the black-box nature and complex structures of DRL models make safety validation a challenging task, a multi-agent validation framework based on Adaptive Stress Testing (AST) is proposed. The validation framework detects system failures caused by unsafe actions, improper environment configurations, and their combinations based on DRL and Bayesian Optimization methods. We refer to our framework as "Multi-Agent Hybrid Adaptive Stress Testing" (for short MAHAST).To address the third problem that the output of DRL models is provided with no explanations and users cannot understand the decision-making in real-time, an explainable safety separation assurance framework is designed. The framework distills a complex DRL model into a shallow Soft Decision Tree (SDT) and uses the distilled knowledge in SDT to provide visual explanations of agent behaviors in each step. The framework helps users understand agent behaviors and build trust in decision-making. We refer to our framework as "Stepwise Explainable Separation Assurance MEthod" (for short SESAME).By integrating the DODA safety enhancement module, the MAHAST validation framework, and the SESAME explainable framework, the dissertation provides safe and trustworthy DRL models for aircraft separation assurance. Through extensive numerical experiments in complex and challenging case studies, we demonstrate the effectiveness of our proposed frameworks: (1) The proposed DODA module helps improve the safety of DRL agents significantly in unseen environments by mitigating the harmful impact of model and state uncertainties. (2) The proposed MAHAST framework can effectively validate the safety of DRL models for aircraft separation assurance systems by detecting failures due to multiple reasons under high-density air traffic. (3) The proposed SESAME framework can help improve users' trust in DRL models for aircraft separation assurance by providing stepwise explanations of model output. In addition, these proposed frameworks can be extended to DRL models for other applications.
- All rights reserved
Notice to Authors
If you are the author of this work and you have any questions about the information on this page, please use the Contact form to get in touch with us.