Uncertainty Quantification in Deep Learning Model and Nested Data Problem
Open AccessUncertainty quantification is an essential topic in statistical analysis. In deep learning models, the uncertainty quantification (UQ) of CNN has been largely overlooked. Among the few existing UQ approaches that have been proposed for deep learning, few have theoretical consistency that can guarantee the uncertainty quality. Lack of efficient UQ tools severely limits the application of CNN in certain areas, such as medicine, where prediction uncertainty is critically important. Apart from that, adversarial attacks have emerged as a new challenge, which can seriously impair the trustworthiness of artificial neural networks. To address these issues, we first propose a novel bootstrap-based framework for estimating prediction uncertainty, where the inference procedure relies on convexified neural networks to establish the theoretical consistency of bootstrap. The proposed approach has a significantly less computational load than its competitors, as it relies on warm-starts at each bootstrap that avoids refitting the model from scratch. At the same time, the proposed approach intrinsically can determine the neural networks' output distributions, which can be employed for adversarial attack detection. Further, we explore a novel transfer learning method so our framework can work on arbitrary neural networks. The comprehensive experimental results demonstrate our approach outperforms other baseline CNNs and state-of-the-art approaches significantly and can serve as a robust, efficient, and accurate solution to detect adversarial attacks.Another related topic is UQ in nested data problem. Nested data contains pathologists' reading scores from multiple cases and multiple regions of interest (ROIs), which are nested in each case. To evaluate pathologists' diagnostic performance in nested data problem, we propose to use the area under the receiver operating characteristic curve (AUC) as the tool. To understand the UQ in AUC estimator, we need to find corresponding AUC variance and AUC covariance between paired readers, which is still challenging. To date, little attention has been paid to the AUC estimation and corresponding UQ analysis for nested data. In this paper, we identify two types of AUC estimators in nested data problems. Specifically, one is the within-case AUC estimator to evaluate the reading performance of distinguishing between positive and negative ROIs within the same patient; and the other is between-cases AUC estimator to evaluate the reading performance of distinguishing between positive and negative ROIs across different patients. An existing method estimates the AUC without separating these two types of AUC, which ends up with a biased AUC estimator if correlations exist between positive and negative ROIs. In this paper, we mainly focus on the between-cases AUC estimator showing it is an unbiased AUC estimator even if there exist correlations between positive and negative ROIs. Further, we provide its variance estimator and covariance estimator between two readers in detail. Based on our simulation model, we derive and prove the theoretical values for the above estimators and verify their accuracy using both simulation results and the High-Throughput Truthing Project's dataset. Lastly, we make the connection between our applied simulation model and the linear mixed-effect model, suggesting our between-cases AUC estimator provides a route for the Nested Data AUC estimation problem.
- All rights reserved
Notice to Authors
If you are the author of this work and you have any questions about the information on this page, please use the Contact form to get in touch with us.