Machine Learning in Augmented Reality for Medical Training: An Intelligent Training Framework
Open AccessAs a relatively new paradigm in medical education, simulation-based training (SBT) has gained wide acceptance to offer experiential learning and deliberate practice for facilitating medical skill acquisition. However, there is an increasing pressure to solve the problems that hinder the improvement of training efficiency: Training opportunities are resource-intensive due to the limited availability of instructors who have heavy clinic duties. Different clinical experiences and preferences of instructors hamper the consistency of training outcomes even with a unified training platform. In addition, current task-trainer simulators are less effective in providing satisfactory procedural feedback for accurate assessment due to the lack of see-through visualization. Considering the increasing attention on disruptive technologies such as augmented reality (AR) and machine learning (ML), it is necessary to explore solutions that integrate these technologies with SBT to boost training quality and efficiency.In this dissertation, we present a new ML in AR training framework to address the issues. An AR intelligent training framework for neonatal endotracheal intubation (ETI) is developed to validate the idea, which provides trainees with complete visualization of the ETI procedure for real-time guidance and assessment. Specifically, the framework can capture the motions of the laryngoscope and the manikin and offer 3D see-through visualization rendered to the head-mounted display (HMD). To develop an effective machine learning model for assessment, we firstly develop a multinomial regression model with manual-crafted features that is based on domain-specific knowledge. To avoid domain-specific knowledge and explicit feature selection in the framework, a dilated Convolutional Neural Network (CNN) model is proposed to evaluate the ETI performance automatically without any intervention. Kinematic multivariate time-series (MTS) data, as the model input, facilitates the robustness of the framework. Furthermore, the attention-integrated CNN model is developed to assess the ETI performance as well as identify regions of motions that significantly contribute to the performance evaluation. In addition, we developed a new auxiliary-based training approach for developing effective automated assessment models with limited medical training data by utilizing the auxiliary tasks that imitate the evaluation rubrics. Lastly, augmented user-friendly feedback is delivered with interpretable results with the ETI scoring rubric through the color-coded motion trajectory classifications that highlight regions that need more practice. Our solution can offer complete 3D procedure visualization with augmented performance information to address poor situational awareness and provide consistent and automated assessments with trainees to learn with a consistent and comprehensive evaluation, including both formative and summative feedback. This empowers trainees to practice with ML insights to precisely improve their movements for parts of the procedure that need more practice. Our framework has the potential to transform into a training solution for other medical procedures with systematic and algorithmic generalization, providing an alternative research direction in the medical training domain for exploring innovative SBT solutions with resource-intensive training data.
- All rights reserved
Notice to Authors
If you are the author of this work and you have any questions about the information on this page, please use the Contact form to get in touch with us.