A Novel Self-Supervised Multimodal Deep Learning Robotic Framework for Real-Time Behavior Analysis and Stress Mitigation
Open Access DepositedBridging Perception and Intervention in Autism Care
Autism Spectrum Disorder (ASD) often accompanies developmental conditions such as communication challenges, restricted interests, and repetitive behaviors—factors that complicate both diagnosis and ongoing care. To address these complexities, this dissertation proposes a novel pipeline that integrates multimodal Transformer models and deep Reinforcement Learning (RL) to provide real-time behavior analysis and robot-assisted interventions. To achieve this goal, we list three research aims. First, we establish a unimodal baseline by comparing state-of-the-art video-based deep learning techniques, including SlowFast networks and a vision transformer, as well as zero-shot learning through large language models. These preliminary findings highlight both the strengths and limitations of single-modality recognition for detecting and classifying interaction styles in autistic children. Building on this groundwork, secondly, we incorporate multimodal sensor data and a self-supervised learning paradigm to develop the Audio-Visual Family Observation Schedule (AV-FOS) model capable of more robust and accurate classification of challenging behaviors. Comprehensive experiments on a specialized Revised Family Observation Schedule 3rd Edition (FOS-III-R) dataset confirm that the multimodal approach substantially outperforms unimodal baselines in both accuracy (0.86 vs. 0.77) and robustness (0.59 vs. 0.33 in F1 Score), even in data-limited clinical environments. Finally, the system transitions from passive perception to active intervention by employing a Deep Q-Network (DQN) to guide a social robot’s actions in real time. This approach leverages objective physiological metrics—such as electrodermal activity, blood volume pulse, and skin temperature—to design a reward function aimed at reducing stress or anxiety in children with ASD. Through repeated interactions and feedback-driven learning, the robot refines its behavior to better support children’s emotional well-being, leading to a 59% reduction in anxiety scores (p < 0.001) and a 42% improvement in perceived engagement (p = 0.005). By bridging perception and intervention, this unified framework demonstrates a promising path toward scalable, data-driven therapies and offers new perspectives on leveraging advanced Artificial Intelligence (AI) for individualized autism care.
- All rights reserved
Notice to Authors
If you are the author of this work and you have any questions about the information on this page, please use the Contact form to get in touch with us.