Abstract
Children with autism spectrum disorder (ASD) have social-emotional deficits that lead to difficulties in recognizing emotions as well as understanding and responding to social interactions. This study presents EMooly, a tablet game that actively involves caregivers and leverages augmented reality (AR) and generative AI (GenAI) to enhance social-emotional learning for autistic children. Through a year of collaborative effort with five domain experts, we developed EMooly that engages children through personalized social stories, interactive and fun activities, and enhanced caregiver participation, focusing on emotion understanding and facial expression recognition. Compared with a baseline, a controlled study with 24 autistic children and their caregivers showed EMooly significantly improved children’s emotion recognition skills and its novel features were preferred and appreciated. EMooly demonstrates the potential of AI and AR in enhancing social-emotional development for autistic children via prompt personalizing and engagement, and highlights the importance of caregiver involvement for optimal learning outcomes.
What is EMooly?
EMooly is an AI and AR-powered tablet game that enhances social-emotional learning for autistic children by actively involving caregivers. By integrating personalized storytelling, augmented reality activities, and interactive exercises, EMooly addresses the challenges of traditional interventions with a fresh, engaging approach.
Why EMooly?
Traditional interventions often lack personalization, engagement, and caregiver participation. EMooly fills these gaps by leveraging cutting-edge technologies to create a holistic, accessible, and fun learning environment for autistic children.
gathering to the final evaluation.
Key Features
- Personalized Storytelling
- AI-generated social stories based on children’s preferences and surroundings.
- Scenarios tailored for different developmental needs.
- Dynamic AR Activities
- Emotion recognition tasks using AR to make learning immersive and engaging.
- Familiar objects integrated into the activities to foster real-world connections.
- Caregiver Involvement
- Clear guidance for caregivers to collaborate actively.
- Interactive turn-taking exercises to improve emotional understanding.
How Does It Work?
- Customization: Children personalize characters and scenarios by incorporating real-world items.
- Comprehension: AI-crafted stories guide emotional understanding with caregiver support.
- Observation & Imitation: Turn-taking activities promote mimicking emotional expressions.
- Recognition: AR-based games help children identify emotions in real-world contexts.
- Reinforcement: Reflective questions bridge gameplay with daily life.
objects into AI-generated narratives with a selected character and target emotion. (b) Comprehension: The child reads and
understands the generated story enhanced with visual illustrations and follow-up questions, accompanied by the caregiver. (c)
Observation & Imitation: The child and the caregiver perform turn-taking activities to facilitate the practice and replication
of emotional expressions. (d) Recognition: The child is engaged in a dynamic AR activity to identify the target emotion
from various emotional expressions overlayed in the real-world environment. (e) Reinforcement: The child and the caregiver
conclude the session with reflective questions that bridge the target emotion with real-life experiences. (f) Communication
tips are available at various places of the user interface during the session.
Proven Impact
- Study Results:
- Significant improvement in emotion recognition (+1.5 points on average).
- Superior user experience and usability compared to traditional methods.
- Caregiver Feedback:
- Increased engagement and collaboration.
- Stories and games seen as more intuitive and enjoyable than traditional slides or flashcards.
DOI: 10.1145/3699738
@article{lyu2024emooly,
title={EMooly: Supporting Autistic Children in Collaborative Social-Emotional Learning with Caregiver Participation through Interactive AI-infused and AR Activities},
author={Lyu, Yue and Liu, Di and An, Pengcheng and Tong, Xin and Zhang, Huan and Katsuragawa, Keiko and Zhao, Jian},
journal={Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies},
volume={8},
number={4},
pages={1--36},
year={2024},
publisher={ACM New York, NY, USA}
}