Back close

Thrivaad: A Multilingual, Predictive Eye-Sign-Based AAC System Powered by Optimized Deep Learning

Publication Type : Conference Paper

Publisher : MDPI AG

Source : Sensors

Url : https://doi.org/10.3390/s26144503

Campus : Amritapuri

Center : Humanitarian Technology (HuT) Labs

Department : Electronics and Communication

Year : 2026

Abstract : Around 1.5% of the global population is suffering from speech impairments; the major causes for this are cerebral palsy and ALS, and the only way for these individuals to communicate is through Augmentative and Alternative Communication (AAC). These systems are either electronic or non-electronic. Based on new study developments, electronic methods, such as Brain–Computer Interaction (BCI) and eye-gaze-based communication, are assessed as the best choices, but they have their own limitations, incorporating limited adaptability to changing conditions, such as setup variations and user fatigue, which reduces the system‘s robustness. Our previous study, Netravad, shows potential for addressing these gaps, but it lacks multilingual support and will not yield the same results under changing lighting conditions. This study, Thrivaad, provides multilingual support and text prediction and integrates optimized deep learning to accurately capture eye movements even in varying environmental lighting conditions. Thrivaad uses eye movements as input from a webcam, and the Optuna-optimized YOLOv5 model is used to detect the eye direction accurately. Then communication is established in English, Malayalam, and Hindi. The text-prediction feature of this system improves communication by reducing the number of eye gestures required to form a message. This study included a total of 60 participants across three age groups with 35,263 eye-sign images collected. With this data, the YOLOv5 model is trained and then optimized by Optuna. The proposal system provides accurate eye direction, text prediction, multilingual support, and improved adaptability to changing conditions for eye-based AAC.

Cite this Research Publication : Rajesh Kannan Megalingam, Sakthiprasad Kuttankulangara Manoharan, Dhilna Cheriyan Manjooran, Dhanaraj Kamble, Thrivaad: A Multilingual, Predictive Eye-Sign-Based AAC System Powered by Optimized Deep Learning, Sensors, MDPI AG, 2026, https://doi.org/10.3390/s26144503

Admissions Apply Now