Publication

Abstract : Sign Language is indispensable for communication for speech or hearing-impaired individuals, but is little known to the rest of the general public. Social isolation and communication deficits are common among signers because of unawareness. For filling this gap, this study puts forth a sensor-based system that recognizes Sign Language gestures as words, to enable efficient communication between non-signers and signers. The system records time-series data from flex and motion sensors embedded in a wearable platform, tracking finger articulation and hand movement that record every dynamic gesture at high fidelity and consistency. Four deep learning models: Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), a heatmap-based Convolutional Neural Network, and a ranking-based gesture matching technique, are utilized to achieve robust and effective gesture recognition. Model performances were evaluated on accuracy, precision, recall, and F1 score. Both LSTM and GRU outperformed in detecting intricate temporal patterns of gestures. The heatmap-based method was suitable for encoding spatial correlations among sensor inputs. Additionally, the ranking-based method offered a lightweight, efficient, and simple solution suitable for real-time low-resource deployment in resource-constrained environments. As compared to vision-based approaches, the camera-free sensor-based solution achieved herein is not privacy-concerned and light, and less noisy. The system is thus stable, scalable, cost-effective, and deployable in real-world settings.