Publication

Abstract : Sign Language Recognition (SLR) enables communication between deaf and hearing individuals. Usually, SLR systems often depend on single-modality inputs, viz., sensor data, RGB videos, etc., which do not address issues such as signer variability and background, affecting the robustness and generalization of these systems. To address these challenges, we propose a novel framework, viz., Visual, Pose, and Motion-based Sign Language Recognition (ViPoMo-SLR), which incorporates RGB video frames, optical flow, and skeletal data—three complementary modalities. RGB frames are extracted to represent spatial features, while optical flow captures motion dynamics, and skeletal data provides hand and body joint positions. We select the crucial frames independently from both the RGB and skeleton streams to focus on the most informative segments. The extracted features are subsequently fused using a fusion technique and processed through a Bidirectional Long Short-Term Memory (BiLSTM) network to model temporal dynamics. Our empirical analysis shows that ViPoMo-SLR significantly outperforms unimodal methods, offering enhanced accuracy, real-time performance, and a scalable solution for robust SLR.