Publication

Abstract :

India is one of the countries with an increasing deaf population. The needs and problems of deaf communities are not adequately addressed. The majority of the previous approaches on Indian Sign Language (ISL) Recognition are based on convolutional neural networks (CNNs), whose performance depends on factors like lighting and the clothing of the sign language performer. The facial key points along with arm and body key points play a significant role in interpreting the ISL. The proposed approach identifies the facial key points and other body key points using the OpenPose model to generate pose video, which removes information unfavorable to CNN networks. The generated pose video is used by the pre-trained CNN model to extract spatial features and recurrent neural networks (RNNs) to extract temporal features necessary for ISL interpretation. We have used pre-trained models like Inception V3, DenseNet121 and MobileNetV2 to extract spatial features along with GRU, LSTM and bidirectional LSTM networks to extract the temporal features. Our model using the pre-trained MobileNetV2 model combined with GRU gave an accuracy of 99%. We also extended our approach using a combination of pose videos with background information.