Back close

Optimized Multi-Modal Conformer-Based Framework for Continuous Sign Language Recognition

Publication Type : Journal Article

Publisher : Institute of Electrical and Electronics Engineers (IEEE)

Source : IEEE Open Journal of the Computer Society

Url : https://doi.org/10.1109/ojcs.2025.3564828

Campus : Amritapuri

School : School of Computing

Center : AmritaCREATE

Year : 2025

Abstract : This study introduces Efficient ConSignformer, a novel framework advancing Continuous Sign Language Recognition (CSLR) by optimizing the Conformer-based CSLR model, ConSignformer. Central to this advancement is the Sign Query Attention (SQA) module, a computationally efficient self-attention mechanism that enhances both performance and scalability, resulting in the Efficient Conformer. Efficient ConSignformer integrates video embeddings from dual-modal CNN pipelines that process heatmaps and RGB videos, along with temporal learning layers tailored for each modality. These embeddings are further refined using the Efficient Conformer for the fused data from two modalities. To improve recognition accuracy, we employ an innovative task-adaptive supervised pretraining strategy for Efficient Conformer on a curated dataset of continuous Indian Sign Language (ISL). This strategy enables the model to effectively capture intricate data relationships during end-to-end training. Experimental results highlight the significant contributions of the SQA module and the pretraining strategy, with our model achieving competitive performance on benchmark datasets, PHOENIX-2014 and PHOENIX-2014 T. Notably, Efficient ConSignformer excels in recognizing longer sign sequences, leveraging a computationally lightweight Conformer backbone.

Cite this Research Publication : Neena Aloysius, Geetha M, Prema Nedungadi, Optimized Multi-Modal Conformer-Based Framework for Continuous Sign Language Recognition, IEEE Open Journal of the Computer Society, Institute of Electrical and Electronics Engineers (IEEE), 2025, https://doi.org/10.1109/ojcs.2025.3564828

Admissions Apply Now