Publication

Abstract : Sign Language Recognition (SLR) is a key task for enabling seamless communication between deaf and hearing individuals. Traditional recognition systems often rely on frame-level analysis, which may not fully capture the spatial and temporal nuances of SL gestures. We present NST-MRN, a novel Spatio-Temporal Memory Routing Network, for isolated dynamic SL word recognition. Instead of processing input data sequentially, as in most existing work, NST-MRN emulates a human-like cognitive system and selectively routes important motion and posture information within a memory-enhanced spatiotemporal system. The framework starts with a module that selects the crucial frames and motion regions to extract salient spatio-temporal information from the optical flow and skeleton data. Then, a dual-stream encoder processes the skeleton motion via a Graph Convolutional Neural Network, and the motion in focus via capsule-inspired dynamic encoders. Both are routed via a Memory Routing Layer, which captures the important features to connect key transitions in sign gestures. A temporal token aggregator then routes features to compressed representations for classification. We demonstrate the architecture’s evaluation on WLASL-100, Include-50, and SMILE-DSGS datasets, with significantly increased accuracy and robustness.