Publication

Abstract : ISLR (Isolated Sign Language Recognition) models face the challenge of optimizing accuracy while managing the limited computing resources available on the systems on which they are implemented. In this paper, we propose a new hybrid framework for the lightweight and efficient integration of MobileNetV2-based spatial features and transformer encoders for isolated sign word classification. When the model is given a 224×224 RGB image input, it generates a spatial convolutional encoding and compresses it to a lower-dimensional convolutional representation. It then passes the representation to the transformer encoders for contextual modeling. The global and recurrent (temporal) structures are managed by the first and second transformer encoder blocks, respectively. Representation refinement is achieved through a fusion of the shallow and feature concatenation. The model as a whole is comprised of approximately 4.2 million parameters, (less than) 353 million FLOPs, and 30 to 35 milliseconds latency per prediction on average. Most of the (computational) load is attributed to MobileNetV2. Most of the additional load is attributed to the transformer block’s computational, reliable, and real-time interactive sign classification.