Real-time isolated recognition of Chilean sign language using Deep Learning

Abstract

Sign language is a fundamental part of deaf culture, with its own linguistic and grammatical rules allowing perfect communication among its speakers (signers). In computer science, various automatic mechanisms have been studied to facilitate this communication, one being Sign Language Recognition (SLR), encompassing the complete process of tracking and identifying signs and converting them into words and expressions. This work develops a dataset for Isolated Chilean Sign Language Recognition (LSCh) by manually segmenting videos of deaf students, preprocessing them with a pose estimation tool, and classifying them using deep learning models based on Transformers. It also explores the impact of data augmentation on model performance and compares it with results from the Argentine sign language corpus (LSA64). Results show the importance of data augmentation in smaller datasets and demonstrate that a single model can learn from signs made by both left- and right-handed signers. Experimental metrics indicate an accuracy of 0.75 and 0.94, and recall of 0.74 and 0.94 for LSCh and LSA64 respectively, using the best-performing models (Mirror + Angle).