Hand recognition technologies to promote inclusion: automatic interpretation of Chilean sign language

Abstract

There are individuals who endure hearing or speech impairments, resulting in undeniable communication challenges in their lives. These individuals can communicate through sign language; however, this language is not well-known to a large part of the population that uses verbal or written means of communication. This work focuses on using hand recognition technologies to interpret sign language, specifically adopting Chilean Sign Language, by employing computer vision techniques and Machine Learning (ML) with various object recognition libraries. These were utilized alongside a dataset of 2D hand images, organized into a CSV file. Mediapipe was also used to detect key points on the hands, achieving high accuracy in gesture detection.

OpenCV, Roboflow, and Mediapipe Hands were used to detect characteristic points on the hands, thus identifying the left or right hand and allowing processing to continue. The specialized Mediapipe Hands module within the Mediapipe framework uses pre-trained convolutional neural network models to identify 21 key points on the hands, facilitating real-time gesture and hand detection, which is essential for detecting hands and working with them in static images or video. OpenCV was employed to capture images per second in RGB color and to render static images. Roboflow was used to train the dataset to recognize Chilean Sign Language.