Sign Language Recognition and Translation: A Multi-Modal Approach Using Computer Vision and Natural Language Processing (2023)

Abstract: Sign-to-Text (S2T) is a hand gesture recognition program in the American Sign Language (ASL) domain. The primary objective of S2T is to classify standard ASL alphabets and custom signs and convert the classifications into a stream of text using neural networks. This paper addresses the shortcomings of pure Computer Vision techniques and applies Natural Language Processing (NLP) as an additional layer of complexity to increase S2T’s robustness.

## introduction globally, sign language is one of the main languages for those who cannot communicate verbally. despite its global presence, not many people understand it or use it. in 2020, 48 million people in the united states alone experience some form of hearing loss, with less than 500,000 -about 1% -of them that drive sign language regularly @xcite . the world health organization (who) estimates that the number of individuals with hearing loss will affect nearly 2.5 billion by 2050 @xcite . with these setbacks, signers may find it challenging to communicate with other individuals not akin to their mode of communication. while mild hearing loss can be remedied with hearing aids and rehabilitation, these solutions may often be too expensive. individuals can alternatively learn sign language. hand gestures are a form of non-verbal communication used by individuals in conjunction with speech to communicate. with the increasing use of technology, hand-gesture recognition is considered an essential aspect of human-machine interaction (hmi), allowing the machine to capture and interpret the user's intent and respond accordingly. the ability to discriminate between human gestures can help in several applications that range from virtual and augmented reality to healthcare services @xcite . as technology becomes easier to use and accessible, many people can likely perform simple commands with computer devices, such as typing text and video streaming. to address the problem statements, we propose s2t -a solution to close the sign language knowledge gap by translating simple hand gestures into text. ## related work this research will explore nlp techniques and apply them to s2t to enhance the translation quality after making prior classifications in computer vision. we know that the research field combining nlp and asl is limited. however, it is noted that nlp can be applied to asl applications when provided with some consumable input, such as text. in the field of nlp, immense research has been put into autocorrect, context awareness, and machine translation. since s2t can be broken into two parts (autocorrect and machine translation), we treat each part as an individual entity. autocorrect algorithms can vary in performance depending on their use case. however, they all follow a similar pattern by cross-referencing an accurate corpus to identify misspelled words. textblob is a standard open-source library launched in 2013 and has been widely used as a standard autocorrect tool @xcite . a study on textblob shows that it can correct 54.6875% of the mistakes in a given prompt. this low score can be due to textblob's over-correcting behavior and lack of information to correct it to the target word @xcite . there are also many machine translation algorithms and architectures that each perform best depending on the specific application. transformer models commonly show great success and have been a standard in many nlp tasks since google introduced them in 2017 @xcite . ## sign-to-text v2 s2t is equipped with computer vision techniques to translate sign language into text. we propose nlp as a second layer of data processing to enhance translation accuracy and introduce an extra translation feature to make the program more accessible. this additional layer will address the main drawbacks of computer vision as a standalone solution. ## conclusions and future work this paper proposes a multi-modal approach to improve sign language recognition and translation by combining computer vision and nlp techniques. by applying autocorrect as a fail-safe for computer vision classification, our team was able to fix 63.25% of the errors present in our dataset, which beats the baseline model by 9.94%. this improvement in word correction provides the machine translation layer to perform better as it can retain the context closest to the intended meaning. however, the nmt model implemented in this study performed slightly subpar compared to the original transformer for english-to-french translation from different datasets. the evaluations conducted for smt also show poor performance on the selected database. more extensive tuning and training on perhaps another corpus, such as those from past wmt conferences or opus, would benefit all methods selected here. this may also align the results of our implementation closer to those of related works utilizing the same architectures. as mt relies on the results of autocorrect, our plan plans to investigate more into improving the implementation of autocorrect. the root of misclassifications primarily comes from the results of computer vision first. while these misclassifications are due to the similarity between each gesture, not all gestures are utterly similar. this suggests that autocorrect can benefit from emphasizing weights for each classification group. by applying an additional bias per classification group, autocorrect can achieve increased correction accuracy overall. further improvements to autocorrect focus on an improved method of context awareness. the current implementation uses the spacy-contextualspellcheck pipeline. while it already improves upon standard autocorrect algorithms, the overall performance is still not substantial enough to be reliably used. our team researched using the viterbi algorithm to improve spacy by better determining the best corrections using part-of-speech tagging and hidden markov models. we can further enhance spacy by directly implementing a bert model step into the pipeline, allowing for more accurate predictions. despite mt results in this work underperforming, we are looking to merge the sequencing capabilities of the attention-based neural network and the purely linguistic nature of the statistical approach to improve translation quality. our future work seeks to leverage these approaches into a confidence-driven hybrid approach -justifying nmt outputs and resolving tokens estimated to have high uncertainty through smt @xcite .