Automatic Gloss Dictionary for Sign Language Learners (2022)

Abstract: A multi-language dictionary is a fundamental tool for language learning, allowing the learner to look up unfamiliar words. Searching an unrecognized word in the dictionary does not usually require deep knowledge of the target language. However, this is not true for sign language, where gestural elements preclude this type of easy lookup. This paper introduces GlossFinder, an online tool supporting 2, 000 signs to assist language learners in determining the meaning of given signs. Unlike alternative systems of complex inputs, our system requires only that learners imitate the sign in front of a standard webcam. A user study conducted among sign language speakers of varying ability compared our system against existing alternatives and the interviews indicated a clear preference for our new system. This implies that GlossFinder can lower the barrier in sign language learning by addressing the common problem of sign finding and make it accessible to the wider community.

## introduction unlike most language systems, which are composed of their written and spoken forms, sign languages (e.g., american sign language (asl) and the australian auslan language) used by the deaf or hard-of-hearing (dhh) community are represented by the rich inputs including facial and gesture movements. as of the year 2020, 430 million people worldwide have developed hearing lossthat is, one in every ten people-and it is estimated that this number may increase to 700 million by 2050 @xcite . sign languages are also used by people suffering the loss of ability to speak (e.g., aphasia) or brain stroke. they are spoken by individuals with various relational connections to sign language speakers, e.g., family members or co-workers. additionally, a substantial and growing number of people are learning a sign language as a second language, e.g., among u.s. university students @xcite . despite the efforts made in building tools to support their learning @xcite @xcite @xcite , many sign language learners have limited means of seeking assistance, and are restricted to class offerings or relying on other experienced sign language speakers. it is therefore increasingly important to support the sign language learner community to facilitate better education and communication. as a fundamental tool in language study, a dictionary is more than a tool to assist sign language learners in searching unfamiliar words. the rich content present in current online dictionaries (e.g., example pronunciation recordings and visual materials) also provide positive feedback to foster the learner's understanding and proficiency in the target language @xcite . most existing sign language dictionaries (e.g., aslsearch @xcite , handspeak @xcite , and signing savvy @xcite ) are text-based and centered on one spoken language, with signs presented in an alphabetical order of their corresponding gloss, i.e., the spoken language counterpart. this does not serve the important scenario when someone encounters an unfamiliar sign and does not know its spoken language translation. another issue with the text-based dictionary is the fact a one-toone correlation between sign and spoken language words does not always exist, and no standard convention exists for handling these discrepancies. the absence of these types of dictionary for sign language learner is due to the difficulty of processing visual input and the lack of intuitive alphabetics assumed in most language dictionaries. an early effort made towards a sign-centric dictionary is @xcite where researchers use pre-defined handshapes (finger poses) to formalise the signs so that they can be arranged similar to a conventional dictionary. follow-up work @xcite @xcite parameterised the signs by key properties (e.g., handshape, position of hands, and whether the sign involves repetitive movement) to make a filtering-based search system. in addition, @xcite used the microsoft kinect to collect human body movements from the sign language speaker and match the performed signs against the database. modern advancements in deep learning algorithms enable processing of unstructured video inputs, and these algorithms have been applied to sign language. progress has been made in identifying isolated @xcite al. (2020a,c); albanie et al. (2020); sincan and keles (2020); momeni et al. (2020) ) or continuous signs ( li et al. (2020b); zhou et al. (2021); bull et al. (2021); duarte et al. (2021); chen et al. (2022) ) from a video. this presents opportunity to develop a dictionary system, which accepts direct video inputs from a user performing a sign, and attempts to return the meaning of that sign. one of the early attempts on such videobased system is @xcite where the author discussed some characteristics in the design and evaluation metrics regarding the user satisfaction. notably, the work did not build an actual automatic recognition technique and the users are only presented with a predetermined set of results during the study. in this paper, we present the platform of gloss-finder, our new video-based sign dictionary, where users directly provide videos of the target sign by performing it to their webcam or via uploaded clips, and the system will retrieve matched signs without any extra input. to the best of our knowledge, it is the first attempt of user study with a functioning system built. the study identifies some key considerations in designing for this specific sign language context. it also verifies sign language learners' frustration when using previous sign dictionaries, either due to the steep learning curve or the poor quality of results. ## conclusion we construct, to the best of our knowledge, the first automatic sign dictionary digesting direct video capture as its inputs. our user study validates the improved usability from the new system. the participants describe it as less demanding to learn in comparison to the existing parameter-based systems. retrieved results are said to be more accurate and able to accommodate the varying video capture quality. enriched results include example videos and explanations are agreed to largely help the user in correctly locating and refining the search. overall, the reported success rate in reaching the searched sign is on average 66% from glossfinder, significantly surpassing the benchmarks. we also conduct analysis to compare different views for presenting the results. it is favored by the participants for the system to include more examples of varieties even at the cost that less glosses can be shown in a single page. our study strengthens the belief that the sign language dictionary design should be visual-based to imitate the practical form of actual sign language teaching and learning. we hope it can also motivate the related research to make sign language learning increasingly accessible to a broader community. as one of the early attempts in building such system, we notice some limitations in the current study: