Sign and Search: Sign Search Functionality for Sign Language Lexica (2021)

Abstract: Sign language lexica are a useful resource for researchers and people learning sign languages. Current implementations allow a user to search a sign either by its gloss or by selecting its primary features such as handshape and location. This study focuses on exploring a reverse search functionality where a user can sign a query sign in front of a webcam and retrieve a set of matching signs. By extracting different body joints combinations (upper body, dominant hand’s arm and wrist) using the pose estimation framework OpenPose, we compare four techniques (PCA, UMAP, DTW and Euclidean distance) as distance metrics between 20 query signs, each performed by eight participants on a 1200 sign lexicon. The results show that UMAP and DTW can predict a matching sign with an 80% and 71% accuracy respectively at the top-20 retrieved signs using the movement of the dominant hand arm. Using DTW and adding more sign instances from other participants in the lexicon, the accuracy can be raised to 90% at the top-10 ranking. Our results suggest that our methodology can be used with no training in any sign language lexicon regardless of its size.

## introduction sign language lexica are a valuable source for people learning sign languages, teachers and parents who need to communicate in signs with their deaf children as well as researchers studying the languages in question. these lexica allow the user to submit a query containing a unique identifier that by definition refers to a sign (commonly referred to as gloss) and retrieve a video or an image of that sign. in addition to this functionality, some lexica let the user define the formal parameters of the target sign (i.e. its location, handshape, or movement) and retrieve all the signs that contain these features. it is then at the users' discretion to view all the provided signs and select the desired ones. these search functionalities are particularly useful as sign languages, contrary to spoken languages, do not have any unified notation system for sign representation. even though a sign search functionality which is based on formal parameters is a userfriendly option in sign language lexica, it still requires manual annotation. dictionary compilers have to manually link these values to the individual videos of signs. this is a time consuming and prone to errors task and, as @xcite discusses, it is the reason why only a few of such dictionaries exist to date. more importantly, according to zwitserlood, these dictionaries are unidirectional "giving only signed translations of words from a spoken language in a one-to-one relation" @xcite . furthermore, as the retrieved results contain only the parameters selected by the user, the signs are presented in no particular order. in this paper, we describe a methodology and its experimental results for multi-directional search functionality for sign language lexica. our proposed method, extending on previous efforts by @xcite and @xcite , utilizes either the uniform manifold approximation and projection for dimension reduction (umap) technique or the dynamic time warping (dtw) algorithm to measure the distance between a query sign and all the signs in a lexicon. both techniques require no training, thus making our methodology applicable to any sign language. requiring training would add further obstacles to making dictionaries widely available, similar to how the need for manual annotation is limiting dictionary availability. finally, these methods have been compared to two other techniques namely principal component analysis (pca) and euclidean distance. the paper is organized as follows: in section 2 we discuss the research which has been conducted in relation to search functionality for sign language lexica or has the potential to be applied in that domain. in section 3 we describe our methodology regarding the extraction of the body joint coordinates as well as the methods and algorithms compared in this study. in section 4 we present the results of our experiments. we discuss them in section 5 and conclude and motivate future research in section 6. ## related work over the last decade, many research projects have examined the use of computer-vision techniques to allow a user to search a sign in a database or lexicon by performing it in front of a camera or sensor. @xcite have developed a system for semi-automatic search functionality. in their system, a user marks the start and end frames of a sign and denotes whether the sign is one-or two-handed. consequently, the system detects the hands on the basis of skin color and motion. the user can correct, if needed, the detected hand locations and pass the query to the system. using dynamic time warping their approach computes the similarity between the query sign and all the signs in the database. their results suggest a 78% accuracy on the top-10 retrieved signs on a 1113 sign lexicon. while the accuracy rate is high enough, the user still needs to indicate the handedness feature (one-or two-handed) as well as the duration of the sign. additionally, the data-set used in this study has been recorded under studio conditions posing the question of applicability on noisy real-life conditions on the video query. @xcite have used dynamic time warping to match a sign on an american sign language dictionary. using microsoft's kinect they detect the hand positions and perform sign matching. their results suggest an accuracy of 77.3% on the top-50 retrieved signs. a major advantage over @xcite implementation is that this system does not require the intervention of the user. @xcite have developed a framework that analyzes handshape, orientation, location, and motion trajectories to recognize 350 asl signs. by passing the extracted features into hidden conditional ordinal random fields (hcorf) they achieve a top-1 accuracy of 93.3% and a top-5 accuracy of 97.9%. vidalón and martino (2016) have created a system for brazilian sign language recognition using dynamic time warping, a nearest-neighbor classifier and kinect. on a data-set of 107 signs, they have reported an accuracy of approximately 98%. a major drawback of their results is the fact that their data-set is user-dependent. the majority of the aforementioned studies use either a depth sensor or computer-vision techniques. these techniques primarily rely on color and motion detection algorithms, as feature extraction methods, which imposes additional problems. such techniques can be prone to errors and most importantly require studio conditions in order to predict the required features such as the face and the hands. while such conditions can be true for the videos in a lexicon they cannot be assumed for the query videos. searching a lexicon can be done in any possible space and lighting conditions, thus it is important that the technique used to capture the required features to be as much as inclusive as possible. in @xcite presented a framework for multi-person 2d pose estimation, openpose. this framework can efficiently detect body, foot, hand and facial key-points from a simple rgb video or picture. its high accuracy, performance and easy implementation make it the ideal framework to parse sign language and gestural videos. its output consists of multiple json (or differently formatted) files containing all the pixel x, y coordinates of the body, hand and face joints. most studies use openpose to pre-process the videos and use its output to further train or compare machine and deep learning models. @xcite have used openpose as well as dtw and nearest-neighbor algorithm to perform classification of six gestures. their results suggested an accuracy of 77.4%. most recently, @xcite have used openpose and dtw to predict a sign recorded using a webcam from a 100 signs lexicon. their method predicted the matching sign with an 87% and 74% accuracy at the top-10 and top-5 retrieved signs by using the path of the dominant hand's wrist. this study extends on previous efforts for efficient sign ranking for sign language lexica by: another limitation posed in our study is that sign language lexica do not often contain multiple instances of a particular sign. while various studies using deep learning techniques have shown high accuracy in predicting different signs @xcite @xcite , they cannot be used in our case. these techniques often require vast amount of data in order to be trained which might not be available on all sign language lexica. our main goal is to develop a system that can be easily used in any sign language lexicon regardless of the amount of data in it and most importantly the language itself. however, in this study, we explore the possibility of having a few additional sign instances in the lexicon and their potential benefit to successful sign retrieval. ## data-sets and methods in this section we describe the data pre-processing as well as the data-sets and methods used in this study. ## results highest accuracy is apparent at a top-50 level at 95% using the umap algorithm and the joints of the dominant hand arm. furthermore, top-20 rank shows an adequate accuracy at 80% again using umap and the dominant hand arm coordinates. figure 2 presents the visualizations of the umap algorithm for each of the skeletal condition for one participant. the results of the calculated euclidean distances on the original high-dimensional space show an adequate accuracy of approximately 68% at the top-50 rank in both dominant hand arm and wrist data-sets. principal component analysis (pca) performed, on average, better using the wrist coordinates and showed the highest accuracy at the top-50 at approximately 41%. dtw showed the highest accuracy at 79% at top-50 rank using the data of the dominant hand wrist and 77% using the dominant hand arm. on average, dtw had the best accuracy at around 70% at the top-20 retrieved signs regardless of the skeletal condition used, with a slight increase noticed using the dominant hand arm data. figure 3 presents the top-1 and top-10 accuracy levels using dtw and umap that have been computed by incrementally adding other participants' data in the lexicon. it can be observed that by adding more sign instances from 6 different participants, the accuracy reached a 90% level at the top-10 retrieved signs using dtw and the upper body and dominant hand wrist data. furthermore, a raise of approximately 15% can be noticed at the top-1 rank on dtw using the upper body and dominant hand wrist joints by adding the data of just 2 participants (figure 3a ). on the other hand, umap did not show any adequate raise at the top-1 accuracy regardless of the added participants' data and skeletal condition. however, an increase, of approximately 35%, can be seen at the top-10 ranking level using the data of 2 participants (figure 3b ). ## discussion in this study we have investigated the use of openpose and four different implementations as distance metrics for an efficient ranking pipeline to retrieve matching signs from a sign language lexicon. the results demonstrated that, on a large vocabulary of 1200 signs, such a task can be achieved with an adequate accuracy rate using the uniform manifold approximation and projection for dimension reduction (umap) or dynamic time warping (dtw) and the dominant arm joints' coordinates. with regard to the visualizations produced by the umap algorithm, a few observations can be made. firstly, by using the upper body joints umap produces discrete clusters. these clusters seem to reflect an abstract representation of the movement of each sign. this behavior has been observed also using the dominant hand arm coordinates, although it is more noticeable using all the upper-body joints. we have observed that signs that have similar movement but different handshapes are grouped close to each other. however, special consideration needs to be made when viewing the visualizations produced by umap, especially the one using the upper body joints. the distances between the noticeable clusters, as well as their size relative to each other, do not hold any particular meaning. this is because of the use of local distances by the algorithm when constructing the graph. however, our results using the euclidean distances on the high-dimensional space suggest that umap preserves the original global distances. finally, it is worth mentioning that dtw performs equally well irrespective of the skeletal condition used at around 70% at the top-20 rank. overall, it produces the most stable and consistent accuracy at the top-10 retrieved signs at around 65%. this accuracy level can be further raised reaching 90% by adding 6 more sign instances (from different signers) into the original lexicon. this attribute can be further explored by lexicographers by asking users of their lexica to submit their own versions of signs. this process can significantly boost the performance of dtw in its ability of retrieving the closest matching sign. a broad benefit of using such an algorithm is the fact that lexica compilers do not need to re-train any model if more signs or sign instances are added to their lexica. in general, while our accuracy does not reach the ones reported by @xcite (77.4% top-1 and 74% top-5 accuracy respectively) using similar algorithms and frameworks, our methods have been applied on a far larger lexicon (1200 signs instead of 6 and 100 respectively). as a result, we provide a better approximation on how these methods can actually be used in real world lexica. ## conclusions to sum up, we have obtained satisfactory results demonstrating that umap and dtw, in combination with the pre-trained pose estimation framework openpose, can be used as an efficient sign ranking and retrieval system. our method can effectively be applied to any sign language lexicon without any training process involved. to the best of our knowledge, this is the first study using umap as a dimensionality reduction technique within the sign language domain and showcasing the strength of such algorithm compared to other implementations. future work will focus on exploring additional deep learning implementations for an efficient handshape and pose recognition. their use, as well as supplementary hyper-parameter optimization for the techniques used in this study, could lead to an increase in accuracy. whilst as we argued that for the availability of dictionaries it will be good to focus on zero training approaches, in future work we intend to run comparative analysis to understand the impact of training based approaches on performance. we propose that further research should also be undertaken in order to assess the application of our method on different datasets and languages. on a wider level, the techniques used in this study could be further explored to measure variation in different sign languages. the results from the use of umap and dtw on a large vocabulary suggest that these techniques might be well suited for variation measurement tasks, broadening their use beyond the search functionality for sign language lexica.