Investigating Dictionary Expansion for Video-based Sign Language Dictionaries (2025)

Abstract: Like most languages, sign languages evolve over time. It is important that sign language dictionaries’ vocabularies are updated over time to reflect these changes, such as by adding new signs. However, most dictionary retrieval methods based upon machine learning models only work with fixed vocabularies, and it is unclear how they might support dictionary expansion without retraining. In this work, we explore the feasibility of dictionary expansion for sign language dictionaries using a simple representation-based method. We explore a variety of dictionary expansion scenarios, e.g., varying number of signs added as well as amount of data for these newly added signs. Through our results, we show how performance varies significantly across different scenarios, many of which are reflective of real-world data challenges. Our findings offer implications for the development & maintenance of video-based sign language dictionaries, and highlight directions for future research on dictionary expansion.

## introduction dictionaries are important resources for sign languages, offering a way to document the many different signs comprising the intricate vocabularies of these languages. in addition to documentation, dictionaries are particularly helpful to novices or language learners, allowing them to easily look up signs they are unfamiliar with. it is crucial that these dictionaries have mechanisms to stay up-to-date with changes in the language, such as the creation and adoption of new signs by signing communities. here we consider the problem of dictionary expansion, where the vocabulary of an established sign language dictionary is updated to incorporate new signs. a key factor that makes sign language dictionaries (and thereby their expansion) unique is that sign languages are visual-manual languages. this means that sign language dictionaries typically use video entries to represent each sign in their vocabulary, and need video-based approaches for users to look up signs and query the dictionary. in a videobased dictionary, a user demonstrates a sign to a camera, and the dictionary returns a ranked list of entries from the dictionary that might correspond to that sign. recent advancements in sign language datasets and modeling show exciting promise in making video-based dictionary retrieval a reality @xcite . for example, many state-ofthe-art isolated sign language recognition models (which are trained to recognize single signs from vocabularies of 2k+ signs) currently achieve recall@10 above 90% (i.e., queried sign is in top-10 results) -this high performance makes it possible to deploy these models for dictionary retrieval. however, a key limitation of almost all such modeling approaches is that it is unclear how they might support dictionary expansion. the technologies underlying most video-based retrieval methods are often machine learning classifier models trained on a fixed vocabulary, where each sign corresponds to a label in a fixed label set. incorporating new vocabulary into these models would typically require retraining them from scratch -however, this is can be computationally expensive and relies on the availability of a sufficient amount of high-quality training data, which may be difficult in practice. it also means that third-parties can't adapt existing dictionaries to their needs (e.g., representing local dialects or specialized lexicons). an ideal approach would support vocabulary expansion without retraining the existing model, but this class of approach remains largely unexplored. in this work, we investigate the feasibility of dictionary expansion without retraining for videobased sign language dictionaries. to overcome fixed vocabulary constraints encountered with current classification approaches, we propose a simple method that instead uses the representations learned by deep learning models for dictionary retrieval, rather than just their final classification layer. doing so allows us to work with unseen signs through nearest neighbor-type approaches. we explore how this type of approach performs across a range of circumstances, simulating dictionary expansion in ideal settings as well as more realistic and challenging scenarios. we separately evaluate the performance of expanded models on core signs (those in the original dictionary) as well as newly added signs. we find that our method performs well when adding a small number of new signs (∼ 1 -100), each with many examples (∼ 15 per sign), to a dictionary with an existing large vocabulary -maintaining performance compared to the pre-expansion for both new and old signs. however, performance degrades substantially when we constrain the number of examples for new signs, or attempt to add many new signs to the vocabulary, which we argue are likely scenarios for realworld dictionary expansion. our work is the first to systematically explore computational challenges dictionary expansion, providing directions for future methods development grounded in real-world needs of sign language dictionaries and their users. these opportunities for future research not only offer a space for technical discovery, but also further the exploration of sign languages as languages @xcite . ## background sign languages and dictionaries. sign languages are visual languages, with each sign composed of distinct handshapes, movements, nonmanual markers, and other phonological features. there are over 300 sign languages in the world. our work focuses on american sign language (asl), which is the most common in north america. asl is culturally significant to deaf community in the continent, and is also learned by many as a second language @xcite . asl dictionaries are a valuable resource for this group. part of what makes dictionary retrieval in sign language challenging is that two different signs might share nearly all the same phonological features (which can cause them to look visually similar) but have very different meanings. for example in asl, sorry and please only differ in handshape, sunrise and sunset only differ in movement (examples of minimal pairs). dictionary retrieval methods need to be robust to such dense lexical neighborhoods. existing sign language dictionaries support querying in english, through a set of descriptive features, or by video demonstration. searching by english word or gloss (e.g., signingsavvy, lifeprint) is a valuable approach for those looking to learn a sign from an english word (i.e., englishto-asl translation). however, users cannot leverage these dictionaries to look up the meaning of an unfamiliar sign (i.e., asl-to-english). allowing users to navigate dictionary search directly in sign languages is complicated as sign languages do not have standardized written forms. one approach is feature-based search (e.g., handspeak & @xcite ) that allows users to search by describing features of a sign (e.g., handshapes, movements). video-based search allows users to search by demonstrating a sign (returning all dictionary entries that may match the search video). video-based dictionaries also allow native signers to navigate dictionaries in their primary languagetheir development is an important avenue of work. ## results overall, we find that the performance of our dictionary expansion method varies greatly across different dictionary settings -figure 2 provides a snapshot of our results. in the following subsections, we walk through the results in each setting and our corresponding analysis. the ranking of settings was consistent across all metrics so we report dcg, but full tables reporting each metric for each experiment can be found in the appendix. ## discussion in this work, we explored sign language dictionary expansion across a variety of settings. our results highlight the feasibility of expanding sign language dictionaries using feature representation-based approach, and demonstrate that performance is contingent on the type and quantity of data available for new vocabulary. given enough examples of a sign, we show that incorporating it into an existing dictionary is quite feasible. this is valuable for when small changes need to be made to a dictionary quickly (e.g., recording a new variation or documenting a newly emerging and rapidly adopted sign, like the sign for 'coronavirus'). when adding multiple signs with multiple examples, we show that models are generally robust, albeit at a slightly lower performance than when just a single sign is added. this can inform the frequency at which we retrain models in response to changing vocabularies. a comparison of numbers from the small dictionary experiments (100 core signs) to the large dictionary experiments (1000 core signs) suggest models may need to be retrained at more frequent intervals initially, but less often once they've develop robust representations. on the other hand, we find that performance degrades when added new signs that have only a limited number of samples. as we would expect, this is the most difficult, but also the most realistic, expansion scenario. this illuminates an important direction for future research: examining new methods for incorporating signs into a dictionary in data-constrained scenarios. we focused on classifiers trained using deep learning in this work; however, alternative training approaches (e.g., meta learning) may result in more robust feature representations for core and added signs. we encourage researchers to take up this line of inquiry. looking forward, we emphasize the importance of addressing dictionary expansion for sign languages. compared to written languages, existing sign language dictionaries represent only a small fraction of the true language lexicon. while many crowdsourcing and documenting initiatives are underway, gathering a snapshot of sign languages that is representative of the many different regional and contextual variations continues to be a challenge. in addition to variations in everyday vernacular, different fields of study (such as stem disciplines) frequently produce new signs to represent new concepts. dictionary expansion then offers a crucial pathway to building sustainable and scalable sign language dictionaries. our results also offer interesting implications for the maintenance of current crowdsourced dictionaries. examples available for each sign are expected to grow over time with contributions from users. a representation-based approach like ours (that works with any number of signs and any number of examples per sign) aligns well to the fluctuating vocabulary size and data availability encountered in these dictionaries. initially, dictionary administrators could work to incorporate a sign with any available sample, and then switch to centroid for these added signs as more examples are collected. while this results in poor added sign performance initially, it allows the sign to be incorporated into the video-based dictionary and solicit further interaction and contributions from dictionary users. to maximize initial performance, dictionary administrators could also work to sustain long-term contributors to record most, if not all, signs being added to the dictionary (as this shows best added sign performance with limited data). overall, our work outlines directions for sign language research aligned with community needs and real world settings and furthers the exploration of sign languages as languages in their own right. ## limitations in this work, we reported results on dictionary expansion using only an st-gcn model. while this architecture is frequently used in sign language recognition and has the benefit of being lightweight, it is not the only established approach. appearancebased approaches to sign language recognition (like i3d) might fare differently under expansion scenarios. exploring dictionary expansion across a variety of architectures as well as robustness of their learned feature representations could be an interesting direction for future research. to simulate dictionary expansion, we created core-unseen splits from asl citizen. this meant that many of the contributors in the unseen split had previously been seen by the model -while this is reflective of a real-world scenario (long-term dictionary contributors), using a completely new dataset for unseen vocabulary would have also allowed us to simulate a dictionary setting with a completely new contributor scenario for added signs. in our results, we note that the centroid approach outperforms the classifier even in the base dictionary setting -to the best of our knowledge, this has not been discussed in prior literature. while valuable, a core limitation of this approach is that the centroid representation requires multiple training examples to become reliable, which may be difficult leverage with data scarcity. our analysis surfaces large variance in dictionary retrieval performance across contributors (section 4.3), we do not investigate these disparities in depth. we hypothesize these disparities could be explained by variations in video quality for these different signers or biases in the underlying model (or both). disentangling these aspects and more systematically exploring dictionary expansion performance across different contributors is valuable direction for future work. we also highlight differences in dictionary expansion trends at smaller vs. larger core vocabulary sizes. however, our results are limited to comparing two specific vocabulary sizes (100 vs. 1000) -a more systematic investigation across different vocabulary sizes may have better unearthed overarching trends. additionally, with our smaller vocabularies we note sensitivity to lexical composition, but our analysis reports aggregates which prevents us from investigating sign-level nuances and conducting a linguistically-informed analysis. lastly, we position our work related to evolution of sign languages, but only focus on adding signs to dictionaries and not replacing or removing signs. while our method could be easily adapted to address these, further research is required to investigate impacts on overall dictionary retrieval performance.