Searching for fingerspelled content in A merican S ign L anguage (2022)

Abstract: Natural language processing for sign language video—including tasks like recognition, translation, and search—is crucial for making artificial intelligence technologies accessible to deaf individuals, and is gaining research interest in recent years. In this paper, we address the problem of searching for fingerspelled keywords or key phrases in raw sign language videos. This is an important task since significant content in sign language is often conveyed via fingerspelling, and to our knowledge the task has not been studied before. We propose an end-to-end model for this task, FSS-Net, that jointly detects fingerspelling and matches it to a text sequence. Our experiments, done on a large public dataset of ASL fingerspelling in the wild, show the importance of fingerspelling detection as a component of a search and retrieval model. Our model significantly outperforms baseline methods adapted from prior work on related tasks.

## introduction sign languages are a type of natural language which convey meaning through sequences of handshapes and gestures as well as non-manual elements, and are a chief means of communication for about 70 million deaf people worldwide. 1 automatic sign language technologies would help to bridge the communication barrier between deaf and hearing individuals, and would make deaf video media more searchable and indexable. automatic sign language processing has recently received growing interest in the computer vision (cv) and natural language processing (nlp) communities. yin et al. (2021) make several recommendations for the study of sign languages in nlp research, including greater emphasis on real-world data. most studies on sign language are based on data collected in a controlled environment, either 1 from https://wfdeaf.org/our-work/ in a studio setting @xcite or in a specific domain @xcite . the challenges involved in real-world signing videos, including various visual conditions and different levels of fluency in signing, are not fully reflected in such datasets. automatic processing of sign language videos "in the wild" has not been addressed until recently, and is still restricted to tasks like isolated sign recognition @xcite @xcite and fingerspelling recognition @xcite . in this work we take a step further and study search and retrieval of arbitrary fingerspelled content in real-world american sign language (asl) video (see figure 1 ). fingerspelling video search (fvs) for searching for sign language videos that include a fingerspelled query word/phrase. the sign language videos are untrimmed, i.e. they include regular signs in addition to fingerspelling, and are downsampled here for visualization. fingerspelling is a component of sign language in which words are signed by a series of handshapes or movements corresponding to single letters (see the appendix for the asl fingerspelling alphabet). fingerspelling is used mainly for lexical items that do not have their own signs, such as proper nouns or technical terms, and has an important place in sign language. for example, fingerspelling accounts for 12-35% of asl @xcite . since important content like named entities is often fingerspelled, the fingerspelled portions of a sign language video often carry a disproportionate amount of the content. most prior work on fingerspelling has focused on recognition @xcite , that is, transcription of a fingerspelling video clip into text. however, automatic recognition assumes that the boundaries of fingerspelled segments are known at test time, and may not be the end goal in real-world use cases. in addition, complete transcription may not be necessary to extract the needed information. fingerspelling search, such as retrieving sign language videos based on a query word, is a more practical task, and is an important component of general video search involving sign language. in addition to introducing the task, we address the research question of whether the explicit temporal localization of fingerspelling can help its search and retrieval, and how best to localize it. as fingerspelling occurs sparsely in the signing stream, explicit detection of fingerspelling could potentially improve search performance by removing unrelated signs. to this end, we propose an end-to-end model, fss-net, which jointly detects fingerspelling from unconstrained signing video and matches it to text queries. our approach consistently outperforms a series of baselines without explicit detection and a baseline with an off-theshelf fingerspelling detector by a large margin. ## related work in existing work on sign language video processing, search and retrieval tasks have been studied much less than sign language recognition (mapping from sign language video to gloss labels) @xcite and translation (mapping from sign language video to text in another language) @xcite . work thus far on sign language search has been framed mainly as the retrieval of lexical signs rather than fingerspelling. @xcite ; @xcite employ mouthing to detect keywords in sign-interpreted tv programs with coarsely aligned subtitles. tamer and saraçlar (2020a,b) utilize whole-body pose estimation to search for sign language keywords (gloss or translated word) in a german sign language translation dataset phoenix-2014t @xcite . all prior work on keyword search for sign language has been done in a closed-vocabulary setting, which assumes that only words from a pre-determined set will be queried. searching in an open-vocabulary setting, including proper nouns, typically requires searching for fingerspelling. some related tasks in the speech processing literature are spoken term detection (std) and queryby-example search, which are the tasks of automatically retrieving speech segments from a database that match a given text or audio query @xcite @xcite . in terms of methodology, our model also shares some aspects with prior work on moment retrieval @xcite @xcite , which also combines candidate generation and matching components. however, we incorporate additional task-specific elements that consistently improve performance. ## tasks we consider two tasks: fingerspelled word search (fws) and fingerspelling-based video search (fvs). fws and fvs respectively consist of detecting fingerspelled words within a given raw asl video stream and detecting video clips of interest containing a given fingerspelled word. 2 given a query video clip v and a list of n words w 1:n , fws is the task of finding which (if any) of w 1:n are present in v. conversely, in fvs the input is a query word w and n video clips v 1:n , and the task consists of finding all videos containing the fingerspelled word w. we consider an openvocabulary setting where the word w is not constrained to a pre-determined set. the two tasks correspond to two directions of search (video-→text and text-→video), as is standard practice in other retrieval work such as video-text search @xcite @xcite . ## model we propose a single model, fss-net (for "finger-spelling search network"), summarized in fig- • : text embedding, 2 : visual embedding. the colors correspond to different input fingerspelling sequences. as pictured, this is the training time model, where the pairing between text and video segments is known. at test time, the labels (colors) of the visual embeddings are unknown and we do not filter the proposals. ure 2, to address the two aforementioned search tasks. fss-net receives a pair of inputs-a raw asl video clip, and a written text sequence-and produces a score indicating the degree of match between the video clip and the text. the text is encoded into an embedding vector via a learned encoder. the visual branch of fss-net generates a number of fingerspelling segment proposals and each proposed visual segment is encoded into a feature space shared with the text embeddings. paired embeddings from both modalities are drawn towards each other in the shared embedding space during training. image encoding the input image frames are encoded into a sequence of feature vectors via an image encoder, which consists of the vgg-19 (simonyan and zisserman, 2015) convolutional layers followed by a bi-lstm. 3 we use raw rgb images as input, instead of signer pose as used in some prior work (tamer and saraçlar, 2020b,a) on sign language search, as estimating pose for hands is particularly hard for signing videos in the wild (see section 6 for details). temporal proposal generation suppose the visual feature sequence is f 1:t , where t is the number of frames in the video clip. the purpose of temporal proposal generation is to produce a number of candidate fingerspelling segments h(i 1:t ) = {(s i , t i )} 1≤@xmath0:t )| from f 1:t , where s i ,t i are the start and end frame indices of the i th proposed segment. below we use h as a shorthand for h(i 1:t ). here we adopt the strategy in @xcite , which is commonly used to generate proposals for action detection. briefly, the model assigns a probability p det of each proposal being fingerspelling. see @xcite for more details. we denote the detection loss as l det . note that the training requires known groundtruth fingerspelling boundaries. in the fingerspelling datasets we use here @xcite , the fingerspelling boundaries are already annotated, so no further annotation is needed. filtering a visual embedding is produced for each segment. we denote a labeled fingerspelling segment (shortened as fingerspelling segment below) as a tuple (s, t, w), where s, t and w represent the start frame index, the end frame index, and the written text it represents. a naive approach would be to use only the ground-truth fingerspelling segments p @xmath1, t i , w i )} 1≤@xmath2. however, this approach does not take into account the potential shifts (errors) that may exist at test time between the ground-truth and generated segment proposals. the embeddings produced by the fingerspelling encoder should be robust to such shifts. to this end, we incorporate proposals in forming positive pairs at training time. formally, let the set of time intervals from the temporal proposal generator be @xmath3, t i )} 1≤@xmath4. we sample k intervals from p t to form the set of generated fingerspelling segments: ## results and analysis 6.1 main results ## conclusion our work takes one step toward better addressing the need for language technologies for sign languages, by defining fingerspelling search tasks and developing a model tailored for these tasks. these tasks are complementary to existing work on keyword search for lexical signs, in that it addresses the need to search for a variety of important content that tends to be fingerspelled, like named entities. fingerspelling search is also more challenging in that it requires the ability to handle an open vocabulary and arbitrary-length queries. our results demonstrate that a model tailored for the task in fact improves over baseline models based on related work on signed keyword search, fingerspelling detection, and speech recognition. however, there is room for improvement between our results and the maximum possible performance. one important aspect of our approach is the use of explicit fingerspelling detection within the model. an interesting avenue for future work is to address the case where the training data does not include segment boundaries for detector training. finally, a complete sign language search system should consider both fingerspelling and lexical sign search. youtube deafvideo misc 16, 18, 20, 24, 32, 40, 60, 75}, according to the typical fingerspelling lengths in the two datasets. the positve/negative threshold of the anchors are 0.6/0.3 respectively. δ iou /δ is are 1.0/0.8 (chosen from {0.4, 0.6, 0.8, 1.0}). the fs-encoder and text encoder are 3-layer/1-layer bilstm with 256 hidden units respectively. the margin m, number of negative samples in n v and n w are tuned to be 0.45, 5 and 5. the model is trained for 25 epochs with adam @xcite at initial learning rate of 0.001 and batch size of 32. the learning rate is halved if the mean average precision on the dev set does not improve for 3 epochs. λ det in equation 4 is 0.1 (chosen from {0.1, 0.5, 1.0}). at test time, we generate @xmath5.7. β is tuned to 1 (chosen from {0.5, 1, 2, 3}).