WLASL - LEX : a Dataset for Recognising Phonological Properties in A merican S ign L anguage (2022)

Abstract: Signed Language Processing (SLP) concerns the automated processing of signed languages, the main means of communication of Deaf and hearing impaired individuals. SLP features many different tasks, ranging from sign recognition to translation and production of signed speech, but has been overlooked by the NLP community thus far. In this paper, we bring to attention the task of modelling the phonology of sign languages. We leverage existing resources to construct a large-scale dataset of American Sign Language signs annotated with six different phonological properties. We then conduct an extensive empirical study to investigate whether data-driven end-to-end and feature-based approaches can be optimised to automatically recognise these properties. We find that, despite the inherent challenges of the task, graph-based neural networks that operate over skeleton features extracted from raw videos are able to succeed at the task to a varying degree. Most importantly, we show that this performance pertains even on signs unobserved during training.

## introduction around 200 languages in the world are signed rather than spoken, featuring their own vocabulary and grammatical structures. for example the american sign language (asl) is not a mere translation of english into signs and is unrelated to the british sign language (bsl). their non-textual nature introduces many challenges to their automated processing, compared with purely textual nlp. research on sign language processing (slp) encompasses tasks such as sign language detection, i.e. recognising if and which signed language is performed @xcite and sign language recognition (slr) @xcite , i.e. the identification of signs either in isolation or in continuous speech. other tasks concern the translation from signed to spoken (or written) @xcite flexion: stacked major location: head minor location: mouth movement: backandforth selected fingers: index signtype: symmetric neural network classifier figure 1 : we annotate asl sign videos with their corresponding phonological information and skeleton features of the speakers, and train neural networks to recognise the former from the latter. finally, these phonological properties have a strong discriminatory power when determining signs. for example, in asl-lex @xcite , a lexicon which also captures phonology information, the authors report that more than 50% of its 994 described signs have a unique combination of only six phonological properties and more than 80% of the signs share their combination with at most two other signs. by relying on this phonological information from resources such as asl-lex, many signs can be uniquely determined. this means that well performing classifiers can leverage this information to predict signs without having encountered them during training. this is a capability that current data-driven approaches to slr lack by design @xcite . thus, in combination, mature approaches to phonology recognition can facilitate the development of sign language resources, for example by providing first-pass silver annotations for new sign languages based on their phonological properties. this is an important task for both documenting low-resource sign languages as well as rapid developing of large-scale datasets, and for fully harnessing data-driven cv approaches. to spur research in this direction, we extend the preliminary work by @xcite and introduce the task of phonological property recognition (ppr). more specifically, with this paper, we contribute (i) wlasllex2001, a large-scale, automatically constructed ppr dataset, (ii) an analysis of the dataset quality, and (iii) an empirical study of the performance of different deep-learning based baselines thereon. ## methodology we address ppr as a classification problem based on features extracted from videos of people speaking sl. although manual annotation approaches are widely adopted, these are time consuming and require expert knowledge. instead, we rely on automated dataset construction. on a high level, we cross-reference a large-scale asl slr dataset with an asl lexicon and annotate videos of signs with their corresponding phonological properties. we then extract skeletal features, by taking advantage of pre-trained deep models from the computer vision community @xcite . finally, we train several deep models to classify them as phonological classes. ## results and discussion the upper half of the lower half of often, crowd sourced @xcite or automatically constructed datasets such as ours, have a performance ceiling, possibly due to incorrectly assigned ground truth labels or low quality of input data @xcite . to investigate the former, we measure the agreement on videos that all models misclassify using fleiss' κ. intuitively, if models consistently agree on a label different than the ground truth, the ground truth label might be wrong. we find that averaged across the six tasks, the agreement is negligible: 0.09 ± 0.06 and 0.11 ± 0.09 for phoneme and gloss split, respectively. similarly, for the latter, if all models consistently fail to assign any correct label for a given video (e.g. all models err on a video appearing in the test sets of movement and flexion), this can hint at low quality of the input, making it impossible to predict anything correctly. we find that this is not the case with wlasl-lex2001, as videos appearing in test sets of different tasks tend to have a low mutual misclassification rate: 1% and 0.7% of videos appearing in test sets of two and three tasks were misclassified by all models for all associated tasks for the phoneme split. for the gloss split the numbers are 3 and 0% for two and three tasks, respectively. together, these observations suggest that the models presented in this paper are unlikely to reach the performance ceiling on wlasl-lex2001 and more advanced approaches could obtain even higher accuracy scores. ## conclusion in this paper, we discuss the task of phonological property recognition (ppr). we automatically construct a dataset for the task featuring six phonological properties and analyse it extensively. we find that there is potential for improvement over our presented data-driven baseline approaches. researchers pursuing this direction can focus on developing better-performing models, for example by relying on jointly learning all properties, as labels for different properties can be mutually dependent. another possible avenue is to investigate the feasibility of using prr to perform tokenisation of continuous sign language speech, by decomposing it into multiple phonemes, which is identified as one of the big challenges of slp @xcite .