A Dataset for Linguistic Understanding, Visual Evaluation, and Recognition of Sign Languages: The K- RSL (2020)

Abstract: The paper presents the first dataset that aims to serve interdisciplinary purposes for the utility of computer vision community and sign language linguistics. To date, a majority of Sign Language Recognition (SLR) approaches focus on recognising sign language as a manual gesture recognition problem. However, signers use other articulators: facial expressions, head and body position and movement to convey linguistic information. Given the important role of non-manual markers, this paper proposes a dataset and presents a use case to stress the importance of including non-manual features to improve the recognition accuracy of signs. To the best of our knowledge no prior publicly available dataset exists that explicitly focuses on non-manual components responsible for the grammar of sign languages. To this end, the proposed dataset contains 28250 videos of signs of high resolution and quality, with annotation of manual and non-manual components. We conducted a series of evaluations in order to investigate whether non-manual components would improve signs’ recognition accuracy. We release the dataset to encourage SLR researchers and help advance current progress in this area toward real-time sign language interpretation. Our dataset will be made publicly available at https://krslproject.github.io/krsl-corpus

## introduction there exist over 300 sign languages around the world that are native to 70 million deaf people @xcite . sign languages are comprised of hand gestures, arms and body movements, head position, facial expressions, and lip patterns @xcite . while automatic speech recognition has progressed to being commercially available, automatic sign language recognition (slr) is still in its infancy @xcite . to date, more than half of published visionbased research utilizes isolated sign language data with a vocabulary size of less than 50 signs @xcite . but the real-world utility of slr solutions requires continuous recognition, which is significantly more challenging than recognising individual signs due to co-articulation (the ending of one sign affecting the start of the next), depiction (visually representing or enacting content), epenthesis effects (insertion of extra features into signs), generalization, and so on @xcite . as a result, realistic, generalisable, and large datasets are necessary to advance slr. current efforts in slr do not address the complexities of sign language linguistics, and thus have a limited real-world value @xcite . @xcite highlight the importance of non-manual components of sign languages. for example, they can change meaning of a verb, or differentiate between objects and people. according to @xcite , there is an overall lack of nonmanual parameters that are included in medium and larger vocabulary recognition systems. for example, many computer vision approaches focus on the signers' hands only and tend to ignore the rich channel of information conveyed by non-manual articulators: facial expressions, mouthing, movement and position of the head and body conveying important grammatical and lexical information. in addition, many datasets allowed novice or nonnative contributions (i.e. students) in addition to slower signing and simplifying the style and the vocabulary to make the computer vision problem easier but of no real value @xcite . for the progress in slr, interdisciplinary efforts are required with an involvement of native signers and sign language linguists. beyond targeting the local need of creating the first corpus within cis (commonwealth of independent states) region suitable for machine learn- ## related work this section discusses related work on sign language datasets, state of the art in slr, and the importance of non-manual features for sign languages. ## the proposed k-rsl corpus given the important role of non-manual markers, in this paper we present a corpus which is motivated by the importance of both manual and non-manual features. we focus on specific cases where nonmanual markers play a vital role in differentiating between similar signs or similar sentences. ## baseline methods signing recognition can be considered as a variation of action recognition or human pose estimation tasks. keypoint detection library openpose @xcite enables us to evaluate both manual (hand keypoints) and nonmanual features (face and pose keypoints). one of the latest works in action recognition @xcite introduces a new spatiotemporal convolutional block r(2+1)d that achieves state-of-the-art results. in order to analyze and classify collected dataset we employ both approaches as a baseline models for isolated sign recognition. we have extracted isolated clips from the statement-question subset of following signs: 'what', 'who', 'which', 'which one', 'when', 'where (direction)', 'where (location)', 'why', 'how', and 'how much'. we distinguish them to twenty classes (as ten words have a pair in both statement and question form). ## experimental results a series of experiments was conducted in order to investigate whether non-manual features would improve recognition accuracy. all experiments were performed on isolated signs extracted from the question-statement subset and divided into 20 classes (10 signs as statement and questions). the first experiment was the classification of 20 classes. for this reason we trained two baseline models: a logistic regression model using only manual features and with non-manual features as an input, and a r(2+1)d model on full frames as an input. evaluation of each model was repeated 10 times with random train/test splits to avoid extreme cases. ## conclusion this paper presents the k-rsl dataset motivated by the need to create sl datasets for interdisciplinary purposes e.g. for computer vision and computational linguistics research. due to the challenging nature of slr, the proposed dataset aims to attract the attention of the computer vision community with the k-rsl dataset being linguistically rich. the data was carefully selected to find various cases when manual gestures will not provide good performance and will stress the need to include nonmanual components into consideration. in addition to computer vision community, this dataset can be utilized by the linguistics community to explore research questions and computationally prove their hypotheses. future work will include expanding the vocabulary of the corpus in addition to diversifying and increasing the number of signers recorded in noisy environmental conditions (e.g. outside of the office environment).