Using Computer Vision to Analyze Non-manual Marking of Questions in KRSL (2021)

Abstract: This paper presents a study that compares non-manual markers of polar and wh-questions to statements in Kazakh-Russian Sign Language (KRSL) in a dataset collected for NLP tasks. The primary focus of the study is to demonstrate the utility of computer vision solutions for the linguistic analysis of non-manuals in sign languages, although additional corrections are required to account for biases in the output. To this end, we analyzed recordings of 10 triplets of sentences produced by 9 native signers using both manual annotation and computer vision solutions (such as OpenFace). We utilize and improve the computer vision solution, and briefly describe the results of the linguistic analysis.

## introduction non-manual marking, that is, linguistically significant use of the body, head, facial features, and eye gaze, is a prominent feature of sign languages @xcite . for instance, in most sign languages, polar questions are accompanied with raised eyebrows, and some type of head movement @xcite . while non-manual markers in many sign languages have been previously studied, many other sign languages have not been analyzed before, including kazakh-russian sign language (krsl), which we discuss in this study. current developments in computer vision provide an opportunity for a large-scale quantitative research on non-manual markers. in this study, we evaluate whether computer vision solutions can be utilized for the analysis of non-manual marking present in sign language video recordings. the objective of this work is to compare non-manual markers in statements and questions in krsl, and to test a computer vision solution against manual annotations. ## applying computer vision the field of computer vision is actively developing thanks to the advancements in deep learning. achievements in this area allow computers to process a large amount of visual information, such as pictures, photos or videos. one of the tasks of computer vision is landmark detection. it can be described as an object recognition task with localization: the algorithm needs not only to detect an object on the image, but also to estimate the position of this object. we planned to use face and head landmarks recognition tools to estimate the eyebrow and head movement. openface is a toolkit for facial landmark detection, head pose estimation, facial action unit recognition and other facial behaviour analysis @xcite @xcite . openface is able to estimate 3d landmarks position from 2d points using point distribution model, which parametrizes the shape of a face using a limited set of parameters, such as scaling, rotation, translation and individual deformations of the face. we used openface 2.2.0 featureextraction model for getting face landmarks position in 3d and head pose estimation for every frame. face landmarks and head coordinates were in x,y,z-coordinates in mm, and the head rotation angle was in radians with the camera being the origin. the model also provided us with a confidence score for each frame. by outputting head rotation, openface directly provides a measure of forward/backward head tilt that we are interested in for the analysis. however, the estimation of eyebrow position from openface output is less straightforward. when we visually investigated the openface output, we noticed that the results conflicted with our initial observations. we expected polar questions to have the largest eyebrow raise but openface output was even smaller than in statements. furthermore, we saw correlation between rotation angle and eyebrow distance, meaning that, probably, openface predictions are prone to bias by the head rotation. to demonstrate this bias, we plotted 3 frames from our test video of the forward to backward head tilt without eyebrow movement. we rotated the face keypoints in the reverse angle of the computed head rotation and centred them on the bridge of the nose keypoint number 27. from figure 1 , we can see that the face points, especially eyebrow points, are bending. hence, the 3d model that openface deduces is likely to be distorted in the presence of head movement. we therefore had to attempt to eliminate such distortions. general 3d reconstruction from a single camera is a really challenging problem and it is outside of our area of competence, therefore we were not able to modify the openface model itself. moreover, we do not have the specifically annotated data with facial landmarks to retrain the model. that is why we will deal with the computed output instead of the model. to deal with the bias we created a machine learning model that would predict the eyebrows distance depending on the head rotation when the eyebrows were not raised or lowered, i.e. the default eyebrow distance with the influence of tilting. we trained linear regression with l2 regularization and alpha 0.001 from sklearn library for python @xcite on statements that contained no eyebrow raises: 63 sentences (4414 frames). the target of the model was the distance from the eyebrow points (18, 23 -inner, 20, 25 -outer) to the eye line -a line between points 36, 45. the mean inner eyebrow distance in the subset was 29.7 and the outer was 27.1. our choice of the model was based on the observation that the distortion seems to be linear and consistent across signers -pearson correlation coefficient between vertical head angle and the eyebrow distance to the eye line in sentences with no eyebrow raise is -0.42 for the inner distance and -0.48 for the outer distance. moreover, the training data set is small and thus a simple model should be sufficient. the features of the models that we used are shown in the first model performed with an mse of about 4 for each eyebrow distance. however we wanted a better result. we added meta information as one-hot encoded vectors and it significantly increased the quality on the cross-validation. we believe that this allowed the model to learn the individual mean eyebrow height for each signer. we also noticed that blinking affected the distance estimation, so we added eye aperture features, which slightly reduced the error. 4 after training the final model on statements, the eyebrow distance was predicted for all sentences, and then subtracted from the distance computed on openface output directly. thus, we subtracted the changes in distance caused by the tilts from the distance based on the output to get the unbiased distance measurement. to check that this approach produced reasonable results, we looked manually at 35 sentences and compared the predictions with the annotations. in general, the output of the model agreed with our annotations. furthermore, the correlation between the vertical head tilt and the new eyebrow distance is lower than on the original distance (-0.27 for inner, -0.25 for outer). having demonstrated that this method of adjusting the eyebrow distance for head movement works well, we conducted the subsequent analysis using it. ## statistical analysis one of the main advantages of applying computer vision to analysis of non-manuals in sign languages is that it opens the possibility of consequent advanced statistical analysis, instead of relying on qualitative observations. thus, we use the current data set to showcase a possible statistical analysis of the output of the proposed computer vision approach. we analyzed the data in r (version 3.6.3) using r studio (version 1.0.143) (r @xcite . for this study, we averaged the eyebrows distance in each video in the parts of sentence areas for inner eyebrows points (20,23) and for outer eyebrows points (18,25) (averaging the left and right eyebrow distance). thus, we had four eyebrow measurements for all the sentences (inner and outer mean distance for the subject and inner and outer mean distance for the verb) and separately we had six eyebrow measurements for the wh-questions (inner and outer mean distance for the wh-sign, subject, and verb). besides, we calculated the mean head rotation angle on the x-axis (vertical tilts) again for two signs for all the sentences and for three signs for the wh-questions. a mixed-effects multivariate linear regression model was picked for the analysis @xcite . we made 9 models with the outcome variables of internal and external eyebrow distances and head tilt angle on the subjects, verbs, and wh-questions. the fixed predictor variables for the first 6 models were sentence type (categorical, three levels: statement, polar question, wh-question), group (categorical, deaf vs. hearing), and all the interactions between the two predictors. for the last three models, the fixed predictor variables were part of sentence (categorical, three levels: wh-word, subject, verb), group (categorical, deaf vs. hearing), and all the interactions between the two predictors. we used orthogonal coding of contrasts for the predictors with three levels. finally, for all models, the random variables were participant (with a random slope for sentence type or part of sentence), and sentence (with a random slope for group). for the models, we used the lme4 package @xcite with the help of the blme package to achieve convergence with a small number of levels for the random effects @xcite . the significance of the contribution of the factors was computed with the anova function from the car package @xcite . ## results firstly, we examined the results visually. from figure 2 , we can see that: polar questions have tilting forward on the subject and verb and eyebrows raise on the whole question, while whquestions have tilting backwards on the wh-word and eyebrows raise in the beginning on the wh-word, which is slowly declining to the end of the sentence. in all the statistical analyses below, the effect of group and interactions between the group and the other effects were never significant, so we do not discuss them further. ## conclusions this paper presents the analysis of non-manual marking of simple polar and wh-questions in krsl produced by nine native krsl signers for a dataset for automatic sign language translation. to this end, we firstly annotated the data set manually, and then applied computer vision techniques to automate extraction of non-manual marking from video recordings. our findings suggest that polar questions in the krsl dataset are marked by an eyebrow raise on the whole sentence, and by consecutive forward head tilts on the subject and the verb. in addition, wh-questions are marked by backward head tilts on the wh-sign, and by an eyebrow raise on the wh-sign that can spread over the whole sentence. additionally, we demonstrated the utility of computer vision solutions, specifically, open-face @xcite @xcite that can be applied to sign language data for the purpose of linguistic analysis of non-manual marking. however, we also discovered that head movement leads to distortion of the facial features even though openface reconstructs a 3d model of the face to account for such movement. we addressed this problem with a machine learning solution.