Towards Performance Improvement in I ndian S ign L anguage Recognition (2020)

Abstract: Sign language is a complete natural language used by deaf and dumb people. It has its own grammar and it differs with spoken language to a great extent. Since people without hearing and speech impairment lack the knowledge of the sign language, the deaf and dumb people find it difficult to communicate with them. The conception of system that would be able to translate the sign language into text would facilitate understanding of sign language without human interpreter. This paper describes a systematic approach that takes Indian Sign Language (ISL) video as input and converts it into text using frame sequence generator and image augmentation techniques. By incorporating these two concepts, we have increased dataset size and reduced overfitting. It is demonstrated that using simple image manipulation techniques and batch of shifted frames of videos, performance of sign language recognition can be significantly improved. Approach described in this paper achieves 99.57% accuracy on the dynamic gesture dataset of ISL.

## introduction according to @xcite , sign language can be considered as a collection of gestures, movements, postures, and facial expressions corresponding to letters and words in spoken languages. comparing sign language with spoken language, main difference exists on modality. deaf and dumb people use sign language as means of communication to express their thoughts and emotions. each country has its own sign language with high degree of grammatical variations. indian sign language is largely dominated by object features, and is different than other sign languages. trained sign language interpreters are needed during medical and legal appointments, educational and training sessions, to provide interpreting services. over the past few years, there has been an increasing demand for human interpreters. but in india, approximately 300 certified human interpreters are available @xcite . given a shortage of interpreting services, a system can be designed that offers flexible alternative when human interpreters are not available. sign language recognition is an approach that converts input sign gesture(s) into text or speech. sign language recognition is a very challenging task since this task involves interpretation between visual and linguistic information. deep neural networks (dnn) perform remarkably well for image recognition task @xcite . but these networks are heavily reliant on big data, otherwise they lead to overfitting. overfitting refers to the phenomenon when a network learns a function with very high variance such as to perfectly model the training data @xcite . strategies to avoid overfitting and to increase generalization performance of dnns are dropout, batch normalization, transfer learning, one-shot learning and pretraining. in contrast to these techniques, data augmentation approaches problem of overfitting and generalization from the root of the problem, the training dataset. we have adopted this concept on input videos for isl recognition to make dataset inflated. in order to increase dataset size, we have created sequence of frames in batches systematically. we have proposed this approach to generate more instances for isl recognition task in order to achieve better model performance. in particular, we have addressed the issue of one research question: how to use dnn with very small amount of input videos in order to incorporate both left-handed and righthanded signs without hurting recognition performance of isl sentences. the rest of the paper is organized as follows: in section 2 we have discussed work related to indian sign language recognition. section 3 explains steps of our proposed approach in details. section 4 shows dataset description and experimental results along with analysis. section 5 provides concluding remarks and directions for future work. ## related work there are two approaches for sign language recognition: 1) device based approach and 2) vision based approach. device based approach hinders the natural movement of hands. also, signer has to wear gloves that are connected through computer through cables. so device based approaches appear to be complex, costly and difficult to deploy. considering limitations of device based approach, we have discussed related work with respect to vision based approach in isl recognition. @xcite proposed an approach that recognized 66 isl dynamic samples of isl alphabets. these gestures were classified using dynamic time warping and approach achieved 77.2% accuracy. @xcite presented an approach for recognizing isl sentences in isl with 91% accuracy with hidden markov model. authors recorded single handed and double handed dynamic samples from 10 sentences. recognition rate of 90.17% was achieved by @xcite for 580 samples. though these samples were not recorded according to isl grammar, this approach worked for manual components of isl. @xcite used front camera of the mobile phone for collecting dynamic signs. the authors achieved 90% accuracy by extracting hand and head contour energies from the collected dynamic signs. issues like hand segmentation from upper half of the body image, boundary changes depending on hand shape of various signers are solved by @xcite with 85% accuracy. authors have used otsu thresholding method for segmentation, mel sec frequency cepstral coefficients (mfcc) for feature extraction of 8 dynamic samples. 92% accuracy was achieved by @xcite using concept of possibility theory on 20 different videos of continuous gestures. these videos were captured in different backgrounds like black, red, multiple objects with black full sleeves dress. @xcite used pseudo 2-dimensional hidden markov model (p2dhmm) for feature extraction, which was proven better than simple hidden markov model. for converting recognized signs in english text, lalr parser was used and 90% accuracy was achieved. (muthu mariappan and go-mathi, 2019) proposed an approach that extracted head and hand contour energies using front camera of the mobile for collecting signs.approach presented by @xcite used skin color segmentation and artificial neural network (ann). this approach worked for specific signer as ann was trained by taking face and hand features of specific signer. as far as reserach in isl is concerned, after nearly 35 years of research, isl is still in infancy when compared to other international sign languages. all the approaches described here work under controlled laboratory setting and use feature extraction techniques. this motivated us to work for isl recognition by doing generalized settings in very limited amount of input data without decreasing accuracy of isl recognition. ## method the details of the dataset are described in the section 4. in this section we describe the steps of the approach we followed to convert isl video into text. figure 1 shows the general framework of our approach. ## experimental results and analysis in this section, we present the details of the dataset and the experimental results with analysis. 1 . ## conclusion and future work it becomes a challenging task when we want to achieve more accuracy with less number of samples in generalized environment. deep learning gives promising results than other traditional algorithms in computer vision task as they learn features from gestures, but they require huge dataset. to overcome the problem of overfitting generated by deep learning models on less amount of data, image augmentation can be used before training data. image augmentation also increases accuracy of test data. in this work, we have empirically proven that simple image manipulation techniques and pretrained model with frame sequence generator creates great impact on the accuracy on isl recognition than using very limited amount of data in training. we have proposed an approach that uses pretrained model mobilenetv2 to learn features from augmented frame sequences of isl gestures using batch of shifted frames to provide decayed sequences for the same gesture. we are working on extending our work from lexical level analysis to machine translation to generate isl sentences. we are also in the process of creating new dataset of isl sentences using the dictionary launched by indian sign language research and training centre (islrtc). in future, we will compare results of mobilenetv2 model with other pretrained models on our dataset.