A Bidirectional Interactive System of Sign Language and Visual Speech Based on Portable Devices At present, the natural communication between deaf and normal people is a major challenge both in theoretical research and application. In addition, in the field of HumanMachine Interaction, there is little work on bidirectional communication based on sign language and visual speech. In this paper, a portable bidirectional interactive system of sign language and visual speech was proposed to help deaf people communicate naturally under certain circumstances. In the section of modeling, Network in Network model was employed to classify sign language words, while the method using a network combined DenseNet and LSTM together was used for lip reading. By wearing the portable device, people can achieve bidirectional translation and communication of visual speech and sign language. We built our own Chinese sign language database containing more than 100 categories of words and conducted experiments in the context of the airport for experimental verification. Eventually, the average time of each round of conversation is within 15 seconds, faster than normal communication without system assistance, which verified the effectiveness and advantage of the purposed system.

A Virtual Sign Language Translator on Smartphones Sign languages are natural languages used primarily by the deaf communities. Unfortunately, it is difficult for people outside of the communities to understand the languages. As smartphones are now ubiquitous, it will be very convenient for people from different communities to communicate if smartphones can help translate the languages. In this paper, a smartphone application called Sign Language Translator (SLT) is introduced, which records the person signing and translates the hand gestures into texts. SLT integrates two pretrained models proposed by previous works and a classification model developed in this study. Given a frame showing the person making the sign, SLT first gets the body skeleton of the person, from which the positions of the hands are identified and the fingersu0027 skeleton is then detected. The classification model is next used to extract features from the skeleton to map to the specific words that are expressed. In the experiments, we chose three specific words in American Sign languages for evaluation. Both training and testing datasets are the videos with the corresponding signs. The accuracy of translation into right words in the testing dataset is more than 80 percent. It shows that SLT can effectively help people to communicate with deaf people.