Sign Language Recognition Analysis using Multimodal Data Voicecontrolled personal and home assistants (such as the Amazon Echo and Apple Siri) are becoming increasingly popular for a variety of applications. However, the benefits of these technologies are not readily accessible to Deaf or HardofHearing (DHH) users. The objective of this study is to develop and evaluate a sign recognition system using multiple modalities that can be used by DHH signers to interact with voicecontrolled devices. With the advancement of depth sensors, skeletal data is used for applications like video analysis and activity recognition. Despite having similarity with the wellstudied human activity recognition, the use of 3D skeleton data in sign language recognition is rare. This is because unlike activity recognition, sign language is mostly dependent on hand shape pattern. In this work, we investigate the feasibility of using skeletal and RGB video data for sign language recognition using a combination of different deep learning architectures. We validate our results on a largescale American Sign Language (ASL) dataset of 12 users and 13107 samples across 51 signs. It is named as GMUASL51. 1 We collected the dataset over 6 months and it will be publicly released in the hope of spurring further machine learning research towards providing improved accessibility for digital assistants.

Action Recognition Based on 3D Skeleton and RGB Frame Fusion Action recognition has wide applications in assisted living, health monitoring, surveillance, and humancomputer interaction. In traditional action recognition methods, RGB videobased ones are effective but computationally inefficient, while skeletonbased ones are computationally efficient but do not make use of lowlevel detail information. This work considers action recognition based on a multimodal fusion between the 3D skeleton and the RGB image. We design a neural network that uses a 3D skeleton sequence and a single middle frame from an RGB video as input. Specifically, our method picks up one frame in a video and extracts spatial features from it using two attention modules, a selfattention module and a skeletonattention module. Further, temporal features are extracted from the skeleton sequence via a BILSTM subnetwork. Finally, the spatial features and the temporal features are combined via a feature fusion network for action classification. A distinct feature of our method is that it uses only a single RGB frame rather than an RGB video. Accordingly, it has a lightweighted architecture and is more efficient than RGB videobased methods. Comparative evaluation on two public datasets, NTURGBD and SYSU, demonstrates that, our method can achieve competitive performance compared with stateoftheart methods.