pwc-archive/papers-with-abstracts huggingface dataset:
Zero-Shot Learning Based Approach For Medieval Word Recognition Using Deep-Learned Features
Authors: Sukalpa Chanda, Jochem Baas, Daniël Haitink, Sebastien Hamely, Dominique Stutzmanny, Lambert Schomaker
Date: 2018-10-01 00:00:00
Proceeding: 16th International Conference on Frontiers in Handwriting Recognition (ICFHR 2018) 2018 10
Tasks: Attribute, Generalized Zero-Shot Learning, Language Modelling, Optical Character Recognition (OCR), Retrieval, Zero-Shot Learning

Historical manuscripts reflect our past. Recently digitization of large quantities of historical handwritten docu- ments is taking place in every corner of the world, and are being archived. From those digital repositories, automatic text indexing and retrieval system fetch only those documents to an end user that they are interested in. A regular OCR technology is not capable of rendering this service to an end user in a reliable manner. Instead, a word recognition/spotting algorithm performs the task. Word recognition based systems require enough labelled data per class to train the system. Moreover, all word classes need to be taught beforehand. Though word spotting could evade this drawback of prior training, these systems often need to have additional overheads like a language model to deal with “out of lexicon” words. Zero-shot learning could be a possible alternative to counter such situation. A Zero-shot learning algorithm is capable of handling unseen classes, provided the algorithm has been fortified with rich discriminating features and reliable “attribute description” per class during training. Since deeply learned features have enough discriminating power, a deep learning framework has been used here for feature extraction purpose. To the best of our knowledge, this is probably the first work on “out of lexicon” medieval word recognition using a Zero-Shot Learning framework. We obtained very encouraging results(accuracy ≈57% for “out of lexicon” classes) while dealing with 166 training classes and 50 unseen test classes.
Exploiting Implicit Trust and Geo-social Network for Recommendation
Authors: Feiyang Li School of Computer Science and Technology/Suzhou Institute for Advanced Study, University of Science and Technology of China, P.R.china mmmwhy@mail.ustc.edu.cn
Date: 2018-12-06 00:00:00
Proceeding: journal 2018 12
Tasks: Collaborative Filtering, Recommendation Systems

Abstract—Recommender system (RS) seeks to predict the rating or preference a user would give to an item, this system
often relies on collaborative filtering (CF). CF suffers from the problems of data sparsity, cold start and location insensitive.
Existing RSs do not consider the spatial extent of users, we analyze the users’ location data from four commercial websites, and conclude that people with close social relationships prefer to purchase in places that are also physically close.State-of-the-art recommendation algorithm TrustSVD extends RS with social trust information, we propose Trust-location SVD (TLSVD) by incorporating the location information and implicit trust into TrustSVD. The improved TLSVD helps to quantitatively analyze the spatial closeness and preference similarity between users. Experimental results indicate that the accuracy of our method is better than other multiple counterparts’, especially when active users have location information or few ratings.
An Integrated Recommender Algorithm for Rating Prediction
Authors: Yefeng Ruan, Tzu-Chun Lin
Date: 2016-08-05 00:00:00
Tasks: Collaborative Filtering, Prediction, Recommendation Systems

Recommender system is currently widely used in many e-commerce systems, such as Amazon, eBay, and so on. It aims to help users to find items which they may be interested in. In literature, neighborhood-based collaborative filtering and matrix factorization are two common methods used in recommender systems. In this paper, we combine these two methods with personalized weights on them. Rather than using fixed weights for these two methods, we assume each user has her/his own preference over them. Our results shows that our algorithm outperforms neighborhood-based collaborative filtering algorithm, matrix factorization algorithm and their combination with fixed weights.
Adaptive RNN Tree for Large-Scale Human Action Recognition
Authors: Wenbo Li, Longyin Wen, Ming-Ching Chang, Ser Nam Lim, Siwei Lyu
Date: 2017-10-22 00:00:00
Proceeding: The IEEE International Conference on Computer Vision (ICCV), 2017 2017 10
Tasks: Action Recognition, Skeleton Based Action Recognition, Temporal Action Localization

In this work, we present the RNN Tree (RNN-T), an adaptive learning framework for skeleton based human action recognition. Our method categorizes action classes and uses multiple Recurrent Neural Networks (RNNs) in a tree-like hierarchy. The RNNs in RNN-T are co-trained with the action category hierarchy, which determines the structure of RNN-T. Actions in skeletal representations are recognized via a hierarchical inference process, during which individual RNNs differentiate finer-grained action classes with increasing confidence. Inference in RNN-T ends when any RNN in the tree recognizes the action with high confidence, or a leaf node is reached. RNN-T effectively addresses two main challenges of large-scale action recognition: (i) able to distinguish fine-grained action classes that are intractable using a single network, and (ii) adaptive to new action classes by augmenting an existing model. We demonstrate the effectiveness of RNN-T/ACH method and compare it with the state-of-the-art methods on a large-scale dataset and several existing benchmarks.
Skeletal quads: Human action recognition using joint quadruples
Authors: Georgios Evangelidis, Gurkirt Singh, Radu Horaud
Date: 2014-08-24 00:00:00
Proceeding: 2014 22nd International Conference on Pattern Recognition 2014 8
Tasks: Action Recognition, Skeleton Based Action Recognition, Temporal Action Localization

Recent advances on human motion analysis have made the extraction of human skeleton structure feasible, even from single depth images. This structure has been proven quite informative for discriminating actions in a recognition scenario. In this context, we propose a local skeleton descriptor that encodes the relative position of joint quadruples. Such a coding implies a similarity normalisation transform that leads to a compact (6D) view-invariant skeletal feature, referred to as skeletal quad. Further, the use of a Fisher kernel representation is suggested to describe the skeletal quads contained in a (sub)action. A Gaussian mixture model is learnt from training data, so that the generation of any set of quads is encoded by its Fisher vector. Finally, a multi-level representation of Fisher vectors leads to an action description that roughly carries the order of sub-action within each action sequence. Efficient classification is here achieved by linear SVMs. The proposed action representation is tested on widely used datasets, MSRAction3D and HDM05. The experimental evaluation shows that the proposed method outperforms state-of-the-art algorithms that rely only on joints, while it competes with methods that combine joints with extra cues.
Skeletonnet: Mining deep part features for 3-d action recognition
Authors: Qiuhong Ke, Senjian An, Mohammed Bennamoun, Ferdous Sohel, Farid Boussaid
Date: 2017-03-31 00:00:00
Proceeding: IEEE Signal Processing Letters ( Volume: 24 , Issue: 6 , June 2017 ) 2017 3
Tasks: Action Recognition, Deep Learning, Skeleton Based Action Recognition, Time Series, Time Series Analysis, Translation

This letter presents SkeletonNet, a deep learning framework for skeleton-based 3-D action recognition. Given a skeleton sequence, the spatial structure of the skeleton joints in each frame and the temporal information between multiple frames are two important factors for action recognition. We first extract body-part-based features from each frame of the skeleton sequence. Compared to the original coordinates of the skeleton joints, the proposed features are translation, rotation, and scale invariant. To learn robust temporal information, instead of treating the features of all frames as a time series, we transform the features into images and feed them to the proposed deep learning network, which contains two parts: one to extract general features from the input images, while the other to generate a discriminative and compact representation for action recognition. The proposed method is tested on the SBU kinect interaction dataset, the CMU dataset, and the large-scale NTU RGB+D dataset and achieves state-of-the-art performance.
Rolling Rotations for Recognizing Human Actions from 3D Skeletal Data
Authors: Raviteja Vemulapalli, Rama Chellappa
Date: 2016-06-27 00:00:00
Proceeding: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2016 6
Tasks: Action Recognition, General Classification, Skeleton Based Action Recognition, Temporal Action Localization

Recently, skeleton-based human action recognition has been receiving significant attention from various research communities due to the availability of depth sensors and real-time depth-based 3D skeleton estimation algorithms. In this work, we use rolling maps for recognizing human actions from 3D skeletal data. The rolling map is a well-defined mathematical concept that has not been explored much by the vision community. First, we represent each skeleton using the relative 3D rotations between various body parts. Since 3D rotations are members of the special orthogonal group SO3, our skeletal representation becomes a point in the Lie group SO3 × ... × SO3, which is also a Riemannian manifold. Then, using this representation, we model human actions as curves in this Lie group. Since classification of curves in this non-Euclidean space is a difficult task, we unwrap the action curves onto the Lie algebra so3 × ... × so3 (which is a vector space) by combining the logarithm map with rolling maps, and perform classification in the Lie algebra. Experimental results on three action datasets show that the proposed approach performs equally well or better when compared to state-of-the-art.
3D CNNs on Distance Matrices for Human Action Recognition
Authors: Alejandro Hernandez Ruiz, Lorenzo Porzi, Samuel Rota Bulò, Francesc Moreno-Noguer
Date: 2017-10-23 00:00:00
Proceeding: MM '17 Proceedings of the 25th ACM international conference on Multimedia 2017 10
Tasks: Action Recognition, Skeleton Based Action Recognition, Temporal Action Localization

In this paper we are interested in recognizing human actions from sequences of 3D skeleton data. For this purpose we combine a 3D Convolutional Neural Network with body representations based on Euclidean Distance Matrices (EDMs), which have been recently shown to be very effective to capture the geometric structure of the human pose. One inherent limitation of the EDMs, however, is that they are defined up to a permutation of the skeleton joints, i.e., randomly shuffling the ordering of the joints yields many different representations. In oder to address this issue we introduce a novel architecture that simultaneously, and in an end-to-end manner, learns an optimal transformation of the joints, while optimizing the rest of parameters of the convolutional network. The proposed approach achieves state-of-the-art results on 3 benchmarks, including the recent NTU RGB-D dataset, for which we improve on previous LSTM-based methods by more than 10 percentage points, also surpassing other CNN-based methods while using almost 1000 times fewer parameters.
Learning Spatio-Temporal Structure from RGB-D Videos for Human Activity Detection and Anticipation
Authors: Hema S. Koppula, Ashutosh Saxena
Date: 2013-02-01 00:00:00
Proceeding: Proceedings of Machine Learning Research volume 28 2013 2
Tasks: Action Detection, Activity Detection, Skeleton Based Action Recognition

We consider the problem of detecting past activities as well as anticipating which activity will happen in the future and how. We start by modeling the rich spatio-temporal relations between human poses and objects (called affordances) using a conditional random field (CRF). However, because of the ambiguity in the temporal segmentation of the sub-activities that constitute an activity, in the past as well as in the future, multiple graph structures are possible. In this paper, we reason about these alternate possibilities by reasoning over multiple possible graph structures. We obtain them by approximating the graph with only additive features, which lends to efficient dynamic programming. Starting with this proposal graph structure, we then design moves to obtain several other likely graph structures. We then show that our approach improves the state-of-the-art significantly for detecting past activities as well as for anticipating future activities, on a dataset of 120 activity videos collected from four subjects.
Unstructured Human Activity Detection from RGBD Images
Authors: Jaeyong Sung, Colin Ponce, Bart Selman, Ashutosh Saxena
Date: 2012-06-28 00:00:00
Proceeding: 2012 IEEE International Conference on Robotics and Automation 2012 6
Tasks: Action Detection, Activity Detection

Being able to detect and recognize human activities is essential for several applications, including personal assistive robotics. In this paper, we perform detection and recognition of unstructured human activity in unstructured environments. We use a RGBD sensor (Microsoft Kinect) as the input sensor, and compute a set of features based on human pose and motion, as well as based on image and point-cloud information. Our algorithm is based on a hierarchical maximum entropy Markov model (MEMM), which considers a person's activity as composed of a set of sub-activities. We infer the two-layered graph structure using a dynamic programming approach. We test our algorithm on detecting and recognizing twelve different activities (CAD-60 / CAD-120) performed by four people in different environments, such as a kitchen, a living room, an office, etc., and achieve good performance even when the person was not seen before in the training set
Learning clip representations for skeleton-based 3d action recognition
Authors: Qiuhong Ke, Mohammed Bennamoun, Senjian An, Ferdous Sohel, Farid Boussaid
Date: 2018-03-05 00:00:00
Proceeding: IEEE Transactions on Image Processing ( Volume: 27 , Issue: 6 , June 2018 ) 2018 3
Tasks: 3D Action Recognition, Action Recognition, Skeleton Based Action Recognition

This paper presents a new representation of skeleton sequences for 3D action recognition. Existing methods based on hand-crafted features or recurrent neural networks cannot adequately capture the complex spatial structures and the long-term temporal dynamics of the skeleton sequences, which are very important to recognize the actions. In this paper, we propose to transform each channel of the 3D coordinates of a skeleton sequence into a clip. Each frame of the generated clip represents the temporal information of the entire skeleton sequence and one particular spatial relationship between the skeleton joints. The entire clip incorporates multiple frames with different spatial relationships, which provide useful spatial structural information of the human skeleton. We also propose a multitask convolutional neural network (MTCNN) to learn the generated clips for action recognition. The proposed MTCNN processes all the frames of the generated clips in parallel to explore the spatial and temporal information of the skeleton sequences. The proposed method has been extensively tested on six challenging benchmark datasets. Experimental results consistently demonstrate the superiority of the proposed clip representation and the feature learning method for 3D action recognition compared to the existing techniques.
Early action prediction by soft regression
Authors: Jian-Fang Hu, Wei-Shi Zheng, Lianyang Ma, Gang Wang, Jian-Huang Lai, Jian-Guo Zhang
Date: 2018-08-06 00:00:00
Proceeding: IEEE Transactions on Pattern Analysis and Machine Intelligence 2018 8
Tasks: Early Action Prediction, Prediction, regression, Skeleton Based Action Recognition

We propose a novel approach for predicting on-going action with the assistance of a low-cost depth camera. Our approach introduces a soft regression-based early prediction framework. In this framework, we estimate soft labels for the subsequences at different progress levels, jointly learned with an action predictor. Our formulation of soft regression framework 1) overcomes a usual assumption in existing early action prediction systems that the progress level of on-going sequence is given in the testing stage; and 2) presents a theoretical framework to better resolve the ambiguity and uncertainty of subsequences at early performing stage. The proposed soft regression framework is further enhanced in order to take the relationships among subsequences and the discrepancy of soft labels over different classes into consideration, so that a Multiple Soft labels Recurrent Neural Network (MSRNN) is finally developed. For real-time performance, we also introduce "local accumulative frame feature (LAFF)", which can be computed efficiently by constructing an integral feature map. Our experiments on three RGB-D benchmark datasets and an unconstrained RGB action set demonstrate that the proposed regression-based early action prediction model outperforms existing models and the early action prediction on RGB-D sequence is more accurate than that on RGB channel.
Learning Shape-Motion Representations from Geometric Algebra Spatio-Temporal Model for Skeleton-Based Action Recognition
Authors: Yanshan Li, Rongjie Xia, Xing Liu, Qinghua Huang
Date: 2019-07-08 00:00:00
Proceeding: 2019 IEEE International Conference on Multimedia and Expo (ICME) 2019 7
Tasks: Action Recognition, Skeleton Based Action Recognition

Skeleton-based action recognition has been widely applied in intelligent video surveillance and human behavior analysis. Previous works have successfully applied Convolutional Neural Networks (CNN) to learn spatio-temporal characteristics of the skeleton sequence. However, they merely focus on the coordinates of isolated joints, which ignore the spatial relationships between joints and only implicitly learn the motion representations. To solve these problems, we propose an effective method to learn comprehensive representations from skeleton sequences by using Geometric Algebra. Firstly, a frontal orientation based spatio-temporal model is constructed to represent the spatial configuration and temporal dynamics of skeleton sequences, which owns the robustness against view variations. Then the shape-motion representations which mutually compensate are learned to describe skeleton actions comprehensively. Finally, a multi-stream CNN model is applied to extract and fuse deep features from the complementary shape-motion representations. Experimental results on NTU RGB+D and Northwestern-UCLA datasets consistently verify the superiority of our method.
Cross-modal Learning by Hallucinating Missing Modalities in RGB-D Vision
Authors: Nuno C. Garcia, Pietro Morerio, Vittorio Murino
Date: 2019-01-01 00:00:00
Proceeding: Multimodal Scene Understanding Algorithms, Applications and Deep Learning 2019 1
Tasks: Action Recognition, Hallucination, Multimodal Activity Recognition, Skeleton Based Action Recognition, Temporal Action Localization

Diverse input data modalities can provide complementary cues for several tasks, usually leading to more robust algorithms and better performance. However, while a (training) dataset could be accurately designed to include a variety of sensory inputs, it is often the case that not all modalities are available in real life (testing) scenarios, when the model is to be deployed. This raises the challenge of how to learn robust representations leveraging multimodal data in the training stage, while considering limitations at test time, such as noisy or missing modalities. This chapter presents a new approach for multimodal video action recognition, developed within the unified frameworks of distillation and privileged information, named generalized distillation. We consider the particular case of learning representations from depth and RGB videos, while relying on RGB data only at test time. Our approach consists in training a hallucination network that learns to distill depth features through multiplicative connections of spatiotemporal representations, leveraging soft labels and hard labels, and the euclidean distance between feature maps. We report state-of-the-art or comparable results on video action recognition on the largest multimodal dataset available for this task, the NTU RGB+D, as well as on the UWA3DII and Northwestern-UCLA.
View invariant human action recognition using histograms of 3D joints
Authors: Lu Xia, Chia-Chih Chen, J. K. Aggarwal
Date: 2012-07-16 00:00:00
Proceeding: 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops 2012 7
Tasks: Action Recognition, Skeleton Based Action Recognition, Temporal Action Localization

In this paper, we present a novel approach for human action recognition with histograms of 3D joint locations (HOJ3D) as a compact representation of postures. We extract the 3D skeletal joint locations from Kinect depth maps using Shotton et al.'s method [6]. The HOJ3D computed from the action depth sequences are reprojected using LDA and then clustered into k posture visual words, which represent the prototypical poses of actions. The temporal evolutions of those visual words are modeled by discrete hidden Markov models (HMMs). In addition, due to the design of our spherical coordinate system and the robust 3D skeleton estimation from Kinect, our method demonstrates significant view invariance on our 3D action dataset. Our dataset is composed of 200 3D sequences of 10 indoor activities performed by 10 individuals in varied views. Our method is real-time and achieves superior results on the challenging 3D action dataset. We also tested our algorithm on the MSR Action 3D dataset and our algorithm outperforms Li et al. [25] on most of the cases.
Group sparsity and geometry constrained dictionary learning for action recognition from depth maps.
Authors: Jiajia Luo, Wei Wang, Hairong Qi
Date: 2014-03-03 00:00:00
Proceeding: 2013 IEEE International Conference on Computer Vision 2014 3
Tasks: Action Recognition, Dictionary Learning, Multimodal Activity Recognition, Temporal Action Localization

Human action recognition based on the depth information provided by commodity depth sensors is an important yet challenging task. The noisy depth maps, different lengths of action sequences, and free styles in performing actions, may cause large intra-class variations. In this paper, a new framework based on sparse coding and temporal pyramid matching (TPM) is proposed for depth-based human action recognition. Especially, a discriminative class-specific dictionary learning algorithm is proposed for sparse coding. By adding the group sparsity and geometry constraints, features can be well reconstructed by the sub-dictionary belonging to the same class, and the geometry relationships among features are also kept in the calculated coefficients. The proposed approach is evaluated on two benchmark datasets captured by depth cameras. Experimental results show that the proposed algorithm repeatedly achieves superior performance to the state of the art algorithms. Moreover, the proposed dictionary learning method also outperforms classic dictionary learning approaches.