1 Abstract-Egocentric activity recognition in first-person videos has an increasing importance with a variety of applications such as lifelogging, summarization, assisted-living and activity tracking. Existing methods for this task are based on interpretation of various sensor information using pre-determined weights for each feature. In this work, we propose a new framework for egocentric activity recognition problem based on combining audio-visual features with multi-kernel learning (MKL) and multi-kernel boosting (MKBoost). For that purpose, firstly grid optical-flow, virtual-inertia feature, log-covariance, cuboid are extracted from the video. The audio signal is characterized using a 'supervector', obtained based on Gaussian mixture modelling of frame-level features, followed by a maximum a-posteriori adaptation. Then, the extracted multi-modal features are adaptively fused by MKL classifiers in which both the feature and kernel selection/weighing and recognition tasks are performed together. The proposed framework was evaluated on a number of egocentric datasets. The results showed that using multi-modal features with MKL outperforms the existing methods. Index Terms-egocentric, first-person vision, activity recognition, multi-kernel learning I. INTRODUCTION NCREASING use of social media has led to a huge amount of multimedia content sharing where videos are as important as text and audio-based posts. The main driving factor behind the increase in video is the widespread use of handheld cameras, sports and action cameras (i.e. GoPro), mobile phones and wearable cameras as accessories (i.e. Snap Spectacles, Google Glass). For this reason, first-person (i.e. egocentric) videos, in which the actors are getting involved in the activities or events, have become popular. In first-person videos, the world is seen from the perspective of the actor within the context of the actor's activities and goals. Some of the application areas of egocentric video analysis are: activity recognition, egocentric video summarization, attention localization and daily care of elderly and disabled patients. Most of these fields are still being studied as research problems given the unstructured nature and motion dynamics of first-person videos. Popular use of egocentric videos in daily life has created the need of summarizing, organizing and analyzing them. A recent survey on summarization of egocentric videos reveals the diversity and abundance of the objectives, approaches and evaluation strategies [1] . Current activity recognition On the other hand, egocentric videos have different characteristics than third-person videos (i.e. fast scene transitions, rapid change in illumination, motion blur and high ego-motion) and require particular approaches. The studies on egocentric activity recognition can be grouped as object and motion-based approaches [6] . In object-based methods, activity recognition is performed using the object(s) detected in videos (i.e. detection of cheese and bread objects imply "making cheese sandwich" activity) [7] . The performance of object-based approaches is directly dependent on the performance of object recognition methods and they are vulnerable to occlusions. Motion-based approaches make use of the assumption that different types of activities such as running, walking, stair climbing, and writing involve different body motions, and these motion patterns can be used for recognizing activities [8, 9] . To obtain this motion pattern, various types of sensors are used such as inertial measurement units (IMUs), GPS or eye-trackers with more specialized equipment (such as SenseCam, Google Glass) in addition to optic and audio sensors. However, these special devices are not widely used as they are limited in battery life and storage capacities. The methods for egocentric activity recognition use various types of features and present different strategies to learn the activities such as supervised or unsupervised learning. One of the most prominent approaches is to combine multiple features using the data generated by different sensors. In this method, each feature extracted locally or globally is considered as a separate channel having equal weights [10] Peter Jančovič is with Electronic, Electrical and Systems Engineering,
Authors in REF propose a multi-modal method for egocentric activity recognition where audio-visual features are combined with multi-kernel learning and boosting.
Multi-modal Egocentric Activity Recognition using Audio-Visual Features
ArXiv
ArXiv
Computer Science