R ECOGNIZING A CTIONS USING O BJECT S TATES
A BSTRACT
Object-centric actions cause changes in object states, including their visual appearance and their immediate context.
We propose a computational framework that uses only two object states, start and end , and learns to recognize the underlying actions.
Our approach has two modules that learn subtle changes induced by the action and suppress spurious correlations.
We demonstrate that only two object states are sufficient to recognize object-centric actions.
Our framework performs better than approaches that use multiple frames and a relatively large model.
Moreover, our method generalizes to unseen objects and unseen video datasets.
I NTRODUCTION
Objects in real world exists in different color, shape and structure.
For instance, an apple can be red or green, cut or peeled, raw or riped.
Growing interest in object state transformations is still limited to image based datasets such as Isola et al.
and Yu & Grauman (2014), which study a set of object-state transformations for edible objects and shoes.
However, using objects states for videos is still uncommon.
In Fathi & Rehg (2013), actions are detected based on changes in object states.
Similar to Fathi & Rehg (2013), we hypothesize that most object-centric actions induce specific changes in object states, where state includes object’s appearance and immediate context. Humans are adept at identifying such actions simply based on the state transformations.
In the commonsense reasoning literature, this notion of state transformation is also known as fluent of an object Liu et al.
; Fire & Zhu (2015); Mueller (2006), defined as the status of an object which varies over time.
For instance, a lemon’s fluent changes from whole → pieces over time, associated with cut action.
In this work, we propose a computational model that embodies the intuition of leveraging object state transformations, or fluent of an object, to reveal the underlying action being performed.
In fact, we limit our focus to the question: Can we infer actions just from emphtwo distinct object states – start and end ?
Note that we do not claim that more intermediate states will not be beneficial for modeling transformations ( e.g , see Wang et al.
(2016b)).
However, for object-centric actions, we show that only using two states can outperform standard approaches using many more frames.
Moreover, prior works in action recognition domain Zhou et al.
(2018a); Simonyan & Zisserman (2014);
Yang et al.
(2020a) use training and test splits based on actions, i.e , for label cut apple , few videos are part of train and some are testing videos. However, we focus to learn the action or transformation of any object. Hence, we create dataset splits where for an action, objects are split up as seen and unseen objects as training and test set respectively. Firstly, our proposed method selects two bounding boxes, representing start and end states, and then extracts their representations.
Our manipulation module uses these representations and learns a fluent representation used to classify actions.
We use cooking video datasets: EpicKitchen
Damen et al.
, YouCook2 Zhou et al. (2018b), and EpicKitchens100 Damen et al.
(2020), and firstly train on just EpicKitchen55 and show that our model can generalize to YouCook2 and
EpicKitches100.
Second, we hold out several unseen objects for all actions while training and generalize to transformations of these unseen objects in test set.
We show that our approach is able to improve performance for both object-manipulating and context-manipulating actions.
To summarize, our contributions are as follows: (a) developing a manipulation module with object and context modules, that can understand object state changes, (b) demonstrating that amplifying relevant information from just two states is enough to outperform architectures that use several frames, and (c) demonstrating the generalization of our approach to unseen objects datasets. [Figure] [Figure]
A PPROACH
In this work, our goal is to recognize object-centric actions using start → end object state changes.
A state manipulation can correspond to object-manipulating actions that change object’s visual appearance and context-manipulating actions that change its context.
Our approach is inspired by attention correlation between labels and images Xu et al.
Our Manipulation Module shown in Figure 1 are designed to amplify similarity between these states to suppress spurious signals and amplify the differences to leverage subtle changes between start and end states.
Currently, no video dataset exists with specific object states before and after an action.
Hence, as a pre-processing step, we first extract object states, or regions of the object in two distinct frames before and after the action has been performed.
More details of are in provided in the appendix A.
Once two distinct object states are captured, we propose our method using Manipulation Module, as shown in Figure 1.
Hence, the input to our model are the start ( I s ) and end ( I t ) states.
We use a pre-trained deep neural network to extract features f s and f t ( f s , f t ∈ R n ; n = 2048) from I s and
I t , respectively.
We then pass f s and f t through an FC layer with ReLU non-linearity to obtain g s and g t ( g s , g t ∈ R n ).
The features g s and g t are then passed through the Manipulation Module for further processing.
Our hypothesis is that the similarities and differences between deep features g s and g t capture the manipulation in objects and its context.
Hence, our manipulation module consists of -– Context
Module and Object Module .
The object module captures and difference in object states, whereas context module learns to amplify the subtle similarity between object’s context or environment that can be used for classification.
Both module take g s and g t as input to produce representations f c and f o respectively.
The two features f c and f o are then fused to produce the final representation.
We describe one of the modules (the context module) in detail below and the other module is implemented in a similar fashion with minor changes.
Context Module captures the similarity between the start and end states.
We first take the outer product of g s and g t denoted as O =
g s ⊗ g t ( O ∈ R n × n ).
Element o i,j represents the similarity between i th element of g s with j th element of g t .
Moreover, let o i ∗ and o ∗ j represent the i th row and j th column of O respectively.
Then, o i ∗ captures the similarity of all the elements in g t with respect to i th element of g s .
To know the most similar element among g t with respect to i th element of g s , we can take a row-wise softmax over O .
Similarly, for j th element of g t , column o ∗ j represents the similarity with all the elements of g s .
Using a column-wise softmax , we can interpret the most similar and least similar element of g s with respect to j th element of g t .
Therefore, by applying column-wise and row-wise softmax , we get two matrices, A and A ′
( A , A [-(cid:
 )-]′ ∈ R n × n , n = 2048 ). A
[Equation]
i
[Equation]
and A ′ [
[Equation]
[Table]
where γ is the inverse temperature parameter.
We compute row and column sum for A and A ′ respectively, to get raw state masks, m t and m s in Eq.1 To get the final masks, we normalize with
ReLU (Eq.2).
We compute the final representation f c of the context module in Eq.
3, where ⊙ is the elementwise multiplication and W ∈ R 2 n × n and b ∈ R n .
[Equation]
Similarly, we can get the object representation f o , except instead of using similarity between states ( A and A ′ ), use distance measure ( D and D ′ ) computed as
D = 1 ./
A and D ′
=
A ′ where 1 ./ ∗ is the element-wise inversion of matrices.
The final representation is the output of maxpooling of f c and f o .
We train using a linear classifier in an end-to-end fashion using softmax
Cross-Entropy loss function.
E XPERIMENTS AND R ESULTS 3. D ATASET AND E VALUATION
We use Epic-kitchens-55 (EK-55) Damen et al.
(2018) which has kitchen activities captured with an egocentric camera.
Among all actions, only 13 actions are object-transforming and 30 actions are context-transforming.
In total, we use 7342 videos, 2120 for object-transforming and 5222 for context-transforming actions.
We create five train/val/test splits of the dataset and report the mean performance across splits.
Each split has all 13 object-transforming actions and randomly chosen 13 context-transforming actions.
Note that the test set, referred to as ‘unseen object-actions’ in each split contains actions involving objects unseen during training.
We report overall accuracy and Mean accuracy.
More details are provided in appendix A.
3. Q UANTITATIVE R ESULTS
We compare our model with two types of action recognition models:
Spation-temporal Baselines.
Our setup is RGB based, hence we compare with the best RGB-only model for action recognition Yang et al.
(2020a)
; Zhou et al.
(2018a).
TPN Yang et al.
(2020a)
[Figure]
Object States Baselines.
We compare with three other baselines which use exactly as input as ours, i.e , 2 object states.
Fathi et al.
Fathi & Rehg (2013).
detect changing regions in video and train
SVMs for each action.
We use our implementation for Fathi & Rehg (2013), with deep features for regions to replicate the results.
We propose two more object states baselines: MaxpoolResFeats uses maxpool ( f s , f t ) and ConcatResFeats uses concatenation ( f s , f t ) for action classification.
Note that we follow the notation explained in section 2, where f s and f t are ResNet features for start and end states.
For a fair comparison, these object states baseline models have around the same number of trainable parameters as our model.
To keep it consistent with TPN Yang et al.
(2020a)
and TRN Zhou et al.
(2018a), we use
ResNet50 He et al.
as our backbone network and all setups follow three-crop testing evaluation protocol.
The results are reported in Table 1 for unseen object actions (test set).
Fordifferent splits for action classes, we denote the final accuracy as mean and confidence interval with 95% confidence, for object-transforming and context-transforming actions separately as well.
Our model outperforms all the baselines for 3 and 8 frames and bounding boxes setup, not only on majority classes but also performs better for minority classes (Table 1).
We show the of our object and context modules for capturing the similarity and dissimilarity, to significantly improve objecttransforming actions, as compared to using max of ResNet features (MaxpoolResFeats baseline) and concatenation baseline.
3. A BLATION S TUDIES
We also do an ablation study on one of the validation set split (Seen object actions), to justify our architectural choices Table 2.
Object Baselines: Using only ResNet features for action recognition, either maxpool (MaxpoolResFeats in Table 2) or concatenation (ConcatResFeats) of object states, also performs better thanand 8 frames models.
( start → end )
Attention: This is akin to using a single module between start and end states.
We use a non-spatial attention block, inspired by Wang et al.
; He et al.
More details on how [Table] this attention is different than our method are in appendix.
This setup shows we need ways to amplify context and object features between start and end states.
Module Alternative: For understanding the significance of our proposed Manipulation Module, we model object manipulating actions as difference between features for object states and mean of features for context transforming actions.
Fusion of the two is done in the same way with maxpool.
This experiment shows need for two separate object and context modules.
3. Q UALITATIVE R ESULTS
We show qualitatively how our method performs on out-of-domain data, such as Epic Kitchens100 Damen et al.
and
YouCook2 Zhou et al. (2018b).
Although, our state detector only finds a limited set of seen objects for these datasets, we achieve, 65.60% Top1 accuracy and 90.10% Top accuracy for Epic Kitchens-100, for unseen and seen actions.
Some examples of correctly classified actions are shown in Figure 2.
Note that these samples are classified correctly by our model among top 5 predictions.
C ONCLUSION
We presented a framework for recognizing object-centric actions using object states.
We show that by leveraging object states object-manipulating and context-manipulating actions can be learnt.
We propose a novel manipulation module, with object and context blocks, that capture the subtle similarities and differences between object states.
We also show generalization to unseen objects and out-of-domain datasets.
In future work, we intend to extend object states for procedure planning and action segmentation.
A A PPENDIX A. R ELATED W ORK
The study of object states and how they transform from one to the other has recently garnered interest in the computer vision community Isola et al.
; Yu & Grauman (2014); Nagarajan & Grauman (2018); Chen et al.
Nan et al.
; Atzmon et al.
; Yang et al.
(2020b)
; Misra et al.
(2017); Wei et al.
; Bertasius & Torresani (2020);
Liu et al.
; Fire & Zhu (2015); Zhou & Berg (2016).
Understanding these states could enable an object recognition system to generalize better to unseen states, as studied in Bertasius & Torresani (2020); Misra et al.
However, most studies have focused on image-based datasets Isola et al.
; Yu & Grauman (2014); Misra et al.
(2017), where the goal is to capture the underlying distribution of states ( e.g , different sizes of heels in shoes, old and new laptops).
Some of these states are just attributes of objects ( e.g , red car and blue car) and are generally static over time; whereas other states are transformations of objects over time ( e.g , raw → ripe, whole → pieces, on table → on plate).
These different types of states provide very different information.
In this work, we leverage the state transformations that happen over time for understanding transforming actions.
For the rest of this paper, we will refer to changes in object states (including visual appearance and context) over time as object state transformations .
For action recognition, most modern approaches can be grouped into two categories: (a) models utilizing temporal and spatial convolutions over videos to classify the actions Girdhar et al.
Simonyan & Zisserman (2014); Feichtenhofer et al.
; Karpathy et al.
(2014); Lin et al.
Wang et al.
(2016a)
; Yang et al.
(2020a)
; Zhou et al.
(2018a), and (b) models which rely on humanobject interactions over time to recognize actions Gupta et al.
; Khan et al.
; Shen et al.
; Wu et al.
; Tan et al.
Xiao et al.
In both categories, the focus is generally to leverage as much temporal information as possible; this is evident with the focus on recent works to utilize as many frames as computationally feasible to understand actions.
Similar to our approach, Bakr et al.
; McCandless & Grauman (2013); Fathi & Rehg (2013) propose using state transformations for action recognition; e.g , Bakr et al.
(2019) uses five keyframes for modeling transformations for action recognition.
Closest to our approach is the work of Fathi and Rehg Fathi & Rehg (2013), which also uses two object bounding boxes as states for action recognition.
However, our approach explicitly amplifies affinity and differential signals between states to extract fluent representations, which are used for action recognition.
A. D ATASET S PLITS E XTENSION
We use Epic-kitchens-55 (EK-55) Damen et al.
as our primary dataset.
EK-55 has kitchen activities captured with an egocentric camera.
In total, it has 352 sparsely annotated bounding boxes for objects and 125 actions.
Among all actions, only 13 actions are object-transforming andactions are context-transforming.
We choose these actions such that each action has atleast 40 videos in total.
Our subset is also impacted by pre-processing step (object bounding boxes detection), since we require two bounding boxes of the object of interest either from the detector or the ground truth.
This drastically reduces the number of videos with a long-tailed distribution over the classes.
In total, we use 7342 videos, 2120 for object-transforming and 5222 for context-transforming actions.
Due to imbalanced number of object-manipulating and context-manipulating actions, we create five train/val/test splits of the dataset and report the mean performance across splits.
Each split has all 13 object-transforming actions and randomly chosen 13 context-transforming actions.
The test set in each split contains actions involving objects unseen during training.
This is different from the typical test set provided for EK-55 Damen et al.
We argue that using those splits do not provide evaluation of the model learning actions or latching onto object features.
Therefore, by having different objects in train and test set inspired by Nagarajan & Grauman (2018), we ensure that the model learns the action, and does not rely only on object visual features.
We refer to test set as “unseen object-actions” and validation set as “seen object-actions” set.
A. P REPROCESSING : E XTRACTING OBJECT STATES
As discussed above, analyzing object state provides useful information for inferring actions.
For this purpose, we first identify two frames in a video, one before and the other after the action has [Table] been performed, corresponding to different states of the object.
We choose the highest confidence bounding box, extracted using an object detector, of the object of interest from each frame as the inputs to our model.
We refer to these two regions as start and end states.
In particular, we leverage the temporal annotations provided with the video to identify clips where the action is performed.
We extract bounding boxes using an object detector on the first N frames of the first clip and the last N frames from the last clip, and choose the box with highest class confidence score for the object under consideration as start and end states, respectively.
A. I MPLEMENTATION D ETAILS
We use ego-centric kitchen video dataset, Epic-Kitchen-55 (EK-55) Damen et al.
(2018) for training.
This dataset provides temporal action localization labels and has object-centric actions.
Each video can have multiple clips of same object-action pair.
We consider first and last clip for the same objectaction in proximity, so that accurate start and end states are captured for objects.
After merging the first and last clips from the video, we use one merged clip for each object-action pair from a video.
For extracting object states, use a Faster-RCNN Ren et al.
(2015) detector, pre-trained on EK55 Damen et al.
(2018), which has sparsely annotated ground truth bounding boxes for 295 object classes.
We apply it on two sets of images from an object-action clip with 30fps: first and lastframes.
From each set, the object’s bounding box with the highest confidence is chosen, which should be higher than 65%.
This gives use start and end object states.
Features are extracted using
ResNet50 He et al.
(2016) pretrained on ImageNet Deng et al.
(2009) from the penultimate layer, and are of dimensions 1 × 2048 .
Extracted features are then passed through our fluent Attention module, and classified for all actions.
For our model, we use learning rate 1e-5, Adam optimizer and ReLU activation.
Since our dataset follows long-tail distribution, we use weighted cross-entropy loss over the actions.
We empirically choose a temperature value as 100.
We use L2-normalization on output of both context and object, before fusion.
A. R ESULTS AND A BLATION
We also show results on seen object in Table 1.
Similar to unseen objects, our model outperforms previous models with a margin for all categories.
( start → end )
Attention: We use a non-spatial attention block, inspired by Wang et al.
; He et al.
Given an input feature ( start / end state, denoted as g s / g t ), three linear projections are applied to compute query ( q ) with respect for before state g s , and key ( k ) and value ( v ) with respect to after state g t .
The attention ( a ) is weighted sum of value representation, where the weight ( M ) is dot product between the query and the corresponding key, followed by a softmax normalization as
[Equation]
where o represents attention output for start state with respect to end state, and λ is the temperature parameter.
Note that g s , q, k, v, o, a ∈ R n ; M ∈ R n × n , where n = 2048 .
This experiment shows we need different ways to amplify similarities and differences between start and end states.