Time-Evolving Conditional Character-centric Graphs for Movie Understanding
Abstract
Temporal graph structure learning for long-term human-centric video understanding is promising but remains challenging due to the scarcity of dense graph annotations for long videos.
It is the desired capability to learn the dynamic spatio-temporal interactions of human actors and other objects implicitly from visual information itself.
Toward this goal, we present a novel Time-Evolving Conditional cHaractercentric graph (TECH) for long-term human-centric video understanding with application in Movie QA.
TECH is inherently a recurrent system of the queryconditioned dynamic graph that evolves over time along the story and follows throughout the course of a movie clip.
As aiming toward human-centric video understanding, TECH uses a two-stage feature reﬁnement process to draw attention to human characters and their interactions while treating the interactions with non-human objects as contextual information.
Tested on the large-scale TVQA dataset, TECH clearly shows advantages over recent state-of-the-art models.
Introduction
Capturing the dynamic story in long-term human-centric video presents a powerful testbed for temporal graph modeling.
An important setting is Movie QA, where the questions are to “probe” a certain aspect of the story – answering them would demonstrate a high degree of understanding.
Here the challenges lie in the analysis of, and reasoning about the long-term temporal dynamic relationships of the human characters and the surroundings with the guidance of the question being asked.
As movies are long and contain an abundant amount of information, it is also key to selectively attend to the most relevant visual entities in relation to the query.
Current methods [4, 5, 7, 6, 10, 11, 12, 17] fall short of meeting these challenges.
Dynamic graph learning offers a natural scheme to solve these problems since graphs are coherent structures to represent the human-centric relationship within a movie clip.
Toward this end, we propose a novel T ime-
E volving C onditional c H aracter-centric graph (TECH) that inherits key advanced properties of temporal graph learning for long-term human-centric video understanding with application in Movie QA task.
Unlike other approaches where graphs often stay static with pre-existed relations, TECH is a recurrent system of query-conditioned dynamic graphs that evolves along the time dimension.
These dynamic structures allow TECH to effectively model the progressive nature of information in movie data.
In addition, TECH introduces a two-stage representation reﬁnement process that allows it to focus on the interactions between human characters given the contextual information of their interactions with the surrounding environment.
Arguably, understanding the rela- tionships across human characters is key in human-centric video understanding as these relationships are often the topic of interest by human queries.
Tested on the large-scale TVQA dataset, we clearly demonstrate the modeling beneﬁts of TECH against the most recent state-of-the-art models in Movie
QA.
TECH: Time-Evolving Conditional Character-centric Graph
We present a novel method for solving Movie QA by representing the visual content of the movie as a dynamic query-conditioned graph of character-object that evolves over time.
Following [2], we detect and track visual features of entities (human characters and non-human objects) in a video, assuming that they live throughout the course of the video of S shots.
This results in N connected sequences of entities over shots V = { z n,s } N,Sn =1 ,s =1 .
Details of visual processing are provided in the supplementary.
The query and answer are encoded similar to those in [4, 11]: Each query is represented as contextual embedding matrix Q = { Q p ∈ R 1 × d } L Q p =1 , and a global vector q g =
mean_pooling (Q) ∈ R d .
Give these visual and linguistic features, we build our model termed T
ime- E volving C onditional c H aracter-centric graph (TECH) to capture the dynamic interactions between human characters and their surroundings in response to the query.
Our TECH is composed of three major modules: (1) TimeEvolving Conditional Graphs over Video Shots, (2) Query-speciﬁc Temporal Information Aggregation and (3) Hierarchical Character-centric Graph.
The readout of TECH is a joint representation ready for answer prediction.
Fig.
1 illustrates how these components interact with each other.
2.1 Time-Evolving Conditional Graphs over Video Shots
Shot-based Entities Graph:
We build a dynamic graph G s ( V s , E s ) for each video shot s based on the interactions between entities living in the same shot.
Vertices V s are a set of N entities
V s = { z n ( s ) } Nn =1 taken from the representative frame of the shot s .
For the sake of readability, we adhere refer to z n as the feature of object n at shot s .
The edges E s ∈ R N × N are an adjacency matrix implying the relationships between the entities.
We obtain E s as a query-induced correlation matrix of the vertices’ features:
[Equation]
where norm is a normalization operator which is the softmax function in our implementation.
W a is learnable parameters.
Entity-based Graphs Evolving over Time: Recall that a video typically contains multiple video shots where each shot s is represented as the above deﬁned graph G s
( V s , E s ) .
It is crucial to effectively connect these shot-based graphs to reﬂect the continuous ﬂow of the spatio-temporal nature of movie clips.
In this work, we design a recurrent graph network that iteratively reﬁnes the representation of each vertex at the present shot s by the information received from its neighbors within the current shot and the information of itself at the previous shots.
Mathematically, assuming H 0 s = { z n ( s ) ∈ R 1 × d } Nn =1 as the representation of the vertices at iteration 0 , the reﬁned representations of the vertices at iteration l are given by:
[Equation]
where l ∈ [1 , L ] , E s is the adjacency matrix given by Eq.
H Ls − 1 is outputs of the reﬁned representations of entities at the previous video shot.
At the end of this stage, we have a set of S spatio-temporal representations for each of human characters and non-human objects across video shots ¯ Z n = (cid:8) H Ln,s | H Ln,s ∈ R 1 × d (cid:9)
S s =1 .
2.2 Query-speciﬁc Temporal Information Aggregation
While visual content in a given movie clip may contain a large amount of information, the information relevant to a query is more speciﬁc.
Thus, we design a query-speciﬁc temporal information aggregation module to retrieve visual moments that are only relevant to the query:
[Equation]
where Q p is the p -th word in the contextual query Q of length L Q , and the attention function is deﬁned in Eq.
2.3 Hierarchical Character-centric Graph
In human-centric video understanding, there are two types of entity-level relations of interest: character-character and character-object.
While one would naturally be more interested in the relationship between characters and how it develops throughout the course of a movie clip, the interactions between the characters with their surroundings often provide contextual information.
Inspired by this observation, we design a hierarchical character-centric graph to characterize the different levels of interest in the character-object relationships as respect to human queries.
In particular, we treat the visual entities ˜ Z
= { ˜ z n } Nn =1 obtained by the aggregation function described above as of two different types: human characters C =
{ c i } Ii =1 ∈ R I × d , and non-human objects
O = { o j } Pj =1 ∈ R P × d , where ˜ Z = C
∪ O .
The graph module operates through two stages of feature reﬁnement: object-to-character , and character-to-character .
Object-to-Character Reﬁnement For each
detected human character c i in C , we build a dynamic graph G c i , whose nodes are the human character c i and all other detected non-human objects in O .
Let X i
= { c i , O } as the features of the nodes, we follow the procedure of building dynamic graphs with shot-based features in Sec.
2.1 to build the graph
G c i of node features X i and an adjacency matrix
A O2C i .
Finally, we reﬁne the representation of the nodes using Deep Graph Convolutional
Network (DGCN) [9]: ¯ X i = DGCN
X i , A O2C i ) .
Character-to-Character Reﬁnement Let ¯
C
=
{ ¯ x i } Ii =1 be the the character embedding after the the last step.
We then build a dynamic graph among only the characters with node features ¯ C and the relationships between the characters are denoted by an adjacency matrix A C 2 C .
We then again reﬁne the representation of the characters with DGCN:
ˆ C = DGCN ( ¯ C, A C2C ) .
Finally, we retrieve a joint representation, ready for answer decoding: r = 1 L Q (cid:80) L Q p =1 Attention ( Q p , ˆ C, ˆ C ) ∈ R d .
The answer decoder is presented in the supplemental material.
Experiments
We evaluate the effectiveness of TECH on TVQA dataset [10].
As our main focus is to understand characters and the relationships with their surroundings solely from a visual perspective, we do not use subtitles and timestamp annotations in our experiments.
The dataset contains a total of 21 ,clips from six American TV series.
Each movie clip is associated with seven multi-choice-related questions with ﬁve candidate answers, resulting in 152 K questions.
[Table]
Comparison against state-of-the-art methods Tablepresents the performance of our proposed method TECH against most recent state-of-the-art (SOTA) methods on the TVQA validation set.
For fair comparisons, we do not mention models that are extensively pretrained on largescale video-text data or models that use external dense captions such as [1, 3, 4, 18, 19].
As showed, TECH sets a new state-of-the-art result by achieving 47.79% in QA accuracy on the validation set, which is around 2 absolute points better than the latest advances in Movie QA.
The results conﬁrm the beneﬁts of utilizing structured data by
TECH compared to existing methods in movie understanding task.
Ablation studies To provide more insights of TECH, we conduct multiple ablation studies to examine the contributions of individual modules in TECH.
Details of the experimental results are given in Table 2.
[Table]
[Table]
Effectiveness of the query-induced correlation matrix : We examine the contribution of our query-guided correlation matrix as described in Sec. 2.1.
In particular, we deliberately set the values of the edges of the graphs with uniform values.
This results in a considerable decrease in performance, conﬁrming that TECH effectively selects meaningful connections between characters and their surrounding objects toward answering a given query
(cid:66) Effectiveness of the time-evolving conditional
graphs
To verify the impact of our time-evolving conditional graphs, we ablate the recurrent graph reﬁnement step as in Sec.
2.1 from the full model.
This degrades the performance of TECH by around 0.8 points.
The result clearly demonstrates the signiﬁcance of modeling the evolving of information over time in movie understanding.
(cid:66) Effectiveness of the query-speciﬁc temporal information aggregation : Rather than using the aggregation operator as in Sec.
2.2, we utilize the mean pooling operation instead
Results show that doing this would lead to a marginal decrease in performance
(cid:66) Effectiveness of hierarchical character-centric
graph
: We investigate the effectiveness of our hierarchical character-centric graph by
replacing the structure as in Sec 2.3 with the simple average pooling operator to get the representation as an input of the answer decoder.
Results suggest that this would hurt the model’s performance by nearly 0.7 points.
Conclusion
This paper introduced a T ime-
E volving C onditional c H aracter-centric graph (TECH) for long-term human-centric video understanding with application in Movie QA.
Built upon sequences of human characters and non-human objects living in space-time, we designed TECH as a recurrent system of query-dependent dynamic graphs that allow information to effectively ﬂow from early points to later points in time.
The design of TECH has shown that modeling temporal graphs was crucial for movie understanding as it helped reﬂect the evolution of events in movie clips.
TECH also showed the beneﬁts of paying attention to human characters and their interactions within a movie clip over the interactions with other non-human objects.
With advantages in model design, TECH clearly advances recent state-of-the-art Movie QA models on the large
A Supplemental Material A.1 Model Details
Visual features A movie clip typically contains multiple video shots by nature.
Hence, we divide each video input into shots using the common shot transition detection algorithm TransNet V2 [14].
For each video shot, we parse it into visual entities of human characters and non-human objects using object detection.
These detected human characters and non-human objects are tracked and linked across shots using their visual salience and geometrical positions within respective video frames following [2].
Formally, given a video with S shots, we only take one key frame per shot to reduce computation cost.
We now represent the given video as a total of N sequences of entities with appearance features denoted by Z a = { z an,s | z an,s ∈ R 1024 } N,Sn =1 ,s =1 and positional features denoted by Z p = { z pn,s | z pn,s ∈ R 4 } N,Sn =1 ,s =1 , then obtain position-aware visual features of the detected entities at shot s using the following multiplicative gating mechanism:
[Equation]
where σ ( · ) ∈ (0 , 1) is the sigmoid function and (cid:12) denotes element-wise product between two vectors.
Attention function We utilize the attention mechanism [15] that takes as input a triplet of query q ∈ R 1 × d , keys K ∈ R M × d , and values
V ∈ R M × d :
Attention ( q, K, V ) :=
[Equation]
[Equation]
where W k , W q and W v are learnable parameters.
Answer decoder As the task Movie QA in TVQA is deﬁned as a multiple choice test, given input as the joint representation r and a set Q g = { q g,i } 5 i =1 represented ﬁve question-answer choice pairs, we use a MLP network with two fully-connected layers, then simply normalize the output by the softmax function to rank answer candidates:
[Equation]
Finally, the predicted answer is taken as ¯ y
=
argmax i (cid:18)(cid:110)
y prob i (cid:111)i
[Equation]
(cid:19) .
Loss function In this paper, the cross-entropy is selected as the loss function to train our model.
A.2 Implementation Details
We use Faster-RCNN trained on MS-COCO for frame-wise character-object detection and DeepSort for multiple object tracking.
The RoBERTa pretrained model [13] is utilized to embed the QA pairs into vectors with a dimension size of 768 .
The number of characters and objects per video is 4 and 6 , respectively.
The dimension of hidden features in our model is set to 128 .
The default setting of
GCN layers is 2.
We apply the Adam optimizer [8] with an initial learning rate of 10 − 4 .
A.3 Qualitative Analysis
Figure 2 provides examples where TECH handles successfully while DenseCap struggles.
Keys to answering these questions lie in the capability to capture the long-term temporal dynamics of human characters with their surrounding environment.
DenseCap without explicit long-term temporal dependencies modeling struggles to predict correct answers.