Multimodal Video Understanding using Graph Neural Network
Abstract
Majority of existing semantic video understanding methods process every video independently without considering the underlying inter-video relationships.
However, videos uploaded by users on social media platforms like YouTube,
Instagram etc.
exhibit inter-video relationship which are a reflection of the interest, geography, culture etc. of the users.
In this work, we explicitly attempt to model this inter-video relationship, originating from the creators of these videos using Graph Neural Networks (GNN) in a multimodal setup.
We perform video classification by leveraging the creators of the videos and semantic similarity between videos for creating edges and observe improvements of 4% in accuracy.
Introduction
Semantic understanding of videos has received lot of attention from both the research community and industry.
Advancements in this field have been accelerated by the presence of large scale video understanding datasets like [14, 3, 8, 20, 17, 25, 16, 5, 4, 13, 29, 18] and deep learning models like [2, 6, 7, 12, 23].
Majority of these tasks process every video sample in isolation during the training and inference stage.
The core assumption of this approach is that the videos are independent from each other and underlying inter-video relationships, if any, would be implicitly modelled by the network.
However, in practical scenarios, videos originating from similar sources are often correlated, especially on social media platforms.
For example, videos uploaded by an individual on a social media platform would be related by the interest or skill-set of the individual.
An individual with a deep interest in comedy videos has higher probability of uploading and creating comedy videos.
Similarly, an expert in cooking might create more food related videos.
This inter-video relationship alludes to the presence of latent relationship existing between different videos which could be modelled for improving video understanding.
While some of the previous works have modelled the intra-video spatio-temporal relationships [21, 28, 24, 19, 22], the efficacy of modelling inter-video correlations for improved semantic video understanding has not been explored.
[1] explored the inter-video relationship but mainly from the semantic similarity perspective in an unsupervised settings.
In this work, we explore the inter-video relationship between videos using
Graph Neural Networks (GNN) in a multimodal setup using the 3MASSIV dataset.
3MASSIV dataset has been sourced from short-video application platform Moj and contains annotated social media videos along with masked identity of the uploaders of these videos.
We leverage the creators of the videos and semantic similarity between them for connecting the videos to create the graph.
To the best of our knowledge, our work is the first attempt towards performing multimodal video understanding by modelling inter-video relations.
We perform extensive experiments using Graph
Convolutional Network (GCN) [15] and GraphSage [11] in unimodal and multimodal setup and demonstrate the advantages of modelling inter-video relationships.
Our Method 2.1 Task
We model the semantic understanding of video as a node classification problem where every video v i ∈ V is a node in the graph G and is annotated with a category y i ∈ Y .
Two videos, v i and v j , created by the same creator are connected with each other using an edge e ij ∈ E .
By leveraging the semantic content encapsulated in the videos and inter-video relationships, the task is to classify each video v i into one of the categories y i ∈ Y .
We model this graph G = <
V, Y, E > using
Graph Convolutional Network ( GCN ) [15] and GraphSage [11] and use cross-entropy loss ( L ce ) for classification.
We present the details of GraphSage in next section and GCN in Section A.1.
2.2 GraphSAGE
We represent the videos (node) of the graph G = <
V, Y, E > by d -dimensional semantic features.
Let, H be the feature matrix formed by stacking d -dimensional features ( h 1 , h 2 , ..., h n ) of n nodes or videos.
Let, W ( l ) represents the weights of the l th layer and N represent the neighbourhood function where, N ( i ) gives the list of nodes belonging to the neighbourhood of v i node.
For every node, GraphSage uniformly samples nodes from the neighbourhood and aggregates them using an aggregator function ϕ as shown in Equation 1.
[Equation]
The aggregated features are then concatenated with the features of the node and transformed using weight matrix W ( l ) and passed to the next layer after applying an activation function φ and normalization.
[Equation]
where, h ( l +1) i represents the transformed features of node i at layer l + 1 and ∥ h ( l +1) i ∥ 2 represents the norm of the features.
2.3 Video Representations
We represent the videos as nodes with features h ⊂ =
R d extracted from pretrained spatio-temporal and audio models.
We used ResNext50-3D [27] (pretrained on Kinetics400) for extracting the spatiotemporal features and CLSRIL-23 [9] models for representing audio.
These features are averaged across the temporal dimension for normalizing the temporal dimension.
More details about feature extraction is present in Appendix Section A.2.
2.4 Edge Creation
We generate the edges between videos for the graph using two constraints:
(a) Semantic Similarity: We create an edge between two videos if the cosine similarity between the features of the nodes exceeds a predefined threshold.
The optimum value of this threshold is selected using cross validation.
(b) Creator Information: An edge is created between two nodes if they belong to the same creator and semantic similarity is greater than the threshold.
Dataset
We experiment with 3MASSIV [10] dataset which has been sourced from popular short-video platform
Moj .
3MASSIV contains the masked identities of the creators of the videos along with human annotated
[Table] [Table]
labels.
3MASSIV contains 50K annotated videos belonging to 34 popular social media concepts and comprises of 11 Indic languages with an average duration of around 20 seconds.
The videos in this dataset have been contributed by more than 20K social media users, thus providing rich and diverse graphical information for our task.
Architecture and Training Details
We use 2-layered graphical networks for our experiments.
We used ReLU as the non-linearity and added a batch normalization layer between the two layers.
We trained our models on A100 GPUs for 150 epochs with a learning rate of 0.001 and Adam Optimizer using 2048 as batch size.
We used
DGL [26] library.
We use the training, validation and test split provided with the dataset for our experiments and report the mean and standard deviation of Top-1, Top-3 and Top-5 accuracy overrandom runs .
[Figure]
[Table]
Results and Discussion 5.1 Unimodal Experiments
In Table 1, we report the performance using visual features.
We train the models ( GCN base and
GraphSage base ) without edges to baseline the results in absence of inter-video relations.
On using the semantic similarity ( GCN cos and GraphSage cos ) for creating edges between videos, we do not observe gains.
However, on leveraging the creator relationships ( GCN cre and GraphSage cre ) between the videos, we observe substantial gains.
This demonstrates the efficacy of the creator edges towards improving semantic understanding of videos.
Overall, we observe that across all the configurations, GraphSage shows improved performance over GCN .
5.2 Multimodal Experiments
Videos are multimodal in nature due to presence of visual and audio channel.
We explore the multimodality in different ways.
(a) Firstly, we use visual features for node representation and generate edges using similarity between the visual features with creator identifier (b)
In second setting, we used the visual features for node representation but generate edges using similarity between audio features along with creator identifier (c)
Then, we generate the edges using concatenation of both visual and audio features and creator identifier while representing the node using visual features.
All these approaches showed similar performance but using the audio features for edge creation improved the Top-1 performance by 0.2% over using only visual features.
(d) Next, we concatenate the visual and audio information together for node representations and edge creation with creator information.
We can see that this shows the best results and improves the accuracy by more than 1%.
We show these quantitative results in Table 2.
5.3 Inductive and Transductive Settings
In Table 3, we compare transductive and inductive settings of training GNNs.
In transductive setting, we use all the dataset splits for generating the training graph.
However, we mask the labels of non-train set videos so that only the labels of train set are used for training.
In the inductive setting, we use only the training videos and labels for creating the training graph and training the model.
We observe that transductive setting shows higher performance than inductive settings as it processes all the splits of the dataset during graph creation.
5.4 Ablation
We study the effect of different aggregators like pool , gcn and mean on GraphSage in Figure 1 (a) and observe the best result using "mean" aggregator.
Thus, we use "mean" aggregator for all our experiments.
We study the impact of varying the cosine distance threshold.
Higher threshold allows less number of edges while lower threshold results in more number of edges.
From Figure 1 (b), we observe that 0.4 shows the best performance and we use it for other experiments.
Conclusions
In this work, we explored the inherent inter-video relationship that exists between the videos uploaded on social media platforms.
We studied GCN and GraphSage under inductive and transductive settings and observed substantial performance gains by exploiting the creator information.
To the best of our knowledge, this inter-video information has not been explored previously in the context of video understanding.
Further, we explored the impact of using both visual and audio modalities as input and for creating graphical edges and observe improvement in video classification. A Appendix A.1 Graph Convolutional Network (GCN)
GCN works on the spectral domain of the graph.
Lets say we have n nodes representing n videos through n features h 1 , h 2 , ..., h n of d dimensions.
These features can be stacked together to form a feature matrix X .
We have an Adjacency matrix A which tells us about the edge connection between the nodes.
We also enforce the self-edge connection between the nodes via ˜
A
=
A + I where, I represents identity matrix.
We normalise ˜
A with ˜ D where ˜ D represent degree matrix correponding to graph generated by ˜ A .
˜ D ij means row wise summation of ˜ A .
The way normalization of ˜ A is down is shown below:
[Equation]
Then the update rule for GCN is:
[Equation]
where, H ( l ) and W ( l ) means feature matrix and weight matrix at l th stage.
Here, H ( l ) =
X .
σ represents the non-linearity added after each layer.
A.2 Preprocessing
We used ResNext50-3D [27] (pretrained on Kinetics700) for extracting spatio-temporal visual features from the 3MASSIV videos.
We sample the videos at 25 frame per second and scale the frames by resizing the shortest side to 240 pixels while maintaining the aspect ratio.
We use 16 centrally cropped frames of dimension 112 x 112 x 3 as clips for extracting 2048-dimensional features from the last convolution layer and average the clip representations to generate a 2048-dimensional representation for the video.
For audio features, we extracted the audio channel as mono-channel from our videos using ffmpeg 2 .
We sample the audios at 16 kHz and use CLSRIL-23 [9] models for audio feature extraction.
The audio features are averaged across the clip to get a 512 -dimensional vector.
B Results
Edge Type: We experiment with using the semantic similarity between the videos as edge weight and binary edges.
In binary edge, if the edge exist, edge weight will be 1.0 else 0.
From the experiments, we found that edge type does not impact model performance although semantic similarity weights shows slight improvement.
We report the results of our ablation study in detail in Table 4, Tableand Table 6.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]