B OUNDARY
AWARE
S
ELF -S UPERVISED L EARNING FOR V IDEO S CENE S EGMENTATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Self-supervised learning has drawn attention through its effectiveness in learning in-domain representations with no ground-truth annotations; in particular, it is shown that properly designed pretext tasks (e.g., contrastive prediction task) bring signiﬁcant performance gains for a downstream task (e.g., classiﬁcation task).
Inspired from this, we tackle video scene segmentation, which is a task of temporally localizing scene boundaries in a video, with a self-supervised learning framework where we mainly focus on designing effective pretext tasks.
In our framework, we discover a pseudo-boundary from a sequence of shots by splitting it into two continuous, non-overlapping sub-sequences and leverage the pseudo-boundary to facilitate the pre-training.
Based on this, we introduce three novel boundary-aware pretext tasks: 1) Shot-Scene Matching (SSM), 2) Contextual Group Matching (CGM) and 3) Pseudo-boundary Prediction (PP); SSM and CGM guide the model to maximize intra-scene similarity and inter-scene discrimination while PP encourages the model to identify transitional moments.
Through comprehensive analysis, we empirically show that pre-training and transferring contextual representation are both critical to improving the video scene segmentation performance.
Lastly, we achieve the new state-of-the-art on the MovieNet-SSeg benchmark.
The code will be released.
I NTRODUCTION
The video scene segmentation is a task of identifying scene boundaries from a video where a scene is deﬁned as a semantic unit for making a story and is composed of a series of semantically cohesive shots—a set of frames captured by the same camera during an uninterrupted period of time—in the same context.
Localizing scene boundaries is a signiﬁcant step towards the high-level video understanding because dividing a long video into a set of meaningful scenes enables models to inspect the individual incidents from complex story.
One of the biggest challenges with temporal semantic segmentation is that it is not achieved simply by detecting changes in visual cues.
As shown in Figure 1(a), we present an example of nine shots, all of which belong to the same scene, where two characters are talking on the phone.
We can see that the overall visual cues within the scene do not stay the same but rather change repeatedly when each character appears.
On the other hand, as presented in Figure 1(b), the other example shows two different scenes which contain visually similar shots (highlighted in blue) where the same character appears in the same place.
Therefore, it is expected that two adjacent scenes which share shots with similar visual cues need to be contextually discriminated.
From this observation, it is important for the video scene segmentation task to model contextual relation between shots by maximizing 1) intra-scene similarity ( i .
e ., the shots belonging to the same scene should be close to each other), and 2) inter-scene discrimination across two adjacent scenes ( i .
e ., two neighbor shots across the scene boundary should be distinguishable).
Supervised learning approaches ( e .
g .,
Rao et al.
(2020)) are clearly limited due to the lack of largescale datasets with reliable ground-truth annotations.
Recently, self-supervision (Chen et al., 2020a;
Caron et al., 2020; He et al., 2020; Roh et al., 2021) is spotlighted through its effectiveness in learning in-domain representation without relying on costly ground truth annotations.
The selfsupervised learning methods (Chen et al., 2021; Feichtenhofer et al., 2021; Dave et al., 2021; Qian et al., 2021) in the video domain are often designed to learn spatio-temporal patterns in short clips
[Figure]
( e .
g ., shots in movies).
This kind of learned representation is generic and can be applied to many video understanding tasks ( e .
g ., action classiﬁcation).
However, such representation is not sufﬁcient for video scene segmentation because this task requires not only a good representation for individual shots but also contextual representation considering neighboring shots at a higher level as illustrated in Figure 1.
Motivated by this, we set our main goal to design effective self-supervised objectives ( i .
e ., pretext tasks) that maximize intra-scene similarity as well as discriminate shots from different scenes.
For the purpose, this raises a penetrating question: how can we design boundary-relevant pretext tasks without access to ground truth boundary annotations?
We introduce a novel B oundary- a ware S elf- S upervised L earning (BaSSL) framework.
Our main idea of BaSSL is to localize a pseudo-boundary, which is obtained by dividing the input sequence of shots into two semantically disjoint sub-sequences, and use it to deﬁne pretext tasks that are beneﬁcial to the video scene segmentation task.
On top of the discovered two sub-sequences and a pseudo-boundary, three boundary-aware pretext tasks are proposed: 1) Shot-Scene Matching (SSM); 2) Contextual Group Matching (CGM); and 3) Pseudo-boundary Prediction (PP).
Note that SSM and CGM encourage the model to maximize intra-scene similarity and inter-scene discrimination while PP enables the model to learn the capability of identifying transitional moments.
In addition, we perform Masked Shot Modeling (MSM) task inspired by Sun et al.
(2019a) to further learn temporal relationship between shots.
The comprehensive analysis demonstrates the effectiveness of the proposed framework ( i .
e ., pre-training of contextual relationship between shots) as well as the contribution of the proposed individual components ( i .
e ., the algorithm for pseudo-boundary discovery and boundary-aware pretext tasks).
Our main contributions are summarized as follows: ( i )
We introduce a novel boundary-aware pre-training framework which adopts dynamic time warping (DTW) algorithm to identify pseudoboundaries and use them as self-supervision to facilitate the pre-training; ( ii ) we propose three boundary-aware pretext tasks, which are carefully designed to learn essential capabilities required for the video scene segmentation task; ( iii ) we perform extensive ablations to demonstrate the effectiveness of the proposed framework, including the observation that our framework is complementary to the existing framework; ( iv ) we achieve the new state-of-the-art on the MovieNet-SSeg benchmark while outperforming existing self-supervised learning-based methods by large margins.
R ELATED W ORK
Video scene segmentation approaches formulate the task as a problem of temporal grouping of shots.
In this formulation, the optimal grouping can be achieved by clustering-based (Rui et al., 1998; Rasheed & Shah, 2003; 2005; Chasanis et al., 2008), dynamic programming-based (Han &
Wu, 2011; Tapaswi et al., 2014; Rotman et al., 2017) or multi-modal input-based (Liang et al., 2009;
Sidiropoulos et al., 2011) methods.
However, the aforementioned methods have been trained and evaluated on small-scale datasets such as OVSD (Rotman et al., 2016) and BBC (Baraldi et al., 2015) which can produce a poorly generalized model.
Recently, Huang et al.
(2020) introduce a largescale video scene segmentation dataset ( i .
e ., MovieNet-SSeg) that contains hundreds of movies.
Training with large-scale data, Rao et al.
(2020) proposes a strong supervised baseline model that performs a shot-level binary classiﬁcation followed by grouping using the prediction scores.
In addition, Chen et al.
(2021) proposes a shot contrastive pre-training method that learns shot-level representation.
We found ShotCoL (Chen et al., 2021) to be the most similar work to our method.
However, our method is different from ShotCoL in that we speciﬁcally focus on learning contextual representations by considering the relationship between shots.
We refer interested readers to the supplementary material for a more detailed analysis of this.
Action segmentation in videos is one of the related works for video scene segmentation, which identiﬁes action labels of individual frames, thus divides a video into a series of action segments.
Supervised methods (Lea et al., 2016; Farha & Gall, 2019) proposed CNN-based architectures to effectively capture temporal relationship between frames in order to address an over-segmentation issue.
As frame-level annotations are prohibitively costly to acquire, weakly supervised methods (Chang et al., 2019; Li et al., 2019; Li & Todorovic, 2020; Souri et al., 2021;
Shen et al., 2021; Zhukov et al., 2019; Fried et al., 2020) have been suggested to use an ordered list of actions occurring in a video as supervision.
Most of the methods are trained to ﬁnd (temporal) semantic alignment between frames and a given action list using an HMM-based architecture (Kuehne et al., 2018),
a dynamic programming-based assignment algorithm
(Fried et al., 2020) or a DTW-based temporal alignment method (Chang et al., 2019).
Recently, unsupervised methods (Kumar et al., 2021;
Wang et al., 2021; Kukleva et al., 2019; Li & Todorovic, 2021; VidalMata et al., 2021) have been further proposed; in a nutshell, clustering-based prototypes are discovered from unlabeled videos, then the methods segment the videos by assigning prototypes (corresponding to one of the actions) into frames.
In contrast to the action segmentation task that is limited to localizing segments each of which represents a single action within an activity, video scene segmentation requires localizing more complex segments each of which may be composed of more than two actions (or activities).
Self-supervised learning in videos has been actively studied for the recent years with approaches proposing various pretext tasks such as future frame prediction (Srivastava et al., 2015; Vondrick et al., 2016;
Ahsan et al., 2018), temporal ordering of frames (Misra et al., 2016; Lee et al., 2017; Xu et al., 2019), geometric transformations prediction (Jing & Tian, 2018), colorization of videos (Vondrick et al., 2018) and contrastive prediction (Feichtenhofer et al., 2021;
Qian et al., 2021; Dave et al., 2021).
In addition, CBT (Sun et al., 2019a;b) proposes a pretext task of masked frame modeling to learn temporal dependency between frames (or clips).
Note that since most of those methods are proposed for the classiﬁcation task, they would be sub-optimal to the video scene segmentation task.
On the other hand, BSP (Xu et al., 2020) proposes boundary-sensitive pre-text tasks based on synthesized pseudo-boundaries that are obtained by concatenating two clips sampled from different videos.
However, strictly speaking, BSP is not a self-supervised learning algorithm since it requires video-level class labels to synthesize pseudo-boundaries; the proposed pretext tasks are not applicable to videos such as movies that are hard to deﬁne semantic labels.
Also, note that we empirically show that pseudo-boundaries identiﬁed by our method are more effective for pre-training than synthesized pseudo-boundaries.
B OUNDARY - AWARE S ELF - SUPERVISED L EARNING (B A SSL)
In this section, we introduce our proposed approach, Boundary-aware Self-Supervised Learning (BaSSL).
We start with the problem formulation followed by the model overview.
Then, we describe our novel boundary-aware pretext tasks for pre-training.
3. P ROBLEM F ORMULATION
Terminologies A video ( e .
g ., documentaries, TV episodes and movies) is a sequence of scenes, deﬁned as a semantic unit for making a story.
A scene is a series of shots, which is a set of frames physically captured by the same camera during an uninterrupted period of time.
Problem Deﬁnition Given a video, which contains a series of N shots { s 1 , ..., s N } with class labels { y 1 , ..., y N } where y i ∈ { 0 , 1 } indicating if it is at the scene boundary (more precisely, if it is the last shot of a scene), the video scene segmentation task is formulated as a simple binary classiﬁcation problem at individual shot level.
By deﬁnition, a scene boundary is where the semantic of a shot is considerably different from its (one-way) neighbors.
Thus, it is in nature important to capture and leverage contextual transition across the scenes.
Consequently, it is a common practice that the information of the neighbor shots are leveraged together when determining scene boundaries.
With this formulation, existing supervised learning approaches typically train a parameterized ( θ ) model by maximizing the expected log-likelihood:
[Figure]
P s e ud o - b o und a r y
D i s c
o
v e r y
[Figure]
P s e ud o - b o und a r y
D i s c
o
v e r y
[Figure] [Figure]
[Equation]
(1) where S n = { s n − K , ..., s n , ..., s n + K } is a set of 2 K + 1 shots centered at n th shot s n , and K is the number of neighbor shots before and after s n .
Note that each shot s is given by a set of N k keyframes, resulting in a tensor with size of ( N k , C, H, W ) where C , H and W are the RGB channels, the frame height and the frame width, respectively.
3. M ODEL O VERVIEW
Our method is based on two-stage training following common practice (Chen et al., 2021): pretraining on large-scale unlabeled data with self-supervision and ﬁne-tuning on relatively small labeled data via transfer learning.
Our main focus is in the pre-training stage, aiming at designing effective pretext tasks for video scene segmentation.
As illustrated in Figure 2, our model consists of two main components: 1) shot encoder embedding a shot by capturing its spatio-temporal patterns, and 2) contextual relation network (CRN) capturing relationship between shots.
Given a sequence S n = { s n − K , ..., s n , ..., s n + K } of shots centered at s n , two-level representations are extracted as follows:
[Equation]
where f ENC : R N k × C × H × W → R D e and f CRN : R (2 K +1) × D e
→ R (2 K +1)
× D c represent the shot encoder and the contextual relation network while D e and D c mean dimensions of encoded and contextualized features, respectively.
e n is an encoding of shot s n by f ENC while E n = { e n − K , ..., e n , ..., e n + K } and C n = { c n − K , ..., c n , ..., c n + K } correspond to the input and output feature sequence for f CRN , respectively.
On top of encoded shot representations, BaSSL extracts a pseudo-boundary (left box of Figure 2) to self-supervise the model instead of relying on ground-truth annotations.
To be speciﬁc, we leverage the dynamic time warping technique to divide the input sequence of shots into two semantically disjoint sub-sequences and output a pseudo-boundary.
(See Section 3.3 for more details.)
Then, as presented in Figure 3, using the discovered pseudo-boundary, we devise three novel boundary-aware pretext tasks in Section 3.4: 1) Shot-Scene Matching to match shots with their associated scenes, 2) Contextual Group Matching to align shots whether they belong to the same scene or not and 3) Pseudo-boundary Prediction to capture semantic changes.
In addition, we adopt the masked shot modeling in CBT (Sun et al., 2019a) to further learn temporal relationship between shots.
After trained with the four pretext tasks, the model is ﬁne-tuned with labeled video scene segmentation data.
(See Section 3.5 for more details.)
3. P SEUDO - BOUNDARY D ISCOVERY
During the pre-training, no scene boundary annotation is available for the input sequence S n of shots.
Thus, we infer pseudo-boundaries which serve as if they are ground truth labels for boundaryrelevant pretext tasks.
In addition, we simplify the problem to always extract a single boundary given an input sequence.
This is reasonable because even a sequence of shots without strong scenelevel semantic transition, there always exists a shot within the sequence across which the semantic transition is maximum, and we use this shot as a pseudo-boundary.
To support our claim, we show examples in Figure 4 where we intentionally infer a pseudo-boundary based on a sequence of shots sampled from the same scene (that is, no scene boundary exists between shots according to the ground truth); we observe that the resulting two sub-sequences are cognitively distinguishable.
This simpliﬁed problem, dividing the input sequence S n into two continuous, non-overlapping subsequences S left n and S right n , can be seen as a temporal alignment problem between S n and S slow n ; speciﬁcally, observing the ﬁrst shot should belong to S left n and the last one to S right n , we deﬁne S slow n = { s n − K , s n + K } , which can be seen as a same video with S n with lower sampling frequency.
Finally, the task becomes aligning intermediate shots either to S left n or S right n preserving continuity.
Under the problem setting, we adopt dynamic time warping (DTW) (Berndt & Clifford, 1994) to ﬁnd the optimal temporal alignment between S n and S slow n .
In detail, DTW solves the following optimization problem using dynamic programming to maximize semantic coherence of the resulting two sub-sequences among all possible boundary candidates:
b
[Equation]
where b is the candidate boundary offset, b ∗ is the optimal boundary offset, and sim ( x , y )
=
x (cid:62) y (cid:107) x (cid:107)(cid:107) y (cid:107) computes cosine similarity between encodings of the given two shots.
Two sub-sequences are inferred as S left n = { s n − K , ..., s n + b ∗ } and S right n = { s n + b ∗ +1 , ..., s n + K } .
s n + b ∗ is the pseudoboundary shot, which is the last shot of S left n .
More examples of pseudo-boundaries identiﬁed by our algorithm is presented in Figure 7 of the supplementary material.
The results are used for learning boundary-aware pretext tasks, which will be described below.
3. P RE - TRAINING O BJECTIVES
As illustrated in Figure 3, we adopt four pretext tasks for pre-training—1) shot-scene matching, 2) contextual group matching, 3) pseudo-boundary prediction, and 4) masked shot modeling—and the ﬁnal pre-training loss is deﬁned by
[Equation]
Shot-Scene Matching (SSM)
The objective of this task is to make the representations of a shot and its associated scene similar to each other, while the representations of the shot and other scenes dissimilar.
In other words, SSM encourages the model to maximize intra-scene similarity, while minimizing inter-scene similarity.
Considering the splitted two sub-sequences ( S left n and S right n ) as pseudo-scenes, we train the model using the InfoNCE loss (Oord et al., 2018):
[Equation]
where h SSM is a SSM head of a linear layer, τ is a temperature hyperparameter and r ( S ) means a scene-level representation; we use the averaged encoding of shots in the sub-sequence S .
N e
and
N r in Eq.
(6) are constructed using other shots and pseudo-scenes in a mini-batch, respectively.
Contextual Group Matching (CGM) Since directly matching representations of shots and scenes would not be effective when the scenes are composed of visually dissimilar shots, CGM is introduced to bridge this gap.
Similar to SSM, CGM is also designed to maximize intra-scene similarity and inter-scene discrimination.
However, CGM measures semantic coherence of the shots rather than comparing visual cues.
With CGM, the model learns to decide if the given two shots belong to the same group ( i .
e ., scene) or not.
In detail, we use the center shot s n in the input sequence as the anchor and construct a triplet of ( s n , s pos , s neg ).
We sample each shot from S left n and S right n ; the one sampled within the same sub-sequence with s n is used as the positive shot s pos , while the other as the negative s neg .
The CGM loss is deﬁned using a binary cross-entropy loss as follows:
[Equation]
where h CGM is a CGM head taking two shots as input and predicting a matching score.
c n , c pos and c neg are the contextualized features by f CRN for the center, positive and negative shots, respectively.
Pseudo-boundary Prediction (PP) Through the above two pretext tasks
, our model learns the contextual relationship between shots.
In addition to these, we design an extra pretext task, PP, which is more directly related to boundary detection; PP makes the model have a capability of identifying transitional moments that semantic changes.
Based on the pseudo-boundary shot and one randomly sampled non-boundary shot, the PP loss is deﬁned as a binary cross-entropy loss:
[Equation]
where h PP is a PP head that projects the contextualized shot representation to a probability distribution over binary class.
c n + b ∗ and c ¯ b indicate the contextualized representation from f CRN for the pseudo-boundary shot s n + b ∗ and randomly sampled non-boundary shot s ¯ b , respectively.
Masked Shot Modeling (MSM)
Inspired by masked frame modeling (Sun et al., 2019a;b), we adopt the MSM task whose goal is to reconstruct the representation of masked shots based on the their surrounding shots.
In this task, given a set of encoded shot representations, we randomly apply masking each of them with a probability of 15%.
For a set M of masked shot offsets, we learn to regress the output on each masked shot to its encoded shot representation, which is given by
[Equation]
where h MSM is a MSM head to match the dimension of contextualized shot representation with that of encoded one.
e m and c m denote the encoded representation by f ENC and contextualized representation by f CRN for a masked shot s m , respectively.
3. F INE - TUNING FOR S CENE B OUNDARY D ETECTION
Recall that we formulate the video scene segmentation as a binary classiﬁcation task to identify contextual transition across the scene.
Different from the pre-training stage, given an input sequence of shots S n , we employ a scene boundary detection head h SBD to infer a prediction from the contextualized representation ( c n ) for the center shot s n .
Following Chen et al.
(2021), we freeze the parameters of the shot encoder and then train only the contextual relation network and the scene boundary detection head using a binary cross-entropy loss with the ground truth label y n as follows:
[Equation]
Note that individual shots are decided to be a scene boundary when its prediction score is higher than a pre-deﬁned threshold (set to 0.5).
[Table]
E XPERIMENT 4. E XPERIMENTAL S ETTINGS
Dataset We evaluate our proposed method on the MovieNet-SSeg dataset (Huang et al., 2020) that is a sub-dataset of MovieNet, containing 1,100 movies with 1.6M shots.
Note that only 318 out of 1,100 movies have scene boundary annotations, which are divided into 190, 64, and 64 movies for training, validation, and test split, respectively.
Following Chen et al.
(2021), we use the entire 1, movies with no ground truth labels for the pre-training and ﬁne-tune the model on the training split.
The performance is measured on the test split.
Metric
Following Huang et al.
, we compare algorithms using Average Precision (AP) and mIoU that measures the averaged intersection over union (IoU) between predicted scene segments and their closest ground truth scene segments.
Also, we adopt F1 score and AUC-ROC as additional evaluation metrics.
Note that contrary to the previous works (Rao et al., 2020; Chen et al., 2021) that report recall, we use F1 score to consider for balanced comparison between precision and recall.
In addition, we report Meta-Sum metric inspired by the works (Chen et al., 2020b; Li et al., 2021) for easy and straightforward comparison of algorithms.
Implementation details We employ ResNet-50 (He et al., 2016) and Transformer (Vaswani et al., 2017) networks as the shot encoder and the contextual relation network, respectively.
For both pre-training and ﬁne-tuning stages, we cross-validate the number of neighbor shots among K = { 4 , 8 , 12 , 16 } and K = 8 is selected due to its good performance and computational efﬁciency.
In all experiments, given a pre-trained model, we ﬁne-tune the model 5 times with different random seeds and report their average score and standard deviation.
Note that more details are presented in supplementary material.
4. C OMPARISON WITH S TATE - OF - THE - ART M ETHODS
We compare our method, BaSSL, with 1) supervised methods including Siamese (Baraldi et al., 2015), MS-LSTM (Huang et al., 2020) and LGSS (Rao et al., 2020), 2) unsupervised methods including GraphCut (Rasheed & Shah, 2005), SCSA (Chasanis et al., 2008), DP (Han & Wu, 2011),
StoryGraph (Tapaswi et al., 2014) and Grouping (Rotman et al., 2017), and 3) self-supervised methods including ShotCoL (Chen et al., 2021).
Without ﬁne-tuning on the downstream task, BaSSL can be seen as an unsupervised model in that it is trained to predict the pseudo-boundary by the PP task.
Table 1 summarizes comparison against state-of-the-art methods.
BaSSL without ﬁne-tuning shows competitive or outperforming performance based only on basic visual features compared to competing unsupervised methods.
Furthermore, ﬁne-tuning BaSSL with ground-truth scene bound-
[Table]
aries, AP is improved by 24.71%p and BaSSL outperforms all other algorithms.
Finally, through longer pre-training (40 epochs), BaSSL surpasses the state-of-the-art method ( i .
e ., ShotCoL) by a large margin (4.00%p in AP).
4. C OMPARISON WITH P RE - TRAINING B ASELINES
We perform extensive experiments to compare BaSSL with the other pre-training baselines that learn shot-level representation by f ENC .
In the experiments, we compare the following three types of pretraining approaches; The ﬁrst group (M1-2) trains f ENC using image-level supervision with object labels on ImageNet (Deng et al., 2009) or place labels on Places365 (Zhou et al., 2017).
The second group (M3-5) trains f ENC through shot-level contrastive learning ( i .
e .
, SimCLR proposed by Chen et al.
(2020a))
with different positive pair sampling strategies.
Speciﬁcally, Instance (M3) takes an instance of the center shot with different augmentation, Temporal (M4) takes one randomly sampled neighbor shot as positive pair in local temporal window, and Nearest Neighbor (NN) (M5) takes the most visually similar shot among the neighbor shots as positive pair, which is also known as
ShotCoL (Chen et al., 2021).
The last group (M6-8) learns both f ENC and f CRN through boundaryaware pretext tasks proposed in this paper.
Given pre-trained representations, we train a video scene segmentation model with three different types of f CRN including MLP (Chen et al., 2021), MSLSTM (Huang et al., 2020) 1 and Transformer.
For fair comparison, all pre-training methods employ
ResNet-50 (He et al., 2016) as the shot encoder f ENC and we pre-train the models for 10 epochs.
In Table 2, we found the following observations.
First, when transferring pre-trained shot representation, employing MS-LSTM and Transformer as f CRN is more effective than using MLP, as they are favorably designed to capture contextual relation between shots (see M1-6).
Second, BaSSL (M7) outperforms all competing baselines (M1-5) through learning contextual representation during pre-training.
Also, it turns out that transferring the representation through f CRN is important for the boundary detection task where it leads to a performance gain of 5.44%p in AP (see M6-7).
Finally, learning shot-level and contextual representations is complementary to each other; that is, incorporating ShotCoL (M5) and our framework (M7) provides further improved performance (M8).
4. A BLATION S TUDIES
Impact of individual pretext tasks We
ﬁrst investigate the contribution of individual pretext tasks.
In this experiment, we train models by varying the combinations of the pretext tasks.
From
Table 3, we can obtain following two observations.
First, when training a model with a single pretext task (P1-4), the MSM task leads to the worst performance compared to the others.
This result indicates that boundary-aware pretext tasks ( i .
e ., SSM, CGM and PP) to learn contextual relation between shots is indeed important for video scene segmentation.
Second, the more pretext tasks we include during pre-training, the better the performance is, and the best performance is obtained when using all tasks (P15).
This means all tasks are complementary to each other, contributing to performance gain.
[Table] [Table]
Psuedo-boundary discovery method To check the effectiveness of DTW-based pseudo-boundary discovery
, we train three models with different pseudo-boundary decision strategies—1)
Random deﬁning one randomly sampled shot in the input sequence as pseudo-boundary, 2) Fixed always taking the center shot as pseudo-boundary, and 3) Synthesized , inspired by Xu et al.
(2020), synthesizing the input sequence by concatenating two sub-sequences sampled from different movies and using the last shot of the ﬁrst sub-sequence as a pseudo-boundary.
Table 4(a) summarizes the results.
Random and Fixed pseudo-boundaries hinder the learning and degenerate the boundary detection performance.
It is notable that BaSSL with Synthesized pseudo-boundaries also outperforms the pre-training baselines in Table 2, which shows the effectiveness of our framework.
Finally, adopting DTW to ﬁnd pseudo-boundaries achieves the best performance.
Hyperparameters We analyze the impact of two key hyperparameters: 1) the number of neighbor shots K and 2) pre-training epochs.
Table 4(b) shows that we achieve higher performance with more neighbor shots, saturating around K = 12 .
Table 4(c) shows the impact of longer pre-training.
We ﬁnd that performance increases until certain numbers ( i .
e ., 40 epochs) and decrease afterward.
We conjecture that this is partly due to overﬁtting to noise from incorrect pseudo-boundaries.
C ONCLUSION
We present BaSSL, a novel self-supervised framework for video scene segmentation, especially designed to learn contextual relationship between shots.
Through the pseudo-boundary discovery, we can deﬁne and conduct boundary-aware pretext tasks that encourage the model to learn the contextual relational representation and a capability of capturing transitional moments.
Comprehensive experiments demonstrate the effectiveness of our framework and we achieve outstanding performance in the MovieNet-SSeg dataset.
Ethics statement In this paper, we introduce a novel self-supervised learning algorithm especially for video scene segmentation.
The video scene segmentation technique could be applied to a wide range of applications, including preview generation for fast contents discovery, trailer generation, and minimally disruptive video-ads insertion.
In addition, our proposed boundary-aware self-supervised learning algorithm can be applied to learn contextual representation of untrimmed, long videos.
This would drive future research in videos to move towards leveraging the extremely large number of raw videos on the web to learn video representations.
However, as a negative effect of this, it would consume large amount of computational resources.
Also, as any machine learning algorithm is highly likely to be biased for training data, the learned representation would be biased and lead to an issue related to discrimination because the training data for self-supervised learning is often not validated by humans.
Reproducibility statement To make our algorithm reproducible, we present implementation details in Section 4.1 of the main paper and in Section B of the supplementary material; we also provide a pseudocode for DTW-based pseudo-boundary discovery method.
In addition, we will release the code and parameters of trained models
A D ATASET S TATISTICS
Table 5 shows the data statistics of different video scene segmentation datasets.
We found the limited number of datasets that provide the scene boundary annotations and, as far as we know, the
MovieNet-SSeg (Huang et al., 2020) is the largest-scale video scene segmentation dataset.
[Table]
B A DDITIONAL I MPLEMENTATION D ETAILS
Additional details of the shot encoder ( i .
e ., ResNet-50) and the contextual relation network ( i .
e .,
Transformer) are as follows.
For Transformer, the dimensions are set to ( L = 2 , H = 768 , A = 8 ) where L , H and A mean the number of stacked transformer blocks, the dimension of hidden activation and the number of attention heads, respectively.
We apply the Dropout technique (Srivastava et al., 2014) on hidden states and attention weights with a probability of 10% and use
GELU (Hendrycks & Gimpel, 2016) as an activation function.
For the shot encoder, each shot is given by three key-frames ( i .
e ., N k = 3 ) and a shot encoding e is computed by the averaged feature after inferring individual three key-frames by ResNet-50; note that, to speed up the training, we use randomly sample one key-frame out of three during the pre-training.
For data augmentation of key-frames in a shot, we adopt PyTorch’s torchvision package.
Given a sequence of shots, we apply random crop (with resize), random ﬂip, random color jitter and random
Gaussian blur.
In detail, ﬁrstly, the cropping is performed with a random size ( i .
e ., scales between [0.14, 1.0] of the original size) and a random aspect ratio (between 3/4 to 4/3), and then the cropped one is resized to (224,224).
Secondly, we apply a random horizontal ﬂip with a probability of 50%.
Thirdly, as a color augmentation, we perform a random color jitter (with a probability of 80%) and a random color dropping to gray scale (with a probability of 20%).
The color strength parameters for jittering are set to { brightness , contrast , saturation , hue } = (0 . 2 , 0 . 2 , 0 . 2 , 0 . 05) .
Finally, Gaussian blur is applied with a probability of 50% where a standard-deviation of spatial kernel is set to [0.1, 2.0].
Note that the same augmentations are applied to all key-frames in the input sequence S n of shots while different color jittering is applied on individual shots.
Also, for S slow n , we perform a different augmentation compared to that applied on S n .
During the pre-training stage, the model parameters are randomly initialized and then trained using the proposed pretext tasks.
We use LARS (You et al., 2017) to learn the model (except for parameters of bias and Batch-Normalization) with a mini-batch of 256 shot sequences, a base learning rate of 0.3, momentum of 0.9, weight decay of 10 − 6 and trust coefﬁcient of 0.001.
We pre-train the model for 10 epochs with a linear warm-up strategy for 1 epoch followed by learning rate decaying with a cosine schedule.
The temperature τ in Eq.
(5) is set to 0.1.
In the ﬁne-tuning stage, we initialize the parameters of the shot encoder and the contextual relation network by that of the pre-trained ones, however, we freeze the parameters of the shot encoder following Chen et al.
We ﬁne-tune the contextual relation network and the scene boundary detection head for 20 epochs using Adam (Kingma & Ba, 2015) with a learning rate of 10 − 5 and a mini-batch of 1024 training examples.
The learning rate is decayed with a cosine schedule without a warm-up stage.
C C OMPARISON WITH S HOT - LEVEL S ELF - SUPERVISED L EARNING
As mentioned in the main paper, our approach is distinguishable from the shot-level pre-training approach (Chen et al., 2021;
Qian et al., 2021) in that the objectives used in our approach (BaSSL)
In contrast, our boundary-aware pre-training method focuses on learning contextual representation by taking neighbor shots into account.
Thus, our method can learn both the shot encoder ( f ENC ) and the contextual relation network (CRN; f CRN ) and transfer their parameters during
[Table]
is to learn contextual representations by taking neighbor shots into account.
Figure 5 provides a clear summary of comparison between shot-level pre-training and our BaSSL.
Firstly, shot-level pre-training takes a pair of two shots as an input while BaSSL takes a sequence of shots.
Secondly, shot-level pre-training aims to train shot encoder ( f ENC ) only, while BaSSL trains both the shot encoder and the contextual relation network ( f ENC and f CRN ).
In contrast to the shot-level pre-training that requires to train f CRN from scratch during the ﬁne-tuning stage, BaSSL beneﬁts from weight transfer by pre-training the parameters of f CRN with large-scale in-domain data in advance.
Note that the results (M6-7) in Table 2 show that the weight transfer of f CRN is important to improve the video scene segmentation performance.
Finally, the contrastive learning objective in shot-level pre-training locates the representations of two shots (query and positive) to be close to each other, whereas Shot-Scene Matching objective in our approach performs the same task but with a shot (query) and its associated scene (positive; a sequence of shots).
The Table 6 summarizes the aforementioned comparisons.
D A LGORITHM FOR P SEUDO - BOUNDARY D ISCOVERY
In this section, we describe the details of pseudo-boundary discovery method applying DTW on
S n and S slow n .
In practice, S n is given as a mini-batch resulting in a tensor with a shape of ( B , S ,
N k , C , H , W ) where individuals mean the batch size, the number of shots in S n ( i .
e ., 2 K
+ 1 ), the number of key-frames in a shot, channels, frame height and frame width, respectively.
Then, we obtain S slow n ∈ R B × 2 × N k × C × H × W that is composed of the ﬁrst and last shots in S n .
We apply two different augmentation functions into key-frames in S n and S slow n , respectively.
Next, we compute encoded representation of S n and S slow n using f ENC .
Note that during the pre-training stage, we randomly sample one key-frame among N k candidates in a shot and then reshape the input tensor as ( B * S , C , H , W ) or ( B *2, C , H , W ) to be forwarded by the shot encoder f ENC ; thus the tensor shape of the encoded shot representation is given by ( B , S , D e ) or ( B , 2, D e ) after apply reshaping, where D e means the dimension of encoded feature.
Finally, given two sequences of encoded representation for S n and S slow n , DTW provides two sub-sequences S left n and S right n and a pseudo boundary shot s n + b ∗ .
The algorithm 1 illustrates the details.
In addition, to demonstrate the simplicity of the alignment computation using DTW, we include the PyTorch code in Listing 1.
The implementation of DTW can be done in 5 lines of python code using tslearn package.
tslearn import metrics sequence 9: E slow n ← { E slow n ; e i } // append 10: end for 11: S left n , S right n , b ∗ ← DTW( E n , E slow n ) // apply dynamic time warping 12: Output : Two continuous non-overlapping sub-sequences S left n and S right n and a pseudo boundary shot s n + b ∗ . E R ESULTS ON ADDITIONAL DATASETS We further compare BaSSL with shot-level pre-training baselines on additional two datasets—BBC and OVSD.
Note that the train and test splits are not available and the dataset size is extremely limited (11 and 21 videos in BBC and OVSD, respectively); in addition, 2 out of 21 videos in OVSD is not available.
Thus, we infer predictions using models trained on MovieNet-SSeg without ﬁnetuning on BBC and OVSD.
The results are summarized in Table 7.
The result shows the superiority of our method compared to shot-level pre-training baselines.
[Table]
[Table]
[Table]
F M EASURING R EPRESENTATION Q UALITY AT P RE - TRAINING S TAGE
The normalized mutual information (NMI) is a metric for clustering algorithms ( e .
g .
, K-Means), which measures the clustering quality.
Since clustering with good representations forms clear boundaries between different classes, NMI can be considered as a proxy to measure the quality of our pre-trained models.
Speciﬁcally, we randomly sample 100 scenes from the test split of
MovieNet-SSeg while we vary the length of scenes N c ∈ { 8 , 16 , 32 } .
Then, we perform K-Means clustering on N c × 100 shot representations extracted by the pre-trained model with the number of classes K =100.
We intend to form a single cluster for each scene in this formulation, assuming that high-quality representation for movie scene segmentation would locate the shot features within the same scene close to each other.
Considering the randomness in the K-Means clustering and scene sampling, we report the averaged score from ﬁve trials.
In Table
[Table]
we compare the NMI score between different pre-trained models; SimCLR (NN) is our SimCLR version implementation of ShotCoL.
The result shows that BaSSL outperforms the shot-level pre-training baselines and the model pre-trained using ImageNet dataset.
With respect to different scene lengths ( N c ; the number of shots included in a single scene), we found our BaSSL is more robust than the other baselines.
Since the visual diversity across the shots increases as the scenes become longer ( N c =8 → 32), it is natural that the NMI score for each baseline is degraded.
However, it is remarkable that, by increasing the number of shots from 8 to 32, the performance of
BaSSL drops only -0.68% while the other baselines suffer from severe degradation.
This demonstrates the effectiveness of BaSSL in maximizing intra-scene similarity.
In addition, we perform ablation study of our algorithm by adding pretext tasks one by one, and measure the corresponding NMI scores.
The result in Table 9 shows that better NMI score is achieved as more pretext tasks are combined together.
This tendency is also observed in our ablation in Table 3, which indicates the NMI score of pre-trained models is highly correlated with the ﬁnal performance after the ﬁne-tuning.
G Q UALITATIVE A NALYSIS
Visualization of similarities between consecutive shots To qualitatively check the effect of individual pretext tasks, we visualize the matrix of cosine similarity between shot representations from the randomly sampled 16 consecutive shots in Figure 6.
The shot representations are computed by models without the ﬁne-tuning in order to solely focus on the behavior of each objective at the pretraining stage.
When the MSM is used only, approximately three clusterings are shown, but similarity around boundaries is smoothed.
Next, when we add PP, dissimilarities around the boundaries are to be sharpened. Then, with additional CGM, the clusters are more clearly obtained. Finally, adding SSM makes the similarity of shots within the same cluster higher (i.e., more yellow ones). Pseudo-boundaries We compare the quality of discovered pseudo-boundaries with the ground truth scene boundaries in Figure 7.
In most cases, we observe the pseudo-boundaries identiﬁed by the DTW algorithm are successfully located in close distance with the ground truth ones.
This result validates our idea considering the problem of discovering pseudo-boundary as a temporal alignment problem between two sequences with different frequencies ( S n and S slow n ).
At the same time, we illustrate the failure cases.
Although discovered pseudo-boundary does not match the ground truth in this case, we ﬁgure the determined boundary is not always arbitrary.
For example, the mismatch is often caused by the noise existing in the ground truth (see the ﬁrst row in the failure cases).
On the other hand, in case all shots are visually similar (see the third row in the failure cases), the DTW solely relying on the visual modality fails to ﬁnd the correct boundary.
The ﬁgure 8 illustrates the scene boundary predictions of different models.
Comparing with the baselines, we observe that our approach, BaSSL, shows qualitatively better performance for video scene segmentation.
On the other hand, we observe the oversegmentation issue in many cases using any compared methods (including ours).
Our ﬁnding implies that achieving the highest recall only does not guarantee the highest performance in practice.
We reckon that further studies on this over-segmentation problem would be a highly important topic when it comes to real-world application.