I NTER -BMV: I NTERPOLATION WITH B LOCK M OTION V ECTORS FOR F AST S EMANTIC S EGMENTATION ON V IDEO
A BSTRACT
Models optimized for accuracy on single images are often prohibitively slow to run on each frame in a video.
Recent work exploits the use of optical ﬂow to warp image features forward from select keyframes, as a means to conserve computation on video.
This approach, however, achieves only limited speedup, even when optimized, due to the accuracy degradation introduced by repeated forward warping, and the inference cost of optical ﬂow estimation.
To address these problems, we propose a new scheme that propagates features using the block motion vectors (BMV) present in compressed video (e g. H.264 codecs), instead of optical ﬂow and bi-directionally warps and fuses features from enclosing keyframes to capture scene context on each video frame.
Our technique, interpolation-BMV, enables us to accurately estimate the features of intermediate frames, while keeping inference costs low.
We evaluate our system on the CamVid and Cityscapes datasets, comparing to both a strong single-frame baseline and related work.
We ﬁnd that we are able to substantially accelerate segmentation on video, achieving near real-time frame rates (20+ frames per second) on large images (e g. 960 ×pixels), while maintaining competitive accuracy.
This represents an improvement of almost 6 × over the single-frame baseline and 2 .
5 × over the fastest prior work
I NTRODUCTION
Semantic segmentation, the task of assigning each pixel in an image to a semantic object class, is a problem of long-standing interest in computer vision.
Like models for other image recognition tasks (e.g. classiﬁcation, detection, instance segmentation), semantic segmentation networks have grown drastically in both layer depth and parameter count in recent years, in the race to segment more complex images, from larger, more realistic datasets, at higher accuracy.
As a result, state-of-the-art segmentation networks today require between 0.5 to 3.0 seconds to segment a single , high-resolution image (e.g. 2048 × 1024 pixels) at competitive accuracy (Zhu et al.
; Gadde et al.
Meanwhile, a new target data format for segmentation has emerged: video.
The motivating use cases include both batch applications, where video is segmented in bulk to generate training data for other models (e.g. autonomous control systems), and streaming settings, where high-throughput video segmentation enables interactive analysis of live footage (e.g. at surveillance sites).
Video in these contexts consists of long image sequences, shot at high frame rates (e.g. 30 fps) in complex environments (e.g. urban cityscapes) on modern, high-deﬁnition cameras.
Segmenting individual frames at high accuracy still calls for the use of competitive image models, but their inference cost precludes their na¨ıve deployment on every frame in a raw multi-hour video stream.
A deﬁning characteristic of realistic video is its high level of temporal continuity.
Consecutive frames demonstrate signiﬁcant spatial similarity, which suggests the potential to reuse computation across frames.
Building on prior work, we exploit two observations: 1) higher-level features evolve more slowly than raw pixel content in video, and 2) feature computation tends to be much more expensive than task-speciﬁc computation across a range of vision tasks (e.g. detection, segmentation) (Shelhamer et al.
; Zhu et al.
Accordingly, we divide our semantic segmentation model into a deep feature network and a cheap, shallow task network (Zhu et al.
We com-
[Figure]
the feature maps with frame-to-frame motion estimates.
The task network is executed on all frames.
Given that feature warping and task computation is much cheaper than feature extraction, a key parameter we aim to optimize is the interval between designated keyframes.
Here we make two key contributions.
First, noting the high level of data redundancy in video, we successfully utilize an artifact of compressed video, block motion vectors (BMV), to cheaply propagate features from frame to frame.
Unlike other motion estimation techniques, which require specialized convolutional networks, block motion vectors are freely available in modern video formats, making for a simple, fast design.
Second, we propose a novel feature estimation technique that enables the features for a large fraction of video frames to be inferred accurately and efﬁciently (see
Fig. 1).
In particular, when computing the segmentation for a keyframe, we also precompute the features for the next designated keyframe.
Features for all subsequent intermediate frames are then computed as a fusion of features warped forward from the last visited keyframe, and features warped backward from the incoming keyframe.
This procedure implements an interpolation of the features of the two closest keyframes.
We then combine the two ideas, using block motion vectors to perform the feature warping in feature interpolation.
The result is a scheme we call interpolation-BMV .
We evaluate our framework on the CamVid and Cityscapes datasets.
Our baseline consists of running a competitive segmentation network, DeepLab (Chen et al.
, on every frame, a setup that achieves published accuracy (Dai et al.
(2017)), and throughput of 3.6 frames per second (fps) on
CamVid and 1.3 fps on Cityscapes.
Our improvements come in two phases.
First, our use of motion vectors for feature propagation allow us to cut inference time on intermediate frames by 53%, compared to approaches based on optical-ﬂow, such as Zhu et al.
Second, our bi-directional feature warping and fusion scheme achieves substantial accuracy improvements, especially at high keyframe intervals.
Together, the two techniques allow us to operate at over twice the average inference speed as the fastest prior work, at any target level of accuracy.
For example, if we are willing to tolerate no worse than 65 mIoU on our CamVid video stream, we are able to operate at a throughput of 20.1 fps, compared to the 8.0 fps achieved by the forward ﬂow-based propagation from Zhu et al.
Overall, even when operating in high accuracy regimes (e.g. within 3% mIoU of the baseline), we are able to accelerate segmentation on video by a factor of 2 - 6 × .
R ELATED W ORK 2. I MAGE S EMANTIC S EGMENTATION
Semantic segmentation is a classical image recognition task in computer vision, originally studied in the context of statistical inference.
The approach of choice was to propagate evidence about pixel class assignments through a probabilistic graphical model (Felzenszwalb & Huttenlocher (2004);
Shotton et al.
(2009)), a technique that scaled poorly to large images with numerous object classes (Kr¨ahenb¨uhl & Koltun (2011)).
In 2014, Long et al.
(2015) proposed the use of fully convolutional neural networks (FCNs) to segment images, demonstrating signiﬁcant accuracy gains on several key datasets.
Subsequent work embraced the FCN architecture, proposing augmentations such as dilated (atrous) convolutions (Yu & Koltun (2016)), post-processing CRFs (Chen et al.
and pyramid spatial pooling (Zhao et al.
(2017)) to further improve accuracy on large, complex images.
2. E FFICIENT V IDEO S EMANTIC S EGMENTATION
The recent rise of applications such as autonomous driving, industrial robotics, and automated video surveillance, where agents must perceive and understand the visual world as it evolves , has triggered substantial interest in the problem of efﬁcient video semantic segmentation.
Shelhamer et al.
(2016) and Zhu et al.
(2017) proposed basic feature reuse and optical ﬂow-based feature warping, respectively, to reduce the inference cost of running expensive image segmentation models on video.
Recent work explores adaptive feature propagation, partial feature updating, and adaptive keyframe selection as techniques to further optimize the scheduling and execution of optical-ﬂow based warping (Zhu et al.
; Li et al.
Xu et al.
In general, these techniques fall short in two respects: (1) optical ﬂow computation remains a computational bottleneck, especially as other network components become cheaper, and (2) forward feature propagation fails to account for other forms of temporal change, besides spatial displacement, such as new scene content (e.g. new objects), perspective changes (e.g. camera pans), and observer movement (e.g. in driving footage).
As a result, full frame features must still be recomputed frequently to maintain accuracy, especially in video footage with complex dynamics, fundamentally limiting the attainable speedup.
2. M OTION AND C OMPRESSED V IDEO
Wu et al.
(2018) train a network directly on compressed video to improve both accuracy and performance on video action recognition.
Zhang et al.
(2016) replace the optical ﬂow network in the classical two-stream architecture (Simonyan & Zisserman (2014)) with a “motion vector CNN”, but encounter accuracy challenges, which they address with various transfer learning schemes.
Unlike these works, our main focus is not efﬁcient training, nor reducing the physical size of input data to strengthen the underlying signal for video-level tasks, such as action recognition.
We instead focus on a class of dense prediction tasks, notably semantic segmentation, that involve high-dimensional output (e.g. a class prediction for every pixel in an image) generated on the original uncompressed frames of a video.
This means that we must still process each frame in isolation.
To the best of our knowledge, we are the ﬁrst to propose the use of compressed video artifacts to warp deep neural representations, with the goal of drastically improved inference throughput on realistic video.
S YSTEM O VERVIEW 3. N ETWORK A RCHITECTURE
We follow the common practice of adapting a competitive image classiﬁcation model (e.g. ResNet101) into a fully convolutional network trained on the semantic segmentation task (Long et al.
(2015); Yu et al.
; Chen et al.
We identify two logical components in our ﬁnal model: a feature network , which takes as input an image i ∈ R 1 × 3 × h × w and outputs a representation f i ∈
R 1 ×
A × h 16 × w 16 , and a task network , which given the representation, computes class predictions for each pixel in the image,
p i ∈ R 1 × C × h × w .
The task network N task is built by concatenating three blocks: (1) a feature projection block, which reduces the feature channel dimensionality to A 2 , (2) a scoring block, which predicts scores for each of the C segmentation classes, and (3) an upsampling block, which bilinearly upsamples the score maps to the resolution of the input image.
3. B LOCK M OTION V ECTORS
MPEG-compressed video consists of two logical components: reference frames, called I-frames, and delta frames, called P-frames.
Reference frames are still RGB frames from the video, usually represented as spatially-compressed JPEG images.
Delta frames, which introduce temporal compression to video, consist of two subcomponents: block motion vectors and residuals.
Block motion vectors, the artifact of interest in our current work, deﬁne a correspondence between pixels in the current frame and pixels in the previous frame.
They are generated using block motion compensation , a standard procedure in video compression algorithms (Richardson (2008)):
Divide the current frame into a non-overlapping grid of 16x16 pixel blocks.
For each block in the current frame, determine the “best matching” block in the previous frame.
A common matching metric is to minimize mean squared error between the blocks.
For each block in the current frame, represent the pixel offset to the best matching block in the previous frame as an ( x, y ) coordinate pair, or motion vector .
The resulting grid of ( x, y ) offsets forms the block motion vector map for the current frame.
For a 16 M × 16 N frame, this map has dimensions M × N .
The residuals then consist of the pixel-level difference between the current frame, and the previous frame transformed by the motion vectors.
3. F EATURE P ROPAGATION
Many cameras compress video by default as a means for efﬁcient storage and transmission.
The availability of a free form of motion estimation at inference time, the motion vector maps in MPEGcompressed video, suggests the following scheme for fast video segmentation (see Algorithm 1 ). Algorithm 1: input 2: for frame
I
{ I } do 3: if i mod n then 4: f i ← N feat
( I i )
S i ← N task
( f i ) 6: else
7: f i ← WARP
S i ← N task
( f i ) 9: end if 10: f c ← f i
11: end for 12: output : frame segmentations { S i }
Choose a keyframe interval n .
On keyframes (every n th frame), execute the feature network N feat to obtain a feature map.
Cache these computed features, f c , and then execute the task network
N task to obtain the keyframe segmentation.
On intermediate frames, extract the motion vectors mv
[ i ] corresponding to the current frame index.
Warp the cached features f c one frame forward via bilinear interpolation with − mv [ i ] .
(To warp forward, we apply the negation of the vector map.)
Here we employ the differentiable, parameter-free spatial warping operator proposed by Jaderberg et al.
Finally, execute N task on the warped features to obtain the current segmentation.
3.3. I NFERENCE R UNTIME A NALYSIS
Feature propagation is effective because it relegates feature extraction, the most expensive network component, to select keyframes.
Of the three remaining operations performed on intermediate frames – motion estimation, feature warping, and task execution – motion estimation with optical ﬂow is the most expensive (see Fig. 2).
By using block motion, we eliminate this remaining bottleneck, accelerating inference times on intermediate frames for a DeepLab segmentation network (Chen et al.
from 116 ms per frame ( F + W + N task ) to 54 ms per frame ( W + N task ).
For keyframe interval n , this translates to a speedup of 53% on n − 1 n of the video frames.
Note that for a given keyframe interval n , as we reduce inference time on intermediate frames to zero, we approach a maximum attainable speedup factor of n over a frame-by-frame baseline that runs the full model on every frame.
Exceeding this bound, without compromising on accuracy, requires an entirely new approach to feature estimation, the subject of the next section.
Incidentally, we also benchmarked the time required to extract block motion vectors from raw video (i.e. H.264 compression time), and found that ffmpeg takes 2.78 seconds to compress 1,
Cityscapes video frames, or 2.78 ms per frame.
In contrast, optical ﬂow computation on a frame pair takes 62 ms (Fig. 2).
We include this comparison for completeness: since compression is a default behavior on modern cameras, block motion extraction is not a true component of inference time. [Figure] F
[Figure]
F EATURE I NTERPOLATION
Given an input video stream, our goal is to compute the segmentation of every frame as efﬁciently as possible, while preserving accuracy.
In a batch setting, we have access to the entire video, and desire the segmentations for all the frames, as input to another model (e.g. an autonomous control system).
In a streaming setting, we have access to frames as they come in, but may be willing to tolerate a small delay of keyframe interval n frames ( n 30 seconds at 30 fps) before we output a segmentation, if that means we can match the throughput of the video stream and maintain high accuracy.
We make two observations.
First, all intermediate frames in a video by deﬁnition lie between two designated keyframes, which represent bounds on the current scene.
New objects that are missed in forward feature propagation schemes are more likely to be captured if both past and incoming keyframes are used.
Second, feature fusion techniques are effective at preserving strong signals in any one input feature map, as seen in Feichtenhofer et al.
This suggests the viability of estimating the features of intermediate frames as the fusion of the features of enclosing keyframes.
Expanding on this idea, we propose the following algorithm (see Fig. 1).
On any given keyframe, precompute the features for the next keyframe.
On intermediate frames, warp the previous keyframe’s features, N feat ( I k ) , forward to the current frame
I i using incremental forward motion estimates, − mv [ k : i ] .
Warp the next keyframe’s features, N feat ( I k + n ) , backward to the current frame using incremental backward motion estimates, mv [ k + n : i ] .
Fuse the two feature maps using either a weighted average or learned fusion operator, F .
Then execute the task network N task on the fused features.
This forms Algorithm 2 .
A formal statement is included in Appendix: Sec. 6.1.
To eliminate redundant computation, on keyframes, we precompute forward and backward warped feature maps f f , f b corresponding to each subsequent intermediate frame, { I k +1 , ..., I k + n − 1 } .
For keyframe interval n , this amounts to n − 1 forward and n − 1 backward warped feature maps.
3.4. F EATURE F USION
We consider several possible fusion operators: max fusion, average fusion, and convolutional fusion (Feichtenhofer et al.
We implement max and average fusion by aligning the input feature maps f f , f b ∈ R 1 × C × h × w along the channel dimension, and computing a max or average across each pixel in corresponding channels, a parameter-free operation.
We implement conv fusion by stacking the input feature maps along the channel dimension [ f f , f b ] C = f s ∈ R 1 × 2 C × h × w , and applying a bank of learned, 1x1 conv ﬁlters to reduce the channel dimensionality by a factor of two.
Before applying the fusion operator, we weight the two input feature maps f f , f b by scalars α and 1 − α , respectively, that correspond to feature relevance , a scheme that works very effectively in practice.
For keyframe interval n , and a frame at offsets p and n − p from the previous and next keyframes, respectively, we set α = n − pn and 1 − α = pn , thereby penalizing the input features warped farther from their keyframe.
Thus, when p is small relative to n , we weight the previous keyframe’s features more heavily, and vice versa.
In summary, the features for intermediate frame
I i are set to: f i = F ( n − pn f f , pn f b ) , where p = i mod n .
This scheme is reﬂected in Alg. 2.
E XPERIMENTS 4.0. D ATASETS
We train and evaluate our system on CamVid (Brostow et al.
and Cityscapes (Cordts et al.
(2016)), two popular, large-scale datasets for complex urban scene understanding.
CamVid consists of over 10 minutes of footage captured at 30 fps and 960 × 720 pixels.
Cityscapes consists of 30frame video snippets shot at 17 fps and 2048 × 1024 pixels.
On CamVid, we adopt the standard train-test split of Sturgess et al.
On Cityscapes, we train on the train split and evaluate on the val split, following the example of previous work (Yu et al.
; Chen et al.
; Zhu et al.
We use the standard mean intersection-over-union (mIoU) metric to evaluate segmentation accuracy, and measure throughput in frames per second (fps) to evaluate inference performance.
4.0. A RCHITECTURE
For our segmentation network, we adopt a variant of the DeepLab architecture called Deformable
DeepLab (Dai et al.
(2017)), which employs deformable convolutions in the last ResNet block (conv5) to achieve signiﬁcantly higher accuracy at comparable inference cost to a standard DeepLab model.
DeepLab (Chen et al.
(2017)) is widely considered a state-of-the-art architecture for semantic segmentation, and a DeepLab implementation currently ranks ﬁrst on the PASCAL VOC object segmentation challenge (Aytar (2018)).
Our DeepLab model uses ResNet-101 as its feature network, which produces intermediate representations f i ∈ R 1 × 2048 ×
h 16 × w 16 .
The DeepLab task network outputs predictions p i ∈ R 1 × C × h × w , where C is 12 or 20 for CamVid and Cityscapes respectively.
4.0. T RAINING
To train our single-frame DeepLab model, we initialize with an ImageNet-trained ResNetmodel, and learn task-speciﬁc weights on the CamVid and Cityscapes train sets.
To train our video segmentation system, we sample at random a labeled image from the train set, and select a preceding and succeeding frame to serve as the previous and next keyframe, respectively.
Since motion estimation with block motion vectors and feature warping are both parameter-free, feature propagation introduces no additional weights.
Training feature interpolation with convolutional fusion, however, involves learning weights for the 1x1 conv fusion layer, which is applied to stacked feature maps, each with channel dimension 2048 .
For both schemes, we train with SGD on an AWS
EC2 instance with 4 Tesla K80 GPUs for 50 epochs, starting with a learning rate of 10 − 3 .
4. R ESULTS 4.1. B ASELINE
For our accuracy and performance baseline, we evaluate our full DeepLab model on every labeled frame in the CamVid and Cityscapes test splits.
Our baseline achieves an accuracy of 68.6 mIoU on CamVid, at a throughput of 3.7 fps.
On Cityscapes, the baseline model achieves 75.2 mIoU, matching published results for the DeepLab architecture we used (Dai et al.
, at 1.3 fps.
4.1. P ROPAGATION AND I NTERPOLATION
In this section, we evaluate our two main contributions: 1) feature propagation with block motion vectors ( prop-BMV ), and 2) feature interpolation, our new feature estimation scheme, implemented with block motion vectors ( inter-BMV ).
We compare to the closest available existing work on the problem, a feature propagation scheme based on optical ﬂow (Zhu et al.
( prop-ﬂow ).
We evaluate by comparing accuracy-runtime curves for the three approaches on CamVid and Cityscapes (see Fig. 3).
These curves are generated by plotting accuracy against throughput at each keyframe interval in Appendix: Tables 4 and 5, which contain comprehensive results.
First, we note that block motion-based feature propagation ( prop-BMV ) outperforms optical ﬂowbased propagation (prop-ﬂow) at all but the lowest throughputs.
While motion vectors are slightly less accurate than optical ﬂow in general, by cutting inference times by 53% on intermediate frames (Sec. 3.3.1), prop-BMV enables operation at much lower keyframe intervals than optical ﬂow to achieve the same inference rates.
This results in a much more favorable accuracy-throughput curve.
Second, we ﬁnd that our feature interpolation scheme ( inter-BMV ) strictly outperforms both feature propagation schemes.
At every keyframe interval, inter-BMV is more accurate than prop-ﬂow and prop-BMV; moreover, it operates at similar throughput to prop-BMV.
This translates to a consistent advantage over prop-BMV, and an even larger advantage over prop-ﬂow (see Fig. 3).
On CamVid, inter-BMV actually registers a small accuracy gain over the baseline at keyframe intervals 2 and 3, utilizing multi-frame context to improve on the accuracy of the single-frame DeepLab model.
[Figure]
[Figure] [Figure]
[Figure] [Figure]
Metrics.
We also distinguish between two metrics: the standard average accuracy, results for which are plotted in Fig.
3, and minimum accuracy, which is a measure of the lowest frame-level accuracy an approach entails, i.e. accuracy on frames farthest away from keyframes.
Minimum accuracy is the appropriate metric to consider when we wish to segment a video as efﬁciently as possible, while ensuring that all frame segmentations meet some threshold level of accuracy.
As an example, at an accuracy target of 65 mIoU, feature interpolation enables operation at 20.1 fps on CamVid (see Table 4).
This is 2 . 5 × faster than achievable inference speeds with feature propagation alone, using either optical ﬂow (8.0 fps) or block motion vectors (9.3 fps).
In general, feature interpolation achieves over twice the throughput as Zhu et al.
(2017) on CamVid and Cityscapes, at any target accuracy.
Minimum accuracy plots (Fig. 5) are included in the Appendix.
Baseline.
We also compare to our frame-by-frame DeepLab baseline, which offers low throughput but high average accuracy.
As Figures 3a and 3b indicate, even at average accuracies above 68 mIoU on CamVid and 70 mIoU on Cityscapes, ﬁgures competitive with contemporary single-frame models (see Table 1 and Table 2), feature interpolation offers speedups of 4 .
5 × and 4 .
2 × , respectively, over the baseline.
Notably, at key interval 3, interpolation obtains a 2 .
5 × speedup over the baseline on CamVid, at slightly higher than baseline accuracy (see Fig. 3a and Table 4). [Table]
[Table]
Delay.
Recall that feature interpolation introduces a delay of keyframe interval n frames, which corresponds to n 30 seconds at 30 fps.
For example, at n = 3 , inter-BMV introduces a delay of [Figure] seconds, or 100 ms.
To put this context, prop-ﬂow (Zhu et al.
(2017)) takes 125 ms to segment a frame at key interval 3, and inter-BMV takes 110 ms.
Thus, by lagging by less than 1 segmentation, we are able to segment 2 .
5 × more frames per hour than the frame-by-frame model (9.1 fps vs. 3.
fps).
This is a suitable tradeoff in almost all batch settings ( .g training data generation, -hoc video analysis)
Fig.
4 depicts a qualitative comparison of interpolation and prop-ﬂow (Zhu et al.
[Figure]
[Figure]
F EATURE F USION
In this second set of experiments, we evaluate the accuracy gain achieved by feature fusion, in order to isolate the contribution of fusion to the success of our feature interpolation scheme.
As
Table 3 demonstrates, utilizing any fusion strategy, whether max, average, or conv fusion, results in higher accuracy than using either input feature map alone.
This holds true even when one feature map is signiﬁcantly stronger than the other (rows 2-4), and for both short and long distances to the keyframes.
This observed additive effect suggests that feature fusion is highly effective at capturing signal that appears in only one input feature map, and in merging spatial information across time.
Forward and Backward refer to the input feature maps (and their direction of warping).
Cityscapes.
[Table]
keyframe(s)
[Table]
C ONCLUSION
We develop interpolation-BMV , a novel segmentation scheme that combines the use of block motion vectors for feature warping, bi-directional propagation to capture scene context, and feature fusion to produce accurate frame segmentations at high throughput.
We evaluate on the CamVid and Cityscapes datasets, and demonstrate signiﬁcant speedups across a range of accuracy levels, compared to both a strong single-frame baseline and prior work.
Our methods are general, and represent an important advance in the effort to operate image models efﬁciently on video.
A PPENDIX
S YSTEM D
We provide the formal statement of feature interpolation, Algorithm 2.
3: for frame do 4: 5: f i ← N feat I i curr keyframe features 6:
S i ← N task f
7: f i + n ← N feat I i n ) keyframe features 8: W f ← PROPAGATE
− mv W b ← PROPAGATE − mv 10: else
11: p ← i mod n
12: f i ← F p n · W f p p n · W b [ n − p
S i ← N task f
14: end if 15: end for 16: output : frame segmentations { S i } 17: function PROPAGATE ( f g g 18:
O ← [ f ] 19: for i = 1 to n do 20: append i
21: end for 22: return O 23: end function
6. R ESULTS
This appendix section includes full tabular results for the CamVid and Cityscapes datasets ( Tableand Table 5 ) and minimum accuracy vs. throughput plots for CamVid and Cityscapes ( Figure 5 ).
[Table]
inter-BMV 3.
[Table]
[Figure]
Data from
[Figure]
[Figure]