  Neural style transfer models have been used to stylize an ordinary video to
specific styles. To ensure temporal inconsistency between the frames of the
stylized video, a common approach is to estimate the optic flow of the pixels
in the original video and make the generated pixels match the estimated optical
flow. This is achieved by minimizing an optical flow-based (OFB) loss during
model training. However, optical flow estimation is itself a challenging task,
particularly in complex scenes. In addition, it incurs a high computational
cost. We propose a much simpler temporal loss called the frame difference-based
(FDB) loss to solve the temporal inconsistency problem. It is defined as the
distance between the difference between the stylized frames and the difference
between the original frames. The differences between the two frames are
measured in both the pixel space and the feature space specified by the
convolutional neural networks. A set of human behavior experiments involving 62
subjects with 25,600 votes showed that the performance of the proposed FDB loss
matched that of the OFB loss. The performance was measured by subjective
evaluation of stability and stylization quality of the generated videos on two
typical video stylization models. The results suggest that the proposed FDB
loss is a strong alternative to the commonly used OFB loss for video
stylization.
