  We propose a hybrid recurrent Video Colorization with Hybrid Generative
Adversarial Network (VCGAN), an improved approach to video colorization using
end-to-end learning. The VCGAN addresses two prevalent issues in the video
colorization domain: Temporal consistency and unification of colorization
network and refinement network into a single architecture. To enhance
colorization quality and spatiotemporal consistency, the mainstream of
generator in VCGAN is assisted by two additional networks, i.e., global feature
extractor and placeholder feature extractor, respectively. The global feature
extractor encodes the global semantics of grayscale input to enhance
colorization quality, whereas the placeholder feature extractor acts as a
feedback connection to encode the semantics of the previous colorized frame in
order to maintain spatiotemporal consistency. If changing the input for
placeholder feature extractor as grayscale input, the hybrid VCGAN also has the
potential to perform image colorization. To improve the consistency of far
frames, we propose a dense long-term loss that smooths the temporal disparity
of every two remote frames. Trained with colorization and temporal losses
jointly, VCGAN strikes a good balance between color vividness and video
continuity. Experimental results demonstrate that VCGAN produces higher-quality
and temporally more consistent colorful videos than existing approaches.
