  This paper addresses the video rescaling task, which arises from the needs of
adapting the video spatial resolution to suit individual viewing devices. We
aim to jointly optimize video downscaling and upscaling as a combined task.
Most recent studies focus on image-based solutions, which do not consider
temporal information. We present two joint optimization approaches based on
invertible neural networks with coupling layers. Our Long Short-Term Memory
Video Rescaling Network (LSTM-VRN) leverages temporal information in the
low-resolution video to form an explicit prediction of the missing
high-frequency information for upscaling. Our Multi-input Multi-output Video
Rescaling Network (MIMO-VRN) proposes a new strategy for downscaling and
upscaling a group of video frames simultaneously. Not only do they outperform
the image-based invertible model in terms of quantitative and qualitative
results, but also show much improved upscaling quality than the video rescaling
methods without joint optimization. To our best knowledge, this work is the
first attempt at the joint optimization of video downscaling and upscaling.
