  Video frame transmission delay is critical in real-time applications such as
online video gaming, live show, etc. The receiving deadline of a new frame must
catch up with the frame rendering time. Otherwise, the system will buffer a
while, and the user will encounter a frozen screen, resulting in unsatisfactory
user experiences. An effective approach is to transmit frames in lower-quality
under poor bandwidth conditions, such as using scalable video coding. In this
paper, we propose to enhance video quality using lossy frames in two
situations. First, when current frames are too late to receive before rendering
deadline (i.e., lost), we propose to use previously received high-resolution
images to predict the future frames. Second, when the quality of the currently
received frames is low~(i.e., lossy), we propose to use previously received
high-resolution frames to enhance the low-quality current ones. For the first
case, we propose a small yet effective video frame prediction network. For the
second case, we improve the video prediction network to a video enhancement
network to associate current frames as well as previous frames to restore
high-quality images. Extensive experimental results demonstrate that our method
performs favorably against state-of-the-art algorithms in the lossy video
streaming environment.
