Over the past decade, machine learning methods based on deep neural networks have provided revolutionary advances across various computer vision applications, in particular for image/video processing and understanding. More recently these have been successfully applied to picture compression, both in terms of enhancing individual tools within conventional codecs, and also in providing new end-to-end compression via auto-encoder architectures [1] .
In the context of learning-based video coding tools, there is a distinct class of methods which provide superior performance by employing Convolutional Neural Network (CNN) processes to enhance the quality of the video reconstructed at the decoder. Typical examples include post-processing (PP) and video format adaptations (with CNN format restoration) [2, 3] . These approaches often employ relatively simple network structures and utilise pixel-wise loss functions, which do not reflect the latest advances in the field. More recently, compression enhancement tools have further improved perceptual quality using Generative Adversarial Networks (GANs) [4, 5] . In this case, the loss functions have typically combined pixel-wise distortions, simple quality metrics and feature map differences with artificially configured weights, which do not offer optimal correlation with visual quality.
In this context, we propose a novel GAN architecture for Compressed Video Enhancement (CVEGAN). The main contributions of this work are summarised below.
(1) We present a novel Mul 2 Res block structure to improve the network cardinality, which employs multiple levels of multiple residual learning branches. The residual blocks in each branch also contain convolutional layers with various kernel sizes. This architecture improves overall enhancement performance compared to conventional residual learning blocks with a fixed kernel size.
(2) We employ enhanced residual non-local blocks (ERNBs) and enhanced convectional block attention modules (ECBAMs), both of which have been modified compared to their original structures [6, 7] . Through extracting non-local features (ERNB) and applying channel and spatial attention mechanisms (ECBAM), the representational capability of the network has been further improved.
(3) We have designed a new training methodology, Relativistic SphereGAN (ReSphereGAN), which embodies an enhanced version of the original SphereGAN [8] . ReSphereGAN considers both north pole-centred geodesic distance and the relativistic geodesic distance between the real and fake feature points in the hypersphere. This stabilises the training process and further minimises the difference between generator output and training target.
(4) We also propose a novel perceptual loss function which optimises video quality during training. This linearly combines the logarithms of four existing losses, where the combining weights have been determined using data from eight subjective video quality databases. (5) We conducted a comprehensive comparison between CVEGAN and several state-of-the-art architectures. This was done for two typical coding enhancement tools: postprocessing and spatial resolution adaptation in the context of the MPEG HEVC coding standard. Results demonstrate that CVEGAN provides superior coding performance compared to its counterparts based on both objective quality assessments and subjective comparisons. Figure 1 shows a subjective comparison of example blocks cropped from the reconstructed frames produced by the anchor HEVC codec, CVEGAN and other state-of-the-art architectures.
With dramatic increases in numbers of users and available content, the introduction of new immersive formats and higher user quality expectations, video content is now by far the greatest consumer of global internet bandwidth. demands are challenging all delivery technologies, not least video compression which plays a key role in the trade-off between limited network capacity and elevated user video quality expectations.
The latest video coding standard, Versatile Video Coding (VVC) [9] , approved by ISO and ITU in 2020, provides 30-40% bitrate savings over its predecessor, the High Efficiency Video Coding (HEVC) [10] . A competitor of HEVC and VVC is the open source and royalty-free codec, AV1, which was released by the Alliance for Open Media (AOM) in 2018. AV1 also achieves significant coding gains compared to HEVC [11] . Despite their evident coding gains, none of these codecs have widely exploited machine learning methods to optimise their architecture or tools. In contrast several learning-based approaches have been published in the open literature which achieve very promising results compared to these standards.
Existing learning-based picture coding algorithms can be classified into two primary categories. The first relates to end-to-end training and optimisation using auto-encoder type architectures [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] . Although solutions in this category are not yet competitive with the latest standardised codecs, such as VVC and AV1, they demonstrates significant potential for the future.
A second class contains algorithms (primarily based on CNNs) that are designed to enhance individual coding tools within a standard codec configuration. Such approaches have been used to optimise tools including: intra prediction [23, 24] , motion estimation [25, 26] , transforms [27, 28] , quantisation [29] , entropy coding [30, 31] , post-processing (PP) [32, 33] , in-loop filtering (ILF) [34, 35] and format adaptation [36] [37] [38] .
Among these CNN-based coding tools, there is one group of methods, which stand out in offering higher coding gains compared to the others [2, 3] . These perform CNN operations at the decoder to enhance the quality of reconstructed video frames. Typical examples include post-processing (PP) [33, 39] , inloop filtering [34, 40] , spatial resolution adaptation (SRA) and bit depth adaptation [41] . While most PP approaches focus on single frame enhancement, some recent contributions have reported multi-frame enhancement methods [5, 42] . However, because these are associated with much higher computational complexity, we address only single frame enhancement here.
Most CNN architectures reported for video enhancement have their origins in single image restoration. Their common architectural features include: (i) concatenated convolutional layers [43] (ii) concatenated residual blocks [41] ; (iii) residual dense connections [38] ; (iv) cascading connections [35] ; and (v) feature review structures [44] . More recently, new advanced structures have also been proposed including channel and spatial attention mechanisms [7] and non-local feature extraction [45] .
The performance of a CNN is generally related to three primary factors: depth (i.e. the depth of networks), width (i.e. the number of feature maps), and cardinality (i.e. the size of transformation sets) [46] [47] [48] . Most of the network architectures described above have been designed to increase the network depth [4, 6, [49] [50] [51] [52] [53] [54] and width [47, 51, 52, [54] [55] [56] [57] , while only a few of them exploit the cardinality characteristic [48, [58] [59] [60] . However, cardinality is widely acknowledged to be a more effective way to improve overall performance and network capacity compared to the other two factors [48] . We also note that the kernel size of convolutional layers in these existing networks is usually set at a fixed value (3 in most cases), which may limit the receptive field size and hence the overall network performance [46] . 
In most deep learning-based coding enhancement tools, 1 or 2 loss is used to train the CNN models with the aim of minimising pixel-wise distortions. Alternative training strategies have been proposed to improve perceptual image quality, typically based on Generative Adversarial Network (GAN) architectures and loss functions combining 1/ 2 loss, feature map differences (e.g. VGG19-54 [51] ) and low-complexity quality metrics (e.g. MS-SSIM and SSIM). Notable examples include approaches using standard GANs [50, 61] , Relativistic average GANs (RaGAN) [4, 38, 39, 51, 54, [62] [63] [64] , conditional GANs (cGAN) [57] , Patch GANs [65] , and Wasserstein GAN-gradient penalty (WGAN-GP) [66] .
Moreover, it is well known that 1 and 2 losses do not correlate well with subjective video quality [67] [68] [69] , and the combined loss functions employed in these GANbased training strategies use artificially configured combining weights, which have never been fully evaluated in terms of their correlation with subjective video quality. These issues inevitably lead to sub-optimal training performance when the networks are utilised for compression application.
The proposed CVEGAN follows the basic GAN framework [70] , combining a generator and a discriminator. Its architecture is described in the following subsections.
The generator, denoted as CVENet, is shown in Figure 2 . This takes a 96×96 YCbCr 4:4:4 compressed image block as the input and outputs a processed image block in the same format, targeting its original uncompressed version. The kernel sizes, number of feature maps and stride values for each convolutional layer of CVENet are shown in Figure 2 .
CVENet has three unique structural features: (1) Mul 2 Res blocks; (2) enhanced residual non-local blocks (ERNB); (3) enhanced convolutional block attention modules (ECBAM). These are described below. It can be seen that a Mish activation function [71] has been employed after all convolutional operations except for the final one. Mish has been previously reported to offer better performance than other commonly used functions such as ReLU, leaky ReLU (LReLU) and parametric ReLU (PReLU) [71] .
In order to further improve information flow between Mul 2 Res blocks, cascading connections [ curves in Figure 2 ) are utilised between the input of the first Mul 2 Res block and the 1×1 convolutional layer after each of the Mul 2 Res blocks, and between the output of each Mul 2 Res block (except the final one) and the 1×1 convolutional layer after each of the subsequent Mul 2 Res blocks. Mul 2 Res Block: The new Mul 2 Res Block structure contains multiple levels of multiple residual learning branches to exploit the cardinality characteristic of networks. Figure 3 illustrates the Mul 2 Res Block structure used in CVEGAN, which has four residual learning branches at two different levels. The number of branches and levels can be adapted for different applications based on the computational resources available.
At the first level, the input of the Mul 2 Res block is fed into four residual learning branches. Each branch has a convolutional layer with various kernel sizes (1×1, 3×3, 5×5 and 7×7). This diversifies the feature maps extracted by the convolutional layers using various receptive field sizes. The ECBAM is also employed at both levels before the final skip connection.
The Mul 2 Res block at the second level also has four residual learning branches. The primary differences include (i) the Mul 2 Res Blocks at the first level are replaced by residual blocks, each of which contains two convolutional layers (with various kernel sizes for different branches) and a Mish activation function; (ii) the output from four branches are concatenated before feeding into the ECBAM. ERNB: Based on the non-local operations proposed in [6, 45] , we have developed an enhanced residual non-local block (ERNB), as show in Figure 4 . This employs concatenation operations and residual blocks to achieve feature fusion, avoiding large matrix multiplication and facilitating training of the feature fusion process. A further modification is the introduction of a long skip connection employed to produce the ERNB output; this is designed to further stabilise the nonlocal learning.
ECBAM: Inspired by attention mechanisms proposed recently [6, 7, 53] , we have designed an enhanced convolutional block attention module (ECBAM) for CVEGAN ( Figure 5 ). This follows the basic structure of the convolutional block attention module (CBAM) [7] , which comprises a channel and a spatial attention module. Rather than directly producing output from the second matrix multiplication as in [7] , we have added a concatenation operation with a convolutional layer to achieve non-linear feature fusion. This has been previously reported to improve information flow and overall performance of the network [52] . Figure 6 illustrates the architecture of the CVEGAN discriminator, which is based on SRGAN [50] . The primary differences include (i) we employ two ERNBs -one after the first convolutional layer, and the other before the final convolutional layer -these extract non-local features and improve the representational capability. (ii) we remove the final dense layer in SRGAN to output high (1024) dimensional feature points rather than a single 1D scalar.
We follow a similar training strategy to that described in [50, 51] , which consists of two stages: (i) CVENet is trained using a combined perceptual loss function (Section IV-A) to obtain an initial model for the second training step; (ii) CVENet is then trained jointly with the discriminator with a new training method ReSphereGAN (Section IV-B) .
As discussed in Section II-D, existing loss functions combining pixel-wise losses, feature map differences and lowcomplexity quality metrics, do not always correlate well with perceived quality. To address this we have employed a loss function comprising a linear combination of the elementary transforms of six commonly used losses, 1 (denoted as L 1 ), 2 (L 2 ), gradient loss [57, 73] (1)
Here f (·) represents an elementary transformation [76] , which can be either a constant, a power, a root, an exponential, a logarithmic, a trigonometric, an inverse trigonometric, a hyperbolic or an inverse hyperbolic function. a i represents the linear combination weights, where a 6 is always set to 1. The range of all these six single losses is between 0 and 1. We have excluded non-linear combinations and combinations of different transformed losses due to their high computational and training complexity.
We have used an eight-fold cross validation method [77] to train the proposed combined loss function (equation (1)) based on eight publicly available subjective video quality databases. These include: the Netflix public database (70 test sequences) [78] , BVI-HD (192) [67] , CC-HD (108) [11] , CC-HDDO (90) [79] , MCL-V (96) [80] , SHVC (32) [81] , IVP (100) [82] , and VQEG-HD3 (72) [83] . All of these contain video sequences compressed using commonly used video codecs (H.264, HEVC, AV1, VVC or MPEG-2).
The eight databases were divided into two sub groupsseven training datasets and one for testing. An exhaustive search was performed among all tested transformation functions and their corresponding weighting parameters, the best of which was selected for this split to achieve the highest average correlation between the combined loss values and subjective scores for all seven training datasets. The Spearman rank order correlation coefficient (SROCC) is employed to quantify the correlation performance. The search range of each parameter is between 0 and 1, with an interval of 0.1. To avoid possible content bias due to a single trainingtesting split, we performed this cross-validation for all eight splits. Table I The transformation function used for all optimal loss functions across all eight training-testing splits, is the natural logarithm ln(·). We use the median values of the corresponding combination parameters, and normalise these to ensure 6 i=1 a i = 1. The final combined loss function L P used to train CVEGAN is given below:
It should be noted that L P remains within the range 0 to 1, and is differentiable, which is required to support backpropagation during training. Since the test sequences were generated using a wide range of video codecs, this loss function should generalise well across image and video compression applications. L P has been used here for training the CVENet (generator) during the first training stage.
The proposed Relativistic SphereGAN training methodology is a modified version of the SphereGAN [8] , which is based on an integral probability metric (IPM). As illustrated in Figure 7 , the original SphereGAN compares the geodesic distances between the north pole N and fake/real feature points in the n-dimensional Euclidean feature space through an inverse stereographic projection. Here n is the feature point number of fake and real data produced by the CVEGAN discriminator, which has a default value of 1024. Inspired by RaGAN [84] , we also calculate the relativistic geodesic distance between the projected real and fake feature points in our loss functions at the second training stage. This modification further optimises the generator by obtaining gradient information from both real and fake data during the adversarial training process. Specifically, the loss functions for generator (L Re gen ) and discriminator (L Re disc ) (in the second training stage) are given below:
Geodesic Distance 
Here , E(·) represents the mean operation. d m (a, b) is the geometric-aware transformation function [8] , which calculates the m-th central geometric moment (geodesic distance) [8] in the hypersphere space between a and b (projected feature points in the hypersphere space). M is set to 3 in this work. T (·) stands for the inverse of stereographic projection [8] from the Euclidean feature space to the hypersphere space.
N is denoted as the north pole in the hypersphere space, and x r and x f are the real and fake feature points respectively in n-dimensional Euclidean feature space. The weight combining the perceptual loss (L P ) and the generator adversarial loss was set to 0.005 based on several previous works [4, 51, 54] on GAN-based image restoration and enhancement.
Gradient Analysis: The gradients generated from the losses during the second training stage are crucial for stabilising the GAN models. It is important to avoid vanishing gradients and explosion problems as pointed out in [46, 70] . The gradients of d m (N, T (x r )) and d m (N, T (x f )) have already been evaluated in [8] , so we just analyse the gradients of the new relativistic geodesic distance d m (T (x r ), T (x f )).
for all m (∇ represents the derivation operation, and · 2 is the Euclidean norm).
Material. This indicates that the proposed ReSphereGAN will ensure stable GAN learning with any moment m.
As mentioned in Section II-B, we have trained and evaluated the proposed network for two coding tools: post-processing (PP) and spatial resolution adaptation (SRA). Other important tools for compression enhancement, e.g. in-loop filtering and effective bit depth adaptation, have similar workflows to PP and SRA but offer lower overall coding gains [41, 85] . These are therefore not presented here due to the limited space available.
Although the first version of VVC has recently been approved by standardisation bodies, we chose to use HEVC Test Model HM 16.20 as the host codec here as proof of concept. This is because HEVC has a low complexity compared to VVC and is hence more suitable for the large scale experimentation employed in this study. Based on our previous investigations, we anticipate that the gains for other codecs (AV1, VVC) will be commensurate with those presented for HEVC. However this remains for future work.
The coding workflows for CNN-based PP and SRA tools are shown in Figure 8 and 9 . The CNN module employed for PP provides an additional quality enhancement on the decoded content, while for SRA, the original video is spatially downsampled (by a factor of 2) before encoding and a CNN-based super resolution operation is performed on the decoded lowresolution content for both up-sampling and quality enhancement. More detailed descriptions of PP and SRA can be found in [86] .
A large training database, BVI-DVC [86] , has been employed to train the proposed network. This contains 800 video clips (10 bit, YCbCr 4:2:0) at various spatial resolutions from 270p to 2160p covering a wide range of content and video texture types. It has been previously reported to provide enhanced training performance over other databases for optimising CNN-based coding enhancement tools. It has recently been adopted as the default training database by JVET Ad-hoc Group 11 (Neural-network-based video coding) [87] .
We followed the same procedure as detailed in [86] to generate training material for CNN-based PP and SRA coding tools using the HEVC HM 16.20 with the Random Access (RA) configuration (Main10 profile) and four base quantisation parameter (QP) values, 22, 27, 32 and 37. This produces two classes of training data for PP and SRA, each of which contains four QP sub-groups. For the SRA class, the nearest neighbour (NN) filter was utilised to up-sample the compressed low resolution video frames to the original resolution, which ensures that the same CVENet architecture can be used for both PP and SRA.
For each QP sub-group in the PP and SRA classes, the compressed (or the NN up-sampled compressed) video frames and their corresponding original counterparts were randomly selected, cropped into 96×96 blocks and converted YCbCr 4:4:4 format. Finally, approx. 195,000 pairs of image blocks for each QP sub-group were generated.
CVEGAN was implemented using the TensorFlow platform (1.8.0). The training process was conducted based on the following parameters: Adam optimisation [88] with the following hyper-parameters: β 1 =0.9 and β 2 =0.999; batch size of 16; 200 training epochs; initial learning rate (0.0001); weight decay of 0.1 for every 100 epochs.
During evaluation, each decoded frame (after NN upsampling for SRA) was converted to YCbCr 4:4:4 format and segmented into 96×96 overlapping blocks with an overlap size of 4 pixels as CVENet (the generator) input. The network output blocks from CVENet were further aggregated following the same pattern and converted to YCbCr 4:2:0 format to form the final reconstructed frame.
Twenty-four popular and state-of-the-art CNN and GAN architectures, which have been widely used in image superresolution, restoration and video compression, have been benchmarked in this paper. All of these have been reimplemented and trained using the same framework (Ten-sorFlow 1.8.0) with identical training material following the same training methodology and loss functions as described in their original literature. During re-implementation, the input and output interfaces of these networks have been modified to satisfy the data format requirements.
All networks under test have been integrated into both PP and SRA coding tools and tested under JVET common test conditions (CTC) [91] using the RA configuration (Main10) with four QP values of 22, 27, 32 and 37. The original HEVC HM 16.20 was used as the host codec and also as the benchmark anchor. The coding performance against HEVC HM is calculated using the Bjøntegaard Delta [92] measurement (BDrate) based on two quality metrics: Peak Signal-to-Noise-Ratio (PSNR, luma channel only) and Video Multimethod Assessment Fusion (VMAF, 0.6.1) [93] . PSNR is the most widely used quality metric for image and video compression, although it does not always correlate well with subjective quality scores [67, 94] . VMAF is a machine learning-based video quality assessment algorithm, increasingly used in industry, which combines various quality metrics and video features using a Support Vector Machine regressor. It has been reported to offer much better correlation performance with subjective opinions on compressed content compared to the PSNR [67, 93, 95] .
The relative computational complexity of all test networks (the generator for the case of GANs) have also been calculated To evaluate their performance, JVET-CTC SDR (19 test sequences) [91] has been employed as the main test dataset due to its content and resolution diversity. The average BD-rates assessed by both PSNR and VMAF were calculated for all tested networks. To further evaluate network generalisation, another two commonly used test databases, UVG (6 test sequences) [96] and AOM main test dataset (21 test sequences) objective-1-fast (o-1-f) 1 [97] , have also been employed to test the proposed CVEGAN and the other three top performers. None of the sequences in these three test datasets were included in the training database, BVI-DVC. It should be noted that only UHD (2160p) content from these databases was used to evaluate the SRA coding tool since, as previously reported [41] , lower resolutions provide only limited and inconsistent coding gains.
We have also conducted a lab-based subjective test using a double stimulus methodology on a selection of network architectures (alongside the anchor HM and CVEGAN) for both PP and SRA. We collected subjective scores from 10 subjects 2 using the reconstructed videos of 12 UHD source sequences (QP 37 only). Further details of the testing configuration can be found in the Supplementary Material.
Five primary contributions have been tested and compared to the state of the art for PP and SRA. All ablation studies are based on the JVET-CTC SDR dataset.
(1) Mul 2 Res Block effectiveness has been evaluated by replacing it with other commonly used convolutional blocks for CNN-based image restoration, which include residual block (RB) [50] , modified residual block (MRB) [4] , residual dense block (RDB) [52] , residual-in-residual dense block (RRDB) 1 We have excluded a few source sequences in UVG and o-1-f datasets, which have been employed in BVI-DVC as training data. 2 Due to the impact of the COVID-19 pandemic, a large lab-based subjective test employing non-expert participants could not be conducted. [51] , residual channel attention block (RCAB) [53] , Xception block [59] (cardinality is 4), ResNeXt block [48] (cardinality is 4) and ResNeSt block [60] (cardinality is 4).
(2) ERNB is substituted by the original non-local block [6] to evaluate its effectiveness.
(3) ECBAM is replaced by the original CBAM [7] for comparison.
(4) ReSphereGAN training has been compared with other commonly used GAN training approaches, including standard GAN [50] , Relativistic average GAN (RaGAN) [51] , PatchGAN [65] , conditional GAN (cGAN) [98] , Wasserstein GAN-gradient penalty (WGAN-GP) [99] and the original SphereGAN [8] .
(5) Perceptual Loss Function proposed in this paper was compared with other commonly used loss functions for GAN training (L 7 -L 11 ), as described in the Section IV-A. Table II summarises the performance for CVEGAN and a selection of the top performing networks, alongside the ablation study results, with more comprehensive results provided in the Supplementary Material. It can be observed that, for both PP and SRA, the proposed CVEGAN outperforms all other tested architectures based on the perceptual quality metric VMAF. It also has a relatively low complexity compared to UDSR, RNAN and RFB-ESRGAN. From the ablation study, we observe that the proposed Mul 2 Res block, ERNB, ECBAM, ReSphereGAN and the new perceptual loss function have contributed at least 1.1%, 1.2%, 1.1%, 1.3% and 1.4% coding gains (assessed by VMAF) when compared to the tested replacements, but with similar or lower complexity. Table III presents the average DMOS (difference of the mean opinion score) values of all evaluated sequences for CVEGAN, HEVC and the three top performing networks. The average DMOS for CVEGAN is lower than that for HEVC anchor and the other networks, providing further evidence of its effectiveness. Figure 1 also shows a subjective comparison between the output of HM 16.20, RNAN [6] , MSRGAN [4] and the proposed CVEGAN. More results can be found in the Supplementary Material. 
In this paper, a novel GAN architecture, CVEGAN, has been proposed for compressed video quality enhancement. This network, when integrated into a conventional video coding system, has enabled significantly improved coding performance compared to many state-of-the-art architectures. This enhanced performance can be attributed to the use of several new features including a novel Mul 2 Res blocks, ERNB, ECBAM, the new ReSphereGAN training methodology and perceptualinspired loss functions. Future work will address network complexity reduction and validation on other standardised codecs. We will also extend our approach to other application scenarios, including image restoration and super-resolution processing.
We would like to acknowledge funding from UK EPSRC (EP/L016656/1 and EP/M000885/1) and the NVIDIA GPU Seeding Grants.