Scalable Neural Video Representations with Learnable Positional Features
Abstract
Succinct representation of complex signals using coordinate-based neural representations (CNRs) has seen great progress, and several recent efforts focus on extending them for handling videos.
Here, the main challenge is how to (a) alleviate a compute-inefficiency in training CNRs to (b) achieve high-quality video encoding while (c) maintaining the parameter-efficiency.
To meet all requirements (a), (b), and (c) simultaneously, we propose neural video representations with learnable positional features (NVP), a novel CNR by introducing “learnable positional features” that effectively amortize a video as latent codes.
Specifically, we first present a
CNR architecture based on designing 2D latent keyframes to learn the common video contents across each spatio-temporal axis, which dramatically improves all of those three requirements.
Then, we propose to utilize existing powerful image and video codecs as a compute-/memory-efficient compression procedure of latent codes.
We demonstrate the superiority of NVP on the popular UVG benchmark; compared with prior arts, NVP not only trains 2 times faster (less than 5 minutes) but also exceeds their encoding quality as 34.07 → 34.57 (measured with the PSNR metric), even using > 8 times fewer parameters.
We also show intriguing properties of NVP, e.g .,
video inpainting, video frame interpolation, etc.
Introduction
Recent advances in coordinate-based neural representations (CNRs) [9, 13, 17, 40, 46] have shown great promise in the field as a new paradigm for representing complex signals, including gigapixel images [29, 34], audios [40], 3D scenes [30, 33, 35] and even large city-scale street views [47].
Instead of storing signal outputs as a coordinate grid ( e.g ., image pixels), whose memory requirement scales unfavorably in terms of resolution and dimension, CNRs represent each signal as a compactly parameterized, continuous neural network; they interpret a signal as a coordinate-to-value function and train a neural network to approximate this mapping.
CNRs enjoy numerous appealing properties, including data compression [11, 12, 59], super-resolution [6], novel view synthesis [18, 22, 33, 54], and generative modeling of complex, high-dimensional data [14, 24, 41, 42, 57, 60], while being parameter-efficient interpretation of a given signal in various scenarios.
In particular, several works have attempted to exploit CNRs to interpret video signals [5, 23, 42, 57] by learning a neural network f : R 3 → R 3 with f ( x, y, t ) =
( r, g, b ) and exhibited their potential as a succinct representation of videos, as well as providing numerous applications including video generative modeling [42, 57], video compression [5, 59], and video super-resolution [7].
They
[Figure]
observe that conventional CNR architectures [40, 46] often fail to encode large-scale videos due to their complex temporal dynamics accompanied by large spatial variations, but the problem can be remarkably mitigated by designing CNR architectures specialized for videos.
For instance, Chen et al.
[5] proposes a CNR structure that focuses on the continuous modeling of the video signal only along the temporal dimension, allowing for more radical variations along spatial axes; it exhibits a comparable encoding quality to existing powerful video codecs ( e.g ., H.264 [51], HEVC [43]) while enjoying lots of intriguing properties ( e.g ., denoising and video frame interpolation).
However, CNRs suffer from a severe compute-inefficiency, 2 limiting their scalability to encode realworld, large-scale videos despite their advantages.
To alleviate this issue, several works [26, 34, 37] have proposed new CNR architectures by separating a CNR f into two parts; f
= h ϕ ◦ g θ for a coordinate-to-latent mapping g θ ( x, y, t ) =
z and a latent-to-RGB mapping h ϕ ( z ) =
( r, g, b ) .
They construct g θ as an embedding function defined with latent grids U θ in which the shape resembles the grid interpretation of a given signal ( e.g ., a 2D array of C -dimensional latent codes U θ ∈
R
H
× W × C for image pixels) rather than as a neural network.
These approaches have shown a promise in compute-efficiency due to the strong locality induced by a grid structure of U θ ; however, they result in another problem: these architectures severely sacrifice the parameter-efficiency since the parameter size of θ can be very large, growing proportionally to both the input coordinate dimension and the signal resolution.
In this paper, we focus on developing video CNRs that are the best of both worlds: achieving high-quality encoding and the compute-/parameter-efficiency simultaneously.
Contribution.
We introduce neural video representations with learnable positional features (NVP), a novel CNR for videos.
NVP avoids requiring a single giant full-dimensional 3D array in g θ by presenting learnable positional features that effectively amortize a given video as “2D and 3D” latent grids with succinct parameters.
Specifically, we decompose the coordinate-to-latent mapping as
[Equation]
where we present two types of latent grids for constructing these mappings (see Figure 2).
◦
Latent keyframes : We first design “image-like” 2D latent grids U
θ
xy , U θ xt , U θ yt for g θ
xy , g θ xt , g θ yt (respectively) that learn the representative video contents across each spatio-temporal axis and dramatically improve the parameter efficiency of NVP.
◦ Sparse positional features : We then introduce a “video-like” 3D latent grid U θ xyt for g θ xyt , whose size is much smaller than the original video pixels
but effectively encodes video details locally.
[Figure]
Moreover, we propose a compute-/memory-efficient compression procedure to further reduce the parameter θ by incorporating existing image and video codecs, e.g ., JPEG [49] for images and
HEVC [43] for videos, respectively.
In particular, we treat 2D and 3D latent grids like the image or video pixels and utilize powerful compression pipelines for them.
Our compression scheme does not require any re-training of trained parameters, which significantly increases compute-efficiency to prior approaches to compressing CNR parameters but remarkably maintains the encoding quality.
We also remark that such a compression approach is not applicable to the hashing-based latent grid of the prior method [34], while ours is “collision-free” and maintains the video- or image-like structures.
Finally, for the choice of h ϕ , we suggest using a modulated network (with respect to the temporal coordinate) to improve the encoding quality of videos that contain dynamic motions.
We verify the effectiveness of our method on the popular UVG benchmark [32].
In particular, NVP achieves the peak signal-to-noise ratio (PSNR; higher is better) metric of 34.57 in 5 minutes (with a single NVIDIA V100 32GB GPU): it is achieved > 2 times faster, even with using > 8 times fewer parameters than the state-of-the-art on compute-efficiency that reaches 34.07 in 10 minutes.
Moreover, compared with prior arts on encoding quality, our method improves the learned perceptual image patch similarity (LPIPS [58]; lower is better) as 0.145 → 0.102 ( + 29.7%) with a similar number of parameters while requiring ∼ 72.5% less training time.
We also show numerous compelling properties of NVP, e.g .,
video inpainting, video frame interpolation, super-resolution, compression, and consistent frame-wise encoding results without deviating quality.
Related work
Coordinate-based neural representations.
Coordinate-based neural representations (CNRs), also known as implicit neural representations or neural fields, have emerged as a new paradigm for representing complex, continuous signals.
They propose to encode signals through a neural network, typically a multilayer perceptron (MLP) combined with high-frequency sinusoidal activations [40] or
Gaussian activations [9].
Prior works have focused on utilizing neural fields on various complicated data, e.g ., gigapixel images [29, 34], 2D videos [5, 42, 57], 3D static scenes, [30, 33, 35], and 3D dynamics scenes [23, 39, 52].
In particular, most approaches have focused on constructing CNR architectures for encoding 3D scenes [3, 15, 47] and exhibited how employing specialized prior knowledge for a given signal domain in the architecture can remarkably boost the encoding quality.
Despite these successes, extending CNRs for videos is yet under-explored.
In this paper, we aim to move toward developing a CNR for videos by exploiting their unique temporal properties.
Hybrid CNRs.
Rather than solely designing a neural network of coordinate-to-RGB mapping, hybrid CNRs incorporate learnable latent codes that follow a grid structure, e.g ., image CNRs combined with 2D latent spatial grids [6, 31], and 3D scene (or shape) CNRs with latent cubic grids [4, 8, 19, 26, 29, 37].
Specifically, they compute a latent vector using the grid-structured latent code and pass it through a neural network to compute the signal output at a given input coordinate.
Such approaches have shown significant efficiencies in training time and encoding quality due to the powerful locality induced by grid-shaped latent codes.
However, the number of parameters required for latent grid-based representations easily grows proportionally to the input coordinate dimension or data resolution, limiting the scalability of hybrid CNRs.
Remarkably, some of the recent approaches have exhibited this inefficiency can be significantly mitigated by considering multi-level (or progressive) structures for latent grids [26, 29, 44, 45].
While prior works primarily focus on encoding images or 3D scenes, we aim to design
NVP: Neural video representations with learnable positional features
We first formulate our problem setup as follows.
Given a video signal v :=
( f 1 , f 2 , . . .
, f T ) consisting of T video frames, the goal is to find a compact neural representation f w with parameters w , from which the original video v can be reconstructed with high quality.
Here, the quality can be defined using various distortion metrics, e.g ., peak signal-to-noise ratio (PSNR) [16] and LPIPS [58], for evaluating a reconstruction quality and a perceptual similarity, respectively.
To achieve this goal, we take an approach based on coordinate-based neural representations (CNRs)— a paradigm where each datum ( e.g ., video) is parameterized as a neural network of coordinate mapping.
In particular, we aim to represent the given video using a neural network f w : R 3 → R 3 , which maps the space-time coordinates ( x, y, t ) of the video to corresponding RGB values ( r, g, b ) , where f w is optimized with reconstruction objectives, e.g ., mean-squared error.
Such an approach has significant potential, as CNRs have shown to encode other continuous, complex signals ( e.g ., 3D scenes) [40, 46] compactly while enjoying lots of intriguing properties, e.g ., super-resolution [6] and denoising [5].
However, CNRs have suffered from tremendous time costs for training, and even in the case of videos, it is difficult to achieve high-quality encodings if one utilizes conventional CNR architectures that overlook the complex spatio-temporal dynamics of videos [5].
Our contribution lies in resolving these issues by designing “learnable positional features” that succinctly encode a video as latent codes with high quality and keeping their compute-/parameter-efficiency intact.
In the rest of this section, we provide a detailed description of each component in NVP.
In Section 3.1, we explain the architecture of NVP.
We then describe our compression procedure in Section 3.2.
3.1 Architecture
We design our video CNR f w as a composition of two functions with a parameterization w :=
( θ, ϕ ) : a coordinate-to-latent mapping g θ and a latent-to-RGB mapping h ϕ .
Here, we decompose the coordinate-to-latent-mapping as g θ =
g θ xy × g θ xt
×
g θ yt
× g θ xyt with g θ ( x, y, t ) =
( z xy , z xt , z yt , z xyt ) (for θ :=
( θ xy , θ xt , θ yt , θ xyt ) ), where each g θ xy , g θ xt , g θ yt is formalized with image-like 2D latent spatial grids U θ xy , U θ xt , U θ yt (respectively) and g θ xyt is designed with a video-like sparse 3D latent grid U θ xyt .
We then present the latent-to-RGB mapping h ϕ
( z xy , z xt , z yt , z xyt ) =
( r, g, b ) to be a multi-layer perceptrion (MLP) modulated by another neural network.
To explain our architecture, we assume all the input coordinate ( x, y, t ) of g θ (and f w ) is in [0 , 1] 3 ⊂ R 3 without loss of generality.
Learnable latent keyframes.
At a high level, learnable latent keyframes U
θ xy , U θ xt , U θ yt are imagelike 2D latent grids learned to capture the common representative contents in a given video v across each t -, y -, x -axis, respectively.
For a given input coordinate ( x, y, t ) , we compute latent vectors z xy , z xt , z yt from U θ
xy , U θ xt , U θ yt individually.
We explain our keyframe only with U θ yt by letting U :
= U yt for simplicity, but note that other keyframes U θ xy , U θ xt operate in the similar manner.
Formally, U is L 2D spatial grids of C -dimensional latent codes u ij , whose resolution is H l × W l :
U :=
( U 1 , . . .
, U L ) ,
[Equation]
Here, the keyframe follows an L -level multi-resolution structure, i.e ., for each level l , the height H l and the width W l become different as H l =
⌊ γ l − 1 H 1 ⌋ and W l =
⌊ γ l − 1 W 1 ⌋ with fixed γ > 1 and
H 1 , W 1 > 0 , where ⌊·⌋ indicates the floor function of the input.
Since U =
U θ yt is shared over the x -axis, we compute the latent vector z yt :=
( z 1 yt , . .
z L yt ) by considering only the value of y and t at a given coordinate ( x, y, t ) .
Specifically, for l = 1 , . . .
, L , each z l yt is a linearly interpolated vector of four vectors in the spatial grid U l , where the indices of these vectors are chosen as the closest ones to the relative position of the input coordinate ( y, t ) ∈ [0 , 1] 2 :
[Equation]
where lerp indicates a linear interpolation operation at the input coordinate between given vectors.
Note that we learn the keyframe as the latent codes, unlike conventional approaches that specify the keyframe in a deterministic manner from T video frames ( f 1 , f 2 , . . .
, f T ) ; it encourages to capture the representative contents of the video over each spatio-temporal direction better.
We also remark that our architecture involves two keyframes U θ xt , U θ yt that are considered across spatial axes.
Considering these keyframes may not be beneficial in the RGB space as the spatial variation of video pixels is often large.
However, in our approach, these keyframes are learned under a more flexible, continuous latent space, and thus the representative frames can even be found in these spatial directions in such a space while encoding the RGB outputs of the video accurately.
Finally, recall that U θ
xy , U θ xt , U θ yt consist of multi-resolution spatial grids, i.e ., the resolution of spatial grids in each latent keyframe grows from coarse to fine.
Since natural scenes often include repeating patterns in various scales, e.g ., a scene of flowers of different sizes, this multi-resolution architecture promotes learning common patterns with reduced memory and computation costs, which is also validated in Müller et al.
Sparse positional features.
Given a space-time coordinate ( x, y, t ) , we compute the latent vector z xyt with U θ xyt that represents the local details of the video at this input position.
Here, U θ xyt :
= ( u ijk ) ∈
R H
× W × S × D is a 3D sparse grid of D -dimensional latent codes, i.e ., the 3D grid size
H × W × S is much smaller than the size of the video pixels of a 3D RGB grid:
[Equation]
for 1 ≤ i ≤ H, 1 ≤ j ≤ W, 1 ≤ k ≤ S.
To evaluate the latent vector z xyt , we concatenate h × w × s latent codes in U θ xyt that their indices are near the relative position of the input ( x, y, t ) :
[Equation]
where h, w, s > 0 are given as hyperparameters.
The locality of U θ xyt as 3D latent codes dramatically alleviates the compute-inefficiency in fitting videos, in contrast to conventional CNRs where the entire parameters are shared (as a neural network) for arbitrary input coordinates ( x, y, t ) and thus require a significant training cost.
Moreover, recall that we construct U θ xyt as a sparse 3D grid of latent codes; remarkably, U θ xyt efficiently captures the video details even if the size is smaller than the number of video pixels since the common contents of a given video are effectively encoded with the latent keyframes U
θ xy , U θ xt , U θ yt .
Note that we design U θ xyt as a sparse 3D grid; each single u ijk should represent a wide area of videos solely without concatenation.
As the single latent vector may lack expressive power to represent such a wide range and may result in a non-smooth transition as the CNR output.
Hence, we mitigate this issue by concatenating multiple vectors near each other (see Figure 10 in Section 4.3).
Instead of directly selecting near latent codes from U θ xyt , one may consider upsampling of U θ xyt using linear interpolation before selecting the close latent codes at a given coordinate.
Such an interpolation further helps each latent code to learn smoother representations and also generalizes to representing video frames at unseen coordinates during training.
Meanwhile, this upsampling requires more computing time compared to the computational cost of other modules in NVP, and thus faces a trade-off between the smoothness and compute-efficiency in training (see Table 4 in Section 4.3).
Modulated implicit function.
With the latent vector z : =
[ z xy , z xt , z yt , z xyt ] evaluated from g θ , a naïve design choice of the latent-to-RGB mapping h ϕ is to utilize a MLP that maps z to the corresponding RGB output ( r, g, b ) .
However, if a video contains temporally dynamic motions, we found such a simple MLP architecture occasionally lacks expressive power and can be difficult to capture the complex dynamics of the given video, even with the large network size of h ϕ .
To circumvent this issue, we design h ϕ to be a K -layer MLP (coined as a synthesizer network) modulated by another modulator network [31]: where the latent vector z and the time coordinate t are passed through the modulator and the synthesizer, respectively.
Here, the modulator network utilizes piecewise linear activations ( e.g ., ReLU), where the synthesizer uses sinusoidal activations.
Specifically, an RGB output ( r, g, b ) of the latent vector z (from ( x, y, t )) is computed as follows:
[Equation]
where A k , b k are weights and biases of k -th layer of the synthesizer, z k is k -th hidden feature of the modulator, and ⊙ denotes an element-wise product.
It helps to achieve high-quality encoding rapidly in early training iterations and often at the convergence than a naive MLP (See Table 3 for details).
indicates applying the method without the corresponding compression scheme.
We report the BPP values of NeRV without compressing parameters if the encoding time is ≤ 1 hour since the NeRV’s compression requires a longer time.
On the other hand, the compression procedure of NVP only takes less than 1 minute, but for a fair comparison, we do not apply it to NVP as well whenever NeRV is not compressed.
[Table]
3.2 Compression procedure
Recall that we aim to find “compact” video CNRs; several works have focused on reducing the number of coordinate-based neural representations parameters (or bits) after training while maintaining their performance.
In particular, they have relied on exploiting existing well-known techniques for neural network compression, e
[Table]
g ., exploiting magnitude pruning [5, 21] or quantization [5, 59], and exhibited considerable results.
However, these approaches mainly involve a re-training of CNR parameters, requiring severe computation costs, and thus are not suitable for practical scenarios.
Instead, we propose a compression pipeline for NVP, which does not require re-training, yet significantly reduces the number of bits while preserving the video quality.
Our main idea is to incorporate existing image and video codecs that have shown their promises for the compression of given pixels.
In particular, we focus on compressing keyframes U θ
xy , U θ xt , U θ yt , and sparse positional features U θ xyt , as the parameter size of the modulated implicit function h ϕ is neglectable compared with those.
Specifically, we quantize U θ xyt
and U
θ xy , U θ xt , U θ yt as 3D/2D grids of 8-bit latent codes and regard them as video and image pixel grids, where the number of the channel becomes the dimension of latent codes.
Based on these interpretations, we compress these latent codes by utilizing existing video and image codecs, e.g ., HEVC [43] for videos and JPEG [49] for images.
Intriguingly, we found this procedure can significantly reduce the parameters while notably maintaining the video quality without any fine-tuning of the latent-to-RGB mapping h ϕ
(See Section 4.3).
Experiments
We verify the effectiveness of our framework on UVG-HD [32], a representative benchmark for evaluating video encodings.
Experimental results demonstrate that our neural video representations with learnable positional features (NVP) simultaneously improves the overall performance by (a) alleviating a compute-inefficiency in training, (b) achieving high-quality video encoding, and (c) maintaining parameter-efficiency.
We also show applications of our NVP, including video inpainting and spatio-temporal interpolation.
Finally, we conduct ablation studies to validate each component.
[Figure]
Evaluation.
We follow similar setups on prior work [5] proposing coordinate-based neural representations (CNRs) for videos.
All reported numbers are averaged over 7 videos in UVG-HD: Beauty,
Bosphorus, HoneyBee, Jockey, ReadySetGo, ShakeNDry, and Yachtride, along with the standard deviations unless otherwise specified.
We also use the Big Buck Bunny video, which is used in
NeRV [5].
For quantitative evaluation, we use the following metrics: peak signal-to-noise ratio (PSNR) for reconstruction quality, LPIPS [58], FLIP [2], and SSIM [50] for perceptual similarity, where all metrics are evaluated in a frame-wise manner and averaged over the whole video.
We evaluate these metrics on different bits-per-pixel (BPP; lower is better) to evaluate the parameter efficiency; see Appendix A.1 for more description of the evaluation.
Implementation details.
All main experiments, including baselines, are processed with a single
GPU (NVIDIA V100 32GB) and 28 instances from a virtual CPU (Intel ® Xeon ® Platinum
CPU @ 2.70GHz), where it takes at most ∼ 11 hours to run our method and ∼ 2 days to run other baselines.
Moreover, we denote NVP-S and NVP-L as the model with latent code dimensions of sparse positional features to be 2 and 4, respectively; see Appendix A.2 for more details.
Baselines.
We compare our method with SIREN [40], and FFN [46], which are well-known signalagnostic CNR architectures, Instant-ngp [34] for state-of-the-art CNRs on compute-efficiency, and
NeRV [5], which is a CNR specialized for videos.
For all of the baseline methods, we follow their reported experimental setups.
In particular, for NeRV [5], we use two configurations provided in the official implementation: NeRV-S and NeRV-L for a small and a large model, respectively.
See
Appendix B for a detailed description of baseline methods.
4.1 Main results
Figure 1, Figure 3, and Table 1 summarize quantitative and qualitative results of NVP and baselines.
Remarkably, as shown in Figure 1, NVP can accurately capture dynamic motions and high-frequency details of a video, e.g ., the legs of a running horse, while prior state-of-the-art CNRs fail to achieve and show a blurry artifact.
We also emphasize such a high-quality encoding is accomplished in “less than 1 minute”, which supports the superior compute-efficiency of NVP in training.
Moreover, Table 1 verifies the effectiveness of NVP with quantitative evaluations, which outperforms all other baselines at varying encoding times from ∼ 5 minutes to > 40 hours.
In particular, NVP significantly improves LPIPS compared with previous methods, demonstrating how the encoded videos are perceptually similar to ground-truth videos.
Such a result is also confirmed in Figure 3;
NeRV shows a distortion that some pixels significantly deviate from the ground-truth outputs, while our method does not suffer from such artifacts.
We also note that the variance of NVP is relatively small compared with other baselines, which shows the robustness of videos with diverse scenes and motion.
See Appendix C and D for video-wise results and discussion on decoding time, respectively.
[Figure]
[Figure] [Figure]
4.2 Applications
In this section we provide several applications of our method, NVP, as video CNRs.
For better, playable illustrations and qualitative results, please refer to our project page.
Video inpainting.
Intriguingly
[Figure]
our method has the capability of video inpainting, i.e ., the desired moving object in the video can be naturally removed by capturing shared video contents with learnable keyframes.
Figure 4 visualizes the illustration of inpainting results from NVP; as shown in this figure, one can see the car is removed where such parts are filled with a natural background.
Video frame interpolation.
Since video CNRs approximate videos as temporally continuous signals, they should interpolate among two different frames at an arbitrary time, even if such a frame does not exist in the training dataset.
To validate the interpolation capability of NVP, we provide the quantitative results on Big Buck Bunny sequences; our method shows better interpolation results measured with various metrics, such as PSNR and LPIPS.
Video super-
resolution.
We remark that NVP encodes a given video as a spatio-temporally continuous signal.
Thus, our method can interpolate the frames across spatial directions, i.e ., frame-wise super-resolution.
Figure 5 exhibits how well NVP smoothly interpolates video frames across spatial directions while preserving the sharp edges, compared with naïve upsampling methods.
Video compression.
Recall that one of the major advantages of CNRs is their succinct encoding to represent a given signal; one may consider utilizing CNRs for video compression [5, 11, 12].
To verify the potential of our method on video compression, we compare the quality of compressed videos from NVP with the ones from the current state-of-the-art video codecs.
As shown in Figure 6, compressed videos from NVP show the comparable reconstruction quality (measured with PSNR metrics) while outperforming perceptual similarity (measured with LPIPS metrics).
4.3 Ablation studies
Effect of architecture components.
To verify the effectiveness of each component, we train our model with all the videos in UVG-HD by removing each component while maintaining the total number of parameters, then measure PSNR metrics from these models.
Table 3 summarizes the effect of three different architecture components.
Without any of the components consisting of our positional features, the reconstruction quality gets dramatically worse, which validates how NVP
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Figure]
[Figure]
succinctly encodes a given video as latent codes.
We also note that the modulation not only improves the final encoding quality but also notably achieves high-quality encoding in its early training epochs.
Compression procedure.
To validate the effectiveness of our compression scheme, we compare the encoding quality of (a) NVP with our compression scheme and (b) NVP compressed through magnitude-based pruning.
Figure 7 shows the proposed compression pipeline outperforms conventional magnitude-based pruning under various BPP values.
We also remark that our compression method does not require re-training and is thus much more time-efficient.
Analysis of non-temporal keyframes.
Recall that we additionally design keyframes across spatial directions, unlike conventional approaches that designate the keyframe over only the temporal direction.
To validate the effect of such keyframes, we compare NVP with (a) NVP without spatial keyframes and (b) NVP without all of the keyframes in Figure 8, where the number of parameters is equally set for a fair comparison.
While utilizing latent learnable keyframes only over temporal direction is already fairly effective for high-quality encoding, one can observe the consideration of keyframes across other directions provides a further improvement.
Consistent frame-wise encoding quality.
Existing keyframe-based compression approaches often suffer from inconsistent encoding quality: the frame-wise quality of the compressed video highly depends on whether it is the designated keyframe or not.
In contrast, NVP learns the keyframes and does not have this problem.
Figure 9 validates the result: our method exhibits consistent encoding performance while conventional popular video codecs show several peaks that the reconstruction quality (measured with PSNR metrics) highly deviates from others.
Effect of the concatenation of latent codes in sparse positional features.
To validate the effectiveness of design choice on extracting latent features from sparse positional features U θ xyt , we compare the reconstructions from NVP with and without concatenation of U θ xyt .
As shown in Figure 10, the concatenation of latent codes u ijk in U θ xyt indeed mitigates non-smooth transitions between latent codes and captures sharp details in a given video better.
In particular, without the concatenation, it results in undesirable artifacts ( e.g ., showing discontinuous borders), validating our concatenation scheme for constructing latent representations from sparse positional features.
Effect of upsampling of sparse positional features.
We also examine the effect of upsampling of the sparse positional features U θ xyt .
Figure 11 shows the result: linear interpolation of U θ xyt exhibits more smooth patterns for unseen coordinates during training.
Meanwhile, we note that the upsampling requires 1.61 times more training time per iteration due to the additional computation bottleneck (see
Table 4); however, regardless of upsampling, we remark that NVP still achieves notable computeefficiency compared with prior state-of-the-art methods (such as Chen et al.
[Figure] [Figure]
[Table]
Discussion and conclusion
We proposed NVP, a new coordinate-based neural representation (CNR) to encode videos as succinct latent codes.
Our main idea is to decompose a video into “image-like” and ”video-like” structures to learn coordinate-to-latent mapping efficiently.
Extensive experiments have verified the effectiveness of NVP on all the parameter-/compute-efficiency and the encoding quality.
We hope our method will facilitate various future research directions in the CNR area.
Limitations and future works.
Each video contains different scenes and motions so that it can be either static or dynamic, yet we utilize the same hyperparameters and architectures for encoding any video.
Although such a video-agnostic design is fairly effective and outperforms prior works, we believe the video-wise consideration of the architecture and hyperparameter can remarkably boost the performance further.
Moreover, we have shown the potential of utilizing powerful image and video codes for compressing latent codes in NVP; extending such codecs to be specialized for the compression of latent codes should be an interesting direction.
Negative social impacts.
A side effect of CNRs is their potential unexpected behavior on encoding; they may cause undesirable artifacts in representing the given signal but are challenging to predict due to the under-explored behavior of training CNRs.
Furthermore, in the case of representing videos, the encoded videos may suffer from severe distortions and conceivably cause ethical problems.
In this respect, such behaviors should be extensively and carefully investigated and mitigated to exploit
CNRs as the standard for encoding videos in real-world situations.
Acknowledgments and Disclosure of Funding
We would like to thank Younggyo Seo, Jihoon Tack, Jongheon Jeong, Sukmin Yun, Jongjin Park,
Junsu Kim, Seong Hyeon Park, Seojin Kim, Changyeon Kim, Jaehyun Nam, and anonymous reviewers for their helpful feedbacks and discussions.
We also appreciate Max Ehrlich for providing the exact results of prior video compression methods.
This work was mainly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2021-0-02068, Artificial
Intelligence Innovation Hub; No.2019-0-00075, Artificial Intelligence Graduate School Program (KAIST); No.2019-0-01906, Artificial Intelligence Graduate School Program (POSTECH)).
This research was partly supported by the Challengeable Future Defense Technology Research and
Development Program (No.915027201) of Agency for Defense Development in 2022.
This work was partially supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No.2022R1F1A1075067).
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section 5.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Section 5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Sectionand supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 4 and supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Section 4.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 4.
(b) Did you mention the license of the assets?
[Yes] See Section 4.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] See the supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] See the supplementary material.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Appendix:
Scalable Neural Video Representations with Learnable Positional Features
[Table] [Table]
[Table]
A More description of experimental setups A.1 Metrics
• PSNR .
To measure the reconstruction quality of models, we use peak signal-to-noise ratio (PSNR), evaluated as − log 10 ( MSE ) , where MSE denotes the mean-squared error between the ground-truth video and the reconstructed video (as 0-1 scale).
• LPIPS .
We use AlexNet [20] pretrained on ImageNet [10], which best performs as a forward metric.
It measures a weighted distance between normalized internal features of the real video and its reconstruction.
• FLIP .
FLIP automates the difference evaluation between alternating images by building on principles of human perception, where we use the official implementation.
A.2 Further implementation details
Training details.
We train the network by adopting mean-squared error as our loss function and using the AdamW optimizer [27] with a learning rate of 0.01.
We utilize HEVC [43] for the compression of our sparse positional features and JPEG [49] for reducing the parameters of keyframes.
Architecture.
For the learnable keyframes, which follow a multi-level structure, we set the number of levels as 16, per level scale γ as 1.35, and the coarsest resolution as 16 × 16 .
To build models with different sizes, we set the number of features per level as 2 and 4 for NVP-S and NVP-L, respectively.
For the sparse positional features, considering the number of video frames, we use a 3D grid size of 300 × 300 × 300 for ShakeNDry and 300 × 300 × 600 for other videos in UVG-HD, which is ∼ 23 × smaller than the video pixels.
We then concatenate 3 × 3 × 1 latent codes in sparse positional features.
Same as the learnable keyframes, we set the latent dimensions of sparse positional features to beand 4 for NVP-S and NVP-L, respectively.
In addition, we design modulated implicit function as a 3-layer multi-layer perceptron (MLP) modulated by another modulator network; both have a hidden size of 128.
For the synthesizer network, we use SIREN [40], which uses sin( σ t z ) with σ t ∈ R as the activation functions, and set the temporal frequency of the first layer as σ t = 30 and the other layers as σ t = 1 .
This network is modulated by a modulator network with LeakyReLU
[53] activation.
We train the network with a batch size of 1,245,184, i.e ., at each iteration, we randomly sample 1,245,184 pixels from entire video pixels.
We initialize the parameters of learnable keyframes (and sparse positional features) from the uniform distribution U ( − 10 − 4 , 10 − 4 ) and use Kaiming normal initialization for the modulator networks.
We train the network using AdamW optimizer [27] with the initial learning rate η = 0 .
01 and weight decay λ = 0 .
We use a cosine annealing learning rate scheduler, where the current learning rate η t at the t -th iteration is defined as follows:
[Equation]
where η min is set to 0.00001, and the total iteration T is set to 100,000 for both NVP-S and NVP-L.
Compression procedure.
Following the prior work[5], we used ffmpeg [48] to extract RGB frames from compressed UVG-HD video files and to compress learnable positional features of NVP.
First, we use the following command to extract RGB videos from the original YUV videos of
UVG-HD:
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
where INPUT is the input file name, and OUTPUT is a directory to save decompressed RGB frames.
Then we use the following commands to compress learnable keyframes:
[Equation]
qscale
[Equation]
SCALE output.
[Equation]
where SCALE is an output option that controls image quality.
We also use the following commands to compress sparse positional features:
[Equation]
where FR and CRF indicate the frame rate and constant rate factor value that controls video quality and compression ratio, respectively.
We summarize the hyperparameters used in the below table:
[Table]
B Description of baseline methods
In this section, we briefly describe the specific parameter we used as baselines of video coordinatebased neural representations (CNRs) for evaluating our framework at a high level.
To compare the parameter efficiency, we consider two situations: the average bits-per-pixel (BPP) of each baseline is either near 0.200 or 0.400.
However, in the case of Instant-ngp [34], to compare the compute efficiency, we set the total number of parameters as similar as NVP-L when encoding time is ≤hour
• SIREN [40], which uses high frequency sine activations ( i.e ., sin( ω 0 z ) with ω
≫ 1 ) and takes spatio-temporal coordinate ( x, y, t ) as input then outputs a corresponding RGB value for each pixel.
We use a 5-layer multi-layer perceptron (MLP) with a hidden size of 2,048, where the frequency ω 0 is set to 30 for all sinusoidal activations.
• FFN [46] leverages random fourier feature (RFF) ( i.e ., [sin(2 π Wz ) , cos(2 π Wz )] ) as a positional embedding layer to encode spatio-temporal coordinates ( x, y, t ) and uses ReLU activation for further layers to output a corresponding RGB value for each pixel.
We use RFF with W ∈ R 3 × 1024 with W ij ∼ N (0 , σ 2 ) with σ = 10 , and a 4-layer MLP with a hidden size of 2,048.
• NeRV [5], a CNR specialized for videos, takes a time index as input and outputs a corresponding RGB image.
We use two configurations provided in the official implementation: 6 NeRV-S and NeRV-L for a small and a large model, respectively.
Specifically, we first apply a 2-layer MLP on the output of the positional encoding layer, and then we stack 5 NeRV blocks with upscale factors 5, 3, 2, 2, 2, respectively.
We set the output channel for the first fully-connected layer as 128 for both models and change the expansion at the beginning of convolutional block size for 4 andfor NeRV-S and NeRV-L, respectively.
After training one model for a single video separately, we prune the trained parameters by removing 15% of the weights, quantize model weights to 8-bit, and apply entropy coding following the compression procedure in the paper.
• Instant-ngp
[34] uses multiresolution hash tables of trainable feature vectors for a given input coordinate ( x, y, t ) .
To be specific, on the UVG-HD benchmark, we set the number of levels as 15, the number of features per level as 2, the maximum entries per level as 2 24 , and the coarsest resolution as 16.
We set per level scale γ as 1.3 for ShakeNDry, and 1.5 for other videos in the UVG-HD.
We use a small neural network with 64 neurons and two hidden layers that use ReLU as output activation for a small latent-to-RGB mapping.
C Video-wise main results
In this section
[Table]
we provide video-wise quantitative results and qualitative illustrations.
[Table]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
D Comparison of video decoding time
We compare the decoding time of NVP with other baselines on a single GPU (NVIDIA V100 32GB).
It was measured on Jockey (600 frames with 1920 × 1080 resolution) in UVG-HD with a single
NVIDIA V100 32GB GPU.
[Table]
NVP requires more decoding time than prior works, mainly due to (a) three learnable keyframes along each spatio-temporal axis (three times more access than the single grids of Instant-ngp [34]) and (b) the modulation architecture while evaluating the corresponding RGB values.
However, note that the computation through each keyframe does not depend on each other; in this respect, the decoding time of NVP can be sped up significantly with a parallel design and implementation, e.g ., following the implementation details from Instant-ngp.
Moreover, while (b) exhibits considerable improvements in encoding complicated videos, we also demonstrate that our method still shows reasonable video encoding without modulations (both are shown in Section 4.3 of the main text); hence, one can control the trade-off between the decoding time and encoding quality by designating the multilayer perceptron (MLP) size and the whether the modulation is applied (or not).
inference.
As the official implementation of Instant-ngp 8 states that the decoding time highly deviates if one uses non-C++/CUDA frameworks, e.g ., Pytorch, we argue the gap of decoding time in Table 7 mainly stems from such different implementation details, and it can be remarkably mitigated by following their implementation.
E More description and visualization of latent keyframes
Our keyframes aim to learn meaningful contents that are shared across a given video along an axis.
For instance, the temporal axis ( U θ xy ) learns common contents in every timeframe, such as background and the watermark in the video, which is invariant to timesteps.
The meanings of the other two keyframes ( U θ xt and U θ xt ) may not be straightforward from their visualizations, as the shared contents across the other two directions in raw RGB space is often ambiguous.
However, we note that we are learning the common contents in “latent space”; although they do not seem straightforward, they indeed play a crucial role in promoting the succinct parametrization of a given video (see Figure 8).
[Figure]
[Figure]
F Additional experiments on different datasets F.1 Experiment on Big Buck Bunny
Following the experimental setup in NeRV [5], we provide additional experimental results in the Big
Buck Bunny video for a more intuitive comparison to prior work
. F.2 Experiment with new videos
We also provide additional experimental results on more complex videos.
In particular, we consider the following three new videos, where all of these videos are collected under the CC0 license:
[Table]
• (Street) A timelapse video of a London street.
People, cars, and buses are moving at different speeds as the traffic signal changes.
[Table]
• (City)
A timelapse video of the city at night.
A lot of cars are moving speedily.• (Surfing)
A video of a man surfing in the ocean.
Huge ocean waves are changing dramatically and fast.
[Table]
9 https://pixabay.com/videos/id-28693/https://pixabay.com/videos/id-19627/https://pixabay.com/videos/id-110734/
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
H Comparison to learning-based video compression
In Figure 28, we compare compressed videos from NVP with well-known video codecs (H.264 [51],
HEVC [43]) and state-of-the-art learning-based video compression methods (DVC [28], STAT-SSFSP [56], HLVC [55], Scale-space [1], Liu et al.
[25], and NeRV [5]) on UVG-HD.
We train NVP for every single video separately, where we use the same architecture and hyperparameters for encoding and compressing all videos in UVG-HD.
Since our primary focus is not on video compression, there exists a gap between the existing compression methods and NVP.
However, we strongly believe that there are numerous future directions to engage NVP for video compression, such as investigating per-video hyperparameter search strategy or designing better CNR architecture more specialized for compression.
[Figure]