Scalable Neural Video Representations with Learnable Positional Features
Abstract
Succinct representation of complex signals using coordinate-based neural representations (CNRs) has seen great progress, and several recent efforts focus onextending them for handling videos.
Here, the main challenge is how to (a) alleviatea compute-inefficiency in training CNRs to (b) achieve high-quality video encodingwhile (c) maintaining the parameter-efficiency.
To meet all requirements (a), (b),and (c) simultaneously, we propose neural video representations with learnable positional features (NVP), a novel CNR by introducing ªlearnable positional featuresºthat effectively amortize a video as latent codes.
Specifically, we first present a
CNR architecture based on designing 2D latent keyframes to learn the commonvideo contents across each spatio-temporal axis, which dramatically improves theall of those three requirements.
Then, we propose to utilize existing powerful imageand video codecs as a compute-/memory-efficient compression procedure of latentcodes.
We demonstrate the superiority of NVP on the popular UVG benchmark;compared with prior arts, NVP not only trains 2 times faster (less than 5 minutes)but also exceeds their encoding quality as 34.00 → 34.43 (measured with the PSNRmetric), even using > 8 times fewer parameters.
We also show intriguing propertiesof NVP, e.g ., video inpainting, video frame interpolation, etc.
Introduction
Recent advances in coordinate-based neural representations (CNRs) [8, 12, 16, 37, 43] have showngreat promise in the field as a new paradigm for representing complex signals, including gigapixelimages [26, 31], audios [37], 3D scenes [27, 30, 32] and even large city-scale street views [44].
Instead of storing signal outputs as a coordinate grid ( e.g ., image pixels), whose memory requirementscales unfavorably in terms of resolution and dimension, CNRs represent each signal as a compactlyparameterized, continuous neural network; they interpret a signal as a coordinate-to-value functionand train a neural network to approximate this mapping.
CNRs enjoy numerous appealing properties,including data compression [10, 11, 54], super-resolution [5], novel view synthesis [17, 21, 30, 51],and generative modeling of complex, high-dimensional data [13, 23, 38, 39, 52, 55], while beingparameter-efficient interpretation of a given signal in various scenarios.
In particular, several works have attempted to exploit CNRs to interpret video signals [4, 22, 39, 52]by learning a neural network f : R 3 → R 3 with f ( x, y, t ) =
( r, g, b ) and exhibited their potentialas a succinct representation of videos, as well as providing numerous applications including videogenerative modeling [39, 52], video compression [4, 54], and video super-resolution [6].
Theyobserve that conventional CNR architectures [37, 43] often fail to encode large-scale videos due totheir complex temporal dynamics accompanied by large spatial variations, but the problem can be remarkably mitigated by designing CNR architectures specialized for videos.
For instance, Chen
[Figure]
et al.
[4] proposes a CNR structure that focuses on the continuous modeling of the video signal onlyalong the temporal dimension, allowing for more radical variations along spatial axes; it exhibits acomparable encoding quality to existing powerful video codecs ( e.g ., H.264 [48], HEVC [40]) whileenjoying lots of intriguing properties ( e.g ., denoising and video frame interpolation).
However, CNRs suffer from a severe compute-inefficiency, 2 limiting their scalability to encode realworld, large-scale videos despite their advantages.
To alleviate this issue, several works [24, 31, 34]have proposed new CNR architectures by separating a CNR f into two parts; f
= h
φ ◦ g θ for acoordinate-to-latent mapping g θ ( x, y, t ) =
z and a latent-to-RGB mapping h φ ( z ) =
( r, g, b ) .
Theyconstruct g θ as an embedding function defined with latent grids U θ in which the shape resembles thegrid interpretation of a given signal ( e.g ., a 2D array of C -dimensional latent codes U θ ∈ R
H
× W × Cfor image pixels) rather than as a neural network.
These approaches have shown a promise incompute-efficiency due to the strong locality induced by a grid structure of U θ ; however, they resultin another problem: these architectures severely sacrifice the parameter-efficiency since the parametersize of θ can be very large, growing proportionally to both the input coordinate dimension and thesignal resolution.
In this paper, we focus on developing video CNRs that are the best of both worlds:achieving high-quality encoding and the compute-/parameter-efficiency simultaneously.
Contribution.
We introduce neural video representations with learnable positional features (NVP),a novel CNR for videos.
NVP avoids requiring a single giant full-dimensional 3D array in g θ bypresenting learnable positional features that effectively amortize a given video as ª2D and 3Dº latentgrids with succinct parameters.
Specifically, we decompose the coordinate-to-latent mapping as
[Equation]
where we present two types of latent grids for constructing these mappings (see Figure 2).
◦
Latent keyframes : We
first
design ªimage-likeº
2D latent
[Equation]
(respectively) that learn the representative video contents across each spatio-temporal axis and
dramatically improve the parameter efficiency of NVP.
◦ Sparse positional features : We then introduce a ªvideo-likeº 3D latent grid U θ xyt for g θ xyt , whose
size is much smaller than the original video pixels, but effectively encodes video details locally.
Moreover, we propose a compute-/memory-efficient compression
procedure
to further reduce
the
parameter θ
by incorporating existing image and video codecs, e.g
, JPEG [46] for images and
HEVC [40]
for videos
, respectively.
In particular, we treat 2D and 3D latent grids like the image or2 Measured with
a
single NVIDIA V100 32GB GPU, NeRV [4] takes at least 15 GPU hours to encode
a
single video of with for the quality.
[Figure]
The latent vector is passed through a neural network video pixels and utilize powerful compression pipelines of them.
Our compression scheme does notrequire any re-training of trained parameters, which significantly increases compute-efficiency thanprior approaches to compressing CNR parameters but remarkably maintains the encoding quality.
We also remark that such a compression approach is not applicable to the hashing-based latent grid ofthe prior method [31], while ours is ªcollision-freeº and maintains the video- or image-like structures.
Finally, for the choice of h φ , we suggest using a modulated network (with respect to the temporalcoordinate) to improve the encoding quality of videos that contain dynamic motions.
We verify the effectiveness of our method on the popular UVG benchmark [29].
In particular, NVPachieves the peak signal-to-noise ratio (PSNR; higher is better) metric of 34.43 in 5 minutes (witha single NVIDIA V100 32GB GPU): it is achieved > 2 times faster, even with using > 8 timesfewer parameters than the state-of-the-art on compute-efficiency that reaches 34.00 in 10 minutes.
Moreover, compared with prior arts on encoding quality, our method improves the learned perceptualimage patch similarity (LPIPS [53]; lower is better) as 0.163 → 0.131 ( + 19.6%) with a similarnumber of parameters while requiring ∼ 20% less training time.
We also show numerous compellingproperties of NVP, e.g .,
video inpainting, video frame interpolation, super-resolution, compression,and consistent frame-wise encoding results without deviating quality.
Related work
Coordinate-based neural representations.
Coordinate-based neural representations (CNRs), alsoknown as implicit neural representations or neural fields, have emerged as a new paradigm forrepresenting complex, continuous signals.
They propose to encode signals through a neural network,typically a multilayer perceptron (MLP) combined with high-frequency sinusoidal activations [37] or
Gaussian activations [8].
Prior works have focused on utilizing neural fields on various complicateddata, e.g ., gigapixel images [26, 31], 2D videos [4, 39, 52], 3D static scenes, [27, 30, 32], and 3Ddynamics scenes [22, 36, 49].
In particular, most approaches have focused on constructing CNRarchitectures for encoding 3D scenes [2, 14, 44] and exhibited how employing a specialized priorknowledge for a given signal domain in the architecture can remarkably boost the encoding quality.
Despite these successes, extending CNRs for videos is yet under-explored.
In this paper, we aim tomove toward developing a CNR for videos by exploiting their unique temporal properties.
Hybrid CNRs.
Rather than solely designing a neural network of coordinate-to-RGB mapping, hybrid
CNRs incorporate learnable latent codes that follow a grid structure, e.g ., image CNRs combined with2D latent spatial grids [5, 28], 3D scene (or shape) CNRs with latent cubic grids [3, 7, 18, 24, 26, 34].
Specifically, they compute a latent vector using the grid-structured latent code and pass it through aneural network to compute the signal output at a given input coordinate.
Such approaches have shownsignificant efficiencies in training time and the encoding quality due to the powerful locality inducedby grid-shaped latent codes.
However, the number of parameters required for latent grid-basedrepresentations easily grows proportionally to the input coordinate dimension or data resolution,limiting the scalability of hybrid CNRs.
Remarkably, some of the recent approaches have exhibitedthis inefficiency can be significantly mitigated by considering multi-level (or progressive) structuresfor latent grids [24, 26, 41, 42].
While prior works primarily focus on encoding images or 3D scenes,we aim to design parameter-efficient hybrid CNRs for videos.
NVP: Neural video representations with learnable positional features
We first formulate our problem setup as follows.
Given a video signal v :=
( f 1 , f 2 , . . .
, f T ) consistingof T video frames, the goal is to find a compact neural representation f w with parameters w , fromwhich the original video v can be reconstructed with high quality.
Here, the quality can be definedusing various distortion metrics, e.g ., peak signal-to-noise ratio (PSNR) [15] and LPIPS [53], forevaluating a reconstruction quality and a perceptual similarity, respectively.
To achieve this goal, we take an approach based on coordinate-based neural representations (CNRs)Ða paradigm where each datum ( e.g ., video) is parameterized as a neural network of a coordinatemapping.
In particular, we aim to represent the given video using a neural network f w : R 3 → R 3 ,which maps the space-time coordinates ( x, y, t ) of the video to corresponding RGB values ( r, g, b ) ,where f w is optimized with reconstruction objectives, e.g ., mean-squared error.
Such an approachhas significant potential, as CNRs have shown to encode other continuous, complex signals ( e.g ., 3Dscenes) [37, 43] compactly while enjoying lots of intriguing properties, e.g ., super-resolution [5] anddenoising [4].
However, CNRs have suffered from tremendous time costs for training, and even in thecase of videos, it is difficult to achieve the high-quality encodings if one utilizes conventional CNRarchitectures that overlook the complex spatio-temporal dynamics of videos [4].
Our contributionlies in resolving these issues by designing ªlearnable positional featuresº that succinctly encode avideo as latent codes with high quality and keeping their compute-/parameter-efficiency intact.
In the rest of this section, we provide a detailed description of each component in NVP.
In Section 3.1,we explain the architecture of NVP.
We then describe our compression procedure in Section 3.2.
3.1 Architecture
[Figure]
We design our video CNR f w as a composition of two functions with a parameterization w :=
( θ, ϕ ) :a coordinate-to-latent mapping g θ and a latent
-to-RGB mapping h φ .
Here, we decompose thecoordinate-to-latent-mapping as g θ =
g θ xy × g θ xt
×
g θ yt
× g θ xyt with g θ ( x, y, t ) =
( z xy , z xt , z yt , z xyt )(for θ :=
( θ xy , θ xt , θ yt , θ xyt ) ), where each g θ xy , g θ xt , g θ yt is formalized with image-like 2D latentspatial grids U θ xy , U θ xt , U θ yt (respectively) and g θ xyt is designed with a video-like sparse 3D latent
grid U θ xyt .
We then present the latent
-to-RGB mapping h φ ( z xy , z xy , z yt , z xyt ) =
( r, g, b ) to be amulti-layer perceptrion (MLP) modulated by another neural network.
To explain our architecture, weassume all the input coordinate ( x, y, t ) of g θ (and f w ) is in [0 , 1] 3 ⊂ R 3 without loss of generality.
Learnable latent keyframes.
At a high level, learnable latent keyframes U
θ xy , U θ xt , U θ yt are imagelike 2D latent grids learned to capture the common representative contents in a given video v acrosseach t -, y -, x -axis, respectively.
For a given input coordinate ( x, y, t ) , we compute latent vectors 136 z xy , z xt , z yt from U θ
xy , U θ xt , U θ yt individually.
We explain our keyframe only with U θ yt by letting 137 U :
= U yt for simplicity, but note that other keyframes U θ xy , U θ xt operate in the similar manner.
Formally, U is L 2D spatial grids of C -dimensional latent codes u ij , whose resolution is H l × W l :
U
[Equation]
=
[Equation]
U 1 , . . .
, U L )
[Equation]
for l = 1 , . . .
, L, u
[Equation]
for 1 ≤ i ≤ H l , 1 ≤ j ≤ W l .
Here, the keyframe follows an L -level multi-resolution structure, i.e ., for each level l , the height H land the width W l become different as H l =
⌊ γ l − 1 H 1 ⌋ and W l =
⌊ γ l − 1 W 1 ⌋ with fixed γ > 1 and
H 1 , W 1 > 0 , where ⌊·⌋ indicates the floor function of the input.
Since U =
U θ yt is shared over thex -axis, we compute the latent vector z yt :=
( z 1 yt , . .
z L yt ) by considering only the value of y and t ata given coordinate ( x, y, t ) .
Specifically, for l = 1 , . . .
, L , each z l yt is a linearly interpolated vector offour vectors in the spatial grid U l , where the indices of these vectors are chosen as the closest ones tothe relative position of the input coordinate ( y, t ) ∈ [0 , 1] 2 :
[Equation]
where lerp indicates a linear interpolation operation at the input coordinate between given vectors.
Note that we learn the keyframe as the latent codes, unlike conventional approaches that specify thekeyframe in a deterministic manner from T video frames ( f 1 , f 2 , . . .
, f T ) ; it encourages to capture the representative contents of the video over each spatio-temporal direction better.
We also remarkthat our architecture involves two keyframes U θ xt , U θ yt that are considered across spatial axes.
Considering these keyframes may not be beneficial in the RGB space as the spatial variation ofvideo pixels is often large.
However, in our approach, these keyframes are learned under a moreflexible, continuous latent space, and thus the representative frames can even be found in these spatialdirections in such a space while encoding the RGB outputs of the video accurately.
Finally, recall that U θ
xy , U θ xt , U θ yt consist of multi-resolution spatial grids, i.e ., the resolution ofspatial grids in each latent keyframe grows from coarse to fine.
Since natural scenes often includerepeating patterns in various scales, e.g ., a scene of flowers of different sizes, this multi-resolutionarchitecture promotes learning common patterns with a reduced memory and computation costs,which is also validated in Müller et al.
Sparse positional features.
Given a space-time coordinate ( x, y, t ) , we compute the latent vector 161 z xyt with U θ xyt that represents the local details of the video at this input position.
Here, U θ xyt :=( u ijk ) ∈
R H
× W × S × D is a 3D sparse grid of D -dimensional latent codes, i.e ., the 3D grid size
H × W × S is much smaller than the size of the video pixels of a 3D RGB grid:
U
[Equation]
u ijk
[Equation]
for 1 ≤ i ≤ H, 1 ≤ j ≤ W, 1 ≤ k ≤ S.
To evaluate the latent vector z xyt , we concatenate h × w × s latent codes in U θ xyt that their indicesare near the relative position of the input ( x, y, t ) : [Equation] −
[Equation]
) where h, w, s > 0 are given as hyperparameters.
The locality of U θ xyt as 3D latent codes dramatically alleviates the compute-inefficiency in fittingvideos, in contrast to conventional CNRs where the entire parameters are shared (as a neural network)for arbitrary input coordinates ( x, y, t ) and thus require a significant training cost.
Moreover, recallthat we construct U θ xyt as a sparse 3D grid of latent codes; remarkably, U θ xyt efficiently captures thevideo details even if the size is smaller than the number of video pixels since the common contents ofa given video are effectively encoded with the latent keyframes U
θ xy , U θ xt , U θ yt .
Modulated implicit function.
With the latent vector z :=
[ z xy , z xt , z yt , z xyt ] evaluated from g θ ,a naïve design choice of the latent-to-RGB mapping h φ is to utilize a MLP that maps z to thecorresponding RGB output ( r, g, b ) .
However, if a video contains temporally dynamic motions, wefound such a simple MLP architecture occasionally lacks an expressive power and can be difficult tocapture the complex dynamics of the given video, even with the large network size of h φ .
To circumvent this issue, we design h φ to be a K -layer MLP (coined as a synthesizer network)modulated by another modulator network [28]: where the latent vector z and the time coordinate tare passed through the modulator and the synthesizer, respectively.
Here, the modulator networkutilizes piecewise linear activations ( e.g ., ReLU), where the synthesizer uses sinusoidal activations.
Specifically, an RGB output ( r, g, b ) of the latent vector z (from ( x, y, t )) is computed as follows:
[Equation]
where A k , b k are weights and biases of k -th layer of the synthesizer, z k is k -th hidden feature of themodulator, and ⊙ denotes an element-wise product.
It helps to achieve high-quality encoding rapidlyin early training iterations and often at the convergence than a naive MLP (see Table 3 for details).
3.2 Compression procedure
Recall that we aim to find ªcompactº video CNRs; several works have focused on reducing the numberof coordinate-based neural representations parameters (or bits) after training while maintaining theirperformance.
In particular, they have relied on exploiting existing well-known techniques for neuralnetwork compression, e.g ., exploiting magnitude pruning [4, 20] or quantization [4, 54], and exhibitedconsiderable results.
However, these approaches mainly involve a re-training of CNR parameters,requiring severe computation costs, and thus are not suitable for practical scenarios.
indicates applying the method without the corresponding compression scheme.
We report the BPP values of NeRV without compressing parameters if the encoding time is ≤ 1 hour since the NeRV’s compression requires a longer time.
On the other hand, the compression procedure of NVP only takes less than 1 minute, but for a fair comparison, we do not apply it to NVP as well whenever NeRV is not compressed.
[Table]
Instead
[Table]
we propose a compression pipeline for NVP
[Table]
which does not require re-training, yet significantly reduces the number of bits while preserving the video quality.
Our main idea is to incorporateexisting image and video codecs that have shown their promises for the compression of given pixels.
In particular, we focus on compressing keyframes U θ
xy , U θ xt , U θ yt , and sparse positional features 197 U θ xyt , as the parameter size of the modulated implicit function h φ is neglectable compared with those.
Specifically, we quantize U θ xyt
and U
θ xy , U θ xt , U θ yt as 3D/2D grids of 8-bit latent codes and regardthem as video and image pixel grids, where the number of the channel becomes the dimension oflatent codes.
Based on these interpretations, we compress these latent codes by utilizing existingvideo and image codecs, e.g ., HEVC [40] for videos and JPEG [46] for images.
Intriguingly, wefound this procedure can significantly reduce the parameters while notably maintaining the videoquality without any fine-tuning of the latent-to-RGB mapping h φ (See Section 4.3).
Experiments
We verify the effectiveness of our framework on UVG-HD [29], a representative benchmark forevaluating video encodings.
Experimental results demonstrate that our neural video representationswith learnable positional features (NVP) simultaneously improves the overall performance by (a)alleviating a compute-inefficiency in training, (b) achieving high-quality video encoding, and (c)maintaining parameter-efficiency.
We also show applications of our NVP, including video inpaintingand spatio-temporal interpolation.
Finally, we conduct ablation studies to validate each component.
Evaluation.
We follow similar setups on prior work [4] proposing coordinate-based neural representations (CNRs) for videos.
All reported numbers are averaged over 7 videos in UVG-HD: Beauty,
Bosphorus, HoneyBee, Jockey, ReadySetGo, ShakeNDry, and Yachtride, along with the standarddeviations unless otherwise specified.
We also use the Big Buck Bunny video, which is used in
NeRV [4].
For quantitative evaluation, we use the following metrics: peak signal-to-noise ratio(PSNR) for reconstruction quality, LPIPS [53], FLIP [1], and SSIM [47] for a perceptual similarity,where all metrics are evaluated in a frame-wise manner and averaged over the whole video.
Weevaluate these metrics on different bits-per-pixel (BPP; lower is better) to evaluate the parameterefficiency; see Appendix A.1 for more description of the evaluation.
[Figure]
Implementation details.
All main experiments, including baselines, are processed with a single
GPU (NVIDIA V100 32GB) and 28 instances from a virtual CPU (Intel ® Xeon ® Platinum
CPU @ 2.70GHz), where it takes at most ∼ 17 hours to run our method and ∼ 3 days to run otherbaselines.
Moreover, we denote NVP-S and NVP-L as the model with latent code dimensions ofsparse positional features to be 2 and 4, respectively; see Appendix A.2 for more details.
Baselines.
We compare our method with SIREN [37], FFN [43], which are well-known signalagnostic CNR architectures, Instant-ngp [31] for state-of-the-art CNRs on compute-efficiency, and
NeRV [4], which is a CNR specialized for videos.
For all of the baseline methods, we follow theirreported experimental setups.
In particular, for NeRV [4], we use two configurations provided inthe official implementation: NeRV-S and NeRV-L for a small and a large model, respectively.
See
Appendix B for a detailed description of baseline methods.
4.1 Main results
Figure 1, Figure 3, and Table 1 summarize quantitative and qualitative results of NVP and baselines.
Remarkably, as shown in Figure 1, NVP can accurately capture dynamic motions and high-frequencydetails of a video e.g ., the legs of a running horse, while prior state-of-the-art CNRs fail to achieveand shows a blurry artifact.
We also emphasize such a high-quality encoding is accomplished in ªlessthan 1 minuteº, which supports the superior compute-efficiency of NVP in training.
Moreover, Table 1 verifies the effectiveness of NVP with quantitative evaluations, which outperformsall other baselines at varying encoding times from ∼ 1 minute to > 40 hours.
In particular, NVPsignificantly improves LPIPS compared with previous methods, demonstrating how the encodedvideos are perceptually similar to ground-truth videos.
Such a result is also confirmed in Figure 3;
NeRV shows a distortion that some pixels significantly deviate from the ground-truth outputs, whileour method does not suffer from such artifacts.
We also note that the variance of NVP is relativelysmall compared with other baselines, which shows the robustness of videos with diverse scenes andmotion.
See Appendix C and D for video-wise results and discussion on decoding time, respectively.
4.2 Applications
In this section, we provide several applications of our method, NVP, as video CNRs.
For better,playable illustrations and qualitative results, please refer to our project page.
Video inpainting.
Intriguingly, our method has the capability of video inpainting, i.e ., the desiredmoving object in the video can be naturally removed by capturing shared video contents with learnablekeyframes.
Figure 4 visualizes the illustration of inpainting results from NVP; as shown in this figure,one can see the car is removed where such parts are filled with a natural background.
[Figure] [Figure]
[Figure]
Video frame interpolation.
Since video CNRs approximate videos as temporally continuous signals,they should interpolate among two different frames at an arbitrary time even if such a frame doesnot exist in the training dataset.
To validate the interpolation capability of NVP, we provide thequantitative results on Big Buck Bunny sequences; our method shows better interpolation resultsmeasured with various metrics
[Figure]
Video super-
resolution.
We remark that NVP encodes a given video as a spatio-temporally continuous signal.
Thus, our method can interpolate the frames across spatial directions, i.e ., frame-wisesuper-resolution.
Figure 5 exhibits how well NVP smoothly interpolates in video frames acrossspatial directions while preserving the sharp edges, compared with naïve upsampling methods.
Video compression.
Recall that one of the major advantages of CNRs is their succinct encoding torepresent a given signal; one may consider utilizing CNRs for video compression [4, 10, 11].
Toverify the potential of our method on video compression, we compare the quality of compressedvideos from NVP with the ones from the current state-of-the-art video codecs.
As shown in Figure 6,compressed videos from NVP show the comparable reconstruction quality (measured with PSNRmetrics) while outperforming perceptual similarity (measured with LPIPS metrics).
4.3 Ablation studies
Effect of architecture components.
To verify the effectiveness of each component, we train ourmodel with the all videos in UVG-HD by removing each component while maintaining the totalnumber of parameters, then measure PSNR metrics from these models.
Table 3 summarizes theeffect of three different architecture components.
Without any of the components consisting of ourpositional features, the reconstruction quality gets dramatically worse, which validates how NVPsuccinctly encodes a given video as latent codes.
We also note that the modulation not only improvesthe final encoding quality but also notably achieves high-quality encoding in its early training epochs.
Analysis of non-temporal keyframes.
Recall that we additionally design keyframes across spatialdirections, unlike conventional approaches that designate the keyframe over only the temporaldirection.
To validate the effect of such keyframes, we compare NVP with (a) NVP without spatialkeyframes and (b) NVP without all of the keyframes in Figure 8, where the number of parametersis equally set for a fair comparison.
While utilizing latent learnable keyframes only over temporaldirection is already fairly effective for high-quality encoding, one can observe the consideration ofkeyframes across other directions provides a further improvement.
[Figure]
[Figure]
[Figure]
Compression procedure.
To validate the effectiveness of our compression scheme, we comparethe encoding quality of (a) NVP with our compression scheme and (b) NVP compressed throughmagnitude-based pruning.
Figure 7 shows the proposed compression pipeline outperforms conventional magnitude-based pruning under various BPP.
We also remark that our compression methoddoes not require re-training and is thus much more time-efficient.
Consistent frame-wise encoding quality.
Existing keyframe-based compression approaches oftensuffer from inconsistent encoding quality: the frame-wise quality of the compressed video highlydepends on whether it is the designated keyframe or not.
In contrast, NVP learns the keyframes anddoes not have this problem.
Figure 9 validates the result: our method exhibits consistent encodingperformance while conventional popular video codecs show several peaks that the reconstructionquality (measured with PSNR metrics) highly deviates from others.
Discussion and conclusion
We proposed NVP, a new coordinate-based neural representation (CNR) to encode videos as succinctlatent codes.
Our main idea is to decompose a video into ªimage-likeº and ºvideo-likeº structures tolearn a coordinate-to-latent mapping efficiently.
Extensive experiments have verified the effectivenessof NVP on all of parameter-/compute-efficiency and the encoding quality.
We hope our method willfacilitate various future research directions in the CNR area.
Limitations and future works.
Each video contains different scenes and motions so that it can beeither static or dynamic, yet we utilize the same hyperparameters and architectures for encoding anyvideo.
Although such a video-agnostic design is fairly effective and outperforms prior works, webelieve the video-wise consideration of the architecture and hyperparameter can remarkably boostthe performance further.
Moreover, we have shown the potential of utilizing powerful image andvideo codes for compressing latent codes in NVP; extending such codecs to be specialized for thecompression of latent codes should be an interesting direction.
Negative social impacts.
A side effect of CNRs is their potential unexpected behavior on encoding;they may cause undesirable artifacts in representing the given signal but are challenging to predictdue to the under-explored behavior of training CNRs.
Furthermore, in the case of representing videos,the encoded videos may suffer from severe distortions and conceivably cause ethical problems.
Inthis respect, such behaviors should be extensively and carefully investigated and mitigated to exploit
CNRs as the standard for encoding videos in real-world situations.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section 4• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.•
Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes] See Section 5(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection 5.(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A](b)
Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Section
and supplementary material.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] See Section 4 and supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Section 4.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 4.
(b) Did you mention the license of the assets?
[Yes] See Section 4.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
See the supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] See the supplementary material.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]