Original: DySR : A DAPTIVE S UPER -R ESOLUTION VIA A LGORITHM AND S YSTEM C O  DESIGN
Revised: DySR : A DAPTIVE S UPER -R ESOLUTION VIA A LGORITHM AND S YSTEM C O - DESIGN

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Super resolution (SR) is a promising approach for improving the quality of low resolution streaming services on mobile devices.
Revised: Super resolution (SR) is a promising approach for improving the quality of low resolution streaming services on mobile devices.

Original: On mobile devices, the available computing and memory resources change dynamically depending on other running applications.
Revised: On mobile devices, the available computing and memory resources change dynamically depending on other running applications.

Original: Due to the high computation and memory demands of SR models, it is essential to adapt the model according to available resources to harvest the best possible model performance while maintaining quality of service (QoS), such as meeting a minimum frame rate and avoiding interruptions.
Revised: Due to the high computation and memory demands of SR models, it is essential to adapt the model according to available resources to harvest the best possible model performance while maintaining quality of service (QoS), such as meeting a minimum frame rate and avoiding interruptions.

Original: Nevertheless, there is no SR model or machine learning system that supports adaptive
Revised: Nevertheless, there is no SR model or machine learning system that supports adaptive

Original: SR, and enabling adaptive SR model on mobile devices is challenging because adapting model can cause significant frame rate drop or even service interruption.
Revised: SR, and enabling adaptive SR model on mobile devices is challenging because adapting model can cause significant frame rate drop or even service interruption.

Original: To address this challenge, we take an algorithm and system co-design approach and propose a Dy namic S uper R esolution framework called DySR that maintains QoS while maximizing the model performance.
Revised: To address this challenge, we take an algorithm and system co-design approach and propose a Dy namic S uper R esolution framework called DySR that maintains QoS while maximizing the model performance.

Original: During the training stage, DySR employs an adaptation-aware one-shot Neural Architecture Search to produce sub-graphs that share kernel operation weights for low model adaptation overhead while striking a balance between performance and frame rate.
Revised: During the training stage, DySR employs an adaptation-aware one-shot Neural Architecture Search to produce sub-graphs that share kernel operation weights for low model adaptation overhead while striking a balance between performance and frame rate.

Original: During the inference stage, an incremental model adaptation method is developed for further reducing the model adaptation overhead.
Revised: During the inference stage, an incremental model adaptation method is developed for further reducing the model adaptation overhead.

Original: We evaluate on a diverse set of hardware and datasets to show that DySR can generate models close to the Pareto frontier while maintaining a steady frame rate throughput with a memory footprint of around 40% less compared to the assembled baseline methods .
Revised: We evaluate on a diverse set of hardware and datasets to show that DySR can generate models close to the Pareto frontier while maintaining a steady frame rate throughput with a memory footprint of around 40% less compared to the assembled baseline methods.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep super-resolution (SR) has been widely used in applications such as medical imaging (Li et al.
Revised: Deep super-resolution (SR) has been widely used in applications such as medical imaging (Li et al.

Original: (2021)), satellite imaging (Shermeyer & Van Etten (2019)), and image restoration (Qiu et al.
Revised: (2021)), satellite imaging (Shermeyer & Van Etten (2019)), and image restoration (Qiu et al.

Original: SR has attracted lots of attentions in recent years due to the surging demands in mobile services such as video conference, content sharing, and video streaming, where it helps provide high-resolution visual content from low-resolution data source (Zhang et al.
Revised: SR has attracted lots of attentions in recent years due to the surging demands in mobile services such as video conference, content sharing, and video streaming, where it helps provide high-resolution visual content from low-resolution data source (Zhang et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: SR models are resource demanding (Li et al.
Revised: SR models are resource demanding (Li et al.

Original: ; Lu & Hu (2022)) and need to meet Quality of
Revised: ; Lu & Hu (2022)) and need to meet Quality of

Original: Service (QoS) standards to provide good user experience in visual services.
Revised: Service (QoS) standards to provide good user experience in visual services.

Original: Examples of QoS including meeting a minimum frame rate and avoiding interruptions so that users perceive smooth motions.
Revised: Examples of QoS including meeting a minimum frame rate and avoiding interruptions so that users perceive smooth motions.

Original: This, however, is challenging for mobile devices where computing and memory resources are limited and the availability of which also depends on other running applications.
Revised: This, however, is challenging for mobile devices where computing and memory resources are limited and the availability of which also depends on other running applications.

Original: To meet QoS for different mobile devices, existing works develop models for specific devices (Liu et al.
Revised: To meet QoS for different mobile devices, existing works develop models for specific devices (Liu et al.

Original: (2021b);
Revised: (2021b);

Original: Lee et al.
Revised: Lee et al.

Original: ; Ayazoglu (2021)) or use Neural Architecture Search (NAS) (Chu et al.
Revised: ; Ayazoglu (2021)) or use Neural Architecture Search (NAS) (Chu et al.

Original: ; Guo et al.
Revised: ; Guo et al.

Original: ; Huang et al.
Revised: ; Huang et al.

Original: to generate multiple hardware-tailored models.
Revised: to generate multiple hardware-tailored models.

Original: However, none of these approaches considers the fluctuating resource environment of mobile devices and often leads to poor QoS. One potential way to achieve good QoS is to dynamically adapt the model based on available resources.
Revised: However, none of these approaches considers the fluctuating resource environment of mobile devices and often leads to poor QoS. One potential way to achieve good QoS is to dynamically adapt the model based on available resources.

Original: The challenges are two folds.
Revised: The challenges are two folds.

Original: First, how to design an adaptive model.
Revised: First, how to design an adaptive model.

Original: Second, how to enable model adaptation in a live inference system.
Revised: Second, how to enable model adaptation in a live inference system.

Original: To enable adaptive model, we employ NAS to generate a set of models with different sizes so that the most profitable model is used under each resource availability situation to ensure a steady frame rate while maximizing the model performance.
Revised: To enable adaptive model, we employ NAS to generate a set of models with different sizes so that the most profitable model is used under each resource availability situation to ensure a steady frame rate while maximizing the model performance.

Original: Unfortunately, none of existing machine learning frameworks supports live model adaptation.
Revised: Unfortunately, none of existing machine learning frameworks supports live model adaptation.

Original: To enable model adaptation in actual system, we explore two ideas.
Revised: To enable model adaptation in actual system, we explore two ideas.

Original: The first idea is to use an assemble method to keep all models loaded in the system at all times to avoid model switching overhead.
Revised: The first idea is to use an assemble method to keep all models loaded in the system at all times to avoid model switching overhead.

Original: However, such a method results in a significantly larger memory footprint, which is unsuitable for mobile devices.
Revised: However, such a method results in a significantly larger memory footprint, which is unsuitable for mobile devices.

Original: The second idea is to load a single model at a time, but the the model switching overhead is high as it interrupts the steaming for 1-3 seconds each time it switches models, leading to even worse
Revised: The second idea is to load a single model at a time, but the the model switching overhead is high as it interrupts the steaming for 1-3 seconds each time it switches models, leading to even worse

Original: QoS.
Revised: QoS.

Original: To achieve low resource consumption while minimizing the model switching overhead, we propose
Revised: To achieve low resource consumption while minimizing the model switching overhead, we propose

Original: DySR 1 , an algorithm and system co-design approach for adaptive SR.
Revised: DySR 1 , an algorithm and system co-design approach for adaptive SR.

Original: To keep a small memory footprint and minimize adaptation overhead, DySR employs an adaptation-aware one-shot NAS approach, where a large meta-graph is trained in one-shot such that sub-graphs share kernel operation weights while exploring the best tradeoffs between performance and frames-per-second (FPS).
Revised: To keep a small memory footprint and minimize adaptation overhead, DySR employs an adaptation-aware one-shot NAS approach, where a large meta-graph is trained in one-shot such that sub-graphs share kernel operation weights while exploring the best tradeoffs between performance and frames-per-second (FPS).

Original: During inference, the meta-graph is fully loaded in the memory and operations are dynamically adapted according to the real-time resource availability in an incremental manner, i.e., only affected operations are swapped or rerouted.
Revised: During inference, the meta-graph is fully loaded in the memory and operations are dynamically adapted according to the real-time resource availability in an incremental manner, i.e., only affected operations are swapped or rerouted.

Original: Since we do not need to load new models from the hard disk, there is no data transfer overhead.
Revised: Since we do not need to load new models from the hard disk, there is no data transfer overhead.

Original: We evaluate DySR against baselines across a wide variety of hardware (from powerful GPUs to lowend mobile processors) using image and video SR datasets (e.g., Urban100 (Huang et al.
Revised: We evaluate DySR against baselines across a wide variety of hardware (from powerful GPUs to lowend mobile processors) using image and video SR datasets (e.g., Urban100 (Huang et al.

Original: and Vimeo90k (Xue et al.
Revised: and Vimeo90k (Xue et al.

Original: Results show that DySR can generate models close to the Pareto frontier of the performance vs. FPS tradeoffs while maintaining a steady frame rate throughput with low memory footprint (40% less compared to ensemble method).
Revised: Results show that DySR can generate models close to the Pareto frontier of the performance vs. FPS tradeoffs while maintaining a steady frame rate throughput with low memory footprint (40% less compared to ensemble method).

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: SR.
Revised: SR.

Original: (Dong et al.
Revised: (Dong et al.

Original: (2014)) is among the first works that employs deep learning models for superresolution.
Revised: (2014)) is among the first works that employs deep learning models for superresolution.

Original: Since then deeper and more complex models such as (Soh et al.
Revised: Since then deeper and more complex models such as (Soh et al.

Original: ; Nazeri et al.
Revised: ; Nazeri et al.

Original: (2019)) were proposed for better performance.
Revised: (2019)) were proposed for better performance.

Original: Generative Adversarial Networks (GANs) (Creswell et al.
Revised: Generative Adversarial Networks (GANs) (Creswell et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: Ahn et al.
Revised: Ahn et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: (2018a))
Revised: (2018a))

Original: and its variations (Prajapati et al.
Revised: and its variations (Prajapati et al.

Original: ; Guo et al.
Revised: ; Guo et al.

Original: Shahsavari et al.
Revised: Shahsavari et al.

Original: (2021)) have been shown to be highly effective in tackling this task.
Revised: (2021)) have been shown to be highly effective in tackling this task.

Original: Attention mechanisms were introduced to SR as well (Zhao et al.
Revised: Attention mechanisms were introduced to SR as well (Zhao et al.

Original: Mei et al.
Revised: Mei et al.

Original: ; Chen et al.
Revised: ; Chen et al.

Original: Methods such as network pruning, knowledge distillation, and quantization have been applied to reduce computational overhead of existing SR deep learning models (Jiang et al.
Revised: Methods such as network pruning, knowledge distillation, and quantization have been applied to reduce computational overhead of existing SR deep learning models (Jiang et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2021b)
Revised: (2021b)

Original: ; Hong et al.
Revised: ; Hong et al.

Original: ; Wang et al.
Revised: ; Wang et al.

Original: However, all the above efforts focus on building a single model for each hardware and do not consider the dynamic resource environment in mobile devices, and thus fall short in meeting streaming QoS for mobile devices.
Revised: However, all the above efforts focus on building a single model for each hardware and do not consider the dynamic resource environment in mobile devices, and thus fall short in meeting streaming QoS for mobile devices.

Original: NAS .
Revised: NAS .

Original: Earlier neural architecture search methods rely on Reinforcement Learning (RL) (Zoph &
Revised: Earlier neural architecture search methods rely on Reinforcement Learning (RL) (Zoph &

Original: Le (2016)) and evolutionary algorithms (Lu et al.
Revised: Le (2016)) and evolutionary algorithms (Lu et al.

Original: (2018); van Wyk & Bosman (2019)) for architecture engineering.
Revised: (2018); van Wyk & Bosman (2019)) for architecture engineering.

Original: However, these methods are extremely resource demanding and often require thousands of GPU hours.
Revised: However, these methods are extremely resource demanding and often require thousands of GPU hours.

Original: Later works such as (Wen et al.
Revised: Later works such as (Wen et al.

Original: (2020); Jin et al.
Revised: (2020); Jin et al.

Original: (2019)) introduce performance prediction, shared weight training, and proxy training to speed up the architecture engineering process.
Revised: (2019)) introduce performance prediction, shared weight training, and proxy training to speed up the architecture engineering process.

Original: DARTS (Liu et al.
Revised: DARTS (Liu et al.

Original: and its followups (Chen et al.
Revised: and its followups (Chen et al.

Original: ; Wu et al.
Revised: ; Wu et al.

Original: (2019)) adopt a differentiable architecture search paradigm.
Revised: (2019)) adopt a differentiable architecture search paradigm.

Original: The once-for-all work (Cai et al.
Revised: The once-for-all work (Cai et al.

Original: (2019)) proposes the idea of generating a single model for multiple hardware deployments though pruning and model swapping is needed for each deployment.
Revised: (2019)) proposes the idea of generating a single model for multiple hardware deployments though pruning and model swapping is needed for each deployment.

Original: One-shot NAS (Bender et al.
Revised: One-shot NAS (Bender et al.

Original: and its variations (Huang & Chu (2021); Zhang et al.
Revised: and its variations (Huang & Chu (2021); Zhang et al.

Original: (2021a)
Revised: (2021a)

Original: ; Zhao et al.
Revised: ; Zhao et al.

Original: (2021)) can generate models with few search iterations and have been explored for SR (Chu et al.
Revised: (2021)) can generate models with few search iterations and have been explored for SR (Chu et al.

Original: ; Guo et al.
Revised: ; Guo et al.

Original: Liu et al.
Revised: Liu et al.

Original: (2021a)
Revised: (2021a)

Original: ; Zhan et al.
Revised: ; Zhan et al.

Original: but existing works only focus on designing a single model and do not consider QoS for streaming on mobile devices.
Revised: but existing works only focus on designing a single model and do not consider QoS for streaming on mobile devices.

Original: In Section 5, we compare DySR with existing SR models.
Revised: In Section 5, we compare DySR with existing SR models.

Original: The results show that our model achieves Pareto optimal performance while meeting QoS.
Revised: The results show that our model achieves Pareto optimal performance while meeting QoS.

Original:  M OTIVATIONS AND C HALLENGES
Revised:  M OTIVATIONS AND C HALLENGES

Original: On mobile devices, SR tasks are often running along with other applications, e.g., users often watch streaming videos while browsing websites or doing online shopping; making video calls while playing games or taking notes.
Revised: On mobile devices, SR tasks are often running along with other applications, e.g., users often watch streaming videos while browsing websites or doing online shopping; making video calls while play- [Figure] models respectively due to being designed specifically for the corresponding hardware [Figure]

Revised: ing games or taking notes.

Original: Therefore, the available computing and memory resources for SR is constantly changing on the already resource constraint mobile devices.
Revised: Therefore, the available computing and memory resources for SR is constantly changing on the already resource constraint mobile devices.

Original: We demonstrate the impact of changing available resources on frame rate for static models running on real mobile devices.
Revised: We demonstrate the impact of changing available resources on frame rate for static models running on real mobile devices.

Original: We test a set of state-of-the-art SR models on the mobile devices they were targeted for and manually[Figure] limit the amount of resource available over time, i.e., by limiting GPU power and adding background loads to CPU.
Revised: We test a set of state-of-the-art SR models on the mobile devices they were targeted for and manually limit the amount of resource available over time, i.e., by limiting GPU power and adding background loads to CPU.

Original: We use a workload trace to show how the utilization changes over time, as shown in
Revised: We use a workload trace to show how the utilization changes over time, as shown in

Original: Figure 1.
Revised: Figure 1.

Original: We observe that reducing the processing power available for the models over time results in significant frames-per-second (FPS) drop.
Revised: We observe that reducing the processing power available for the models over time results in significant frames-per-second (FPS) drop.

Original: This drop at higher utilizations show that the QoS is impacted.
Revised: This drop at higher utilizations show that the QoS is impacted.

Original: Therefore, the state-of-the-art SR models do not perform as expected under practical circumstances, and to maintain QoS (i.e., minimum FPS), models need to be adapted according to the available resources.
Revised: Therefore, the state-of-the-art SR models do not perform as expected under practical circumstances, and to maintain QoS (i.e., minimum FPS), models need to be adapted according to the available resources.

Original: To fill this gap, however, is challenging.
Revised: To fill this gap, however, is challenging.

Original: First, we need to design a set of models that can achieve Pareto-optimal performance under different frame rates.
Revised: First, we need to design a set of models that can achieve Pareto-optimal performance under different frame rates.

Original: However, no existing works can generate such a set of models.
Revised: However, no existing works can generate such a set of models.

Original: Second, we need to adapt models within milliseconds in the real-time system to meet QoS requirements.
Revised: Second, we need to adapt models within milliseconds in the real-time system to meet QoS requirements.

Original: Unfortunately, no existing machine learning frameworks support real-time model adaptation.
Revised: Unfortunately, no existing machine learning frameworks support real-time model adaptation.

Original: One potential solution is to switch between a list of
Revised: One potential solution is to switch between a list of

Original: [Figure]
Revised: [Figure]

Revised: parameters in memory against the number of static models loaded as one assembled set.

Original: static models according to the available resources.
Revised: static models according to the available resources.

Original: We prototype this idea and perform experiments on different devices, as shown in Fig.
Revised: We prototype this idea and perform experiments on different devices, as shown in Fig.

Original: Here, we unload and reload static models as soon as the utilizations change at the intervals 5, 10, and 15 seconds, as shown in (a), (b), and (c) respectively.
Revised: Here, we unload and reload static models as soon as the utilizations change at the intervals 5, 10, and 15 seconds, as shown in (a), (b), and (c) respectively.

Original: Across all devices, we observe an interruption of service during each model swapping period.
Revised: Across all devices, we observe an interruption of service during each model swapping period.

Original: As pointed out in (Liao et al.
Revised: As pointed out in (Liao et al.

Original: , such interruption is the time it takes to read the new model from storage and initialize all the parameters in memory.
Revised: , such interruption is the time it takes to read the new model from storage and initialize all the parameters in memory.

Original: Pre-loading multiple static models as one is a potential solution to avoid such interruption.
Revised: Pre-loading multiple static models as one is a potential solution to avoid such interruption.

Original: However, the assembled model has a significantly higher memory consumption, as shown in Fig.
Revised: However, the assembled model has a significantly higher memory consumption, as shown in Fig.

Original: 2d, which is not practical for resource constraint mobile devices.
Revised: 2d, which is not practical for resource constraint mobile devices.

Original:  A LGORITHM AND S YSTEM C O - DESIGN : DySR
Revised:  A LGORITHM AND S YSTEM C O - DESIGN : DySR

Original: To address the aforementioned challenges, in this section, we adopt an algorithm and system codesign approach and propose DySR .
Revised: To address the aforementioned challenges, in this section, we adopt an algorithm and system codesign approach and propose DySR .

Original: DySR employs adaptive-aware one-shot NAS to create a set of models that achieve Pareto-optimal performance under different frame rate with minimum adaptation overhead during runtime.
Revised: DySR employs adaptive-aware one-shot NAS to create a set of models that achieve Pareto-optimal performance under different frame rate with minimum adaptation overhead during runtime.

Original: DySR also introduces an incremental adaptation approach to further reduce the model adaptation overhead to meet QoS.
Revised: DySR also introduces an incremental adaptation approach to further reduce the model adaptation overhead to meet QoS.

Original: 4. A DAPTATION - AWARE O NE - SHOT N EURAL A RCHITECTURE S EARCH
Revised: 4. A DAPTATION - AWARE O NE - SHOT N EURAL A RCHITECTURE S EARCH

Original: Neural Architecture Search (NAS) allows creating multiple models in an automated fashion, and thus we choose it for model generation.
Revised: Neural Architecture Search (NAS) allows creating multiple models in an automated fashion, and thus we choose it for model generation.

Original: However, for SR, existing NAS methods only target for a finding a single model with the highest performance while ignoring the frame rate and model adaptation overhand.
Revised: However, for SR, existing NAS methods only target for a finding a single model with the highest performance while ignoring the frame rate and model adaptation overhand.

Original: To provide QoS on mobile devices, we design an adaptive-aware one-shot
Revised: To provide QoS on mobile devices, we design an adaptive-aware one-shot

Original: NAS method for generating a set of models that achieves Pareto-optimal performance with frame rate constraint and low model adaptation overhead under different hardware devices and resource availability.
Revised: NAS method for generating a set of models that achieves Pareto-optimal performance with frame rate constraint and low model adaptation overhead under different hardware devices and resource availability.

Original: Following (Liu et al.
Revised: Following (Liu et al.

Original: ; Bender et al.
Revised: ; Bender et al.

Original: , we define the architecture search space as S , which is represented as a single Directed Acyclic Graph (DAG) containing all possible path and operation combinations, called the meta-graph.
Revised: , we define the architecture search space as S , which is represented as a single Directed Acyclic Graph (DAG) containing all possible path and operation combinations, called the meta-graph.

Original: The main idea here is to train a metagraph such that it contains sub-graph models with Pareto-optimal performance across a range of frame rates.
Revised: The main idea here is to train a metagraph’s sub-graph models with Pareto-optimal performance across a range of frame rates.

Original: The sub-graphs are all part of the same meta-graph, so thatduring inference the full meta-graph is deployed and the sub-graphs can be adapted in real time  to maintain QoS using an adaptation policy.
Revised: The subgraphs are all part of the same meta-graph, so that the full meta-graph is deployed and the sub-graphs can be adapted in real time during inference to maintain QoS using an adaptation policy.

Original: Adaptation-aware Sub-graph Selection .
Revised: Adaptation-aware Sub-graph Selection .

Original: The generation of adaptation-aware sub-graphs requires two unique criteria to be fulfilled during the NAS search phase to be effective.
Revised: The generation of adaptation-aware sub-graphs requires two unique criteria to be fulfilled during the NAS search phase to be effective.

Original: First, in order to reduce resource consumption, we need to reduce total memory of the meta-graph to be significantly less than an assembled model.
Revised: First, in order to reduce resource consumption, we need to reduce total memory of the meta-graph to be significantly less than an assembled model.

Original: Second, we need to minimize the sub-graph adaptation time to ensure uninterrupted service.
Revised: Second, we need to minimize the sub-graph adaptation time to ensure uninterrupted service.

Original: In order to keep memory consumption at a minimum, we design our meta-graph space to share operations between sub-graphs wherever possible (e.g. two sub-graphs requiring a 3x3x64 convolutional operation at the same layer will use the same meta-graph operation, and thus have intersecting execution paths).
Revised: In order to keep memory consumption at a minimum, we design our meta-graph space to share operations between sub-graphs wherever possible (e.g. two sub-graphs requiring a 3x3x64 convolutional operation at the same layer will use the same meta-graph operation, and thus have intersecting execution paths).

Original: To keep the switching time between models low, we need to reduce the number of execution path re-routings between sub-graphs to a minimum.
Revised: To keep the switching time between models low, we need to reduce the number of execution path re-routings between sub-graphs to a minimum.

Original: Luckily, we find that both of these properties are related based on the observation that the number of total operations meta-graph and the number of path re-routes are inversely proportional to the number of shared operations .
Revised: Luckily, we find that both of these properties are related based on the observation that the number of total operations meta-graph and the number of path re-routes are inversely proportional to the number of shared operations .

Original: In other words, the more operations that are shared among sub-graphs, the less number of redundant operations there are and so meta-graph memory size is reduced.
Revised: In other words, the more operations that are shared among sub-graphs, the less number of redundant operations there are and so meta-graph memory size is reduced.

Original: At the same time, more shared operations also means more common execution paths between sub-graphs and so less number of paths need to be changed while adapting.
Revised: At the same time, more shared operations also means more common execution paths between sub-graphs and so less number of paths need to be changed while adapting.

Original: We use this observation to develop the custom search policy to generate our adaptive models.
Revised: We use this observation to develop the custom search policy to generate our adaptive models.

Original: Thus the objective of our sub-graph selection policy can be formally defined as the constrained joint optimization problem -
Revised: Thus the objective of our sub-graph selection policy can be formally defined as the constrained joint optimization problem -

Original: [Equation]
Revised: [Equation]

Original: Here, P is the HR quality metric (i.e. PSNR or SSIM), Eval ( A ( α, D )) is the PSNR value of SR output of sub-graph α of meta-graph A on dataset D , µ is the number of operations α has shared with all other sub-graph A − α .
Revised: Here, P is the HR quality metric (i.e. PSNR or SSIM), Eval ( A ( α, D )) is the PSNR value of SR output of sub-graph α of meta-graph A on dataset D , µ is the number of operations α has shared with all other sub-graph A − α .

Original: fps is the frames-per-second and F ρmin is the minimum FPS allowed under available resource ρ .
Revised: fps is the frames-per-second and F ρmin is the minimum FPS allowed under available resource ρ .

Original: Note that ρ can change over time and can be generalizable to any type of computational resource even though in our case we mainly demonstrate using the available processing power.
Revised: Note that ρ can change over time and can be generalizable to any type of computational resource even though in our case we mainly demonstrate using the available processing power.

Original: The implementation details are described in detail below.
Revised: The implementation details are described in detail below.

Original: [Figure]
Revised: [Figure]

Original: Meta-Graph Design .
Revised: Meta-Graph Design .

Original: The first step in designing a NAS framework is the Meta-graph.
Revised: The first step in designing a NAS framework is the Meta-graph.

Original: This needs to be diverse enough to generate a wide range of efficient candidate architectures with competitive performance with state-of-the-art.
Revised: This needs to be diverse enough to generate a wide range of efficient candidate architectures with competitive performance with state-of-the-art.

Original: In our case, we focus on designing a meta-graph with a variety of inference FPS with a good set of PSNR values, rather than developing a single novel cell architecture that beats all state-of-the-art SR models.
Revised: In our case, we focus on designing a meta-graph with a variety of inference FPS with a good set of PSNR values, rather than developing a single novel cell architecture that beats all state-of-the-art SR models.

Original: We design our meta-graph as a cell-based architecture ( as shown in Figure 3a).
Revised: We design our meta-graph as a cell-based architecture (as shown in Figure 3a).

Original: The input layer and the last output layer are kept constant, and the architecture between them consists of layers of cells followed by an upsampling layer.
Revised: The input layer and the last output layer are kept constant, and the architecture between them consists of layers of cells followed by an upsampling layer.

Original: During the search phase, we sample paths within each searchable cell such that only one type of cell block is chosen at any one time.
Revised: During the search phase, we sample paths within each searchable cell such that only one type of cell block is chosen at any one time.

Original: Existing works havealready proposed a wide variety of cell blocks which were demonstrably efficient andalso have many operations in common, and so we use them for our search space.
Revised: Existing works have proposed a wide variety of cell blocks which were demonstrably efficient and have many operations in common, and so we use them for our search space.

Original: Specifically, we use the cells from CARN (Ahn et al.
Revised: Specifically, we use the cells from CARN (Ahn et al.

Original: , FALSR (Chu et al. (2021)),

Original: RCAB (Zhang et al.

Original: , AutoGAN (Fu et al.

Original: , WDSR (Yu et al. , ESRGAN (Wang et al. (2018a)) and MoreMNAS (Chu et al.
Revised: (2018)), FALSR ( and MoreMNAS (Chu et al.

Original: since they are well-known, efficient, selfcontained blocks.
Revised: since they are well-known, efficient, self-contained blocks.

Original: Another large benefit is that all these cell blocks share many of the same operation types, making it possible for them to share weights.
Revised: Another large benefit is that all these cell blocks share many of the same operation types, making it possible for them to share weights.

Original: The most commonly shared operations between them are the convolutional and dense layers.
Revised: The most commonly shared operations between them are the convolutional and dense layers.

Original: As such, we set their parameter search spaces as: • 
Revised: As such, we set their parameter search spaces as: • Convolutional

Original: Convolutional Filters - 1x1, 3x3, 5x5, • Convolutional Channels - 3, 16, 32, 64, • Dense - 64, 128, 256.
Revised: Filters - 1x1, 3x3, 5x5, • Convolutional Channels - 3, 16, 32, 64, • Dense - 64, 128, 256.

Original: Cell blocks for RCAB, AutoGAN, ESGRAN and FALSR also have attention and residual connections, which we can enable or disable as paths since we find that while their inclusion result in an increase in performance, it is not significant at the higher end models when considering the FPS increases.
Revised: Cell blocks for RCAB, AutoGAN, ESGRAN and FALSR also have attention and residual connections, which we can enable or disable as paths since we find that while their inclusion result in increased performance, it is not significant at the higher end models when considering the FPS increases.

Original: Layers .
Revised: Layers .

Original: The number of layers is one of the most important factors when determining the overall
Revised: The number of layers is one of the most important factors when determining the overall

Original: FPS and is therefore a very important search parameter.
Revised: FPS and is therefore a very important search parameter.

Original: Additionally, different layer types perform distinctly with different layer numbers (Wang et al.
Revised: Additionally, different layer types perform distinctly with different layer numbers (Wang et al.

Original: (2018a);
Revised: (2018a);

Original: Ahn et al.
Revised: Ahn et al.

Original: Chu et al.
Revised: Chu et al.

Original: Therefore, we set the range of our number of layers of cells from between 15 to 50 with intervals/groups of 5 for every single cell type.
Revised: Therefore, we set the range of our number of layers of cells from between 15 to 50 with intervals/groups of 5 for every single cell type.

Original: Lastly, for our upscaling layer, we keep the choices between four types – RCAN (Zhang et al.
Revised: Lastly, for our upscaling layer, we keep the choices between four types – RCAN (Zhang et al.

Original: , Deconvolutional (Dong et al.
Revised: , Deconvolutional (Dong et al.

Original: , Depth2Space (Bhardwaj et al.
Revised: , Depth2Space (Bhardwaj et al.

Original: and Multi-branch bicubic (Behjati et al.
Revised: and Multi-branch bicubic (Behjati et al.

Original: We have a total of number of possible cell types of 12 when including blocks with and without attention and residual networks and a total number of parameter combinations of 3 * 4 * 3 = 36, giving us a total number of 12 * 36 = 432 cell combinations.
Revised: We have a total of number of possible cell types of 12 when including blocks with and without attention and residual networks and a total number of parameter combinations of 3 * 4 * 3 = 36, giving us a total number of 12 * 36 = 432 cell combinations.

Original: With possible layer combinations of 3 to 10, we have a permutation formula 432 P 10 + 432 P 9 + 432 P 8 + ... 432 P 3 times 4 for the number of possible upscaling layers, which gives us a total of5 . 8 e 10 23 possible networks.
Revised: With possible layer combinations of 3 to 10, we have a permutation formula 432 P 10 + 432 P 9 + 432 P 8 + ... 432 P 3 times 4 for the number of possible upscaling layers, which gives us a total of

Original: Sampling Paths .
Revised: Sampling Paths .

Original: Given such a large number of possible graphs, we need an efficient method for traversal.
Revised: Given such a large number of possible graphs, we need an efficient method for traversal.

Original: Fortunately, our hard constraint of Eq.
Revised: Fortunately, our hard constraint of Eq.

Original: 1 for FPS can play an important part in reducing the space here.
Revised: 1 for FPS can play an important part in reducing the space here.

Original: We first start with setting the number of layers to the minimum possible.
Revised: We first start with setting the number of layers to the minimum possible.

Original: This is a tunable hyperparameter, and we find that below 15 convolutional layer blocks results in all sub-graphs with low PSNR value.
Revised: This is a tunable hyperparameter, and we find that below 15 convolutional layer blocks results in all sub-graphs with low PSNR value.

Original: Therefore, we set it to 15 and start random uniform sampling models without replacement.
Revised: Therefore, we set it to 15 and start random uniform sampling models without replacement.

Original: We profile them on the desired hardware to ensure the FPS is under the F max .
Revised: We profile them on the desired hardware to ensure the FPS is under the F max .

Original: If not, it is permanently discarded.
Revised: If not, it is permanently discarded.

Original: Once we have N l models, we move to the next layer limit and iterate until we have reached the last layer limit I (50 in our current setting) which gives us a total of N l ∗ I models.
Revised: Once we have N l models, we move to the next layer limit and iterate until we have reached the last layer limit I (50 in our current setting) which gives us a total of N l ∗ I models.

Original: Note that for our scenario, we need multiple models across a wide FPS spectrum.
Revised: Note that for our scenario, we need multiple models across a wide FPS spectrum.

Original: In other words, we need multiple choices of models for the same dataset deployable for different hardware and levels of resource availability.
Revised: In other words, we need multiple choices of models for the same dataset deployable for different hardware and levels of resource availability.

Original: Therefore, before starting the training we first determine suitable models by binning them.
Revised: Therefore, before starting the training we first determine suitable models by binning them.

Original: We select how many bins of FPS we need as B with each bin having a window size of 200ms, and we bin our profiled models within each bin.
Revised: We select how many bins of FPS we need as B with each bin having a window size of 200ms, and we bin our profiled models within each bin.

Original: We then train and evaluate each model within each bin for 5 epochs and evaluate them which gives us a sufficient estimate of final model performance.
Revised: We then train and evaluate each model within each bin for 5 epochs and evaluate them which gives us a sufficient estimate of final model performance.

Original: We now need to use an evaluation criteria to rank N b models per bin.
Revised: We now need to use an evaluation criteria to rank N b models per bin.

Original: For this method, we set a desired PSNR threshold for bin b decided by the Pareto frontier.
Revised: For this method, we set a desired PSNR threshold for bin b decided by the Pareto frontier.

Original: For example, from figure 8 we see that models on 1080Ti with 20 FPS needs to have at least 34 PSNR to be competitive, so we select that as the threshold.
Revised: For example, from figure 8 we see that models on 1080Ti with 20 FPS needs to have at least 34 PSNR to be competitive, so we select that as the threshold.

Original: Then we take all models above that threshold and rank them based on the number of intersecting operations with others µ from Eq.1 We iterate this process until all bins B have N b models.
Revised: Then we take all models above that threshold and rank them based on the number of intersecting operations with others µ from Eq.1 We iterate this process until all bins B have N b models.

Original: This way, we have a total of N b ∗ B models for exploration.
Revised: This way, we have a total of N b ∗ B models for exploration.

Original: The parameters N l , N b and B are very important for determining the search and model efficiency for our framework and their choice is discussed later in Section 5.
Revised: The parameters N l , N b and B are very important for determining the search and model efficiency for our framework and their choice is discussed later in Section 5.

Original: We throw out all other models which are not within the bins and redeploy the meta-graph with only the selected model operations.
Revised: We throw out all other models which are not within the bins and redeploy the meta-graph with only the selected model operations.

Original: This reduces the number of models in the search space drastically and reduces weight sharing which further alleviates the co-sharing issue.
Revised: This reduces the number of models in the search space drastically and reduces weight sharing which further alleviates the co-sharing issue.

Original: Training .
Revised: Training .

Original: After reducing the search space to N b ∗ B models, we then train them till convergence.
Revised: After reducing the search space to N b ∗ B models, we then train them till convergence.

Original: As mentioned above, we perform GAN training.
Revised: As mentioned above, we perform GAN training.

Original: Our predictor models are the searched models and for our discriminator we use the pre-trained VGG-16 model.
Revised: Our predictor models are the searched models and for our discriminator we use the pre-trained VGG-16 model.

Original: We use the Adam optimizer with loss function for the generator defined as -
Revised: We use the Adam optimizer with loss function for the generator defined as -

Original: [Equation]
Revised: [Equation]

Original: where x hr and x lr are the HR and LR images respectively, G is the generator network, l vgg is the
Revised: where x hr and x lr are the HR and LR images respectively, G is the generator network, l vgg is the

Original: VGG loss (Wang et al.
Revised: VGG loss (Wang et al.

Original: (2018b)
Revised: (2018b))

Original: and D is the discriminator.
Revised: and D is the discriminator.

Original: This loss is discussed in detail in (Wang et al.

Original: (2018b).

Original: 4. A DAPTIVE S UB -G RAPHS
Revised: 4. A DAPTIVE S UB -G RAPHS

Original: Standard implementations employ static graphs with fixed inputs and outputs pre-set between operations and are not changed during forward passes.
Revised: Standard implementations employ static graphs with fixed inputs and outputs pre-set between operations and are not changed during forward passes.

Original: As a result, traditionally designed meta-graphs cannot change operation sequences in real time.
Revised: As a result, traditionally designed meta-graphs cannot change operation sequences in real time.

Original: Instead, they have to reroute the batches among different model inputs for inference.
Revised: Instead, they have to reroute the batches among different model inputs for inference.

Original: For our case, we have a single large model in the form of a meta-graph.
Revised: For our case, we have a single large model in the form of a meta-graph.

Original: It contains sub-graph architectures which share operations.
Revised: It contains sub-graph architectures which share operations.

Original: In order to switch between sub-graphs, we can change the input-output paths for the operations.
Revised: In order to switch between sub-graphs, we can change the input-output paths for the operations.

Original: We take advantage of the non-static graph property to implement this.
Revised: We take advantage of the non-static graph property to implement this.

Original: We first define the sub-graphs as a network with the nodes representing the operations and the input-output variables as the end-point for paths between them.
Revised: We first define the sub-graphs as a network with the nodes representing the operations and the input-output variables as the end-point for paths between them.

Original: We store each of the meta-graph operations as node objects with its input and output variable references within.
Revised: We store each of the meta-graph operations as node objects with its input and output variable references within.

Original: During inference, we can change the input/output variables of one operation node to point to and from another node.
Revised: During inference, we can change the input/output variables of one operation node to point to and from another node.

Original: This effectively decides the execution path of the input batch, thereby defining the sub-graph architecture itself.
Revised: This effectively decides the execution path of the input batch, thereby defining the sub-graph architecture itself.

Original: Since we have all the operations with their trained weights loaded in memory, we can call every available operation with our desired input values at any time.
Revised: Since we have all the operations with their trained weights loaded in memory, we can call every available operation with our desired input values at any time.

Original: Thus, when we want to execute a specific sub-graph path, we start by first selecting the required operations from the list of all operation nodes.
Revised: Thus, when we want to execute a specific sub-graph path, we start by first selecting the required operations from the list of all operation nodes.

Original: We then create the sub-graph architecture by routing the output variables (which are pointers to Tensors in memory) of one operation to the input of the next desired operation by changing the variable references.
Revised: We then create the sub-graph architecture by routing the output variables (which are pointers to Tensors in memory) of one operation to the input of the next desired operation by changing the variable references.

Original: We can do this by applying the equal operation of the output variable of one operation node to the input variable of the next operation node, which essentially links them together into a direct execution path.
Revised: We can do this by applying the equal operation of the output variable of one operation node to the input variable of the next operation node, which essentially links them together into a direct execution path.

Original: Since we can set the output pointer variables to the input variables at any time between forward passes without disrupting the computation results, we can re-route paths at a per-batch granularity.
Revised: Since we can set the output pointer variables to the input variables at any time between forward passes without disrupting the computation results, we can re-route paths at a per-batch granularity.

Original: Thus, by changing the pointer variables of input-outputs, we can select sub-graphs architectures as per our requirements.
Revised: Thus, by changing the pointer variables of input-outputs, we can select sub-graphs architectures as per our requirements.

Original: This allows us to adapt between sub-graphs in real-time without degrading QoS.
Revised: This allows us to adapt between sub-graphs in real-time without degrading QoS.

Original: 4. M ODEL A DAPTATION P OLICY
Revised: 4. M ODEL A DAPTATION P OLICY

Original: Inputs: Meta-graph after training A ∗ , CPU/GPU utilization % ρ at time c , maximum allowed
Revised: Inputs: Meta-graph after training A ∗ , CPU/GPU utilization % ρ at time c , maximum allowed

Original: FPS F max , empty profile table T , α sel is the list of sub-graphs that meet the selection criteria in descending order .
Revised: FPS F max , empty profile table T , α sel is the list of sub-graphs that meet the selection criteria in descending order.

Original: for each α i in A ∗ 
Revised: for each α i in A ∗ do for each ρ j do

Revised: T i,j = Profile ( α i , ρ j ) end for end for while True do ρ c =

Revised: CurrentResources () , α sel =

Revised: [] for each α i in A ∗ do if T i,c ≤ F max then α sel += α

Revised: i end if end for α sel =

Revised: Sort ( α sel ) based on max PSNR

Original:  We design an adaptation policy to decide the model adaptation strategy in real time for a given resource availability.
Revised: Switch to model α sel [0] end while return None We design an adaptation policy to decide the model adaptation strategy in real time for a given resource availability.

Original: This is done in two steps.
Revised: This is done in two steps.

Original: First, we generate a profiling table which contains the frame rate of each model under different resource availability (e.g., utilization) for a device.
Revised: First, we generate a profiling table which contains the frame rate of each model under different resource availability (e.g., utilization) for a device.

Original: We profile each model using 20 image frames and use the average latency to infer frame rate.
Revised: We profile each model using 20 image frames and use the average latency to infer frame rate.

Original: The profiling cost is no. of models ∗ resource availability granularity .
Revised: The profiling cost is no. of models ∗ resource availability granularity .

Original: This step is a one-time offline effort and thus the profiling cost is not a major overhead.
Revised: This step is a one-time offline effort and thus the profiling cost is not a major overhead.

Original: During the real-time SR task, we keep track of the resource availability of system via resource monitoring tools.
Revised: During the real-time SR task, we keep track of the resource availability of system via resource monitoring tools.

Original: Once we observe a change in resource availability, we adapt the model by using the search criteria α sel where fps ( α sel ) ≤ F max and sort the candidates based on their performance.
Revised: Once we observe a change in resource availability, we adapt the model by using the search criteria α sel where fps ( α sel ) ≤ F max and sort the candidates based on their performance.

Original: We use the adaptive sub-graph mechanism to select the top performing sub-graph and switch to it in real time to maintain QoS. The algorithm is constantly running while the application is active.
Revised: We use the adaptive sub-graph mechanism to select the top performing sub-graph and switch to it in real time to maintain QoS. The algorithm is constantly running while the application is active.

Original: It does not output any specific variable, but constantly controls the switch between sub-graphs (see Alg. 1 for details) .
Revised: It does not output any specific variable, but constantly controls the switch between sub-graphs (see Alg. 1 for details).  E VALUATION Model # of Params. Chkpt.

Original: Model
Revised: Size

Revised: Assembled 11.3M 88MB

Revised: DySR

Original: [Table]
Revised: [Table]

Revised: 44M 67MB

Original: Assembled
Revised: Hardware FLOPS Freq. Snapdragon899G

Original: [Table]
Revised: [Table]

Revised: .8 GHz

Revised: Intel

Revised: i

Revised: [Table]

Revised: 560M 388G

Revised: [Table]

Revised: 9 GHz

Revised: Nvidia 1080Ti 11.3T

Original: E VALUATION 5. T RAINING S ETUP
Revised: [Table] .4 GHz Nvidia A 312T 1.3 GHz 5. T RAINING S ETUP

Original: We implement DySR using PyTorch and perform the search and training using 4 A100 GPUs, taking 21 GPU days to complete per run.
Revised: We implement DySR using PyTorch and perform the search and training using 4 A100 GPUs, taking 21 GPU days to complete per run.

Original: During the deployment, we perform a quick profiling (e.g., a few steps of forward passes) on the target device (e.g. Snapdragon 855) to measure its inference latency and create a profiled database.
Revised: During the deployment, we perform a quick profiling (e.g., a few steps of forward passes) on the target device (e.g. Snapdragon 855) to measure its inference latency and create a profiled database.

Original: We then change our adaptation based on the expected inference latency from this profile.
Revised: We then change our adaptation based on the expected inference latency from this profile.

Original: For training the searched models till convergence, we use Div2K training dataset with 64x64 patch size, batch size of 16 and learning rate of 0.0043.
Revised: For training the searched models till convergence, we use Div2K training dataset with 64x64 patch size, batch size of 16 and learning rate of 0.0043.

Original: For the search parameters, we use the values 15, 2 and 5 for N l , N b and B respectively.
Revised: For the search parameters, we use the values 15, 2 and 5 for N l , N b and B respectively.

Original: For the video superresolution dataset, we train with Vimeo90k (Xue et al.
Revised: For the video super-resolution dataset, we train with Vimeo90k (Xue et al.

Original: The LR images and frames were generated by bi-cubic downsampling them by a factor of 2 and 4.
Revised: The LR images and frames were generated by bi-cubic downsampling them by a factor of 2 and 4.

Original: To generate the Pareto Optimality curve, we use the the models as shown in Figure 6a).
Revised: To generate the Pareto Optimality curve, we use the the models as shown in Figure 6a).

Original: 5. B ASELINES AND P ARAMETERS
Revised: 5. B ASELINES AND P ARAMETERS

Original: [Figure]
Revised: [Figure]

Original: ) and DySR with different numbers of subgraphs generated by the N b and B parameters.
Revised: , Static (CARN) and DySR with different numbers of subgraphs generated by the N b and B parameters.

Original: It is calculated as the sum of memory used for both model and activation (i.e., intermediate results).
Revised: It is calculated as the sum of memory used for both model and activation (i.e., intermediate results).

Original: We now describe our choice of baselines and search parameters.
Revised: We now describe our choice of baselines and search parameters.

Original: Since no other papers (to our knowledge) address this problem in the SR scope, we create our baseline model by combining all the models from Figure 6a (referred to as Assembled ).
Revised: Since no other papers (to our knowledge) address this problem in the SR scope, we create our baseline model by combining all the models from Figure 6a (referred to as Assembled ).

Original: It contains high-performing models over a wide FPS spectrum including the NTIRE 2022 Efficient Super-Resolution Challenge (Li et al.
Revised: It contains high-performing models over a wide FPS spectrum including the NTIRE 2022 Efficient Super-Resolution Challenge (Li et al.

Original: winner ByteESR (Kong et al.
Revised: winner ByteESR (Kong et al.

Original: Our DySR should ideally generate similar models and so makes for a fair comparison.
Revised: Our DySR should ideally generate similar models and so makes for a fair comparison.

Original: As mentioned above, the parameters N b and B are very important in determining the quality of the models.
Revised: As mentioned above, the parameters N b and B are very important in determining the quality of the models.

Original: Together they determine the total number of sub-graphs generated which in turn determines the amount of memory taken up by the metagraph.
Revised: Together they determine the total number of sub-graphs generated which in turn determines the amount of memory taken up by the meta-graph.

Original: It also indirectly determines how much weight is shared by the sub-graphs (more sub-graphs means more chances of operations being shared).
Revised: It also indirectly determines how much weight is shared by the sub-graphs (more sub-graphs means more chances of operations being shared).

Original: In Figure 6b we show the memory consumed when executing DySR , Assembled and the single static CARN model after applying dynamic 8-bit quantization for all of them (Pagliari et al.
Revised: In Figure 6b we show the memory consumed when executing DySR , Assembled and the single static CARN model after applying dynamic 8-bit quantization for all of them (Pagliari et al.

Original: As we tune the N b and B , we get 5, 10,and 20 models (no. of models =
Revised: As we tune the N b and B , we get 5, 10, 15 and 20 models (no. of models =

Original: N b ∗ B ).
Revised: N b ∗ B ).

Original: We see that even with 15 models, DySR has almost equal consumption to Assembled due to weight sharing among sub-graphs.
Revised: We see that even with 15 models, DySR has almost equal consumption to Assembled due to weight sharing among sub-graphs.

Original: With 5 models, DySR consumes around twice as much memory compared to the single model, illustrating the memory efficiency of our weight-shared NAS system.
Revised: With 5 models, DySR consumes around twice as much memory compared to the single model, illustrating the memory efficiency of our weightshared NAS system.

Original: Since Assembled contains 10 models, we selectmodels for our DySR as well for fair comparison for the rest of the experiments.
Revised: Since Assembled contains 10 models, we select 10 models for our DySR as well for fair comparison for the rest of the experiments.

Original: Table 4 shows that with equal number of models, our framework has less number of parameters and size on disk.
Revised: Table 4 shows that with equal number of models, our framework has less number of parameters and size on disk.

Original: 5. P ARETO O PTIMALITY
Revised: 5. P ARETO O PTIMALITY

Original: We now test our searched and fully trained sub-graphs against state-of-the-art models.
Revised: We now test our searched and fully trained sub-graphs against state-of-the-art models.

Original: Figureshows the PSNR vs. FPS performance of each of the 10 sub-networks in DySR for different datasets.
Revised: Figureshows the PSNR vs. FPS performance of each of the 10 sub-networks in DySR for different datasets.

Original: [Figure]
Revised: [Figure]

Original: The sub-graphs are evaluated by passing the input through the execution path of the specific subgraphs and their corresponding inference time measured.
Revised: The sub-graphs are evaluated by passing the input through the execution path of the specific subgraphs and their corresponding inference time measured.

Original: We observe here that across both image super-resolution and video super-resolution datasets for 2X and 4X upscaling, our sub-graphs achieve Pareto Optimality for the PSNR vs. FPS tradeoff created by the state-of-the-art models across the full FPS spectrum, demonstrating the efficiency of our search space.
Revised: We observe here that across both image super-resolution and video super-resolution datasets for 2X and 4X upscaling, our sub-graphs achieve Pareto Optimality for the PSNR vs. FPS tradeoff created by the state-of-the-art models across the full FPS spectrum, demonstrating the efficiency of our search space.

Original: [Figure]
Revised: [Figure]

Original: We next evaluate our generated models against a diverse set of hardware, and the results are shown in
Revised: We next evaluate our generated models against a diverse set of hardware, and the results are shown in

Original: Figure 8.
Revised: Figure 8.

Original: We use a mobile CPU Snapdragon 855, a laptop CPU Intel i5-560M, a desktop grade GPU 1080Ti and a server-grade GPU A100 (Table 5).
Revised: We use a mobile CPU Snapdragon 855, a laptop CPU Intel i5-560M, a desktop grade GPU 1080Ti and a server-grade GPU A100 (Table 5).

Original: This figure shows the results for 2X scaling for the Div2k dataset (other datasets show similar results, so we eliminate them for lack of space).
Revised: This figure shows the results for 2X scaling for the Div2k dataset (other datasets show similar results, so we eliminate them for lack of space).

Original: We search our models on the 1080Ti to be above 15 FPS.
Revised: We search our models on the 1080Ti to be above 15 FPS.

Original: Therefore, the model’s FPS shift for different hardware but still have models above 15 FPS (as was our search criteria).
Revised: Therefore, the model’s FPS shift for different hardware but still have models above 15 FPS (as was our search criteria).

Original: We see that across all these hardware, the Pareto Optimality of our generated sub-graphs are maintained, demonstrating that even with one search run we can generate efficient models for inference for such a wide variety of models.
Revised: We see that across all these hardware, the Pareto Optimality of our generated sub-graphs are maintained, demonstrating that even with one search run we can generate efficient models for inference for such a wide variety of models.

Original: This property allows for model adaptivity since this allows the selection of models with any desired FPS for any hardware, and ensures close to the optimal HR output quality.
Revised: This property allows for model adaptivity since this allows the selection of models with any desired FPS for any hardware, and ensures close to the optimal HR output quality.

Original: Along with the ability to change models on-the-fly, DySR allows for optimal PSNR vs. FPS tradeoff for resource adaptivity as well, as will be demonstrated next.
Revised: Along with the ability to change models on-the-fly, DySR allows for optimal PSNR vs. FPS tradeoff for resource adaptivity as well, as will be demonstrated next.

Original: 5. D YNAMIC R ESOURCE A DAPTIVITY
Revised: 5. D YNAMIC R ESOURCE A DAPTIVITY

Original: To demonstrate the resource adaptivity of our sub-graphs and model selection policy, we use three traces to generate real world resource fluctuation scenarios.
Revised: To demonstrate the resource adaptivity of our sub-graphs and model selection policy, we use three traces to generate real world resource fluctuation scenarios.

Original: We use a bursty trace to simulate extreme changes in resource availability, stable to illustrate a relatively small variation in resource availability and random .
Revised: We use a bursty trace to simulate extreme changes in resource availability, stable to illustrate a relatively small variation in resource availability and random .

Original: Since resource adaptivity has rarely been discussed in the SR context, we create two possible baselines.
Revised: Since resource adaptivity has rarely been discussed in the SR context, we create two possible baselines.

Original: The static baseline contains a single model deployed at any one time from the list of state-of-the-art models used above for generating the Pareto Optimality curve.
Revised: The static baseline contains a single model deployed at any one time from the list of state-of-the-art models used above for generating the Pareto Optimality curve.

Original: The
Revised: The

Original: [Figure]
Revised: [Figure]

Original: Assembled baseline contains all the models from the state-of-the-art list deployed at the same time to represent the scenario of keeping models pre-loaded.
Revised: Assembled baseline contains all the models from the state-of-the-art list deployed at the same time to represent the scenario of keeping models pre-loaded.

Original: Based on available resources, we use our real-time swapping policy to choose the best model for both our DySR as well as the two baselines.
Revised: Based on available resources, we use our real-time swapping policy to choose the best model for both our DySR as well as the two baselines.

Original: Figure 9 shows the results of our experiment using the Div2k dataset on 1080Ti for 2x scaling.
Revised: Figure 9 shows the results of our experiment using the Div2k dataset on 1080Ti for 2x scaling.

Original: The top row shows the trace used, the middle row shows the CDF of the resulting FPS fluctuation over time, and the bottom row shows the the Cumulative Distribution Frequency (CDF) of the PSNR/FPS.
Revised: The top row shows the trace used, the middle row shows the CDF of the resulting FPS fluctuation over time, and the bottom row shows the the Cumulative Distribution Frequency (CDF) of the PSNR/FPS.

Original: It is the probability of the PSNR/FPS being less than a certain value at any given time.
Revised: It is the probability of the PSNR/FPS being less than a certain value at any given time.

Original: Higher y-axis values at lower p values means better performance.
Revised: Higher y-axis values at lower p values means better performance.

Original: Here we observe that across all traces, DySR achieves overall higher values of PSNR compared to the baselines.
Revised: Here we observe that across all traces, DySR achieves overall higher values of PSNR compared to the baselines.

Original: The FPS for Assembled and DySR are somewhat similar due to both having little to no reloading overhead.
Revised: The FPS for Assembled and DySR are somewhat similar due to both having little to no reloading overhead.

Original: However, Assembled performs worse for PSNR due to having less granularity of models available across the full FPS spectrum.
Revised: However, Assembled performs worse for PSNR due to having less granularity of models available across the full FPS spectrum.

Original: For example, between 20 to 25 FPS in Figurefor Div2K with 2X upscaling, we see that our framework generates four models but the Assembled only has two.
Revised: For example, between 20 to 25 FPS in Figurefor Div2K with 2X upscaling, we see that our framework generates four models but the Assembled only has two.

Original: Therefore, Assembled has less options and tends to choose models with lower PSNR in order to be able to maintain its FPS.
Revised: Therefore, Assembled has less options and tends to choose models with lower PSNR in order to be able to maintain its FPS.

Original: The Static models perform the worst due to having many interruption of services as expected.
Revised: The Static models perform the worst due to having many interruption of services as expected.

Original: For the PSNR values, Static has a significant portion of its CDF distribution at 0 and therefore performs much worse than the others, especially at the lower p values.
Revised: For the PSNR values, Static has a significant portion of its CDF distribution at 0 and therefore performs much worse than the others, especially at the lower p values.

Original: Our framework outperforms all baselines by having the highest PSNR probabilities.
Revised: Our framework outperforms all baselines by having the highest PSNR probabilities.

Original: Another observation is that the largest improvement in PSNR and FPS gained from DySR is for the bursty workload.
Revised: Another observation is that the largest improvement in PSNR and FPS gained from DySR is for the bursty workload.

Original: This is mainly due to there being more model switches required here compared to the other two workloads.
Revised: This is mainly due to there being more model switches required here compared to the other two workloads.

Original: This amplifies the benefits derived from the model selection policy and granularity and thus a greater difference is observed.
Revised: This amplifies the benefits derived from the model selection policy and granularity and thus a greater difference is observed.

Original: As the workloads grow more stable, the numbers vary less, illustrating that the more unstable a system the more it should benefit from our solutions.
Revised: As the workloads grow more stable, the numbers vary less, illustrating that the more unstable a system the more it should benefit from our solutions.

Original: Based on all these observations, we conclude that DySR outperforms both baselines and can address the resource adaptivity issue.
Revised: Based on all these observations, we conclude that DySR outperforms both baselines and can address the resource adaptivity issue.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we focus on providing QoS for super resolution used in steaming services of mobile devices.
Revised: In this paper, we focus on providing QoS for super resolution used in steaming services of mobile devices.

Original: We propose an algorithm and system co-design approach DySR , which employs an adaptive-aware one-shot NAS to generate models with Pareto-optimal performance, low switching overhead, and frame rate constraint that can be adapted in real-time via our low cost incremental adaptation method.
Revised: We propose an algorithm and system co-design approach DySR , which employs an adaptive-aware one-shot NAS to generate models with Pareto-optimal performance, low switching overhead, and frame rate constraint that can be adapted in real-time via our low cost incremental adaptation method.

Original: DySR outperforms existing state-of-the-art models as well as the simple assemble method we explored by achieving the best performance while providing a steady frame rate throughput with a memory footprint that is 40% less compared to the 10-model Assembled baseline.  
Revised: DySR outperforms existing state-of-the-art models as well as the simple assemble method we explored by achieving the best performance while providing a steady frame rate throughput with a memory footprint that is 40% less compared to the 10-model Assembled baseline.  A CKNOWLEDGEMENTS

Revised: We are grateful to our anonymous reviewers for their valuable comments and suggestions that significantly improved the paper.

Revised: This work is sponsored by National Science Foundation CAREER-

Original: A. V ISUAL C OMPARISON
Revised: A. V ISUAL C OMPARISON

Original: We compare our DySR outputs against other models in current literature with similar sizes.
Revised: We compare our DySR outputs against other models in current literature with similar sizes.

Original: DySRL,M,S represent models with 1.6, 0.74 and 0.38M parameters respectively.
Revised: DySRL,M,S represent models with 1.6, 0.74 and 0.38M parameters respectively.

Original: CARN Ahn et al.
Revised: CARN Ahn et al.

Original: AutoGAN Fu et al.
Revised: AutoGAN Fu et al.

Original: and FALSR-C Chu et al.
Revised: and FALSR-C Chu et al.

Original: (2021) have 1.5, 0.71 and 0.41M parameters.
Revised: (2021) have 1.5, 0.71 and 0.41M parameters.

Original: We observe that across both datasets, our HR outputs are quite similar to comparable state-of-the-art.
Revised: We observe that across both datasets, our HR outputs are quite similar to comparable state-of-the-art.