Original: A NYTIME D ENSE P REDICTION WITH C ONFIDENCE A DAPTIVITY
Revised: A NYTIME D ENSE P REDICTION WITH C ONFIDENCE A DAPTIVITY

Original: A BSTRACT
Revised: A BSTRACT

Original: Anytime inference requires a model to make a progression of predictions which might be halted at any time.
Revised: Anytime inference requires a model to make a progression of predictions which might be halted at any time.

Original: Prior research on anytime visual recognition has mostly focused on image classification.
Revised: Prior research on anytime visual recognition has mostly focused on image classification.

Original: We propose the first unified and end-toend approach for anytime dense prediction.
Revised: We propose the first unified and end-toend approach for anytime dense prediction.

Original: A cascade of “exits” is attached to the model to make multiple predictions.
Revised: A cascade of “exits” is attached to the model to make multiple predictions.

Original: We redesign the exits to account for the depth and spatial resolution of the features for each exit.
Revised: We redesign the exits to account for the depth and spatial resolution of the features for each exit.

Original: To reduce total computation, and make full use of prior predictions, we develop a novel spatially adaptive approach to avoid further computation on regions where early predictions are already sufficiently confident.
Revised: To reduce total computation, and make full use of prior predictions, we develop a novel spatially adaptive approach to avoid further computation on regions where early predictions are already sufficiently confident.

Original: Our full method, named anytime dense prediction with confidence (ADP-C), achieves the same level of final accuracy, and meanwhile significantly reduces total computation.
Revised: Our full method, named anytime dense prediction with confidence (ADP-C), achieves the same level of final accuracy, and meanwhile significantly reduces total computation.

Original: We evaluate ADP-C on Cityscapes semantic segmentation and MPII human pose estimation: our method enables anytime inference without sacrificing accuracy while also reducing the total FLOPs of its base models by 44.4% and 59.1%.
Revised: We evaluate ADP-C on Cityscapes semantic segmentation and MPII human pose estimation: our method enables anytime inference without sacrificing accuracy while also reducing the total FLOPs of its base models by 44.4% and 59.1%.

Original: We compare with anytime inference by deep equilibrium networks and feature-based stochastic sampling, showing that ADP-C dominates both across the accuracy-computation curve.
Revised: We compare with anytime inference by deep equilibrium networks and feature-based stochastic sampling, showing that ADP-C dominates both across the accuracy-computation curve.

Original: Our code is available at https://github.com/liuzhuang13/anytime .
Revised: Our code is available at https://github.com/liuzhuang13/anytime .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep convolutional networks (Krizhevsky et al., 2017; He et al., 2016) achieve high accuracy but at significant computational cost.
Revised: Deep convolutional networks (Krizhevsky et al., 2017; He et al., 2016) achieve high accuracy but at significant computational cost.

Original: Their computational burden hinders deployment, especially for time-critical or low-resource use cases that for instance require interactivity or inference on a mobile device.
Revised: Their computational burden hinders deployment, especially for time-critical or low-resource use cases that for instance require interactivity or inference on a mobile device.

Original: This efficiency problem is tackled by special-purpose libraries (Chetlur et al., 2014), compression by network pruning (Han et al., 2015; Li et al., 2016; Liu et al., 2019), quantization (Rastegari et al., 2016; Jacob et al., 2018), and distillation (Hinton et al., 2015; Romero et al., 2014).
Revised: This efficiency problem is tackled by special-purpose libraries (Chetlur et al., 2014), compression by network pruning (Han et al., 2015; Li et al., 2016; Liu et al., 2019), quantization (Rastegari et al., 2016; Jacob et al., 2018), and distillation (Hinton et al., 2015; Romero et al., 2014).

Original: These solutions accelerate network computation but the entire network must still be computed; however, a prediction may be needed sooner.
Revised: These solutions accelerate network computation but the entire network must still be computed; however, a prediction may be needed sooner.

Original: Time constraints vary, but the inference time of a standard deep network does not.
Revised: Time constraints vary, but the inference time of a standard deep network does not.

Original: [Figure]
Revised: [Figure]

Original: Anytime inference (Fig. 1) mitigates this issue by bringing flexibility to model computation.
Revised: Anytime inference (Fig. 1) mitigates this issue by bringing flexibility to model computation.

Original: An anytime algorithm (Dean & Boddy, 1988) gradually improves its results as more computation time is given.
Revised: An anytime algorithm (Dean & Boddy, 1988) gradually improves its results as more computation time is given.

Original: It can be interrupted at any point during its computation to return a result as system or user requirements demand.
Revised: It can be interrupted at any point during its computation to return a result as system or user requirements demand.

Original: In this way, the time to the first output is reduced while the quality of the last output is preserved.
Revised: In this way, the time to the first output is reduced while the quality of the last output is preserved.

Original: An anytime model makes a progression of predictions between the first and last.
Revised: An anytime model makes a progression of predictions between the first and last.

Original: This progression continues if time remains, or halts if it is either already satisfactory or out of time.
Revised: This progression continues if time remains, or halts if it is either already satisfactory or out of time.

Original: For example, consider a user on a mobile device: an approximate result could be returned earlier if there is urgency, or the user could monitor the sequence of predictions as time goes by and stop the model once it is good enough.
Revised: For example, consider a user on a mobile device: an approximate result could be returned earlier if there is urgency, or the user could monitor the sequence of predictions as time goes by and stop the model once it is good enough.

Original: Note that anytime inference differs from adaptive or dynamic inference (Veit & Belongie, 2018;
Revised: Note that anytime inference differs from adaptive or dynamic inference (Veit & Belongie, 2018;

Original: Wu et al., 2018; Wang et al., 2018a) where the model decides how much to compute instead of an external decision.
Revised: Wu et al., 2018; Wang et al., 2018a) where the model decides how much to compute instead of an external decision.

Original: Prior research has explored anytime inference by feature selection (Karayev et al., 2014) or ensembling models through boosting (Grubb & Bagnell, 2012).
Revised: Prior research has explored anytime inference by feature selection (Karayev et al., 2014) or ensembling models through boosting (Grubb & Bagnell, 2012).

Original: For end-to-end neural network models, research has focused on classification for anytime inference or adaptive inference.
Revised: For end-to-end neural network models, research has focused on classification for anytime inference or adaptive inference.

Original: In particular, the multi-scale dense network (Huang et al., 2017) is an architecture for resource-efficient classification.
Revised: In particular, the multi-scale dense network (Huang et al., 2017) is an architecture for resource-efficient classification.

Original: The attraction of anytime inference is not limited to classification however, and the additional computation required for dense prediction tasks makes it even more desirable.
Revised: The attraction of anytime inference is not limited to classification however, and the additional computation required for dense prediction tasks makes it even more desirable.

Original: For instance, an autonomous driving system may demand swifter reaction time for safety in the presence of pedestrians, and so an anytime semantic segmentor might sooner recognize their presence.
Revised: For instance, an autonomous driving system may demand swifter reaction time for safety in the presence of pedestrians, and so an anytime semantic segmentor might sooner recognize their presence.

Original: In addition to urgency, an anytime segmentor could help efficiency, by not further processing already confident predictions of street pixels and therefore save power.
Revised: In addition to urgency, an anytime segmentor could help efficiency, by not further processing already confident predictions of street pixels and therefore save power.

Original: In this work, we develop the first single-model anytime approach for dense prediction tasks.
Revised: In this work, we develop the first single-model anytime approach for dense prediction tasks.

Original: We adopt an early exiting framework, where multiple predictors branch off from the intermediate stages of the model.
Revised: We adopt an early exiting framework, where multiple predictors branch off from the intermediate stages of the model.

Original: The exits are trained end-to-end (both the original exit and intermediate exits), and during inference each provides a prediction in turn.
Revised: The exits are trained end-to-end (both the original exit and intermediate exits), and during inference each provides a prediction in turn.

Original: To compensate for differences in depth and spatial dimensions across stages, we redesign the predictors for earlier exits.
Revised: To compensate for differences in depth and spatial dimensions across stages, we redesign the predictors for earlier exits.

Original: For each exit, we choose an encoder-decoder architecture to enlarge receptive fields and smooth spatial noise.
Revised: For each exit, we choose an encoder-decoder architecture to enlarge receptive fields and smooth spatial noise.

Original: Exits might suffice for anytime image classification, but dense prediction tasks have spatial structures.
Revised: Exits might suffice for anytime image classification, but dense prediction tasks have spatial structures.

Original: Simple regions may need less processing while complex ones need more.
Revised: Simple regions may need less processing while complex ones need more.

Original: Standard inference applies an equal amount of computation at every pixel without taking advantage of spatial structure.
Revised: Standard inference applies an equal amount of computation at every pixel without taking advantage of spatial structure.

Original: Our spatially adaptive anytime inference scheme decides whether or not to continue computation at each exit and position.
Revised: Our spatially adaptive anytime inference scheme decides whether or not to continue computation at each exit and position.

Original: We mask the output of each exit by thresholding the confidence of its predictions: the remaining computation for sufficiently confident pixels is then reduced (Fig. 2).
Revised: We mask the output of each exit by thresholding the confidence of its predictions: the remaining computation for sufficiently confident pixels is then reduced (Fig. 2).

Original: For each masked pixel, its prediction will be persisted in the following exits, as it is already sufficiently confident.
Revised: For each masked pixel, its prediction will be persisted in the following exits, as it is already sufficiently confident.

Original: In the following layers, the features for the masked pixel will be interpolated, rather than convolved, and therefore reduce computation.
Revised: In the following layers, the features for the masked pixel will be interpolated, rather than convolved, and therefore reduce computation.

Original: The confidence measure can depend on the task, e.g., in segmentation, it could be the entropy of class predictions.
Revised: The confidence measure can depend on the task, e.g., in segmentation, it could be the entropy of class predictions.

Original: This confidence adaptivity can substantially reduce the total computation while maintaining accuracy.
Revised: This confidence adaptivity can substantially reduce the total computation while maintaining accuracy.

Original: We experiment with two dense prediction tasks:
Revised: We experiment with two dense prediction tasks:

Original: Cityscapes semantic segmentation and MPII human pose estimation.
Revised: Cityscapes semantic segmentation and MPII human pose estimation.

Original: Redesigning the exits and including confidence adaptivity significantly improves across accuracy-efficiency operating points.
Revised: Redesigning the exits and including confidence adaptivity significantly improves across accuracy-efficiency operating points.

Original: Our full approach, named anytime dense prediction with confidence (ADP-C), not only makes anytime predictions, but its final predictions achieve the same level of accuracy as the base model, with 40-60% less total computation.
Revised: Our full approach, named anytime dense prediction with confidence (ADP-C), not only makes anytime predictions, but its final predictions achieve the same level of accuracy as the base model, with 40-60% less total computation.

Original: For analysis, we visualize predictions and confidence adaptivity across exits, and ablate design choices for the exits and masking.
Revised: For analysis, we visualize predictions and confidence adaptivity across exits, and ablate design choices for the exits and masking.

Original:  A PPROACH Anytime Setting
Revised:  A PPROACH Anytime Setting

Original: In an anytime inference setting, the user can stop the inference process based on the input or a current event.
Revised: In an anytime inference setting, the user can stop the inference process based on the input or a current event.

Original: Thus the computation budget for each instance x could be time or input-dependent.
Revised: Thus the computation budget for each instance x could be time or input-dependent.

Original: We use B ( x, t ) to denote the computation budget assigned for instance x at time t , where the time variable t models events that can change the budget.
Revised: We use B ( x, t ) to denote the computation budget assigned for instance x at time t , where the time variable t models events that can change the budget.

Original: B ( x, t ) could be independent of x , i.e., the budget only depends on the time t , for example if a model on a server is asked to make predictions with less budget during high-traffic hours; B ( x, t ) can also be independent of t , meaning the budget is only decided by input x , regardless of external events.
Revised: B ( x, t ) could be independent of x , i.e., the budget only depends on the time t , for example if a model on a server is asked to make predictions with less budget during high-traffic hours; B ( x, t ) can also be independent of t , meaning the budget is only decided by input x , regardless of external events.

Original: The output of the anytime model depends on the budget given, and we denote it as f ( x, B ( x, t )) .
Revised: The output of the anytime model depends on the budget given, and we denote it as f ( x, B ( x, t )) .

Original: Assuming L is the task loss and y is the ground truth, the per-instance loss is L ( f ( x, B ( x, t )) , y ) .
Revised: Assuming L is the task loss and y is the ground truth, the per-instance loss is L ( f ( x, B ( x, t )) , y ) .

Original: This leads to the expected training loss to be E ( x,y ) ∼ ( X,Y ) ,t ∼ T
Revised: This leads to the expected training loss to be E ( x,y ) ∼ ( X,Y ) ,t ∼ T

Original: [ L ( f ( x, B ( x, t )) , y )] , where ( X, Y ) is the inputoutput joint distribution and T is the distribution modeling the time or event variable.
Revised: [ L ( f ( x, B ( x, t )) , y )] , where ( X, Y ) is the inputoutput joint distribution and T is the distribution modeling the time or event variable.

Original: Early Exiting.
Revised: Early Exiting.

Original: Next, we introduce the early exiting framework which has been used in prior works (Huang et al., 2017; Teerapittayanon et al., 2016) for anytime prediction.
Revised: Next, we introduce the early exiting framework which has been used in prior works (Huang et al., 2017; Teerapittayanon et al., 2016) for anytime prediction.

Original: Standard convolutional networks only have one prediction “head” at its final stage.
Revised: Standard convolutional networks only have one prediction “head” at its final stage.

Original: The network takes the input x , forwards it through intermediate layers, and finally outputs the prediction at its head.
Revised: The network takes the input x , forwards it through intermediate layers, and finally outputs the prediction at its head.

Original: The concrete form of
Revised: The concrete form of

Original: [Table]
Revised: [Table]

Original: (cid:51)(cid:85)(cid:72)(cid:71)(cid:76)(cid:70)(cid:87)(cid:76)(cid:82)(cid:81)(cid:3)(cid:21) (cid:41)(cid:72)(cid:68)(cid:87)(cid:88)(cid:85)(cid:72)(cid:86)(cid:3)(cid:73)(cid:82)(cid:85)(cid:3)(cid:73)(cid:76)(cid:81)(cid:68)(cid:79)(cid:3)(cid:72)(cid:91)(cid:76)(cid:87) the head depends on the task.
Revised: (cid:51)(cid:85)(cid:72)(cid:71)(cid:76)(cid:70)(cid:87)(cid:76)(cid:82)(cid:81)(cid:3)(cid:21) (cid:41)(cid:72)(cid:68)(cid:87)(cid:88)(cid:85)(cid:72)(cid:86)(cid:3)(cid:73)(cid:82)(cid:85)(cid:3)(cid:73)(cid:76)(cid:81)(cid:68)(cid:79)(cid:3)(cid:72)(cid:91)(cid:76)(cid:87) the head depends on the task.

Original: For dense prediction, the head is usually one or multiple convolutions that output spatial maps representing pixel-wise predictions.
Revised: For dense prediction, the head is usually one or multiple convolutions that output spatial maps representing pixel-wise predictions.

Original: To obtain an anytime model, we attach multiple heads to the network, branching from its intermediate features (Fig. 2).
Revised: To obtain an anytime model, we attach multiple heads to the network, branching from its intermediate features (Fig. 2).

Original: We call these additional heads early exits , since they allow the network to give early predictions and stop the inference at the current layer.
Revised: We call these additional heads early exits , since they allow the network to give early predictions and stop the inference at the current layer.

Original: Suppose we add k early exits at intermediate layers with layer indices l 1 . . .
Revised: Suppose we add k early exits at intermediate layers with layer indices l 1 . . .

Original: , l k .
Revised: , l k .

Original: We denote the intermediate features at these layers
Revised: We denote the intermediate features at these layers

Original: F l 1 ( x ) . . .
Revised: F l 1 ( x ) . . .

Original: , F l k ( x ) , and the functions represented by the early exits E 1 . . .
Revised: , F l k ( x ) , and the functions represented by the early exits E 1 . . .

Original: , E k .
Revised: , E k .

Original: Note that E i s may be of the same form but they do not share weights.
Revised: Note that E i s may be of the same form but they do not share weights.

Original: The early prediction maps can be denoted as
Revised: The early prediction maps can be denoted as

Original: [Equation]
Revised: [Equation]

Original: = E i ( F l i ( x )) , i = 1 . . .
Revised: = E i ( F l i ( x )) , i = 1 . . .

Original: k .
Revised: k .

Original: Together with the original final prediction ˆ y k +1 , the total loss is:
Revised: Together with the original final prediction ˆ y k +1 , the total loss is:

Original: [Equation]
Revised: [Equation]

Original: where w i is the weight coefficient at exit i .
Revised: where w i is the weight coefficient at exit i .

Original: The original network, together with the added exits, will be trained end-to-end to optimize this total loss function.
Revised: The original network, together with the added exits, will be trained end-to-end to optimize this total loss function.

Original: In experiments, we set all weights equal to 1.
Revised: In experiments, we set all weights equal to 1.

Original: This corresponds to the minimization of the expected loss in Sec.
Revised: This corresponds to the minimization of the expected loss in Sec.

Original: 2 when the exiting probabilities at all exits are equal.
Revised: 2 when the exiting probabilities at all exits are equal.

Original: We find this to be a simple yet effective scheme.
Revised: We find this to be a simple yet effective scheme.

Original: For anytime inference, as the network propagates features through its layers, if the computation budget is reached or the user asks the model to stop, it will output the latest ˆ y i that is already computed.
Revised: For anytime inference, as the network propagates features through its layers, if the computation budget is reached or the user asks the model to stop, it will output the latest ˆ y i that is already computed.

Original: Similar early exiting strategies have been used in resource-efficient image classification (Teerapittayanon et al., 2016; Huang et al., 2017), but dense prediction tasks require further steps detailed in the following subsections.
Revised: Similar early exiting strategies have been used in resource-efficient image classification (Teerapittayanon et al., 2016; Huang et al., 2017), but dense prediction tasks require further steps detailed in the following subsections.

Original: Head Redesign.
Revised: Head Redesign.

Original: Typical convolutional networks have a hierarchical structure that begins with shallow, fine, and more local features and ends with deep, coarse, and more global features.
Revised: Typical convolutional networks have a hierarchical structure that begins with shallow, fine, and more local features and ends with deep, coarse, and more global features.

Original: These deeper features represent more image content by their larger receptive fields.
Revised: These deeper features represent more image content by their larger receptive fields.

Original: For dense prediction, upsampling is done within the network to restore lost resolution during downsampling, and ensure precise spatial correspondence between the input and the output.
Revised: For dense prediction, upsampling is done within the network to restore lost resolution during downsampling, and ensure precise spatial correspondence between the input and the output.

Original: This upsampling can be accomplished in a few (Long et al., 2015) or many (Zhao et al., 2017) layers, but no matter the architecture, the network learns its most local features in its earliest layers.
Revised: This upsampling can be accomplished in a few (Long et al., 2015) or many (Zhao et al., 2017) layers, but no matter the architecture, the network learns its most local features in its earliest layers.

Original: This presents a challenge for the earliest exits, since these features are limited in depth and receptive field.
Revised: This presents a challenge for the earliest exits, since these features are limited in depth and receptive field.

Original: Making direct predictions at these exits with the typical 1 × 1 convolution head produces spatially noisy and inaccurate results.
Revised: Making direct predictions at these exits with the typical 1 × 1 convolution head produces spatially noisy and inaccurate results.

Original: To compensate for these lacking early features, we redesign the prediction heads for the exits E i .
Revised: To compensate for these lacking early features, we redesign the prediction heads for the exits E i .

Original: Each E i first downsamples its input features F l i ( x ) , through a series of pooling and 1 × 1 convolution layers.
Revised: Each E i first downsamples its input features F l i ( x ) , through a series of pooling and 1 × 1 convolution layers.

Original: Each pooling operation
Revised: Each pooling operation

Original: halves the spatial resolution, increasing its output’s receptive fields
Revised: halves the spatial resolution, increasing its output’s receptive fields

Original: The following convolution provides the opportunity to learn new coarser-level features, specifically for that exit’s prediction.
Revised: The following convolution provides the opportunity to learn new coarser-level features, specifically for that exit’s prediction.

Original: After several (denoting this number as D ) “pool-conv” layers, we upsample the features back to the original output resolution, with an equal number ( D ) of bilinear interpolation and 1 × 1 convolution layers.
Revised: After several (denoting this number as D ) “pool-conv” layers, we upsample the features back to the original output resolution, with an equal number ( D ) of bilinear interpolation and 1 × 1 convolution layers.

Original: The output of this “interpolate-conv” sequence will be the prediction ˆ
Revised: The output of this “interpolate-conv” sequence will be the prediction ˆ

Original: y i at this exit.
Revised: y i at this exit.

Original: This is important for ensuring spatial accuracy for pixel-level dense prediction tasks.
Revised: This is important for ensuring spatial accuracy for pixel-level dense prediction tasks.

Original: Our redesigned exits are essentially small “encoder-decoder” modules (Fig. 2), where the encoder downsamples the features, the decoder upsamples them back.
Revised: Our redesigned exits are essentially small “encoder-decoder” modules (Fig. 2), where the encoder downsamples the features, the decoder upsamples them back.

Original: The downsampling ratio at each exit is determined by D , the number of consecutive “pool-conv” layers.
Revised: The downsampling ratio at each exit is determined by D , the number of consecutive “pool-conv” layers.

Original: Intuitively, features at earlier layers are more fine-level, and the exit branching from them can potentially benefit from more downsampling.
Revised: Intuitively, features at earlier layers are more fine-level, and the exit branching from them can potentially benefit from more downsampling.

Original: In experiments, we use an encoder with D =
Revised: In experiments, we use an encoder with D =

Original: N − i downsampling operations at exit i , where N is the total number of exits, including the original last exit.
Revised: N − i downsampling operations at exit i , where N is the total number of exits, including the original last exit.

Original: Empirically we find this strategy works well, and alternative strategies are compared in Sec. 4.
Revised: Empirically we find this strategy works well, and alternative strategies are compared in Sec. 4.

Original: Finally, in all early exits, the first convolution will transform the number of channels to a fixed number for all exits.
Revised: Finally, in all early exits, the first convolution will transform the number of channels to a fixed number for all exits.

Original: By setting the channel width relatively small, we can still save computation while adding layers with this redesigned encoder-decoder head structure.
Revised: By setting the channel width relatively small, we can still save computation while adding layers with this redesigned encoder-decoder head structure.

Original: Confidence Adaptivity.
Revised: Confidence Adaptivity.

Original: For dense prediction tasks, any early prediction ˆ y i is a spatial map consisting of pixel-wise predictions at each position.
Revised: For dense prediction tasks, any early prediction ˆ y i is a spatial map consisting of pixel-wise predictions at each position.

Original: While most convolution networks spend an equal amount of computation at each input position, it is likely that recognition at some regions are easier than others, where the network can make predictions with high confidence even at earlier exits.
Revised: While most convolution networks spend an equal amount of computation at each input position, it is likely that recognition at some regions are easier than others, where the network can make predictions with high confidence even at earlier exits.

Original: For instance, the inner part of a large sky segment may be easy to recognize, whereas the boundary between the bicycle and the person riding it may need more careful delineation.
Revised: For instance, the inner part of a large sky segment may be easy to recognize, whereas the boundary between the bicycle and the person riding it may need more careful delineation.

Original: Once an early prediction is made, we can inspect the “confidence” at each position.
Revised: Once an early prediction is made, we can inspect the “confidence” at each position.

Original: As an example, for semantic segmentation, the maximum probability over all classes can serve as a confidence measure.
Revised: As an example, for semantic segmentation, the maximum probability over all classes can serve as a confidence measure.

Original: If the confidence has passed a pre-defined threshold at certain positions (green squares on predictions in Fig. 2), we may decide these predictions are likely to be correct, and not continue the computation of further layers at this position.
Revised: If the confidence has passed a pre-defined threshold at certain positions (green squares on predictions in Fig. 2), we may decide these predictions are likely to be correct, and not continue the computation of further layers at this position.

Original: Suppose the pixels of the early prediction ˆ y i are indexed by p , we form a mask M i :
Revised: Suppose the pixels of the early prediction ˆ y i are indexed by p , we form a mask M i :

Original: [Equation]
Revised: [Equation]

Original: For any convolution layer between exit i ( E i ) and the next exit i + 

Revised: For any convolution layer between exit i ( E i ) and the next exit i +

Original: ( E i +1 ), we could choose whether to perform or skip computation at position p based on the mask (Fig. 2).
Revised: ( E i +1 ), we could choose whether to perform or skip computation at position p based on the mask (Fig. 2).

Original: Assuming C is a convolution layer with input f in , then by applying the mask, the output f out at position p becomes:
Revised: Assuming C is a convolution layer with input f in , then by applying the mask, the output f out at position p becomes:

Original: [Equation]
Revised: [Equation]

Original: If C ’s output and the mask M i do not share the same spatial size, we interpolate ˆ y i in Eqn.
Revised: If C ’s output and the mask M i do not share the same spatial size, we interpolate ˆ y i in Eqn.

Original: 2 to the size of C ’s output, so that the mask M i is compatible with C in Eqn.
Revised: 2 to the size of C ’s output, so that the mask M i is compatible with C in Eqn.

Original: The output f out could be sparse, with many positions being 0 .
Revised: The output f out could be sparse, with many positions being 0 .

Original: This could potentially harm further convolutional computation.
Revised: This could potentially harm further convolutional computation.

Original: To compensate for this, we spatially interpolate these positions from their neighbors across all channels, using a similar approach as in (Xie et al., 2020).
Revised: To compensate for this, we spatially interpolate these positions from their neighbors across all channels, using a similar approach as in (Xie et al., 2020).

Original: Denoting the interpolation operation as I , the final output feature f ∗ out is
Revised: Denoting the interpolation operation as I , the final output feature f ∗ out is

Original: [Equation]
Revised: [Equation]

Original: Here, the value of I ( f out )( p ) is a weighted average of all the neighboring pixels centered at p within a radius r :
Revised: Here, the value of I ( f out )( p ) is a weighted average of all the neighboring pixels centered at p within a radius r :

Original: [Equation]
Revised: [Equation]

Original: where s indexes p ’s neighboring pixels and Ω( p )
Revised: where s indexes p ’s neighboring pixels and Ω( p )

Original: = { s |∥ s − p ∥ ∞ ≤ r, s ̸ = p } , the neighborhood of p .
Revised: = { s |∥ s − p ∥ ∞ ≤ r, s ̸ = p } , the neighborhood of p .

Original: We set radius r = 7 in our experiments.
Revised: We set radius r = 7 in our experiments.

Original: W ( p,s ) is the weight assigned to point s for interpolating at p , for which we use the RBF kernel, a distance-based exponential decaying weighting scheme:
Revised: W ( p,s ) is the weight assigned to point s for interpolating at p , for which we use the RBF kernel, a distance-based exponential decaying weighting scheme:

Original: [Equation]
Revised: [Equation]

Original: with λ being a trainable parameter.
Revised: with λ being a trainable parameter.

Original: This indicates that the closer s is to p , the larger its assigned weight will be.
Revised: This indicates that the closer s is to p , the larger its assigned weight will be.

Original: Note that masked-out features ( M i ( p ) = 0 ) still participate in the interpolation process as inputs with values of 0.
Revised: Note that masked-out features ( M i ( p ) = 0 ) still participate in the interpolation process as inputs with values of 0.

Original: Replacing filtering by interpolation at these already confident spatial locations ( M i ( p ) = 0 ) could potentially save a substantial amount of computation.
Revised: Replacing filtering by interpolation at these already confident spatial locations ( M i ( p ) = 0 ) could potentially save a substantial amount of computation.

Original: The mask M i will be used for all convolutions between exit i and i + 1 , including the convolutions inside exit i + 1 .
Revised: The mask M i will be used for all convolutions between exit i and i + 1 , including the convolutions inside exit i + 1 .

Original: Once the forward pass arrives at the next exit, to make the prediction ˆ y i +1 , the last prediction at positions where M i ( p ) = 0 will be carried over, having already been deemed confident enough at the last exit and having been skipped during further computations.
Revised: Once the forward pass arrives at the next exit, to make the prediction ˆ y i +1 , the last prediction at positions where M i ( p ) = 0 will be carried over, having already been deemed confident enough at the last exit and having been skipped during further computations.

Original: This means:
Revised: This means:

Original: [Equation]
Revised: [Equation]

Original: The network then calculates a new mask M i +1 based on ˆ y
Revised: The network then calculates a new mask M i +1 based on ˆ y

Original: i +1 , and uses it to skip computation going forward.
Revised: i +1 , and uses it to skip computation going forward.

Original: The process continues until we reach the final exit.
Revised: The process continues until we reach the final exit.

Original: In summary, we incorporate spatial confidence adaptivity into the early exiting network, by not filtering at spatial locations that are already sufficiently confident in the latest prediction.
Revised: In summary, we incorporate spatial confidence adaptivity into the early exiting network, by not filtering at spatial locations that are already sufficiently confident in the latest prediction.

Original: At these positions interpolation is used instead, at much reduced computational cost, to avoid excessive sparsity.
Revised: At these positions interpolation is used instead, at much reduced computational cost, to avoid excessive sparsity.

Original: Unless otherwise specified, confidence adaptivity is used in both training and inference.
Revised: Unless otherwise specified, confidence adaptivity is used in both training and inference.

Original: We dub our full approach as anytime dense prediction with confidence (ADP-C).
Revised: We dub our full approach as anytime dense prediction with confidence (ADP-C).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate ADP-C with two dense prediction tasks: semantic segmentation and human pose estimation.
Revised: We evaluate ADP-C with two dense prediction tasks: semantic segmentation and human pose estimation.

Original: Our experiments are implemented using PyTorch (Paszke et al., 2019).
Revised: Our experiments are implemented using PyTorch (Paszke et al., 2019).

Original: Architectures.
Revised: Architectures.

Original: We use the High-Resolution Network (HRNet) (Wang et al., 2020) architecture as our base model.
Revised: We use the High-Resolution Network (HRNet) (Wang et al., 2020) architecture as our base model.

Original: HRNet is a multi-stage architecture, where each stage adds lower-resolution/largerscale features.
Revised: HRNet is a multi-stage architecture, where each stage adds lower-resolution/largerscale features.

Original: Specifically, we adopt the standard HRNet- { W48,W32 } models and the smaller
Revised: Specifically, we adopt the standard HRNet- { W48,W32 } models and the smaller

Original: HRNet-W18 model.
Revised: HRNet-W18 model.

Original: HRNet-W48 is state-of-the-art for semantic segmentation and HRNet-W32 is suitable for pose estimation.
Revised: HRNet-W48 is state-of-the-art for semantic segmentation and HRNet-W32 is suitable for pose estimation.

Original: HRNet-W18 is highly efficient and has been shown to outperform other efficient networks (Zhao et al., 2018;
Revised: HRNet-W18 is highly efficient and has been shown to outperform other efficient networks (Zhao et al., 2018;

Original: Sandler et al., 2018) in its accuracy-efficiency tradeoff.
Revised: Sandler et al., 2018) in its accuracy-efficiency tradeoff.

Original: The 48/32/18 denotes the number of channels in the bottleneck of the first stage.
Revised: The 48/32/18 denotes the number of channels in the bottleneck of the first stage.

Original: The original head for
Revised: The original head for

Original: HRNet before our redesigning is two consecutive 1 × 1 convolutions.
Revised: HRNet before our redesigning is two consecutive 1 × 1 convolutions.

Original: We attach three exits, one at the end of each stage before the final prediction.
Revised: We attach three exits, one at the end of each stage before the final prediction.

Original: We follow the training/evaluation protocol and hyperparameters of the reference HRNet implementation at (Sun et al., 2019;
Revised: We follow the training/evaluation protocol and hyperparameters of the reference HRNet implementation at (Sun et al., 2019;

Original: Wang et al., 2020) (except that our models include a loss at each exit).
Revised: Wang et al., 2020) (except that our models include a loss at each exit).

Original: Please see Appendix C for more training details.
Revised: Please see Appendix C for more training details.

Original: Baselines.
Revised: Baselines.

Original: HRNet : we compare with a standard HRNet that has only one (final) exit with the same backbone architecture.
Revised: HRNet : we compare with a standard HRNet that has only one (final) exit with the same backbone architecture.

Original: The standard HRNet is not anytime, so we focus on comparing it with our anytime model’s final exit.
Revised: The standard HRNet is not anytime, so we focus on comparing it with our anytime model’s final exit.

Original: MDEQ (Bai et al., 2020) is a recent deep implicit model, which achieves competitive performance on vision tasks without stacking explicit layers, but rather solves an optimization problem for inference.
Revised: MDEQ (Bai et al., 2020) is a recent deep implicit model, which achieves competitive performance on vision tasks without stacking explicit layers, but rather solves an optimization problem for inference.

Original: Its representation z ∗ is an equilibrium point of its learned transformation f
Revised: Its representation z ∗ is an equilibrium point of its learned transformation f

Original: ( z ; x ) , i.e., f ( z ∗ ; x ) =
Revised: ( z ; x ) , i.e., f ( z ∗ ; x ) =

Original: z ∗ where x is the input.
Revised: z ∗ where x is the input.

Original: The representation is obtained by iteratively solving the equation f
Revised: The representation is obtained by iteratively solving the equation f

Original: ( z ; x ) =
Revised: ( z ; x ) =

Original: z , for which the quality of the solution improves with more iterations.
Revised: z , for which the quality of the solution improves with more iterations.

Original: The converged representation is then decoded into a prediction.
Revised: The converged representation is then decoded into a prediction.

Original: We examine anytime prediction with the MDEQ by decoding intermediate iterates of the representation.
Revised: We examine anytime prediction with the MDEQ by decoding intermediate iterates of the representation.

Original: To the best of our knowledge, this is the first study of anytime implicit modeling, as (Bai et al., 2020) only reports the predictions of implicit models at equilibrium, and does not produce or inspect intermediate predictions.
Revised: To the best of our knowledge, this is the first study of anytime implicit modeling, as (Bai et al., 2020) only reports the predictions of implicit models at equilibrium, and does not produce or inspect intermediate predictions.

Original: We use the “small” version of the MDEQ (Bai et al., 2020) and the 4th, 6th, 8th, and 10th iterations of its equilibrium optimization to bound the amount of computation and align its iterations with our architecture’s stages.
Revised: We use the “small” version of the MDEQ (Bai et al., 2020) and the 4th, 6th, 8th, and 10th iterations of its equilibrium optimization to bound the amount of computation and align its iterations with our architecture’s stages.

Original: Feature-Based Stochastic Sampling
Revised: Feature-Based Stochastic Sampling

Original: : we follow Xie et al.
Revised: : we follow Xie et al.

Original: (2020) in using internal features to predict masking positions, with the GumbelSoftmax trick (Jang et al., 2016) used for sampling.
Revised: (2020) in using internal features to predict masking positions, with the GumbelSoftmax trick (Jang et al., 2016) used for sampling.

Original: We use a 3 × 3 convolution upon the features for exits (Fig. 2) for mask prediction until the next exit.
Revised: We use a 3 × 3 convolution upon the features for exits (Fig. 2) for mask prediction until the next exit.

Original: During training, an L 1 sparsity regularization is applied on the stochastic continuous mask values, and only during inference the mask values are discretized.
Revised: During training, an L 1 sparsity regularization is applied on the stochastic continuous mask values, and only during inference the mask values are discretized.

Original: The interpolation procedure is the same as in our method.
Revised: The interpolation procedure is the same as in our method.

Original: This baseline is evaluated on
Revised: This baseline is evaluated on

Original: HRNet-W18.
Revised: HRNet-W18.

Original: [Table]
Revised: [Table]

Original: 3. S EMANTIC S EGMENTATION
Revised: 3. S EMANTIC S EGMENTATION

Original: The Cityscapes dataset (Cordts et al., 2016) consists of 2048 × 1024 images of urban street scenes with segmentation annotations of 19 classes.
Revised: The Cityscapes dataset (Cordts et al., 2016) consists of 2048 × 1024 images of urban street scenes with segmentation annotations of 19 classes.

Original: We train the models with the training set and report results on the validation set.
Revised: We train the models with the training set and report results on the validation set.

Original: The accuracy metric is the standard mean intersection-over-union (mIoU %), and the computation metric is the number of floating-point operations (FLOPs).
Revised: The accuracy metric is the standard mean intersection-over-union (mIoU %), and the computation metric is the number of floating-point operations (FLOPs).

Original: Anytime inference improves with higher accuracy, less computation [Table] and more predictions.
Revised: Anytime inference improves with higher accuracy, less computation, and more predictions.

Original: We evaluate
Revised: We evaluate

Original: HRNet-W48 and HRNet-W

Revised: HRNet-W48 and HRNet-W

Original: for this task.
Revised: for this task.

Original: Redesigned heads (RH) use our encoder-decoder structure for exits.
Revised: Redesigned heads (RH) use our encoder-decoder structure for exits.

Original: Since we have 4 exits in total, we repeat the downsampling operation 3/2/1 times at exit 1/2/3 to generate larger-scale features for earlier exits, as described in Sec. 2.
Revised: Since we have 4 exits in total, we repeat the downsampling operation 3/2/1 times at exit 1/2/3 to generate larger-scale features for earlier exits, as described in Sec. 2.

Original: We set the number of channels at all exits to 128/64 for
Revised: We set the number of channels at all exits to 128/64 for

Original: HRNet-W48/W18.
Revised: HRNet-W48/W18.

Original: For confidence adaptivity (CA), we use the maximum probability among all classes as the confidence measure, and set the confidence threshold in Eqn.
Revised: For confidence adaptivity (CA), we use the maximum probability among all classes as the confidence measure, and set the confidence threshold in Eqn.

Original: 2 to be 99.8% based on cross-validation.
Revised: 2 to be 99.8% based on cross-validation.

Original: For CA, the computation for each input can differ, so we report the average FLOPs across all validation images at each exit.
Revised: For CA, the computation for each input can differ, so we report the average FLOPs across all validation images at each exit.

Original: The results for HRNet-W48 are shown in Table 1.
Revised: The results for HRNet-W48 are shown in Table 1.

Original: We observe that our early exiting model based on HRNet-W48 outperforms the MDEQ model by a large margin, with significantly less FLOPs at each exit.
Revised: We observe that our early exiting model based on HRNet-W48 outperforms the MDEQ model by a large margin, with significantly less FLOPs at each exit.

Original: With RH, we achieve notable accuracy gain in early predictions, especially at the first exit (more than 10%), with roughly the same computation.
Revised: With RH, we achieve notable accuracy gain in early predictions, especially at the first exit (more than 10%), with roughly the same computation.

Original: With CA added, we arrive at our full ADP-C method (RH + CA), which maintains roughly the same accuracy as the RH model but reduces the total computation at exits 3 and 4.
Revised: With CA added, we arrive at our full ADP-C method (RH + CA), which maintains roughly the same accuracy as the RH model but reduces the total computation at exits 3 and 4.

Original: Interestingly ADP-C has slightly higher mIoU at the final exit (81.3 vs. 80.7) with 44.4% less total computation (387.1 vs. 696.2 GFLOPs) compared to the base
Revised: Interestingly ADP-C has slightly higher mIoU at the final exit (81.3 vs. 80.7) with 44.4% less total computation (387.1 vs. 696.2 GFLOPs) compared to the base

Original: HRNet.
Revised: HRNet.

Original: This is possibly due to a potential regularization effect of confidence adaptivity: computing fewer intermediate features exactly may prevent overfitting.
Revised: This is possibly due to a potential regularization effect of confidence adaptivity: computing fewer intermediate features exactly may prevent overfitting.

Original: The same results are plotted in Fig.
Revised: The same results are plotted in Fig.

Original: 3 (left).
Revised: 3 (left).

Original: The plot shows accuracy ( y -axis) and computation ( xaxis) tradeoffs: points to the upper left indicate better anytime performance.
Revised: The plot shows accuracy ( y -axis) and computation ( xaxis) tradeoffs: points to the upper left indicate better anytime performance.

Original: The baseline HRNet is represented by a red cross, while anytime models are plotted as curves with a point for each prediction.
Revised: The baseline HRNet is represented by a red cross, while anytime models are plotted as curves with a point for each prediction.

Original: We plot the results for the smaller HRNet-W18 model in Fig.
Revised: We plot the results for the smaller HRNet-W18 model in Fig.

Original: 3 (middle).
Revised: 3 (middle).

Original: RH improves early prediction accuracy from the basic early exiting model, and CA substantially reduces computation at later exits.
Revised: RH improves early prediction accuracy from the basic early exiting model, and CA substantially reduces computation at later exits.

Original: The full model reaches the same-level of accuracy as the baseline HRNet with much less total computation.
Revised: The full model reaches the same-level of accuracy as the baseline HRNet with much less total computation.

Original: We note that our model with confidence values as mask indicators also outperforms the feature-based mask sampling method in the accuracy-computation tradeoff, demonstrating that confidence is effective at filtering out redundant computation despite its simplicity.
Revised: We note that our model with confidence values as mask indicators also outperforms the feature-based mask sampling method in the accuracy-computation tradeoff, demonstrating that confidence is effective at filtering out redundant computation despite its simplicity.

Original: Our experiments measure computation by FLOPs rather than time.
Revised: Our experiments measure computation by FLOPs rather than time.

Original: Reporting FLOPs is common (Figurnov et al., 2017; Xie et al., 2020;
Revised: Reporting FLOPs is common (Figurnov et al., 2017; Xie et al., 2020;

Original: Huang et al., 2017; Wang et al., 2018a; Liu et al., 2019) and meaningful because it is hardware independent.
Revised: Huang et al., 2017; Wang et al., 2018a; Liu et al., 2019) and meaningful because it is hardware independent.

Original: However, similarly to spatially adaptive computation methods (Figurnov et al., 2017; Xie et al., 2020), our model does not achieve wall clock speedup at this time due to the lack of software/hardware support for sparse convolution with current frameworks and GPU devices.
Revised: However, similarly to spatially adaptive computation methods (Figurnov et al., 2017; Xie et al., 2020), our model does not achieve wall clock speedup at this time due to the lack of software/hardware support for sparse convolution with current frameworks and GPU devices.

Original: To approximate CPU speedup, we conduct a profiling experiment on a multi-threading processor (specifically we measure computation time on a Linux machine with
Revised: To approximate CPU speedup, we conduct a profiling experiment on a multi-threading processor (specifically we measure computation time on a Linux machine with

Original: Intel Xeon Gold 5220R CPUs using 16 threads).
Revised: Intel Xeon Gold 5220R CPUs using 16 threads).

Original: We replace all convolutions with our implementations following (Xie et al., 2020).
Revised: We replace all convolutions with our implementations following (Xie et al., 2020).

Original: ADP-C on HRNet-W48 achieves 1.48 × speedup compared tothe non-anytime baseline, measured in end-to-end latency (wall-clock time). There is a gap between
Revised: ADP-C on HRNet-W48 achieves 1.48 × speedup compared to

Original: [Figure]
Revised: [Figure]

Revised: the non-anytime baseline, measured in end-to-end latency (wall-clock time).

Original:  this measured time and the theoretical 1.80 × speedup measured by FLOPs.
Revised: There is a gap between this measured time and the theoretical 1.80 × speedup measured by FLOPs.

Original: ADP-C and others can benefit from ongoing and future work on efficient sparse convolutions (Graham & van der Maaten, 2017; Choy et al., 2019; Verelst & Tuytelaars, 2020; Elsen et al., 2020).
Revised: ADP-C and others can benefit from ongoing and future work on efficient sparse convolutions (Graham & van der Maaten, 2017; Choy et al., 2019; Verelst & Tuytelaars, 2020; Elsen et al., 2020).

Original: We also refer readers to the
Revised: We also refer readers to the

Original: Hardware Lottery (Hooker, 2021) for a discussion on how hardware compatibility affects progress in AI research.
Revised: Hardware Lottery (Hooker, 2021) for a discussion on how hardware compatibility affects progress in AI research.

Original: Please see the supplement for an anytime inference video where each exit is timed to the computation it requires.
Revised: Please see the supplement for an anytime inference video where each exit is timed to the computation it requires.

Original: 3. H UMAN P OSE E STIMATION
Revised: 3. H UMAN P OSE E STIMATION

Original: For human pose estimation, we evaluate on the MPII Human Pose dataset (Andriluka et al., 2014) of image crops annotated with body joints collected from everyday human activities.
Revised: For human pose estimation, we evaluate on the MPII Human Pose dataset (Andriluka et al., 2014) of image crops annotated with body joints collected from everyday human activities.

Original: The positions of 16 joint types are annotated for the human-centered in each crop.
Revised: The positions of 16 joint types are annotated for the human-centered in each crop.

Original: We report the standard metric (Andriluka et al., 2014) for MPII, the PCKh (head-normalized probability of correct keypoint) score, on its validation set.
Revised: We report the standard metric (Andriluka et al., 2014) for MPII, the PCKh (head-normalized probability of correct keypoint) score, on its validation set.

Original: We use HRNet-W32 for this task and follow the reference settings from (Sun et al., 2019).
Revised: We use HRNet-W32 for this task and follow the reference settings from (Sun et al., 2019).

Original: The standard head for this task is 1 × 1 convolution.
Revised: The standard head for this task is 1 × 1 convolution.

Original: As in segmentation, our redesigned heads are encoder-decoder structures.
Revised: As in segmentation, our redesigned heads are encoder-decoder structures.

Original: The number of channels for all exits is 64.
Revised: The number of channels for all exits is 64.

Original: Pose estimation task is formulated as regression.
Revised: Pose estimation task is formulated as regression.

Original: The HRNet model outputs 16 spatial feature maps, each one regressing the corresponding body joint.
Revised: The HRNet model outputs 16 spatial feature maps, each one regressing the corresponding body joint.

Original: The only positive target for each type is coded as 1; all other points are negatives coded as 0.
Revised: The only positive target for each type is coded as 1; all other points are negatives coded as 0.

Original: Unlike in segmentation, the output at each pixel is not a probability distribution, so we use the maximum value across channels as the confidence measure.
Revised: Unlike in segmentation, the output at each pixel is not a probability distribution, so we use the maximum value across channels as the confidence measure.

Original: A pixel is masked out if the maximum value at that position is smaller than the threshold, marking it unlikely to be a joint prediction.
Revised: A pixel is masked out if the maximum value at that position is smaller than the threshold, marking it unlikely to be a joint prediction.

Original: We choose 0.002 as the threshold by cross-validation, as a larger value makes the mask too sparse and hurts learning.
Revised: We choose 0.002 as the threshold by cross-validation, as a larger value makes the mask too sparse and hurts learning.

Original: The RH + CA model adopts adaptivity afterepochs of normal training, because nearly all outputs are too close to zero in the beginning.
Revised: The RH + CA model adopts adaptivity afterepochs of normal training, because nearly all outputs are too close to zero in the beginning.

Original: [Table]
Revised: [Table]

Original: Fig.
Revised: Fig.

Original: 3 (right) and Table 2 show the results.
Revised: 3 (right) and Table 2 show the results.

Original: We observe a similar trend to segmentation: RH improves accuracy and CA reduces FLOPs.
Revised: We observe a similar trend to segmentation: RH improves accuracy and CA reduces FLOPs.

Original: In this case, ADP-C reduces computation 
Revised: In this case, ADP-C reduces computation by 59.1% (9.49 to 3.

Original: A NALYSIS
Revised: GFLOPs) while accuracy only drops by 0.13% relative to the baseline HRNet. [Figure]

Original: Visualizations.
Revised: Visualizations.

Original: To inspect our anytime predictions and masking on Cityscapes, we visualize ADP-C exit results on a validation image with HRNet-W48.
Revised: To inspect our anytime predictions and masking on Cityscapes, we visualize ADP-C exit results on a validation image with HRNet-W48.

Original: Fig.
Revised: Fig.

Original: 4 shows the predictions, confidence maps, and computation masks across exits.
Revised: 4 shows the predictions, confidence maps, and computation masks across exits.

Original: With each exit, the prediction accuracy improves, especially in more detailed areas with more segments.
Revised: With each exit, the prediction accuracy improves, especially in more detailed areas with more segments.

Original: The confidence maps are shown with high lighter/yellow and low darker/green.
Revised: The confidence maps are shown with high lighter/yellow and low darker/green.

Original: Most unconfident points lie around segment boundaries, and the interior of large stuff segments (road, vegetation) are already confident at early exits.
Revised: Most unconfident points lie around segment boundaries, and the interior of large stuff segments (road, vegetation) are already confident at early exits.

Original: This motivates the use of confidence adaptivity to avoid unnecessary further computations on these areas.
Revised: This motivates the use of confidence adaptivity to avoid unnecessary further computations on these areas.

Original: For computation masks, the correct/incorrect predictions at each exit are marked white/black.
Revised: For computation masks, the correct/incorrect predictions at each exit are marked white/black.

Original: Pixels surpassing the confidence threshold (99.8%) are masked and marked green.
Revised: Pixels surpassing the confidence threshold (99.8%) are masked and marked green.

Original: Many pixels can be masked out in this way, and each exit masks more.
Revised: Many pixels can be masked out in this way, and each exit masks more.

Original: Most of the masked pixels are found in the inner parts of large segments or already correct areas.
Revised: Most of the masked pixels are found in the inner parts of large segments or already correct areas.

Original: In fact, the masked pixels are 100% correct at all exits for this instance, which partly justifies their exclusion from later computation.
Revised: In fact, the masked pixels are 100% correct at all exits for this instance, which partly justifies their exclusion from later computation.

Original: The predictions at these positions are already confident and correct at early exits, and so the only potential harm of skipping their computation later is their possible effect at less confident positions.
Revised: The predictions at these positions are already confident and correct at early exits, and so the only potential harm of skipping their computation later is their possible effect at less confident positions.

Original: See Appendix E for more visualizations.
Revised: See Appendix E for more visualizations.

Original: On average, 19.3%, 38.4%, 63.0% pixels are masked out at exit 1, 2, 3, respectively.
Revised: On average, 19.3%, 38.4%, 63.0% pixels are masked out at exit 1, 2, 3, respectively.

Original: Downsampling at Early Exits.
Revised: Downsampling at Early Exits.

Original: In Sec. 2, we described how many consecutive downsampling operations we use at each exit, by D = N − i , which means we use D = 3 / 2 / 1 consecutive “poolconv” layers for downsampling at exit 1 / 2 / 3 .
Revised: In Sec. 2, we described how many consecutive downsampling operations we use at each exit, by D = N − i , which means we use D = 3 / 2 / 1 consecutive “poolconv” layers for downsampling at exit 1 / 2 / 3 .

Original: Here we compare this strategy with D = 1 / 1 /and 3 / 3 / 3 , where the same level of downsampling and hence the same head structure is used at all exits, on HRNet-W18.
Revised: Here we compare this strategy with D = 1 / 1 /and 3 / 3 / 3 , where the same level of downsampling and hence the same head structure is used at all exits, on HRNet-W18.

Original: Fig.
Revised: Fig.

Original: 5 (left) shows that our adopted D = 3 / 2 / 1 strategy obtains the highest accuracy at all exits among these choices.
Revised: 5 (left) shows that our adopted D = 3 / 2 / 1 strategy obtains the highest accuracy at all exits among these choices.

Original: [Figure]
Revised: [Figure]

Original: Masking Criterion.
Revised: Masking Criterion.

Original: We used the max probability as the confidence measure and a fixed threshold for masking.
Revised: We used the max probability as the confidence measure and a fixed threshold for masking.

Original: Here we consider a few alternatives.
Revised: Here we consider a few alternatives.

Original: One is to mask out the top k % (by max prob.)
Revised: One is to mask out the top k % (by max prob.)

Original: of the pixels at each exit, regardless of their values.
Revised: of the pixels at each exit, regardless of their values.

Original: We also consider thresholding on the entropy of the probability distribution.
Revised: We also consider thresholding on the entropy of the probability distribution.

Original: In addition, we compare them with random masking.
Revised: In addition, we compare them with random masking.

Original: We use HRNet-W18, change the ratio or threshold for a wide range, and present the average mIoU vs. GFLOPs on all exits at Fig.
Revised: We use HRNet-W18, change the ratio or threshold for a wide range, and present the average mIoU vs. GFLOPs on all exits at Fig.

Original: 5 (right).
Revised: 5 (right).

Original: For this ablation, adaptivity is only applied during inference.
Revised: For this ablation, adaptivity is only applied during inference.

Original: First, we notice all three confidence criteria largely outperform random masking.
Revised: First, we notice all three confidence criteria largely outperform random masking.

Original: For max probability, using a threshold performs slightly better than a fixed ratio, possibly because this gives the flexibility for different exits to mask out different amounts of points.
Revised: For max probability, using a threshold performs slightly better than a fixed ratio, possibly because this gives the flexibility for different exits to mask out different amounts of points.

Original: Finally, we observe using entropy as the confidence measure performs similarly as using max probability, but we stick to max probability in our methods because it is trivial to compute.
Revised: Finally, we observe using entropy as the confidence measure performs similarly as using max probability, but we stick to max probability in our methods because it is trivial to compute.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Anytime Inference.
Revised: Anytime Inference.

Original: Anytime algorithms (Zilberstein, 1996; Dean & Boddy, 1988) can be interrupted at any point during computation to return a result, whose quality improves gradually with more computation time.
Revised: Anytime algorithms (Zilberstein, 1996; Dean & Boddy, 1988) can be interrupted at any point during computation to return a result, whose quality improves gradually with more computation time.

Original: In machine learning, anytime inference has been achieved by boosting (Grubb & Bagnell, 2012), reinforcement learning (Karayev et al., 2014), and random forests (Fr ¨ ohlich et al., 2012).
Revised: In machine learning, anytime inference has been achieved by boosting (Grubb & Bagnell, 2012), reinforcement learning (Karayev et al., 2014), and random forests (Fr ¨ ohlich et al., 2012).

Original: Anytime deep networks have been brought to bear on image classification, but not dense prediction.
Revised: Anytime deep networks have been brought to bear on image classification, but not dense prediction.

Original: Branching architectures have been a common strategy (Amthor et al., 2016;
Revised: Branching architectures have been a common strategy (Amthor et al., 2016;

Original: Teerapittayanon et al., 2016)
Revised: Teerapittayanon et al., 2016)

Original: along with other techniques such as adaptive loss balancing (Hu et al., 2019).
Revised: along with other techniques such as adaptive loss balancing (Hu et al., 2019).

Original: While there is work on the tasks of person re-identification (Wang et al., 2019) and stereo depth (Wang et al., 2018b), these techniques are task-specific, while our method applies to multiple dense prediction tasks, as we show with semantic segmentation and pose estimation.
Revised: While there is work on the tasks of person re-identification (Wang et al., 2019) and stereo depth (Wang et al., 2018b), these techniques are task-specific, while our method applies to multiple dense prediction tasks, as we show with semantic segmentation and pose estimation.

Original: Liu et al.
Revised: Liu et al.

Original: (Liu & He, 2016) learn a hierarchy of models for anytime segmentation, but its multiple models complicate training and testing, and require more memory.
Revised: (Liu & He, 2016) learn a hierarchy of models for anytime segmentation, but its multiple models complicate training and testing, and require more memory.

Original: Our work instead augments the base model architecture for simplicity and efficiency.
Revised: Our work instead augments the base model architecture for simplicity and efficiency.

Original: The PointRend method (Kirillov et al., 2020) outputs a initial dense prediction first and then refine it adaptively, but the predictions are all made at full depth.
Revised: The PointRend method (Kirillov et al., 2020) outputs a initial dense prediction first and then refine it adaptively, but the predictions are all made at full depth.

Original: Its majority of the computation is spent before the first output and thus cannot be practically “anytime”.
Revised: Its majority of the computation is spent before the first output and thus cannot be practically “anytime”.

Original: Our method is the first to selectively update anytime predictions across space and layers.
Revised: Our method is the first to selectively update anytime predictions across space and layers.

Original: Adaptive Computation.
Revised: Adaptive Computation.

Original: An adaptive model adjusts its computation to each specific instance during inference.
Revised: An adaptive model adjusts its computation to each specific instance during inference.

Original: For deep networks, this is often done by adjusting which layers to execute, that is, choosing which layers to run or skip.
Revised: For deep networks, this is often done by adjusting which layers to execute, that is, choosing which layers to run or skip.

Original: This can be done by a supervised controller (Veit & Belongie, 2018; Liu & Deng, 2017), a routing policy optimized by reinforcement learning (Wang et al., 2018a;
Revised: This can be done by a supervised controller (Veit & Belongie, 2018; Liu & Deng, 2017), a routing policy optimized by reinforcement learning (Wang et al., 2018a;

Original: Wu et al., 2018; Lin et al., 2017a), or other training strategies (McGill & Perona, 2017).
Revised: Wu et al., 2018; Lin et al., 2017a), or other training strategies (McGill & Perona, 2017).

Original: Rather than choosing layers, spatial adaptivity chooses where to adjust the amount of computation across different spatial positions in the input.
Revised: Rather than choosing layers, spatial adaptivity chooses where to adjust the amount of computation across different spatial positions in the input.

Original: For example, the model could infer spatial masks for feature maps and skip computation on masked areas (Shomron et al., 2019; Dong et al., 2017; Lin et al., 2017b;
Revised: For example, the model could infer spatial masks for feature maps and skip computation on masked areas (Shomron et al., 2019; Dong et al., 2017; Lin et al., 2017b;

Original: Ren et al., 2018; Cao et al., 2019).
Revised: Ren et al., 2018; Cao et al., 2019).

Original: Figurnov et al.
Revised: Figurnov et al.

Original: (2017) maintains a halting score at each pixel and once it reaches a threshold the model will stop inference at those positions for spatially coarse tasks like classification or bounding box detection.
Revised: (2017) maintains a halting score at each pixel and once it reaches a threshold the model will stop inference at those positions for spatially coarse tasks like classification or bounding box detection.

Original: Xie et al.
Revised: Xie et al.

Original: (2020) stochastically sample positions for computation from an end-to-end learned sampling distribution.
Revised: (2020) stochastically sample positions for computation from an end-to-end learned sampling distribution.

Original: Li et al.
Revised: Li et al.

Original: (2017) convert a deep network into a difficulty-aware cascade, where earlier steps handle easier regions and later steps tackle harder regions.
Revised: (2017) convert a deep network into a difficulty-aware cascade, where earlier steps handle easier regions and later steps tackle harder regions.

Original: These spatially adaptive models reduce computation, but are not anytime: they do not make a series of predictions and cannot be interrupted.
Revised: These spatially adaptive models reduce computation, but are not anytime: they do not make a series of predictions and cannot be interrupted.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We propose ADP-C, the first single-model anytime approach for dense visual prediction.
Revised: We propose ADP-C, the first single-model anytime approach for dense visual prediction.

Original: Based on an early-exiting framework, our redesigned exiting heads and confidence adaptivity both improve the accuracy-computation tradeoff.
Revised: Based on an early-exiting framework, our redesigned exiting heads and confidence adaptivity both improve the accuracy-computation tradeoff.

Original: On Cityscapes semantic segmentation and MPII pose estimation,
Revised: On Cityscapes semantic segmentation and MPII pose estimation,

Original: ADP-C achieves 40%-60% FLOPs reduction with the same-level final accuracy, compared against the baseline HRNet.
Revised: ADP-C achieves 40%-60% FLOPs reduction with the same-level final accuracy, compared against the baseline HRNet.

Original: We further analyze confidence adaptivity with visualizations and ablate key design choices to justify our approach to anytime inference with confidence
Revised: We further analyze confidence adaptivity with visualizations and ablate key design choices to justify our approach to anytime inference with confidence

Original: Acknowledgement.
Revised: Acknowledgement.

Original: This work was supported in part by DoD including DARPA’s XAI, LwLL, and/or SemaFor programs, as well as BAIR’s industrial alliance programs
Revised: This work was supported in part by DoD including DARPA’s XAI, LwLL, and/or SemaFor programs, as well as BAIR’s industrial alliance programs

Original: A PPENDIX A A BLATION ON I NTERPOLATION R ADIUS
Revised: A PPENDIX A A BLATION ON I NTERPOLATION R ADIUS

Original: In the main paper
Revised: In the main paper

Original: [Table]
Revised: [Table]

Original: we use a default radius of r = 7 when interpolating masked-out features (Eqn.and following text).
Revised: we use a default radius of r = 7 when interpolating masked-out features (Eqn.and following text).

Original: Here we demonstrate that the radius of 7 is reasonable through an ablation experiment, whose results are listed in Table 3.
Revised: Here we demonstrate that the radius of 7 is reasonable through an ablation experiment, whose results are listed in Table 3.

Original: The experiment is conducted with HRNet-W18 on
Revised: The experiment is conducted with HRNet-W18 on

Original: Cityscapes semantic segmentation, with both redesigned head (RH) and confidence adaptivity (CA).
Revised: Cityscapes semantic segmentation, with both redesigned head (RH) and confidence adaptivity (CA).

Original: We observe that a radius of 7 outperforms lower ones (3 and 5) in the average mIoU slightly, with minimal FLOPs addition, due to the fact that interpolation is done channel-wise.
Revised: We observe that a radius of 7 outperforms lower ones (3 and 5) in the average mIoU slightly, with minimal FLOPs addition, due to the fact that interpolation is done channel-wise.

Original: A further increase of radius to 9 does not bring significant gain in final or average mIoU.
Revised: A further increase of radius to 9 does not bring significant gain in final or average mIoU.

Original: [Table]
Revised: [Table]

Original: B I NFERENCE -O NLY C ONFIDENCE A DAPTIVITY
Revised: B I NFERENCE -O NLY C ONFIDENCE A DAPTIVITY

Original: [Table]
Revised: [Table]

Original: In the experiment section of the main paper, confidence adaptivity is used in both training and inference.
Revised: In the experiment section of the main paper, confidence adaptivity is used in both training and inference.

Original: Here we compare this with the setting where adaptivity is only used for inference in
Revised: Here we compare this with the setting where adaptivity is only used for inference in

Original: Table 4.
Revised: Table 4.

Original: We use HRNet-W48 with redesigned heads (RH) on Cityscapes for this experiment.
Revised: We use HRNet-W48 with redesigned heads (RH) on Cityscapes for this experiment.

Original: “No
Revised: “No

Original: Adaptivity” corresponds to the “EE + RH” row in Table 1.
Revised: Adaptivity” corresponds to the “EE + RH” row in Table 1.

Original: We observe that using adaptivity only at inference hurts the accuracy, compared with using it at both training and inference.
Revised: We observe that using adaptivity only at inference hurts the accuracy, compared with using it at both training and inference.

Original: It also increases the average FLOPs at exit 3 and 4.
Revised: It also increases the average FLOPs at exit 3 and 4.

Original: C E XPERIMENTAL S ETTINGS
Revised: C E XPERIMENTAL S ETTINGS

Original: For Cityscapes semantic segmentation, we follow the training settings at the official codebase (HRNet Semantic Segmentation) of HRNet for semantic segmentation.
Revised: For Cityscapes semantic segmentation, we follow the training settings at the official codebase (HRNet Semantic Segmentation) of HRNet for semantic segmentation.

Original: The HRNet-W18/48 models are pre-trained on ImageNet.
Revised: The HRNet-W18/48 models are pre-trained on ImageNet.

Original: During training, multi-scale and flipping data augmentation is used, and the input cropping size is 512 × 1024 .
Revised: During training, multi-scale and flipping data augmentation is used, and the input cropping size is 512 × 1024 .

Original: The model is trained for 484 epochs, with an initial learning rate of 0.01 and a polynomial schedule of power 0.9, a weight decay of 0.0005, a batch size of 12, optimized by SGD with 0.9 momentum.
Revised: The model is trained for 484 epochs, with an initial learning rate of 0.01 and a polynomial schedule of power 0.9, a weight decay of 0.0005, a batch size of 12, optimized by SGD with 0.9 momentum.

Original: In evaluation, we use single-scale testing without flipping, with input resolution 1024 × 2048 .
Revised: In evaluation, we use single-scale testing without flipping, with input resolution 1024 × 2048 .

Original: For the Feature-Based Stochastic Sampling baseline, we modify the exit weights from (1 , 1 , 1 , 1) to (0 . 5 , 0 . 5 , 0 . 5 , 1) as we find it produces more stable masking values during training.
Revised: For the Feature-Based Stochastic Sampling baseline, we modify the exit weights from (1 , 1 , 1 , 1) to (0 . 5 , 0 . 5 , 0 . 5 , 1) as we find it produces more stable masking values during training.

Original: The L 1 sparsity regularization on masks is set to 0 .
Revised: The L 1 sparsity regularization on masks is set to 0 .

Original: The mask outputs at exit (1 , 2 , 3) have additional weight factors of (1 / 3 , 2 / 3 , 1) to encourage more sparse features towards the end.
Revised: The mask outputs at exit (1 , 2 , 3) have additional weight factors of (1 / 3 , 2 / 3 , 1) to encourage more sparse features towards the end.

Original: For MPII human pose estimation, we follow the training settings at the official codebase (HRNet
Revised: For MPII human pose estimation, we follow the training settings at the official codebase (HRNet

Original: Human Pose Estimation) of HRNet for pose estimation.
Revised: Human Pose Estimation) of HRNet for pose estimation.

Original: The HRNet-32 model we use is also pretrained on ImageNet.
Revised: The HRNet-32 model we use is also pretrained on ImageNet.

Original: The image size for both training and evaluation is 256 × 256 .
Revised: The image size for both training and evaluation is 256 × 256 .

Original: The model is trained for 210 epochs, with an initial learning rate of 0.001, and decaying of 0.1 at epoch 170 and 200.
Revised: The model is trained for 210 epochs, with an initial learning rate of 0.001, and decaying of 0.1 at epoch 170 and 200.

Original: The optimization is done by Adam with γ 1 = 0 .
Revised: The optimization is done by Adam with γ 1 = 0 .

Original: 99 , γ 2 = 0 , a weight decay of 0.0001, and a momentum of 0.9.
Revised: 99 , γ 2 = 0 , a weight decay of 0.0001, and a momentum of 0.9.

Original: The batch size is 128.
Revised: The batch size is 128.

Original: In evaluation, flipping test is used.
Revised: In evaluation, flipping test is used.

Original: D PASCAL-C ONTEXT R ESULTS
Revised: D PASCAL-C ONTEXT R ESULTS

Original: We present results with the PASCAL-Context semantic segmentation dataset (Mottaghi et al., 2014) in Table 5.
Revised: We present results with the PASCAL-Context semantic segmentation dataset (Mottaghi et al., 2014) in Table 5.

Original: It consists of 59 segmentation classes.
Revised: It consists of 59 segmentation classes.

Original: For the Early Exiting baseline, we use weights of (0.33, 0.33, 0.33, 1) for 4 exits, respectively, as we found increasing it to all 1 would hurt the final performance too much.
Revised: For the Early Exiting baseline, we use weights of (0.33, 0.33, 0.33, 1) for 4 exits, respectively, as we found increasing it to all 1 would hurt the final performance too much.

Original: The confidence threshold is set to 99.5%.
Revised: The confidence threshold is set to 99.5%.

Original: We observe RH improve the accuracy in most exits.
Revised: We observe RH improve the accuracy in most exits.

Original: ADP-C outperforms the Early Exiting baseline in average mIoU, despite its final mIoU is worse.
Revised: ADP-C outperforms the Early Exiting baseline in average mIoU, despite its final mIoU is worse.

Original: It also saves more than 20% FLOPs compared with the vanilla Early Exiting baseline.
Revised: It also saves more than 20% FLOPs compared with the vanilla Early Exiting baseline.

Original: [Table]
Revised: [Table]

Original: E M ORE V ISUALIZATIONS
Revised: E M ORE V ISUALIZATIONS

Original: We present more visualizations of the same type as Fig.
Revised: We present more visualizations of the same type as Fig.

Original: 4 in the main paper, in Fig.
Revised: 4 in the main paper, in Fig.

Original: 7 and 8.
Revised: 7 and 8.

Original: Their input and ground truth are shown in Fig.
Revised: Their input and ground truth are shown in Fig.

Original: 6 in the same order.
Revised: 6 in the same order.

Original: We can see the same trend as discussed in the main paper still holds: the model will mask out confident points that are inside large segments (e.g., road, vegetable), which are mostly already predicted correctly in early exits.
Revised: We can see the same trend as discussed in the main paper still holds: the model will mask out confident points that are inside large segments (e.g., road, vegetable), which are mostly already predicted correctly in early exits.

Original: F M ORE MDEQ R ESULTS
Revised: F M ORE MDEQ R ESULTS

Original: In the main paper, we used MDEQ’s “small” model’s 4th, 6th, 8th, and 10th iterations’ results to align with our model’s 4 exits.
Revised: In the main paper, we used MDEQ’s “small” model’s 4th, 6th, 8th, and 10th iterations’ results to align with our model’s 4 exits.

Original: In this section, we provide full results on Cityscapes semantic segmentation at all iterations for both the “small” and the “XL” (extra-large) models at Table 6.
Revised: In this section, we provide full results on Cityscapes semantic segmentation at all iterations for both the “small” and the “XL” (extra-large) models at Table 6.

Original: The original configuration (MDEQ) sets the number of iterations to 26 and 27 for MDEQ-small and MDEQ-XL.
Revised: The original configuration (MDEQ) sets the number of iterations to 26 and 27 for MDEQ-small and MDEQ-XL.

Original: In Fig.
Revised: In Fig.

Original: 9 and 10, we also provide qualitative visualization of results for the same 6 validation images from Fig.
Revised: 9 and 10, we also provide qualitative visualization of results for the same 6 validation images from Fig.

Original: 6, with MDEQ-small at 4th, 6th, 8th, 10th, 14th, 18th, 22th, 26th iterations.
Revised: 6, with MDEQ-small at 4th, 6th, 8th, 10th, 14th, 18th, 22th, 26th iterations.

Original: With the progression of iterations, the predictions get more and more accurate.
Revised: With the progression of iterations, the predictions get more and more accurate.

Original: [Table]
Revised: [Table]