Original: I T N ET : ITERATIVE NEURAL NETWORKS FOR FAST AND EFFICIENT ANYTIME PREDICTION
Revised: I T N ET : ITERATIVE NEURAL NETWORKS FOR FAST AND EFFICIENT ANYTIME PREDICTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep neural networks have usually to be compressed and accelerated for their usage in low-power, e.g. mobile, devices.
Revised: Deep neural networks have usually to be compressed and accelerated for their usage in low-power, e.g. mobile, devices.

Original: Common requirements are high accuracy, high throughput, low latency, and a small memory footprint.
Revised: Common requirements are high accuracy, high throughput, low latency, and a small memory footprint.

Original: A good trade-off between accuracy and latency has been shown by networks comprising multiple intermediate outputs.
Revised: A good trade-off between accuracy and latency has been shown by networks comprising multiple intermediate outputs.

Original: In this study, we introduce a multi-output network that has a tiny memory footprint in terms of its computational graph, which allows its execution on novel, massively-parallel hardware accelerators designed for extremely high throughput.
Revised: In this study, we introduce a multi-output network that has a tiny memory footprint in terms of its computational graph, which allows its execution on novel, massively-parallel hardware accelerators designed for extremely high throughput.

Original: To this end, the graph is designed to contain loops by iteratively executing a single network building block.
Revised: To this end, the graph is designed to contain loops by iteratively executing a single network building block.

Original: These so-called iterative neural networks enable stateof-the-art results for semantic segmentation on the CamVid and Cityscapes datasets that are especially demanding in terms of computational resources.
Revised: These so-called iterative neural networks enable stateof-the-art results for semantic segmentation on the CamVid and Cityscapes datasets that are especially demanding in terms of computational resources.

Original: In ablation studies, the improvement of network training by intermediate network outputs as well as the trade-off between weight sharing over iterations and the network size are investigated.
Revised: In ablation studies, the improvement of network training by intermediate network outputs as well as the trade-off between weight sharing over iterations and the network size are investigated.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: For massively-parallel hardware accelerators (Schemmel et al., 2010; Merolla et al., 2014; Yao et al., 2020; Graphcore, 2020a), every neuron and synapse in the network model has its physical counterpart on the hardware system.
Revised: For massively-parallel hardware accelerators (Schemmel et al., 2010; Merolla et al., 2014; Yao et al., 2020; Graphcore, 2020a), every neuron and synapse in the network model has its physical counterpart on the hardware system.

Original: Usually, by design, memory and computation is not separated anymore, but neuron activations are computed next to the memory, i.e. the parameters, and fully in parallel.
Revised: Usually, by design, memory and computation is not separated anymore, but neuron activations are computed next to the memory, i.e. the parameters, and fully in parallel.

Original: This is in contrast to the rather sequential data processing of CPUs and GPUs, for which the computation of a network model is tiled and the same arithmetic unit is re-used multiple times for different neurons.
Revised: This is in contrast to the rather sequential data processing of CPUs and GPUs, for which the computation of a network model is tiled and the same arithmetic unit is re-used multiple times for different neurons.

Original: Since the computation is performed fully in parallel and in memory, the throughput of massively-parallel accelerators is usually much higher than for CPUs and GPUs.
Revised: Since the computation is performed fully in parallel and in memory, the throughput of massively-parallel accelerators is usually much higher than for CPUs and GPUs.

Original: This can be attributed to the fact that the latency and power consumption for accessing local memory, like for in-memory computing, are much lower than for computations on CPUs and GPUs that require the frequent access to non-local memory like DRAM (Sze et al., 2017).
Revised: This can be attributed to the fact that the latency and power consumption for accessing local memory, like for in-memory computing, are much lower than for computations on CPUs and GPUs that require the frequent access to non-local memory like DRAM (Sze et al., 2017).

Original: However, the network graph has to ﬁt into the memory of the massively-parallel hardware accelerators to allow for maximal throughput.
Revised: However, the network graph has to ﬁt into the memory of the massively-parallel hardware accelerators to allow for maximal throughput.

Original: If the network graph exceeds the available memory, in principle, the hardware has to be re-conﬁgured at high frequency as it is the case for CPUs and GPUs and the throughput would be substantially reduced.
Revised: If the network graph exceeds the available memory, in principle, the hardware has to be re-conﬁgured at high frequency as it is the case for CPUs and GPUs and the throughput would be substantially reduced.

Original: Mixed-signal massively-parallel hardware systems usually operate on shorter time scales than digital ones, e.g. compare Schemmel et al.
Revised: Mixed-signal massively-parallel hardware systems usually operate on shorter time scales than digital ones, e.g. compare Schemmel et al.

Original: (2010) and Yao et al.
Revised: (2010) and Yao et al.

Original: (2020) to Merolla et al.
Revised: (2020) to Merolla et al.

Original: (2014) and Graphcore (2020a), and would allow for even higher throughputs.
Revised: (2014) and Graphcore (2020a), and would allow for even higher throughputs.

Original: In order to achieve neural networks with tiny computational graphs, in which nodes are operations and edges are activations, we heavily re-use a single building block of the network (see the iterative block in Figure 1a).
Revised: In order to achieve neural networks with tiny computational graphs, in which nodes are operations and edges are activations, we heavily re-use a single building block of the network (see the iterative block in Figure 1a).

Original: Not only the structure of computations, i.e. the type of network layers including their dimensions and connectivity, is identical for each iteration of this building block, but also the weights are shared between iterations.
Revised: Not only the structure of computations, i.e. the type of network layers including their dimensions and connectivity, is identical for each iteration of this building block, but also the weights are shared between iterations.

Original: In the computational graphs of these so-called iterative neural networks (ItNet), the re-used building blocks with shared weights can be represented by nodes with self-loops.
Revised: In the computational graphs of these so-called iterative neural networks (ItNet), the re-used building blocks with shared weights can be represented by nodes with self-loops.

Original: Compared to conventional feed-forward networks, loops simplify the graph by reducing the number of unique nodes and, consequently, its computational footprint.
Revised: Compared to conventional feed-forward networks, loops simplify the graph by reducing the number of unique nodes and, consequently, its computational footprint.

Original: However, the restriction of sharing weights usually decreases the number of free parameters and, hence, the accuracy of networks.
Revised: However, the restriction of sharing weights usually decreases the number of free parameters and, hence, the accuracy of networks.

Original: To isolate and quantify this effect we compare networks with weight sharing to networks, for which the parameters of the building blocks are chosen to be independent between iterations of
Revised: To isolate and quantify this effect we compare networks with weight sharing to networks, for which the parameters of the building blocks are chosen to be independent between iterations of

Original: [Figure]
Revised: [Figure]

Original: 75 th shown.

Original: the building block.
Revised: the building block.

Original: In contrast to the above proposal, conventional deep neural networks for image processing usually do not share weights and have no (e.g. Huang et al., 2017) or few (e.g. one building block for each scale like by Greff et al., 2017) layers of identical structure.
Revised: In contrast to the above proposal, conventional deep neural networks for image processing usually do not share weights and have no (e.g. Huang et al., 2017) or few (e.g. one building block for each scale like by Greff et al., 2017) layers of identical structure.

Original: Liao & Poggio (2016) share weights between re-used building blocks, but use multiple unique building blocks.
Revised: Liao & Poggio (2016) share weights between re-used building blocks, but use multiple unique building blocks.

Original: To improve the training of networks, which contain loops in their graphs, and to reduce the latency of networks during inference we use multiple intermediate outputs.
Revised: To improve the training of networks, which contain loops in their graphs, and to reduce the latency of networks during inference we use multiple intermediate outputs.

Original: Multi-output networks that heavily re-use intermediate activations are beneﬁcial for a wide range of applications, especially in the mobile domain.
Revised: Multi-output networks that heavily re-use intermediate activations are beneﬁcial for a wide range of applications, especially in the mobile domain.

Original: In an online manner, they allow to trade off latency versus accuracy with barely any overhead (e.g. Huang et al., 2018).
Revised: In an online manner, they allow to trade off latency versus accuracy with barely any overhead (e.g. Huang et al., 2018).

Original: From an application point of view, the beneﬁt of this trade-off can be best described in the following two scenarios (Huang et al., 2018):
Revised: From an application point of view, the beneﬁt of this trade-off can be best described in the following two scenarios (Huang et al., 2018):

Original: In the so-called anytime prediction scenario, the prediction of a network is progressively updated, whereas the ﬁrst output deﬁnes the initial latency of the network.
Revised: In the so-called anytime prediction scenario, the prediction of a network is progressively updated, whereas the ﬁrst output deﬁnes the initial latency of the network.

Original: In a second scenario, a limited computational budget can be unevenly distributed over a set of samples with different "difﬁculties" in order to increase the average accuracy.
Revised: In a second scenario, a limited computational budget can be unevenly distributed over a set of samples with different "difﬁculties" in order to increase the average accuracy.

Original: Since all nodes in the network graph are computed in parallel on massively-parallel hardware systems (e.g. Esser et al., 2016), the latency for inference is dominated by the depth of the network, i.e. the longest path from input to output (Fischer et al., 2018).
Revised: Since all nodes in the network graph are computed in parallel on massively-parallel hardware systems (e.g. Esser et al., 2016), the latency for inference is dominated by the depth of the network, i.e. the longest path from input to output (Fischer et al., 2018).

Original: Consequently, we prefer networks that compute all scales in parallel (similar to Huang et al., 2018; Ke et al., 2017) and increase their depth for each additional intermediate output to networks that keep the depth constant and progressively increase their width (Yu et al., 2019).
Revised: Consequently, we prefer networks that compute all scales in parallel (similar to Huang et al., 2018; Ke et al., 2017) and increase their depth for each additional intermediate output to networks that keep the depth constant and progressively increase their width (Yu et al., 2019).

Original: Furthermore, multi-scale networks are also beneﬁcial for the integration of global information, as especially required by dense prediction tasks like semantic segmentation (Zhao et al., 2017).
Revised: Furthermore, multi-scale networks are also beneﬁcial for the integration of global information, as especially required by dense prediction tasks like semantic segmentation (Zhao et al., 2017).

Original: To further reduce the latency we also reduce the depth of the building blocks for each scale.
Revised: To further reduce the latency we also reduce the depth of the building blocks for each scale.

Original: In deep learning literature, the computational costs are usually quantiﬁed by counting the parameters and/or the multiply-accumulate operations (MACs) required for the inference of a single sample.
Revised: In deep learning literature, the computational costs are usually quantiﬁed by counting the parameters and/or the multiply-accumulate operations (MACs) required for the inference of a single sample.

Original: For fully convolutional networks, the number of parameters is independent of the spatial resolution of the network’s input and the intermediate feature maps.
Revised: For fully convolutional networks, the number of parameters is independent of the spatial resolution of the network’s input and the intermediate feature maps.

Original: Especially for large inputs as commonly used for semantic segmentation, the number of parameters does not cover the main workload and is, hence, not suited as a measure for computational costs.
Revised: Especially for large inputs as commonly used for semantic segmentation, the number of parameters does not cover the main workload and is, hence, not suited as a measure for computational costs.

Original: MACs have the advantage that they can be easily calculated and are usually a good approximation for the latency and throughput on
Revised: MACs have the advantage that they can be easily calculated and are usually a good approximation for the latency and throughput on

Original: CPUs and even GPUs.
Revised: CPUs and even GPUs.

Original: However, for most novel hardware accelerators, not the MACs, but memory transfers are dominating the computational costs in terms of power consumption (Chen et al., 2016;
Revised: However, for most novel hardware accelerators, not the MACs, but memory transfers are dominating the computational costs in terms of power consumption (Chen et al., 2016;

Original: Sze et al., 2017; Chao et al., 2019).
Revised: Sze et al., 2017; Chao et al., 2019).

Original: These memory transfers are minimized on massivly-parallel hardware systems as long as the network graph ﬁts into the in-computation memory of these systems, i.e. the memory of their arithmetic units.
Revised: These memory transfers are minimized on massivly-parallel hardware systems as long as the network graph ﬁts into the in-computation memory of these systems, i.e. the memory of their arithmetic units.

Original: Since both the power consumption during inference and the production cost scale with the size of this memory, in addition to MACS, we also compare the size of the computational graphs between networks.
Revised: Since both the power consumption during inference and the production cost scale with the size of this memory, in addition to MACS, we also compare the size of the computational graphs between networks.

Original: Note that the practical beneﬁts of ItNets cannot be demonstrated on conventional CPUs or GPUs, since these hardware systems do not support the processing of neural networks in a fully-parallel and, hence, low-latency fashion.
Revised: Note that the practical beneﬁts of ItNets cannot be demonstrated on conventional CPUs or GPUs, since these hardware systems do not support the processing of neural networks in a fully-parallel and, hence, low-latency fashion.

Original: Note that, in this study, we focus on network models in the low-power regime of only few billion
Revised: Note that, in this study, we focus on network models in the low-power regime of only few billion

Original: MACs, while processing large-scale images as commonly used for semantic segmentation (for datasets, see Section 2.4).
Revised: MACs, while processing large-scale images as commonly used for semantic segmentation (for datasets, see Section 2.4).

Original: The key contributions of this study are:
Revised: The key contributions of this study are:

Original: • We introduce efﬁcient networks with tiny computational graphs that heavily re-use a single building block and provide multiple intermediate outputs (Sections 2.1, 2.2 and 2.5).
Revised: • We introduce efﬁcient networks with tiny computational graphs that heavily re-use a single building block and provide multiple intermediate outputs (Sections 2.1, 2.2 and 2.5).

Original: • We search for the best hyperparameters of this model and investigate the effect of multiple outputs and weight sharing on the network training (Sections 2.3, 3.1 and 3.2).
Revised: • We search for the best hyperparameters of this model and investigate the effect of multiple outputs and weight sharing on the network training (Sections 2.3, 3.1 and 3.2).

Original: • To our knowledge, we set the new state-of-the-art in terms of accuracy over the size of the computational graph and discuss the potential beneﬁts for low-power applications.
Revised: • To our knowledge, we set the new state-of-the-art in terms of accuracy over the size of the computational graph and discuss the potential beneﬁts for low-power applications.

Original: We will release the source code upon acceptance for publication.
Revised: We will release the source code upon acceptance for publication.

Original:  M ETHODS
Revised:  M ETHODS

Original: The following networks process images of size x × y × 3 and output N semantic maps m n of size x × y × C with C being the number of classes.
Revised: The following networks process images of size x × y × 3 and output N semantic maps m n of size x × y × C with C being the number of classes.

Original: 2. N ETWORK ARCHITECTURE
Revised: 2. N ETWORK ARCHITECTURE

Original: We are interested in network architectures with a small computational footprint facilitating their application in mobile devices.
Revised: We are interested in network architectures with a small computational footprint facilitating their application in mobile devices.

Original: To this end, we design a neural network that heavily re-uses the intermediate activations and weights (Figure 1a).
Revised: To this end, we design a neural network that heavily re-uses the intermediate activations and weights (Figure 1a).

Original: Conceptionally, the network model can be split into three main building blocks: the data block , the iterative block and the classiﬁcation block (for an overview and details, see Figure 1a and Figure 2, respectively).
Revised: Conceptionally, the network model can be split into three main building blocks: the data block , the iterative block and the classiﬁcation block (for an overview and details, see Figure 1a and Figure 2, respectively).

Original: While the data block is executed only once for each image, the iterative block can be executed multiple times in a row by feeding back its output as the input for the next iteration.
Revised: While the data block is executed only once for each image, the iterative block can be executed multiple times in a row by feeding back its output as the input for the next iteration.

Original: The classiﬁcation block outputs the prediction of the semantic map by processing the intermediate activations of the feedback signal.
Revised: The classiﬁcation block outputs the prediction of the semantic map by processing the intermediate activations of the feedback signal.

Original: While the weights of the iterative block are shared between iterations, the weights of the classiﬁcation block are unique for each iteration.
Revised: While the weights of the iterative block are shared between iterations, the weights of the classiﬁcation block are unique for each iteration.

Original: We consider the size of the computational graph and the number of
Revised: We consider the size of the computational graph and the number of

Original: MACs as meaningful indicators for the computational footprint (see also Section 1).
Revised: MACs as meaningful indicators for the computational footprint (see also Section 1).

Original: To obtain a high-accuracy network architecture under these objectives we reduce the size of the computational graph by introducing loops and optimize the following architectural hyperparameters: the number of scales L , the number of iterations N , and the number of bottleneck residual blocks K .
Revised: To obtain a high-accuracy network architecture under these objectives we reduce the size of the computational graph by introducing loops and optimize the following architectural hyperparameters: the number of scales L , the number of iterations N , and the number of bottleneck residual blocks K .

Original: 2. N ETWORK TRAINING
Revised: 2. N ETWORK TRAINING

Original: For training, we use a joint cost function for all outputs of the network:
Revised: For training, we use a joint cost function for all outputs of the network:

Original: [Equation]
Revised: [Equation]

Original: where c n is the categorical cross entropy between the true labels ˜ m n and the network predictions m n .
Revised: where c n is the categorical cross entropy between the true labels ˜ m n and the network predictions m n .

Original: The weight factors a n are normalized as follows: ¯ a n = a n / ( (cid:80) i a i ) .
Revised: The weight factors a n are normalized as follows: ¯ a n = a n / ( (cid:80) i a i ) .

Original: We use the Adam optimizer with β 1 = 0 .
Revised: We use the Adam optimizer with β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 999 and a learning rate 0 .
Revised: 999 and a learning rate 0 .

Original: 001 that we multiply with 0 .
Revised: 001 that we multiply with 0 .

Original: 1 after 70% and 85% of the number of overall training epochs.
Revised: 1 after 70% and 85% of the number of overall training epochs.

Original: We use a batch size of 8 and train the network for 2000 ( 4000 for Figure 5) and 900 for the CamVid and Cityscapes datasets, respectively.
Revised: We use a batch size of 8 and train the network for 2000 ( 4000 for Figure 5) and 900 for the CamVid and Cityscapes datasets, respectively.

Original: For Figure 3, Figure 4 and the appendix, we report the mean values and the errors of the means across 5 trials.
Revised: For Figure 3, Figure 4 and the appendix, we report the mean values and the errors of the means across 5 trials.

Original: For Figures 1 and 5, we report the trial with the highest peak accuracy overtrials.
Revised: For Figures 1 and 5, we report the trial with the highest peak accuracy overtrials.

Original: For the results shown in Figures 1 and 5, we use dropout with rate 0 .
Revised: For the results shown in Figures 1 and 5, we use dropout with rate 0 .

Original: 1 after the depth-wise convolutions in the bottleneck residual blocks and an L2 weight decay of 10 − 5 in all convolutional layers.
Revised: 1 after the depth-wise convolutions in the bottleneck residual blocks and an L2 weight decay of 10 − 5 in all convolutional layers.

Original: For all other results, we do not use dropout and weight decay.
Revised: For all other results, we do not use dropout and weight decay.

Original: 2. N ETWORK EVALUATION
Revised: 2. N ETWORK EVALUATION

Original: Throughout this study, we measure the quality of semantic segmentation by calculating the mean intersection-over-union (mIoU Jaccard, 1912), which is the ratio of the area of overlap and the area of union
Revised: Throughout this study, we measure the quality of semantic segmentation by calculating the mean intersection-over-union (mIoU Jaccard, 1912), which is the ratio of the area of overlap and the area of union

Original: [Equation]
Revised: [Equation]

Original: averaged over all classes.
Revised: averaged over all classes.

Original: We consider a network to perform well if it achieves a high mIoU while requiring few MACs.
Revised: We consider a network to perform well if it achieves a high mIoU while requiring few MACs.

Original: To this end, we calculate the area under the curve of the mIoU ( y n ) over MACs ( x n ) with output index n as follows:
Revised: To this end, we calculate the area under the curve of the mIoU ( y n ) over MACs ( x n ) with output index n as follows:

Original: [Equation]
Revised: [Equation]

Original: with ( x 0 , y 0 )
Revised: with ( x 0 , y 0 )

Original: = (0 , 0 . 00828) , where y 0 denotes the mIoU at chance level for the CamVid dataset.
Revised: = (0 , 0 . 00828) , where y 0 denotes the mIoU at chance level for the CamVid dataset.

Original: To compensate for different maximum numbers x N of MACs for different sets of hyperparameters, we normalize as follows:
Revised: To compensate for different maximum numbers x N of MACs for different sets of hyperparameters, we normalize as follows:

Original: AUC = AUC x N .
Revised: AUC = AUC x N .

Original: [Figure]
Revised: [Figure]

Original: The size of the computational graph is computed by accumulating the memory requirements of all nodes, i.e. network layers, in the network graph.
Revised: The size of the computational graph is computed by accumulating the memory requirements of all nodes, i.e. network layers, in the network graph.

Original: For each layer, the total required memory is the sum of the memory for parameters, input feature maps and output feature maps.
Revised: For each layer, the total required memory is the sum of the memory for parameters, input feature maps and output feature maps.

Original: The theoretical latency of a network if executed fully in parallel is determined by the length, i.e. the depth, of the path from input to output of this network (see also Section 1).
Revised: The theoretical latency of a network if executed fully in parallel is determined by the length, i.e. the depth, of the path from input to output of this network (see also Section 1).

Original: For both the size of the computational graph and the latency, we only consider convolutional layers like commonly done in literature (e.g. Paszke et al., 2016;
Revised: For both the size of the computational graph and the latency, we only consider convolutional layers like commonly done in literature (e.g. Paszke et al., 2016;

Original: Wu et al., 2018; Mehta et al., 2019).
Revised: Wu et al., 2018; Mehta et al., 2019).

Original: This means, we ignore other network layers like normalizations, activations, concatenations, additions and spatial resizing, for which we assume that they can be fused with the convolutional layers.
Revised: This means, we ignore other network layers like normalizations, activations, concatenations, additions and spatial resizing, for which we assume that they can be fused with the convolutional layers.

Original: 2. D ATASETS
Revised: 2. D ATASETS

Original: The CamVid dataset (Brostow et al., 2008) consists of 701 ( 367 for training, 101 for validation,for testing) annotated images ﬁlmed from a moving car.
Revised: The CamVid dataset (Brostow et al., 2008) consists of 701 ( 367 for training, 101 for validation,for testing) annotated images ﬁlmed from a moving car.

Original: We use the same static pre-processing as in
Revised: We use the same static pre-processing as in

Original: Badrinarayanan et al.
Revised: Badrinarayanan et al.

Original: (2017) to obtain images and semantic labels of size 480 × 360 and normalize the pixel values to the interval [0 , 1] by dividing all pixel values by 255 .
Revised: (2017) to obtain images and semantic labels of size 480 × 360 and normalize the pixel values to the interval [0 , 1] by dividing all pixel values by 255 .

Original: For online data augmentation during training, we horizontally ﬂip the pairs of images and labels at random and use random crops of size 480 × 352 .
Revised: For online data augmentation during training, we horizontally ﬂip the pairs of images and labels at random and use random crops of size 480 × 352 .

Original: The Cityscapes dataset (Cordts et al., 2016) consists of 3475 ( 2975 for training, 500 for validation) annotated images and we use the validation set for testing.
Revised: The Cityscapes dataset (Cordts et al., 2016) consists of 3475 ( 2975 for training, 500 for validation) annotated images and we use the validation set for testing.

Original: We resize the original images and semantic labels to 1024 × 512 and divide all pixel values by 127 . 5 and subtract 1 to obtain values in the interval [ − 1 , 1] .
Revised: We resize the original images and semantic labels to 1024 × 512 and divide all pixel values by 127 . 5 and subtract 1 to obtain values in the interval [ − 1 , 1] .

Original: For online data augmentation during training, we horizontally ﬂip the pairs of images and labels at random.
Revised: For online data augmentation during training, we horizontally ﬂip the pairs of images and labels at random.

Original: For both datasets, pixels with class labels not marked for training are ignored in the cost function.
Revised: For both datasets, pixels with class labels not marked for training are ignored in the cost function.

Original: 2. B ATCH NORMALIZATION
Revised: 2. B ATCH NORMALIZATION

Original: In case of independent parameters between iterations of the iterative block, i.e. w n +1 is independent from w n in Figure 1a, batch normalization improves the network training.
Revised: In case of independent parameters between iterations of the iterative block, i.e. w n +1 is independent from w n in Figure 1a, batch normalization improves the network training.

Original: However, in case of weight sharing, also sharing the parameters of batch normalization between iterations signiﬁcantly worsens network training.
Revised: However, in case of weight sharing, also sharing the parameters of batch normalization between iterations signiﬁcantly worsens network training.

Original: Since not sharing batch-normalization parameters would violate our idea to re-use the identical building block again and again, we place batch-normalization layers between the iterations of the iterative block (see Figure 2b).
Revised: Since not sharing batch-normalization parameters would violate our idea to re-use the identical building block again and again, we place batch-normalization layers between the iterations of the iterative block (see Figure 2b).

Original: For comparability, we use the same setup also for networks without weight sharing, although the average validation mIoU is slightly decreased compared to networks that instead use batch normalization after each convolution (Figure 3).
Revised: For comparability, we use the same setup also for networks without weight sharing, although the average validation mIoU is slightly decreased compared to networks that instead use batch normalization after each convolution (Figure 3).

Original: [Figure]
Revised: [Figure]

Original:  R ESULTS
Revised:  R ESULTS

Original: In order to obtain accurate networks with low computational costs we ﬁrst search for the best set of the architectural hyperparameters L , N and K (Section 3.1).
Revised: In order to obtain accurate networks with low computational costs we ﬁrst search for the best set of the architectural hyperparameters L , N and K (Section 3.1).

Original: Since we are also interested in the trade-off between weight sharing and the networks size we also consider networks without weight sharing in this search.
Revised: Since we are also interested in the trade-off between weight sharing and the networks size we also consider networks without weight sharing in this search.

Original: Then, we investigate the impact of intermediate losses on the network performance to ﬁnd the best set of weight factors for the loss function (Section 3.2).
Revised: Then, we investigate the impact of intermediate losses on the network performance to ﬁnd the best set of weight factors for the loss function (Section 3.2).

Original: Finally, for the found set of hyperparameters and weight factors, we show results of ItNets on the CamVid and
Revised: Finally, for the found set of hyperparameters and weight factors, we show results of ItNets on the CamVid and

Original: Cityscapes dataset (Section 3.3).
Revised: Cityscapes dataset (Section 3.3).

Original: 3. S EARCH FOR ARCHITECTURAL HYPERPARAMETERS
Revised: 3. S EARCH FOR ARCHITECTURAL HYPERPARAMETERS

Original: In order to compare ItNets with and without weight sharing between iterations of the iterative block our goal is to ﬁnd a common set of architectural hyperparameters for both scenarios.
Revised: In order to compare ItNets with and without weight sharing between iterations of the iterative block our goal is to ﬁnd a common set of architectural hyperparameters for both scenarios.

Original: We perform a grid search over the hyperparameters of our network model as described in Figures 1 and 2.
Revised: We perform a grid search over the hyperparameters of our network model as described in Figures 1 and 2.

Original: For each set of hyperparameters, we choose the largest possible number of channels f that results in a network with less than 2 . 2 billion MACs for the last output of the network on single samples of the CamVid dataset.
Revised: For each set of hyperparameters, we choose the largest possible number of channels f that results in a network with less than 2 . 2 billion MACs for the last output of the network on single samples of the CamVid dataset.

Original: Since we are interested in network architectures with a high mean intersection-over-union (mIoU) and a low number of MACs, we sort the architectures by their area under the curve (see AUC in Section 2.3) for both networks with independent and shared weights.
Revised: Since we are interested in network architectures with a high mean intersection-over-union (mIoU) and a low number of MACs, we sort the architectures by their area under the curve (see AUC in Section 2.3) for both networks with independent and shared weights.

Original: Networks with a peak performance max n ( mIoU n ) ≤ mIoU totmax − 0 .
Revised: Networks with a peak performance max n ( mIoU n ) ≤ mIoU totmax − 0 .

Original: 02 , where mIoU totmax is the highest peak performance over all sets of hyperparameters, are considered as not suitable for applications and are discarded.
Revised: 02 , where mIoU totmax is the highest peak performance over all sets of hyperparameters, are considered as not suitable for applications and are discarded.

Original: For comparability between networks with independent and shared weights, we choose the architecture with the lowest average rank across the two classes of networks and obtain the following set of hyperparameters: N = 16 iterations, L = 3 scales and K = 1 bottleneck residual blocks (for the results of the grid search, see Figures 6 and 7 in the appendix).
Revised: For comparability between networks with independent and shared weights, we choose the architecture with the lowest average rank across the two classes of networks and obtain the following set of hyperparameters: N = 16 iterations, L = 3 scales and K = 1 bottleneck residual blocks (for the results of the grid search, see Figures 6 and 7 in the appendix).

Original: Note that many iterations ( N = 16 ) result in a high re-usage of the iterative block, a low latency and a small computational graph.
Revised: Note that many iterations ( N = 16 ) result in a high re-usage of the iterative block, a low latency and a small computational graph.

Original: 3. M ULTI - OUTPUT TRAINING
Revised: 3. M ULTI - OUTPUT TRAINING

Original: We study the impact of additional intermediate network outputs on the network performance by training the same network architecture with different sets of weight factors a n of the loss function (for details, see Section 2.2).
Revised: We study the impact of additional intermediate network outputs on the network performance by training the same network architecture with different sets of weight factors a n of the loss function (for details, see Section 2.2).

Original: In summary, later network outputs beneﬁt from the additional loss
Revised: In summary, later network outputs beneﬁt from the additional loss

Original: [Figure]
Revised: [Figure]

Original: applied to earlier outputs.
Revised: applied to earlier outputs.

Original: This is supported by ablation studies, in which we trained networks with single, only late or thinned out outputs.
Revised: This is supported by ablation studies, in which we trained networks with single, only late or thinned out outputs.

Original: The following results and conclusions are similar for both network types with and without weight sharing and, hence, we discuss them jointly.
Revised: The following results and conclusions are similar for both network types with and without weight sharing and, hence, we discuss them jointly.

Original: The training of single outputs is worse than jointly training all outputs, except for only training the ﬁrst output (see Figure 4a and e).
Revised: The training of single outputs is worse than jointly training all outputs, except for only training the ﬁrst output (see Figure 4a and e).

Original: However, a network with only the ﬁrst output has no practical relevance due to its tiny network size and, consequently, low mIoU.
Revised: However, a network with only the ﬁrst output has no practical relevance due to its tiny network size and, consequently, low mIoU.

Original: The intermediate activations of the early layers are optimized for the potentially conﬂicting tasks of providing the basis for high accuracy at early outputs and, at the same time, top accuracy at late outputs.
Revised: The intermediate activations of the early layers are optimized for the potentially conﬂicting tasks of providing the basis for high accuracy at early outputs and, at the same time, top accuracy at late outputs.

Original: However, our experiments suggest that especially the optimization of the early outputs improves the performance of late outputs (see Figure 4b and f).
Revised: However, our experiments suggest that especially the optimization of the early outputs improves the performance of late outputs (see Figure 4b and f).

Original: Our observation that early losses do not conﬂict with late losses allows to keep these early losses resulting in networks with very low latency (see Figure 5d).
Revised: Our observation that early losses do not conﬂict with late losses allows to keep these early losses resulting in networks with very low latency (see Figure 5d).

Original: In addition, keeping the ﬁrst output to allow for low latency, but decreasing the density of outputs, removes unnecessary constraints and slightly improves the mIoU (see Figure 4c and g).
Revised: In addition, keeping the ﬁrst output to allow for low latency, but decreasing the density of outputs, removes unnecessary constraints and slightly improves the mIoU (see Figure 4c and g).

Original: The removed outputs are likely not required in applications, since the improvement of the mIoU between consecutive outputs is rather small (see Figure 1e).
Revised: The removed outputs are likely not required in applications, since the improvement of the mIoU between consecutive outputs is rather small (see Figure 1e).

Original: So far, we removed sets of outputs from the loss function, but kept the weight factors a n identical for all remaining outputs.
Revised: So far, we removed sets of outputs from the loss function, but kept the weight factors a n identical for all remaining outputs.

Original: For linearly increasing the weight factors over the outputs, the mIoU jointly decreases for all outputs (compare different colors in Figure 4d and h).
Revised: For linearly increasing the weight factors over the outputs, the mIoU jointly decreases for all outputs (compare different colors in Figure 4d and h).

Original: This unexpected decrease in the mIoU for late outputs may be attributed to the observed effect that earlier outputs are crucial for late performance as discussed above.
Revised: This unexpected decrease in the mIoU for late outputs may be attributed to the observed effect that earlier outputs are crucial for late performance as discussed above.

Original: 3. T HE COST OF WEIGHT SHARING
Revised: 3. T HE COST OF WEIGHT SHARING

Original: In the following, we address the cost of sharing weights in the iterative block.
Revised: In the following, we address the cost of sharing weights in the iterative block.

Original: Speciﬁcally, we compare the number of MACs, the number of parameters, the size of the computational graph and the latency between networks with and without weight sharing.
Revised: Speciﬁcally, we compare the number of MACs, the number of parameters, the size of the computational graph and the latency between networks with and without weight sharing.

Original: Compared to networks with independent weights, networks with shared weights have substantially less free parameters that, consequently, results in worse mIoUs.
Revised: Compared to networks with independent weights, networks with shared weights have substantially less free parameters that, consequently, results in worse mIoUs.

Original: To compensate for this decrease in the peak mIoUs we increase the width f of the latter by a factor of 1 .
Revised: To compensate for this decrease in the peak mIoUs we increase the width f of the latter by a factor of 1 .

Original: 76 ( f = 51 with weight sharing compared to f = 29 without weight sharing), which corresponds to approximately three times the number of MACs (Figure 3b).
Revised: 76 ( f = 51 with weight sharing compared to f = 29 without weight sharing), which corresponds to approximately three times the number of MACs (Figure 3b).

Original: Although the number of MACs has to be increased for weight sharing (Figure 5a), the number of parameters is substantially smaller ( 44% ), since parameters are highly re-used (Figure 5b).
Revised: Although the number of MACs has to be increased for weight sharing (Figure 5a), the number of parameters is substantially smaller ( 44% ), since parameters are highly re-used (Figure 5b).

Original: The size of the computational graph is dominated by the size of the feature maps due to their large spatial dimensions.
Revised: The size of the computational graph is dominated by the size of the feature maps due to their large spatial dimensions.

Original: If the weights are shared between iterations of the iterative block the previously unrolled network graph can be collapsed into a single node with a self-loop reducing the size of the computational graph (Figure 5c).
Revised: If the weights are shared between iterations of the iterative block the previously unrolled network graph can be collapsed into a single node with a self-loop reducing the size of the computational graph (Figure 5c).

Original: The latency is identical for all networks, since we use the same architectural hyperparameters (Figure 5d).
Revised: The latency is identical for all networks, since we use the same architectural hyperparameters (Figure 5d).

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: This study investigates the trade-off between the size of the computational graphs and the accuracies of so-called iterative neural networks that show state-of-the-art performance for this trade-off.
Revised: This study investigates the trade-off between the size of the computational graphs and the accuracies of so-called iterative neural networks that show state-of-the-art performance for this trade-off.

Original: Iterative networks require less than half the size for the computational graph compared to networks that are state-of-the-art in terms of the number of MACs (compare 276MB for ItNet to 639MB for ESPNetv to achieve an mIoU of 61 . 1 in Figure 1c).
Revised: Iterative networks require less than half the size for the computational graph compared to networks that are state-of-the-art in terms of the number of MACs (compare 276MB for ItNet to 639MB for ESPNetv to achieve an mIoU of 61 . 1 in Figure 1c).

Original: In addition, the multiple intermediate network outputs allow for any-time prediction and a lower latency that is further reduced by the shallow network design (Figure 1d).
Revised: In addition, the multiple intermediate network outputs allow for any-time prediction and a lower latency that is further reduced by the shallow network design (Figure 1d).

Original: While targeting the same accuracy with an identical network structure, the sharing of weights reduces the size of the graph by a factor larger than 4 (compare 83MB to 384MB to achieve an mIoU of approximately 64 . 2 in Figure 5c), but increases the number of MACs by a factor of approximately 3 (compare 2.0 to 6.2 billion MACs in Figure 5a).
Revised: While targeting the same accuracy with an identical network structure, the sharing of weights reduces the size of the graph by a factor larger than 4 (compare 83MB to 384MB to achieve an mIoU of approximately 64 . 2 in Figure 5c), but increases the number of MACs by a factor of approximately 3 (compare 2.0 to 6.2 billion MACs in Figure 5a).

Original: However, this increase in
Revised: However, this increase in

Original: MACs is expected to be smaller than the increase in throughput by novel, massively-parallel hardware accelerators, especially if the computational graphs ﬁt into the in-computation memory of these hardware systems (see also Section 1).
Revised: MACs is expected to be smaller than the increase in throughput by novel, massively-parallel hardware accelerators, especially if the computational graphs ﬁt into the in-computation memory of these hardware systems (see also Section 1).

Original: For example, Graphcore (2020b) reports a 300 -fold increase in throughput compared to GPUs for recurrent networks of small size and Esser et al.
Revised: For example, Graphcore (2020b) reports a 300 -fold increase in throughput compared to GPUs for recurrent networks of small size and Esser et al.

Original: (2016) report more than 5000 frames per second and Watt for computer vision tasks.
Revised: (2016) report more than 5000 frames per second and Watt for computer vision tasks.

Original: During the training of ItNets, we observe that additional and especially early network outputs improve the performance of late outputs, which was also observed by Zhou et al.
Revised: During the training of ItNets, we observe that additional and especially early network outputs improve the performance of late outputs, which was also observed by Zhou et al.

Original: (2019) on pre-trained networks for age estimation.
Revised: (2019) on pre-trained networks for age estimation.

Original: In principle, the presented methods could also be applied to different tasks, like object classiﬁcation and detection, for which we expect similar observations.
Revised: In principle, the presented methods could also be applied to different tasks, like object classiﬁcation and detection, for which we expect similar observations.

Original: However, these datasets are usually provided in a lower spatial resolution, which reduces the challenge and the need to reduce the size of these already rather small computational graphs.
Revised: However, these datasets are usually provided in a lower spatial resolution, which reduces the challenge and the need to reduce the size of these already rather small computational graphs.

Original: Since massively-parallel hardware systems are not easily accessible at the moment, we cannot provide real benchmark data, but will in the following exemplarily discuss the potential beneﬁts of our network models if executed on such hardware systems.
Revised: Since massively-parallel hardware systems are not easily accessible at the moment, we cannot provide real benchmark data, but will in the following exemplarily discuss the potential beneﬁts of our network models if executed on such hardware systems.

Original: To this end, we describe the execution of our model on the Graphcore (2020a) system, since this system seems to be the most promising in terms of commercialization and availability.
Revised: To this end, we describe the execution of our model on the Graphcore (2020a) system, since this system seems to be the most promising in terms of commercialization and availability.

Original: If the full network graph ﬁts into the in-computation memory of the Graphcore (2020a) system (3.6GB for the IPU-M2000 machine), the input data can be pipelined through this graph in a streaming fashion without the need to reconﬁgure the hardware.
Revised: If the full network graph ﬁts into the in-computation memory of the Graphcore (2020a) system (3.6GB for the IPU-M2000 machine), the input data can be pipelined through this graph in a streaming fashion without the need to reconﬁgure the hardware.

Original: Then, by design, independent nodes in this graph are executed fully in parallel (see also Fischer et al., 2018).
Revised: Then, by design, independent nodes in this graph are executed fully in parallel (see also Fischer et al., 2018).

Original: This results in substantially lower latencies during network inference compared to systems that do not execute network graphs fully in parallel like CPUs and GPUs (an up to 25 times lower latency is reported by Graphcore, 2020b).
Revised: This results in substantially lower latencies during network inference compared to systems that do not execute network graphs fully in parallel like CPUs and GPUs (an up to 25 times lower latency is reported by Graphcore, 2020b).

Original: In contrast, for CPUs and GPUs, the workload of the network graph has to be tiled and the arithmetic units are continuously reconﬁgured for each tile.
Revised: In contrast, for CPUs and GPUs, the workload of the network graph has to be tiled and the arithmetic units are continuously reconﬁgured for each tile.

Original: ItNets signiﬁcantly reduce the footprint of their graphs by introducing loops and, hence, improve the network’s accuracy for the same footprint.
Revised: ItNets signiﬁcantly reduce the footprint of their graphs by introducing loops and, hence, improve the network’s accuracy for the same footprint.

Original: Although the in-computation memory is huge for the
Revised: Although the in-computation memory is huge for the

Original: IPU-M2000 machine that is optimized for data centers
Revised: IPU-M2000 machine that is optimized for data centers

Original: , we expect embedded versions of similar systems to have a considerably smaller in-computation memory that, then, require networks with tiny computational graphs.
Revised: , we expect embedded versions of similar systems to have a considerably smaller in-computation memory that, then, require networks with tiny computational graphs.

Original: Other massively-parallel hardware designs increase the density of the memory in terms of synapses per chip area by aggressive quantization (Merolla et al., 2014) or mixed-signal implementations (Schemmel et al., 2010; Yao et al., 2020) and impose additional challenges for the development of network models.
Revised: Other massively-parallel hardware designs increase the density of the memory in terms of synapses per chip area by aggressive quantization (Merolla et al., 2014) or mixed-signal implementations (Schemmel et al., 2010; Yao et al., 2020) and impose additional challenges for the development of network models.

Original: For the Cityscapes dataset, our ItNet already requires two Nvidia V100 GPU with 32GB memory if the standard training procedure is not modiﬁed.
Revised: For the Cityscapes dataset, our ItNet already requires two Nvidia V100 GPU with 32GB memory if the standard training procedure is not modiﬁed.

Original: Consequently, exploring large-scale
Revised: Consequently, exploring large-scale

Original: ItNets is difﬁcult by using backpropagation on GPUs.
Revised: ItNets is difﬁcult by using backpropagation on GPUs.

Original: However, since ItNets are optimized for their execution on massively-parallel hardware systems, this demonstrates the fundamentally different operation principles between these systems and GPUs as well as highlights the need to think beyond workloads that are tailored for GPUs.
Revised: However, since ItNets are optimized for their execution on massively-parallel hardware systems, this demonstrates the fundamentally different operation principles between these systems and GPUs as well as highlights the need to think beyond workloads that are tailored for GPUs.

Original: In parallel to this study, a novel training technique was introduced by Bai et al.
Revised: In parallel to this study, a novel training technique was introduced by Bai et al.

Original: (2020) that signiﬁcantly improves the scaling of vision networks composed of iteratively executed building blocks with weight sharing.
Revised: (2020) that signiﬁcantly improves the scaling of vision networks composed of iteratively executed building blocks with weight sharing.

Original: Instead of using backpropagation, they optimize the equilibrium point of a ﬁx-point process described by the iterative execution of the building block.
Revised: Instead of using backpropagation, they optimize the equilibrium point of a ﬁx-point process described by the iterative execution of the building block.

Original: They achieved task accuracies comparable to state-of-the-art networks and, to this end, required approximately the same number of parameters compared to the baselines.
Revised: They achieved task accuracies comparable to state-of-the-art networks and, to this end, required approximately the same number of parameters compared to the baselines.

Original: In line with our study, this results in comparably large building blocks that are executed many times and, consequently, require signiﬁcantly more MACs than the baseline networks.
Revised: In line with our study, this results in comparably large building blocks that are executed many times and, consequently, require signiﬁcantly more MACs than the baseline networks.

Original: However, the training method of Bai et al.
Revised: However, the training method of Bai et al.

Original: (2020) can not be applied to networks, for which the weights are allowed to be updated between iterations and, hence, the effect of weight sharing can not be studied in isolation.
Revised: (2020) can not be applied to networks, for which the weights are allowed to be updated between iterations and, hence, the effect of weight sharing can not be studied in isolation.

Original: In addition, their method includes the raw image (8MB in float32 for
Revised: In addition, their method includes the raw image (8MB in float32 for

Original: Cityscapes) into the state of the iterative block, which prevents to shrink their networks to the regime of few billion MACs as shown by our method, for which the state has a size of only 4.8MB.
Revised: Cityscapes) into the state of the iterative block, which prevents to shrink their networks to the regime of few billion MACs as shown by our method, for which the state has a size of only 4.8MB.

Original: The concept of recurrent building blocks has also been previously applied to semantic segmentation ( ? ) and to generate super-resolution images (Cheng et al., 2018).
Revised: The concept of recurrent building blocks has also been previously applied to semantic segmentation ( ? ) and to generate super-resolution images (Cheng et al., 2018).

Original: However, these works share the same network building block between different scales of the feature maps, which does not allow for anytime predictions.
Revised: However, these works share the same network building block between different scales of the feature maps, which does not allow for anytime predictions.

Original: Instead of processing all scales at once like in Bai et al.
Revised: Instead of processing all scales at once like in Bai et al.

Original: (2020) and the work at hand, a full encoder-decoder network can be shared over time.
Revised: (2020) and the work at hand, a full encoder-decoder network can be shared over time.

Original: To predict semantic maps for single images (Wang et al., 2019) or videos (Valipour et al., 2017) the intermediate feature maps of this encoder-decoder network are connected by recurrent units over time.
Revised: To predict semantic maps for single images (Wang et al., 2019) or videos (Valipour et al., 2017) the intermediate feature maps of this encoder-decoder network are connected by recurrent units over time.

Original: Ballas et al.
Revised: Ballas et al.

Original: (2016) apply a similar technique to networks with encoders only.
Revised: (2016) apply a similar technique to networks with encoders only on video tasks.

Original: As an extension of this study, partly releasing the constraint of weight sharing may allow for a signiﬁcant reduction in the required size of the iterative block.
Revised: As an extension of this study, partly releasing the constraint of weight sharing may allow for a signiﬁcant reduction in the required size of the iterative block.

Original: To this end, a ratio between shared and free weights may be chosen close to, but smaller than one.
Revised: To this end, a ratio between shared and free weights may be chosen close to, but smaller than one.

Original: This will require to reconﬁgure the network graph on hardware, which may be costly or not possible at all depending on the hardware system.
Revised: This will require to reconﬁgure the network graph on hardware, which may be costly or not possible at all depending on the hardware system.

Original: An alternative approach to release constraints would be to replace the weights of the building block by a functions that modulates the weights over the iterations of the building block.
Revised: An alternative approach to release constraints would be to replace the weights of the building block by a functions that modulates the weights over the iterations of the building block.

Original: Another interesting open question for further studies is the root cause for the observation that early outputs improve the performance of late outputs.
Revised: Another interesting open question for further studies is the root cause for the observation that early outputs improve the performance of late outputs.

Original: This effect may be attributed to some kind of knowledge distillation and / or to the shortcut for the gradients from network output to input.
Revised: This effect may be attributed to some kind of knowledge distillation and / or to the shortcut for the gradients from network output to input.