Original: C UTTING D OWN T RAINING M EMORY BY R E - FOWARDING
Revised: C UTTING D OWN T RAINING M EMORY BY R E - FOWARDING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Deep Neutral Networks(DNNs) require huge GPU memory when training on modern image/video databases.
Revised: Deep Neutral Networks(DNNs) require huge GPU memory when training on modern image/video databases.

Original: Unfortunately, the GPU memory as a hardware resource is always ﬁnite, which limits the image resolution, batch size, and learning rate that could be used for better DNN performance.
Revised: Unfortunately, the GPU memory as a hardware resource is always ﬁnite, which limits the image resolution, batch size, and learning rate that could be used for better DNN performance.

Original: In this paper, we propose a novel training approach, called Re-forwarding , that substantially reduces memory usage in training.
Revised: In this paper, we propose a novel training approach, called Re-forwarding , that substantially reduces memory usage in training.

Original: Our approach automatically ﬁnds a subset of layers in DNNs, and stores tensors only at these layers during the ﬁrst forward.
Revised: Our approach automatically ﬁnds a subset of layers in DNNs, and stores tensors only at these layers during the ﬁrst forward.

Original: During backward, extra local forwards (called the Re-forwarding process) are conducted to compute the missing tensors between the subset of layers.
Revised: During backward, extra local forwards (called the Re-forwarding process) are conducted to compute the missing tensors between the subset of layers.

Original: The total memory cost becomes the sum of (1) the memory cost at the subset of layers and (2) the maximum memory cost among local re-forwards.
Revised: The total memory cost becomes the sum of (1) the memory cost at the subset of layers and (2) the maximum memory cost among local re-forwards.

Original: Re-forwarding trades training time for memory and does not compromise any performance in testing.
Revised: Re-forwarding trades training time for memory and does not compromise any performance in testing.

Original: We propose theories and algorithms that achieve the optimal memory solutions for
Revised: We propose theories and algorithms that achieve the optimal memory solutions for

Original: DNNs with either linear or arbitrary computation graphs.
Revised: DNNs with either linear or arbitrary computation graphs.

Original: Experiments show that
Revised: Experiments show that

Original: Re-forwarding cuts down up-to 80% of training memory on popular DNNs such as Alexnet, VGG, ResNet, Densenet and Inception net.
Revised: Re-forwarding cuts down up-to 80% of training memory on popular DNNs such as Alexnet, VGG, ResNet, Densenet and Inception net.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The standard DNN training process consists of two alternated stages: forward and backward.
Revised: The standard DNN training process consists of two alternated stages: forward and backward.

Original: Fig.(a) illustrates an example of feed-forward neural networks.
Revised: Fig.(a) illustrates an example of feed-forward neural networks.

Original: In the forward stage, the network takes an input tensor, [ BatchSize × Channel × Width × Height ] , and computes the tensors at each layer until producing the output.
Revised: In the forward stage, the network takes an input tensor, [ BatchSize × Channel × Width × Height ] , and computes the tensors at each layer until producing the output.

Original: In the backward stage, difference between the output and ground truth is passed back along the network to compute the gradients at each layer.
Revised: In the backward stage, difference between the output and ground truth is passed back along the network to compute the gradients at each layer.

Original: The regular training approach saves tensors at all layers during forward, because they are all needed to compute gradients during backward.
Revised: The regular training approach saves tensors at all layers during forward, because they are all needed to compute gradients during backward.

Original: The total memory cost is the sum of cost over all layers.
Revised: The total memory cost is the sum of cost over all layers.

Original: In popular backbone DNNs for feature extraction of images, such as AlexNet (Krizhevsky et al.
Revised: In popular backbone DNNs for feature extraction of images, such as AlexNet (Krizhevsky et al.

Original: , VGG (Simonyan & Zisserman (2014)) and ResNet (He et al.
Revised: , VGG (Simonyan & Zisserman (2014)) and ResNet (He et al.

Original: , the memory cost increases quadratically with the input image resolution and network depth.
Revised: , the memory cost increases quadratically with the input image resolution and network depth.

Original: For example, given an median size input tensor of (32 , 3 , 224 , 224) , ResNet101 requires around 5000 MB.
Revised: For example, given an median size input tensor of (32 , 3 , 224 , 224) , ResNet101 requires around 5000 MB.

Original: In more challenging tasks, DNNs that detect small objects and large number of object categories require input image resolution of more than 600 × 600 (Ren et al.
Revised: In more challenging tasks, DNNs that detect small objects and large number of object categories require input image resolution of more than 600 × 600 (Ren et al.

Original: ; Singh et al.
Revised: ; Singh et al.

Original: ; Redmon & Farhadi (2018)).
Revised: ; Redmon & Farhadi (2018)).

Original: The memory issue is worse for video-based DNNs, such as CDC (Shou et al.
Revised: The memory issue is worse for video-based DNNs, such as CDC (Shou et al.

Original: (2017)), C3D (Ji et al.
Revised: (2017)), C3D (Ji et al.

Original: and 3D-ResNet (Hara et al.
Revised: and 3D-ResNet (Hara et al.

Original: To model complex activities in video, the input tensor may contain 64 frames.
Revised: To model complex activities in video, the input tensor may contain 64 frames.

Original: Moreover, DNN training takes much more memory than testing.
Revised: Moreover, DNN training takes much more memory than testing.

Original: In order to train DNNs with large databases and big learning rate, the batch size can be up to 64 .
Revised: In order to train DNNs with large databases and big learning rate, the batch size can be up to 64 .

Original: In training DNN compositions, such as Generative adversarial networks (GANs), multiple generator and discriminator networks are simultaneously stored in GPU memory.
Revised: In training DNN compositions, such as Generative adversarial networks (GANs), multiple generator and discriminator networks are simultaneously stored in GPU memory.

Original: Existing efforts to address memory issues presented three main approaches: (1) Better single GPUs.
Revised: Existing efforts to address memory issues presented three main approaches: (1) Better single GPUs.

Original: Recent GPUs provide larger memory at the expense of exponentially growing price and power consumption.
Revised: Recent GPUs provide larger memory at the expense of exponentially growing price and power consumption.

Original: For instance, from TitanXp, Quadro P6000 to Tesla V100, for 1-2.7 times increase in memory, the prices increase 2.8-8.5 times.
Revised: For instance, from TitanXp, Quadro P6000 to Tesla V100, for 1-2.7 times increase in memory, the prices increase 2.8-8.5 times.

Original: (2) Parallelization among multiple GPUs (Dean et al.
Revised: (2) Parallelization among multiple GPUs (Dean et al.

Original: (2012); Shi et al.
Revised: (2012); Shi et al.

Original: ; Langford et al.
Revised: ; Langford et al.

Original: ; Mcdonald et al. ; McDonald et al.
Revised: ; Mcdonald et al. ; McDonald et al.

Original: Zinkevich et al.
Revised: Zinkevich et al.

Original: ; Agarwal et al.
Revised: ; Agarwal et al.

Original: (2014); Agarwal & Duchi (2011)), which requires expensive clusters, introduces substantial I/O cost, and does not reduce the total memory cost.
Revised: (2014); Agarwal & Duchi (2011)), which requires expensive clusters, introduces substantial I/O cost, and does not reduce the total memory cost.

Original: (3) Low-level heuristic techniques.
Revised: (3) Low-level heuristic techniques.

Original: Optimization of computation graphs (Aho et al.
Revised: Optimization of computation graphs (Aho et al.

Original: (1986)), which merges inplace operations into non-inplace operations to cut down memory.
Revised: (1986)), which merges inplace operations into non-inplace operations to cut down memory.

Original: Liveness analysis (Aho et al.
Revised: Liveness analysis (Aho et al.

Original: (1986)), which dynamically recycles garbage tensors in training epochs.
Revised: (1986)), which dynamically recycles garbage tensors in training epochs.

Original: These approaches are speciﬁc to certain DNN structures, data and tasks.
Revised: These approaches are speciﬁc to certain DNN structures, data and tasks.

Original: To address above issues, we propose a fundamental approach that explores trade-off between memory and computation power of GPUs.
Revised: To address above issues, we propose a fundamental approach that explores trade-off between memory and computation power of GPUs.

Original: Note that recent affordable GPUs, although limited in memory ( 12GB), provide exceptional improvement in GPU cores and FLOPS.
Revised: Note that recent affordable GPUs, although limited in memory ( 12GB), provide exceptional improvement in GPU cores and FLOPS.

Original: Trading computational time for memory is a very attractive solution that make it possible to train very heavy DNNs with ﬁnite
Revised: Trading computational time for memory is a very attractive solution that make it possible to train very heavy DNNs with ﬁnite

Original: GPU memory.
Revised: GPU memory.

Original: Our approach only saves tensors at a subset of layers during the ﬁrst forward, and conduct only extra local forwards to compute the missing tensors needed during backward.
Revised: Our approach only saves tensors at a subset of layers during the ﬁrst forward, and conduct only extra local forwards to compute the missing tensors needed during backward.

Original: We call the extra forward process as Re-forwarding .
Revised: We call the extra forward process as Re-forwarding .

Original: The total memory cost is the sum of (1) the cost at the subset of layers and (2) the maximum memory cost among local re-forwards.
Revised: The total memory cost is the sum of (1) the cost at the subset of layers and (2) the maximum memory cost among local re-forwards.

Original: Training with Reforwarding, see Fig.
Revised: Training with Reforwarding, see Fig.

Original: 1 (b), leads to substantial memory reduction.
Revised: 1 (b), leads to substantial memory reduction.

Original: We propose sophisticate theories and efﬁcient algorithms that achieve the optimal memory solution of arbitrary computation graphs.
Revised: We propose sophisticate theories and efﬁcient algorithms that achieve the optimal memory solution of arbitrary computation graphs.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: To alleviate the memory pressure from a single GPU processor, many researchers utilized the wellestablished techniques for distributed computation (Dean et al.
Revised: To alleviate the memory pressure from a single GPU processor, many researchers utilized the wellestablished techniques for distributed computation (Dean et al.

Original: ; Shi et al.
Revised: ; Shi et al.

Original: ; Langford et al.
Revised: ; Langford et al.

Original: ; Mcdonald et al. ; McDonald et al.
Revised: ; Mcdonald et al. ; McDonald et al.

Original: ; Zinkevich et al.
Revised: ; Zinkevich et al.

Original: ; Agarwal et al.
Revised: ; Agarwal et al.

Original: (2014); Agarwal & Duchi (2011)).
Revised: (2014); Agarwal & Duchi (2011)).

Original: These techniques distribute memory pressure to possibly inﬁnite
Revised: These techniques distribute memory pressure to possibly inﬁnite

Original: GPUs or server clusters, but do not reduce the total memory cost of DNNs.
Revised: GPUs or server clusters, but do not reduce the total memory cost of DNNs.

Original: Other researchers reduced the memory on ﬁnite hardware by optimizing computation graph of DNN and performing liveness analysis.
Revised: Other researchers reduced the memory on ﬁnite hardware by optimizing computation graph of DNN and performing liveness analysis.

Original: The computation graph of DNNs describes the dependencies of tensors among layers.
Revised: The computation graph of DNNs describes the dependencies of tensors among layers.

Original: Liveness analysis recycles garbage to manage memory.
Revised: Liveness analysis recycles garbage to manage memory.

Original: These ideas were originated from compiler optimization (Aho et al.
Revised: These ideas were originated from compiler optimization (Aho et al.

Original: and has been widely adopted by deep learning frameworks: Theano (Bastien et al.
Revised: and has been widely adopted by deep learning frameworks: Theano (Bastien et al.

Original: ; Bergstra et al.
Revised: ; Bergstra et al.

Original: , MXNet (Chen et al.
Revised: , MXNet (Chen et al.

Original: , Tensorﬂow (Abadi et al. (2016)) and CNTK (Yu et al. (2014)).
Revised: , Tensorﬂow (Abadi et al. (2016)) and CNTK (Yu et al. (2014)).

Original: Some other techniques efﬁciently swap data between CPU and GPU (Wang et al.
Revised: Some other techniques efﬁciently swap data between CPU and GPU (Wang et al.

Original: ; Rhu et al.
Revised: ; Rhu et al.

Original: These techniques usually cost extra I/
Revised: These techniques usually cost extra I/

Original: O time and still do not actually reduce the total memory cost.
Revised: O time and still do not actually reduce the total memory cost.

Original: The closest work to our approach, Chen et al.(Chen et al.
Revised: The closest work to our approach, Chen et al.(Chen et al.

Original: (2016)), uses the gradient checkpoints (similar to the subset of layers in Re-forwarding).
Revised: (2016)), uses the gradient checkpoints (similar to the subset of layers in Re-forwarding).

Original: However, (Chen et al.
Revised: However, (Chen et al.

Original: (2016)) only worked on linear computation graph via a heuristic algorithm.
Revised: (2016)) only worked on linear computation graph via a heuristic algorithm.

Original: Our approach generates optimal solutions for both linear and arbitrary computation graphs .
Revised: Our approach generates optimal solutions for both linear and arbitrary computation graphs .

Original: Our algorithm reduces training memory by manipulating high-level tensors, therefore is generalizable to any DNNs and their compositions.
Revised: Our algorithm reduces training memory by manipulating high-level tensors, therefore is generalizable to any DNNs and their compositions.

Original: All previous techniques are compatible to our approach and can further improve the memory efﬁciency of DNN training.
Revised: All previous techniques are compatible to our approach and can further improve the memory efﬁciency of DNN training.

Original:  L INEAR C OMPUTATION G RAPH (LCG)
Revised:  L INEAR C OMPUTATION G RAPH (LCG)

Original: Denote a computation graph as G = (cid:0) E, V (cid:1) .
Revised: Denote a computation graph as G = (cid:0) E, V (cid:1) .

Original: E = { e i } and V = { v i } are the edges and vertexes in the computation graph, respectively.
Revised: E = { e i } and V = { v i } are the edges and vertexes in the computation graph, respectively.

Original: In deep neural networks, the vertexes represent the tensors and the edges represent operations.
Revised: In deep neural networks, the vertexes represent the tensors and the edges represent operations.

Original: Denote function l ( · ) as a measure of memory cost.
Revised: Denote function l ( · ) as a measure of memory cost.

Original: V R is the subset of vertexes saved during the ﬁrst forward.
Revised: V R is the subset of vertexes saved during the ﬁrst forward.

Original: l ( v i ) is deﬁned as the memory cost of storing vertex v i .
Revised: l ( v i ) is deﬁned as the memory cost of storing vertex v i .

Original: For two adjacent vertexes v i and v j in set V R , the memory cost during re-forwarding from v i to v j is deﬁned as l ( v i , v j )
Revised: For two adjacent vertexes v i and v j in set V R , the memory cost during re-forwarding from v i to v j is deﬁned as l ( v i , v j )

Original: = (cid:80) j − 1 t
Revised: = (cid:80) j − 1 t

Original: = i +1 l ( v t ) , which is the sum of cost over all the vertexes between v i and v j .
Revised: = i +1 l ( v t ) , which is the sum of cost over all the vertexes between v i and v j .

Original: Using these notations, the memory cost of training with re-forwarding is formulated as
Revised: Using these notations, the memory cost of training with re-forwarding is formulated as

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst term is the sum of the memory cost of all the stored tensors, and the second term is the maximal cost among the re-forwards.
Revised: where the ﬁrst term is the sum of the memory cost of all the stored tensors, and the second term is the maximal cost among the re-forwards.

Original: For easy illustration, we start by formulating Re-forwarding on Linear Computation Graphs (LCG) (Fig. 

Revised: For easy illustration, we start by formulating Re-forwarding on Linear Computation Graphs (LCG) (Fig.

Original: (a)).
Revised: (a)).

Original: For LCGs, Eqn. 1 can be solved in two cases.
Revised: For LCGs, Eqn. 1 can be solved in two cases.

Original: Case(1) LCG with Identical Vertex Cost: Suppose a LCG has N vertexes, each of which has the same cost l ( v i ) = 1 N and the total cost is 1 .
Revised: Case(1) LCG with Identical Vertex Cost: Suppose a LCG has N vertexes, each of which has the same cost l ( v i ) = 1 N and the total cost is 1 .

Original: Obviously, the optimal solution is reached when vertexes in V R are distributed evenly in the LCG.
Revised: Obviously, the optimal solution is reached when vertexes in V R are distributed evenly in the LCG.

Original: Suppose the number of vertexes in V R is k .
Revised: Suppose the number of vertexes in V R is k .

Original: The total cost is then kN + 1 k .
Revised: The total cost is then kN + 1 k .

Original: The optimal solution of Eqn.
Revised: The optimal solution of Eqn.

Original: 1 is k = √ N , and the optimal total cost is√ N .
Revised: 1 is k = √ N , and the optimal total cost is√ N .

Original: Case (2) LCG with Non-identical Vertex Cost: When the assumption of identical cost does not hold, the solution to Eqn.
Revised: Case (2) LCG with Non-identical Vertex Cost: When the assumption of identical cost does not hold, the solution to Eqn.

Original: 1 does not have an analytic form.
Revised: 1 does not have an analytic form.

Original: Denote the maximal Re-forward cost max j l ( v j , v j +1 ) as a constant C , and the solution to Eqn. 1 is reduced to solving for min V R (cid:80)
Revised: Denote the maximal Re-forward cost max j l ( v j , v j +1 ) as a constant C , and the solution to Eqn. 1 is reduced to solving for min V R (cid:80)

Original: i l ( v i ) .
Revised: i l ( v i ) .

Original: 1: for each vertex pair ( v i , v j ) in G do 2:
Revised: 1: for each vertex pair ( v i , v j ) in G do 2:

Original: Set the maximal term as l ( v i , v j ) 3:
Revised: Set the maximal term as l ( v i , v j ) 3:

Original: Construct Accessibility Graph 4:
Revised: Construct Accessibility Graph 4:

Original: Find the shortest path in the Accessibility Graph as the solution 5:
Revised: Find the shortest path in the Accessibility Graph as the solution 5:

Original: Compute the actual total cost of the solution 6:
Revised: Compute the actual total cost of the solution 6:

Original: Save the solution if it’s better.
Revised: Save the solution if it’s better.

Original: Suppose the actual max term of this solution is B , and l ( v i , v j ) =
Revised: Suppose the actual max term of this solution is B , and l ( v i , v j ) =

Original: C , skip the loops where
Revised: C , skip the loops where

Original: B ≤ l ( v i , v j ) <
Revised: B ≤ l ( v i , v j ) <

Original: C
Revised: C

Original: All the Re-forward costs in an optimal solution satisfy the constraint l ( v j , v j +1 ) ≤ C .
Revised: All the Re-forward costs in an optimal solution satisfy the constraint l ( v j , v j +1 ) ≤ C .

Original: We solve
Revised: We solve

Original: Eqn. 1 by constructing a new graph, called Accessibility Graph G A = (cid:0) E A , V (cid:1) .
Revised: Eqn. 1 by constructing a new graph, called Accessibility Graph G A = (cid:0) E A , V (cid:1) .

Original: The edges of G A , called Accessibility Edge e Aij , exists between vertex v i and v j if and only if l ( v i , v j ) ≤ C .
Revised: The edges of G A , called Accessibility Edge e Aij , exists between vertex v i and v j if and only if l ( v i , v j ) ≤ C .

Original: Now the problem of solving min V R (cid:80)
Revised: Now the problem of solving min V R (cid:80)

Original: i l ( v i ) is equivalent to ﬁnding the shortest path from the source vertex and the target vertex in the Accessibility Graph.
Revised: i l ( v i ) is equivalent to ﬁnding the shortest path from the source vertex and the target vertex in the Accessibility Graph.

Original: Notice that in the optimal solution, the max term equal the one maximal term among all l ( v i , v i +1 ) terms.
Revised: Notice that in the optimal solution, the max term equal the one maximal term among all l ( v i , v i +1 ) terms.

Original: To traverse all possible max terms, we can simply compute the loss of every vertex pair and use it as a possible max term.
Revised: To traverse all possible max terms, we can simply compute the loss of every vertex pair and use it as a possible max term.

Original: Given a max term
Revised: Given a max term

Original: C , suppose the actual max term of the solution under C is B and B < C .
Revised: C , suppose the actual max term of the solution under C is B and B < C .

Original: It’s obvious that for all the max terms B ≤ max < C , the solution would be the same solution.
Revised: It’s obvious that for all the max terms B ≤ max < C , the solution would be the same solution.

Original: Therefore, these max terms can be skipped.
Revised: Therefore, these max terms can be skipped.

Original: Algorithm 1 summarizes the process for searching an optimal solution for
Revised: Algorithm 1 summarizes the process for searching an optimal solution for

Original: LCG.
Revised: LCG.

Original: Suppose there are N vertexes in the computation graph, the time complexity of Algorithmis O ( N 4 ) 1 .
Revised: Suppose there are N vertexes in the computation graph, the time complexity of Algorithmis O ( N 4 ) 1 .

Original:  A RBITRARY C OMPUTATION G RAPH (ACG)
Revised:  A RBITRARY C OMPUTATION G RAPH (ACG)

Original: As generalization of DNNs with LCG, we present theory 2 and algorithms for DNNs with Arbitrary
Revised: As generalization of DNNs with LCG, we present theory 2 and algorithms for DNNs with Arbitrary

Original: Computation Graphs (ACG), in particular the acyclic directed graphs(Fig.
Revised: Computation Graphs (ACG), in particular the acyclic directed graphs(Fig.

Original: 2 (b)).
Revised: 2 (b)).

Original: 4. A SSUMPTION FOR O PTIMALITY
Revised: 4. A SSUMPTION FOR O PTIMALITY

Original: The optimal solution of Re-forwarding corresponds to an optimal division of ACG, such that memory cost (Eqn. 1) is minimum.
Revised: The optimal solution of Re-forwarding corresponds to an optimal division of ACG, such that memory cost (Eqn. 1) is minimum.

Original: We denote that an ACG is divided into end-to-end segments by a set of vertexes.
Revised: We denote that an ACG is divided into end-to-end segments by a set of vertexes.

Original: These end-to-end segments can have multiple endpoint vertexes, for example, multiple source vertexes and multiple target vertexes.
Revised: These end-to-end segments can have multiple endpoint vertexes, for example, multiple source vertexes and multiple target vertexes.

Original: In this paper, as an assumption and also for simpliﬁcation, these end-to-end segments are narrowed down to those with only one source vertex and one target vertex.
Revised: In this paper, as an assumption and also for simpliﬁcation, these end-to-end segments are narrowed down to those with only one source vertex and one target vertex.

Original: Another assumption in the case of ACG is imposed on the operation that has multiple inputs: one can compute the gradients of output with respect to the gradients of inputs without using the current value of inputs.
Revised: Another assumption in the case of ACG is imposed on the operation that has multiple inputs: one can compute the gradients of output with respect to the gradients of inputs without using the current value of inputs.

Original: Examples of operations that meet this assumption are: concatenation (the gradient of output is also the concatenation of the gradient of input), add (the gradient of output equals the gradient of input), etc.
Revised: Examples of operations that meet this assumption are: concatenation (the gradient of output is also the concatenation of the gradient of input), add (the gradient of output equals the gradient of input), etc.

Original: An example that breaks this assumption is multiplication (the gradient of input depends on the input).
Revised: An example that breaks this assumption is multiplication (the gradient of input depends on the input).

Original: Fortunately, most of the popular networks meet this assumption.
Revised: Fortunately, most of the popular networks meet this assumption.

Original: A simple way to remove this assumption is to store all the input tensors of this multi-input operation.
Revised: A simple way to remove this assumption is to store all the input tensors of this multi-input operation.

Original: However, this is not modeled by our loss function and may lead to sub-optimal solution.
Revised: However, this is not modeled by our loss function and may lead to sub-optimal solution.

Original: In summary, there are only two assumptions in our approach: (1) the segment in a solution only has two endpoints (source and target).
Revised: In summary, there are only two assumptions in our approach: (1) the segment in a solution only has two endpoints (source and target).

Original: (2) the multi-input operation can compute the gradients of output without using the current value of input.
Revised: (2) the multi-input operation can compute the gradients of output without using the current value of input.

Original: Under these two assumptions, our approach is optimal for
Revised: Under these two assumptions, our approach is optimal for

Original: ACGs.
Revised: ACGs.

Original: 4. D EFINITION AND T HEOREM
Revised: 4. D EFINITION AND T HEOREM

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: Close Set: A set s containing vertexes and edges is a close set if and only if it satisﬁes the following 3 properties: 1.
Revised: Closed Set: A set s containing vertexes and edges is a closed set if and only if it satisﬁes the following three properties: 1.

Original: All the vertexes of s have a common ancestor v i and a common descendent v j ; 2.
Revised: All the vertexes of s have a common ancestor v i and a common descendent v j ; 2.

Original: Suppose the vertex subset of s is V and edge subset is E , and the set of edges between two arbitrary vertexes of V ∪ { v i , v j } is E (cid:48)  .
Revised: Denote the vertex subset of s as V , edge subset as E , and the set of edges between two arbitrary vertexes of V ∪ { v i , v j } is E (cid:48) , the edge from v i to v j (if exists) as e ij .

Original: E must either equal E (cid:48) or E (cid:48) − { e ij } , with e ij being the edge from v i to v j (if exists); 3.
Revised: E must either be E (cid:48) or E (cid:48) − { e ij } ; 3.

Original: An arbitrary v 1 ∈ V doesn’t have edge with another arbitrary v 2 / ∈ V ∪ { v i , v j } .
Revised: An arbitrary v 1 ∈ V doesn’t have edge with another arbitrary v 2 / ∈ V ∪ { v i , v j } .

Original: For multiple valid close sets between v i and v j , we denote the largest one as s ij
Revised: For multiple valid closed sets between v i and v j , we denote the largest one as s ij

Original: Deﬁnition 2.
Revised: Deﬁnition 2.

Original: [ s ij ] =
Revised: [ s ij ] =

Original: s ij ∪ { v i , v j } .
Revised: s ij ∪ { v i , v j } .

Original: [ s ij ) =
Revised: [ s ij ) =

Original: s ij ∪ { v i } .
Revised: s ij ∪ { v i } .

Original: ( s ij ] = s ij ∪ { v j }
Revised: ( s ij ] = s ij ∪ { v j }

Original: In the deﬁnition of Close Set , property 1 corresponds to the two endpoint assumption in section 4.
Revised: In the deﬁnition of Closed Set , property 1 corresponds to the two endpoint assumption in section 4.

Original: where the two endpoints become v i and v j in the deﬁnition.
Revised: where the two endpoints become v i and v j in the deﬁnition.

Original: Property 2 conﬁnes the edges of s to be either E (cid:48) or E (cid:48) − { e ij } .The set with E = E (cid:48) and the set with E = E (cid:48) − {
Revised: Property 2 conﬁnes the edge subsets of s to be one of two cases: E (cid:48) or E (cid:48) − { e ij } .

Original: e ij } will be different validclose sets although theymight have same vertexes.
Revised: Both cases are valid although they have different edges.

Original: Property 3 guarantees the independence of such a set s , meaning that the vertexes within s have no connections with other vertexes outside s ∪ { v i , v j } .
Revised: Property 3 guarantees the independence of such a set s , meaning that the vertexes within s have no connections with other vertexes outside s ∪ { v i , v j } .

Original: As there might be multiple valid close sets between v i and v j , which corresponds to the Branched Close Set in Deﬁnition 5, we denote the largest close set between v i and v j as s ij and denote smaller close set with an extra superscript, such as s 1 ij .
Revised: As there might be multiple valid closed sets between v i and v j , which corresponds to the Branched Closed Set in Deﬁnition 5, we denote the largest closed set between v i and v j as s ij and denote smaller closed set with an extra superscript, such as s 1 ij .

Original: Deﬁnition 3.
Revised: Deﬁnition 3.

Original: Splitting Vertex:
Revised: Splitting Vertex:

Original: A vertex v t ∈ s ij is a splitting vertex of s ij if and only if s it exists, s tj exists and s ij =
Revised: A vertex v t ∈ s ij is a splitting vertex of s ij if and only if s it exists, s tj exists and s ij =

Original: s it ∪ s tj ∪ { v t } and s it ∩ s tj = ∅
Revised: s it ∪ s tj ∪ { v t } and s it ∩ s tj = ∅

Original: Deﬁnition 4.
Revised: Deﬁnition 4.

Original: Splittable Close Set (Type 1): close set with at least 1 splitting vertex.
Revised: Splittable Closed Set (Type 1): closed set with at least 1 splitting vertex.

Original: The deﬁnition of Splitting Vertex is to describe whether a close set can be divided into two linearly arranged close set.
Revised: The deﬁnition of Splitting Vertex is to describe whether a closed set can be divided into two linearly arranged closed set.

Original: A close set is splittable if it has at least 1 splitting vertex and is deﬁned asClose
Revised: A closed set is splittable if it has at least 1 splitting vertex and is deﬁned as

Original:  Set Type 1 .
Revised: Closed Set Type 1 .

Original: Deﬁnition 5.
Revised: Deﬁnition 5.

Original: Branched Close Set (Type 2): A close set is branched if it has 0 splitting vertex and can be divided into branches: s ij =
Revised: Branched Closed Set (Type 2): A closed set is branched if it has 0 splitting vertex and can be divided into branches: s ij =

Original: s 1 ij ∪ s 2 ij and s 1 ij ∩ s 2 ij = ∅
Revised: s 1 ij ∪ s 2 ij and s 1 ij ∩ s 2 ij = ∅

Original: Deﬁnition 6.
Revised: Deﬁnition 6.

Original: Non-branched Close Set (Type 3): A close set s ij is non-branched if it has 0 splitting vertex and no branch: (cid:54)
Revised: Non-branched Closed Set (Type 3): A closed set s ij is non-branched if it hassplitting vertex and no branch: (cid:54)

Original: ∃ s 1 ij (cid:36) s ij
Revised: ∃ s 1 ij (cid:36) s ij

Original: Among close set with no splitting vertex, we categorize close set with branches as Close Set Type 2 , and close set without branches as Close Set Type 3 .
Revised: Among closed set with no splitting vertex, we categorize closed set with branches as Closed Set Type 2 , and closed set without branches as Closed Set Type 3 .

Original: The examples of different types of close set are shown in Fig.close
Revised: The examples of different types of closed set are shown in Fig.

Original: Deﬁnition 7.
Revised: Deﬁnition 7.

Original: Maximal Split: { [ s pq ] } is a maximal split of non-branched s ij if [ s ij ] = ∪{ [ s pq ] } and ∀ s ab , s cd ∈ { [ s pq ] } , s ab ∩ s cd = ∅ and (cid:54) ∃{ [ s (cid:48)
Revised: Maximal Split: { [ s pq ] } is a maximal split of non-branched s ij if [ s ij ] = ∪{ [ s pq ] } and ∀ s ab , s cd ∈ { [ s pq ] } , s ab ∩ s cd = ∅ and (cid:54) ∃{ [ s (cid:48)

Original: pq ] } (cid:36) {
Revised: pq ] } (cid:36) {

Original: [ s pq ] } such that ∪{ [ s (cid:48) pq ] } =
Revised: [ s pq ] } such that ∪{ [ s (cid:48) pq ] } =

Original: [ s kt ] (cid:36) [ s ij ]
Revised: [ s kt ] (cid:36) [ s ij ]

Original: Deﬁnition 8.
Revised: Deﬁnition 8.

Original: Division of Close Set: For type 1, its division is the linear segments separated by all its splitting vertexes; for type 2, its division is all its branches, any of which cannot be divided into more branches; for type 3, its division is its maximal split.
Revised: Division of Closed Set: For type 1, its division is the linear segments separated by all its splitting vertexes; for type 2, its division is all its branches, any of which cannot be divided into more branches; for type 3, its division is its maximal split.

Original: For close set type 1, it can be divided into linearly arranged segments.
Revised: For closed set type 1, it can be divided into linearly arranged segments.

Original: For close set type 2, it can be divided into branches.
Revised: For closed set type 2, it can be divided into branches.

Original: So here we investigate the division of close set type 3.
Revised: So here we investigate the division of closed set type 3.

Original: As we don’t want trivial division, for example, division that is formed by every edge in the close set, we deﬁne
Revised: As we don’t want trivial division, for example, division that is formed by every edge in the closed set, we deﬁne

Original: Maximal Split to describe the split such that each member of the split is as large as possible.
Revised: Maximal Split to describe the split such that each member of the split is as large as possible.

Original: An example of maximal split is shown in Fig.
Revised: An example of maximal split is shown in Fig.

Original: 4 (c).
Revised: 4 (c).

Original: In the deﬁnition of maximal split, the term maximal is implied by saying that any subset of this split cannot be combined into a single close set.
Revised: In the deﬁnition of maximal split, the term maximal is implied by saying that any subset of this split cannot be combined into a single closed set.

Original: If it can, then the maximal split will be formed by this larger close set and all the rest of the previous split.
Revised: If it can, then the maximal split will be formed by this larger closed set and all the rest of the previous split.

Original: For close set type 3, we use its maximal split as its division.
Revised: For closed set type 3, we use its maximal split as its division.

Original: Deﬁnition 9.
Revised: Deﬁnition 9.

Original: Division Tree: Division tree is a representation of a computation graph, where the root node is the whole computation graph, the leaf nodes are all the single tensors in the computation graph, and for a non-leaf node, its children is the members of its division.
Revised: Division Tree: Division tree is a representation of a computation graph, where the root node is the whole computation graph, the leaf nodes are all the single tensors in the computation graph, and for a non-leaf node, its children is the members of its division.

Original: With the division of 3 types of close sets, the computation graph can be reorganized into a division tree (Figure 5) where a non-leaf node would be a close set and its children would be its corresponding division.
Revised: With the division of 3 types of closed sets, the computation graph can be reorganized into a division tree (Figure 5) where a non-leaf node would be a closed set and its children would be its corresponding division.

Original: The root node is the whole computation graph, the largest close set, and the leaf nodes would be single tensors in the computation graph.
Revised: The root node is the whole computation graph, the largest closed set, and the leaf nodes would be single tensors in the computation graph.

Original: With division tree, we can apply divide-andconquer to search for optimal solution.
Revised: With division tree, we can apply divide-and-conquer to search for optimal solution.

Original: Theorem 1.
Revised: Theorem 1.

Original: The division tree of a computation graph is unique and complete.
Revised: The division tree of a computation graph is unique and complete.

Original: The uniqueness of the division tree indicates that the optimal solution of the division tree would also be the optimal solution of the whole computation graph.
Revised: The uniqueness of the division tree indicates that the optimal solution of the division tree would also be the optimal solution of the whole computation graph.

Original: The completeness indicates that the division tree has included all the possible members of solution and represents the whole search space for the optimal solution.
Revised: The completeness indicates that the division tree has included all the possible members of solution and represents the whole search space for the optimal solution.

Original: Theorem 1 is proved in the appendix.
Revised: Theorem 1 is proved in the appendix.

Original: 4. A LGORITHM
Revised: 4. A LGORITHM

Original: We search optimal solutions for ACGs by solving several sub-problems using Algorithm 2-4 respectively.
Revised: We search optimal solutions for ACGs by solving several sub-problems using Algorithm 2-4 respectively.

Original: Based on these components, we present our ﬁnal solver as Algorithm 5.
Revised: Based on these components, we present our ﬁnal solver as Algorithm 5.

Original: Algorithm 2 judges whether a vertex is a splitting vertex of a close set.
Revised: Algorithm 2 judges whether a vertex is a splitting vertex of a closed set.

Original: This algorithm mainly follows the Deﬁnition 3 and uses vertex set to check the property of a splitting vertex.
Revised: This algorithm mainly follows the Deﬁnition 3 and uses vertex set to check the property of a splitting vertex.

Original: With this algorithm, we can judge whether a close set is type 1 and get its division if it is.
Revised: With this algorithm, we can judge whether a closed set is type 1 and get its division if it is.

Original: Suppose there are
Revised: Suppose there are

Original: N vertexes in s ij , the time complexity of Algorithm 2 is O ( N 2 ) .
Revised: N vertexes in s ij , the time complexity of Algorithm 2 is O ( N 2 ) .

Original: Algorithm 2 Judge whether a vertex v t is a splitting vertex of close set s ij 1: Let { v in } be the vertexes of all the vertexes within [ s ij ] that have paths to v t .
Revised: Algorithm 2 Judge whether a vertex v t is a splitting vertex of closed set s ij 1: Let { v in } be the vertexes of all the vertexes within [ s ij ] that have paths to v t .

Original: Let { v out } be the vertexes of all the vertexes within [ s ij ] that have paths from v t .
Revised: Let { v out } be the vertexes of all the vertexes within [ s ij ] that have paths from v t .

Original: 2: if { v in }∪{ V out }∪{ v t } = { v | v ∈ [ s ij ] } and { v in }∩{ V out } =
Revised: 2: if { v in }∪{ V out }∪{ v t } = { v | v ∈ [ s ij ] } and { v in }∩{ V out } =

Original: ∅
Revised: ∅

Original: and (cid:54) ∃ v 1 ∈ { v in } , v 2 ∈ { v out } , v 1 , v 2 have connections then 3:
Revised: and (cid:54) ∃ v 1 ∈ { v in } , v 2 ∈ { v out } , v 1 , v 2 have connections then 3:

Original: Return true 4: else 5:
Revised: Return true 4: else 5:

Original: Return False
Revised: Return False

Original: Algorithm 3 examines whether a close set is branched.
Revised: Algorithm 3 examines whether a closed set is branched.

Original: It uses a growing algorithm to check whether an independent subpart of this close set can form a close set.
Revised: It uses a growing algorithm to check whether an independent subpart of this closed set can form a closed set.

Original: If a non-trivial close set s ij has an edge from v i to v j , then it’s branched because this edge itself can be treated as a close set.
Revised: If a non-trivial closed set s ij has an edge from v i to v j , then it’s branched because this edge itself can be treated as a closed set.

Original: Combined with Algorithm 2, we can know the type of a close set and get its division if it’s type 2.
Revised: Combined with Algorithm 2, we can know the type of a closed set and get its division if it’s type 2.

Original: Suppose there are N vertexes in s ij , the time complexity of Algorithm 3 is O ( N 2 ) .
Revised: Suppose there are N vertexes in s ij , the time complexity of Algorithm 3 is O ( N 2 ) .

Original: Algorithm 4 addresses the problem of ﬁnding the maximal split, the division of a close set type 3 s ij .
Revised: Algorithm 4 addresses the problem of ﬁnding the maximal split, the division of a closed set type 3 s ij .

Original: First get all the possible close sets within s ij and use a property of maximal split to judge whether this close set is a member of the maximal split.
Revised: First get all the possible closed sets within s ij and use a property of maximal split to judge whether this closed set is a member of the maximal split.

Original: The property is: there cannot exist another close set s ab (cid:36) s ij but contains any member of this maximal split.
Revised: The property is: there cannot exist another closed set s ab (cid:36) s ij but contains any member of this maximal split.

Original: This property is proved in
Revised: This property is proved in

Original: Lemma 6 of the appendix.
Revised: Lemma 6 of the appendix.

Original: Suppose there are N vertexes in s ij , the time complexity of Algorithm 4 is O ( N 4 ) .
Revised: Suppose there are N vertexes in s ij , the time complexity of Algorithm 4 is O ( N 4 ) .

Original: Algorithm 5 is the solver for ACGs.
Revised: Algorithm 5 is the solver for ACGs.

Original: First, the division tree of the computation graph is built.
Revised: First, the division tree of the computation graph is built.

Original: Similar to the linear solver, a max term list is formed by the cost of all the possible close sets for traverse.
Revised: Similar to the linear solver, a max term list is formed by the cost of all the possible closed sets for traverse.

Original: Given a max term, we propose a greedy idea: for a close set, never expand it unless the its cost exceed the max term.
Revised: Given a max term, we propose a greedy idea: for a closed set, never expand it unless the its cost exceed the max term.

Original: In other word, if the max term doesn’t allow a leap over this close set, we expand it, otherwise, do not expand it.
Revised: In other word, if the max term doesn’t allow a leap over this closed set, we expand it, otherwise, do not expand it.

Original: Because once expanded, some cost of other vertexes inside this close set might be introduced, and the cost will never be smaller than unexpanded.
Revised: Because once expanded, some cost of other vertexes inside this closed set might be introduced, and the cost will never be smaller than unexpanded.

Original: If some children of the close set type 1 are expanded, the rest reforms a few linear segments and still can 
Revised: If some children of the closed set type 1 are expanded, the rest reforms a few linear segments and still can be solved by the linear solver. If some children of the closed set type 2 or 3 are expanded, the other

Original: Algorithm 3 Judge whether s ij is branched
Revised: Algorithm 3 Judge whether s ij is branched

Revised: Return

Revised: 12: else 13:

Revised: 14: else 15:

Revised: Return false

Original: Algorithm 4 Find the maximal split of a non-branched s ij with 0 splitting vertex be solved by the linear solver.
Revised: Algorithm 4 Find the maximal split of a non-branched s ij with 0 splitting vertex there doesn’t exist such kt

Original: If some children of the close set type 2 or 3 are expanded, the other members remain unexpanded and need no changes.
Revised: ij , s kt the maximal split. members remain unexpanded and need no changes.

Original: Suppose there are N vertexes in computation graph, the time complexity of Algorithm 5 is O ( N 4 ) .
Revised: Suppose there are N vertexes in computation graph, the time complexity of Algorithm 5 is O ( N 4 ) .

Original: 1: Get all possible close set and their costs.
Revised: 1: Get all possible closed set and their costs.

Original: Use their costs to form the max term list.
Revised: Use their costs to form the max term list.

Original: 2: Reorganize the computation graph into a division tree: from the root node (the computation graph
Revised: 2: Reorganize the computation graph into a division tree: from the root node (the computation graph), build its children from its division, until all the leaf nodes are single tensors. 3: for each possible

Original: ), build its children from its division the nodes are term m  do 4: 
Revised: max term m in max term list { m } do 4: if current closed set is type 1 then 5: For all the children that have cost larger than current max term. Expand them and solve the next level. All the expanded children have separated the current closed set to linear segments.

Revised: Solve all the linear segments with current max term.

Original: 7: else 8:
Revised: 7: else 8:

Original: For  the children  max term them  solve the  level.
Revised: For all the children that have cost larger than current max term. Expand them and solve the next level.

Original: All the other members remain unexpanded.
Revised: All the other members remain unexpanded.

Original: Summarize the total loss, save the current solution if it’s better.
Revised: Summarize the total loss, save the current solution if it’s better.

Original:  E XPERIMENT
Revised:  E XPERIMENT

Original: We evaluated Re-forwarding on two main groups of neural networks (1) networks with linear structures, such as Alexnet (Krizhevsky et al.
Revised: We evaluated Re-forwarding on two main groups of neural networks (1) networks with linear structures, such as Alexnet (Krizhevsky et al.

Original: and vgg series (Simonyan & Zisserman (2014)).
Revised: and vgg series (Simonyan & Zisserman (2014)).

Original: (2) networks with non-linear structures, such as Resnet series (He et al.
Revised: (2) networks with non-linear structures, such as Resnet series (He et al.

Original: Densenet series (Huang et al.
Revised: Densenet series (Huang et al.

Original: and Inception net (Szegedy et al.
Revised: and Inception net (Szegedy et al.

Original: For each network in Table 1, an computation graph is built such that every vertex is a Float32 tensor, every edge is an operation, and the memory cost of a vertex is its tensor size measured in MB.
Revised: For each network in Table 1, an computation graph is built such that every vertex is a Float32 tensor, every edge is an operation, and the memory cost of a vertex is its tensor size, the number of Float32.

Original: We compared Re-forwarding with Chen (Chen et al.
Revised: We compared Re-forwarding with Chen (Chen et al.

Original: and the regular training approach.
Revised: and the regular training approach.

Original: Note that Chen et al.
Revised: Note that Chen et al.

Original: (2016) only worked on linear computation graphs .
Revised: (2016) only worked on linear computation graphs .

Original: To compare with (Chen et al.
Revised: To compare with (Chen et al.

Original: on non-linear networks , we[Table] manually re-organized all the non-linear computation graphs into linear computation graphs with their splitting vertexes, and fed them to Chen et al.
Revised: on non-linear networks, we manually re-organized all the non-linear computation graphs into linear computation graphs with their splitting vertexes, and fed them to Chen et al.

Original: (see Table 1 “Chen et al.
Revised: (see Table 1 “Chen et al.

Original: (2016) manual  (MB)”).
Revised: (2016) manual [Table] (MB)”).

Original: Our Re-forwarding approach directly works on arbitrary computation graphs.
Revised: Our Re-forwarding approach directly works on arbitrary computation graphs.

Revised: We have also included a customized network on which Chen’s approach is not applicable and our approach can still cut down memory.

Revised: The computation graph of this network is visualized in the appendix.

Original: All experiments were conducted in Pytorch.
Revised: All experiments were conducted in Pytorch.

Original: To remove irrelevant GPU memory cost, such as model and Pytorch CUDA interface cost, all training memory costs were measured with two different input sizes and compute the difference between two measurements.
Revised: To remove irrelevant GPU memory cost, such as model and Pytorch CUDA interface cost, all training memory costs were measured with two different input sizes and compute the difference between two measurements.

Original: For example, to measure the memory cost of Alexnet with input size [ BatchSize, Channel, Width, Height ] =
Revised: For example, to measure the memory cost of Alexnet with input size [ BatchSize, Channel, Width, Height ] =

Original: [16 , 3 , 224 , 224] , we ﬁrst record the training memory of input [16 , 3 , 224 , 224] as r 1 , and input [32 , 3 , 224 , 224] as r 2 .
Revised: [16 , 3 , 224 , 224] , we ﬁrst record the training memory of input [16 , 3 , 224 , 224] as r 1 , and input [32 , 3 , 224 , 224] as r 2 .

Original: The actual memory cost given [16 , 3 , 224 , 224] input is measured as r 2 − r 1 .
Revised: The actual memory cost given [16 , 3 , 224 , 224] input is measured as r 2 − r 1 .

Original: We also measure the training time (time of 1 iteration) for regular approach and our approach.
Revised: We also measure the training time (time of 1 iteration) for regular approach and our approach.

Original: Each time is measured by the average of 20 iterations.
Revised: Each time is measured by the average of 20 iterations.

Original: Our approach has completely the same training time with Chen’s approach, thus we summarize it as space efﬁcient training time in the table.
Revised: Our approach has completely the same training time with Chen’s approach, thus we summarize it as space efﬁcient training time in the table.

Original: To use existing DNN implementations, the input of Inception net is [ BatchSize, 3 , 300 , 300] , and the input of all other networks is [ BatchSize, 3 , 224 , 224] .
Revised: To use existing DNN implementations, the input of Inception net is [ BatchSize, 3 , 300 , 300] , and the input of all other networks is [ BatchSize, 3 , 224 , 224] .

Original: Table.
Revised: Table.

Original: 1 shows that Re-forwarding cuts down huge amount of memory from the regular approach and incurs reasonable time overhead: 26% space off and 40% time overhead for Alexnet, around 40% space off and 40% time overhead for Vgg series.
Revised: 1 shows that Re-forwarding cuts down huge amount of memory from the regular approach and incurs reasonable time overhead: 26% space off and 40% time overhead for Alexnet, around 40% space off and 40% time overhead for Vgg series.

Original: For Resnet series, the deeper network, the more memory was cut down.
Revised: For Resnet series, the deeper network, the more memory was cut down.

Original: On the deepest Resnet152, 80% space off was achieved with only 39% time overhead.
Revised: On the deepest Resnet152, 80% space off was achieved with only 39% time overhead.

Original: For Densenet series, more than 80% space off was achieved with around 40% time overhead.
Revised: For Densenet series, more than 80% space off was achieved with around 40% time overhead.

Original: Notice that, Chen et al.
Revised: Notice that, Chen et al.

Original: (2016) only works on linear networks.
Revised: (2016) only works on linear networks.

Original: Its results on non-linear networks were manually synthesized.
Revised: Its results on non-linear networks were manually synthesized.

Original: Re-forwarding directly works on non-linear networks and constantly outperformed Chen et al.
Revised: Re-forwarding directly works on non-linear networks and constantly outperformed Chen et al.

Original: (2016) and its “manual” version.
Revised: (2016) and its “manual” version.

Original: This supports our claim that
Revised: This supports our claim that

Original: Re-forwarding is optimal.
Revised: Re-forwarding is optimal.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Re-forwarding is a fundamental approach that explores trade-off between memory and computation power of GPUs.
Revised: Re-forwarding is a fundamental approach that explores trade-off between memory and computation power of GPUs.

Original: By saving tensors at a subset of layers during forward, and conducting extra local forwards for backward, Re-forwarding makes it possible to train very heavy DNNs with ﬁnite GPU memory.
Revised: By saving tensors at a subset of layers during forward, and conducting extra local forwards for backward, Re-forwarding makes it possible to train very heavy DNNs with ﬁnite GPU memory.

Original: To our knowledge, our theoretical and algorithmic results are the ﬁrst top-down work that achieve an optimal memory solution for arbitrary computation graphs in DNNs.
Revised: To our knowledge, our theoretical and algorithmic results are the ﬁrst top-down work that achieve an optimal memory solution for arbitrary computation graphs in DNNs.

Original: Re-forwarding can be further embedded and optimized with any low-level techniques such as distributed computing,
Revised: Re-forwarding can be further embedded and optimized with any low-level techniques such as distributed computing,

Original: GPU/CPU swapping, computation graph optimization and liveness analysis.
Revised: GPU/CPU swapping, computation graph optimization and liveness analysis.

Original: A P ROOF A. L EMMAS
Revised: A P ROOF A. L EMMAS

Original: Lemma 1.
Revised: Lemma 1.

Original: If s ij
Revised: If s ij

Original: ∩ s kt (cid:54) = ∅ and s ij (cid:54)⊂ s kt and s kt (cid:54)⊂ s ij , then s ij ∩ s kt =
Revised: ∩ s kt (cid:54) = ∅ and s ij (cid:54)⊂ s kt and s kt (cid:54)⊂ s ij , then s ij ∩ s kt =

Original: s kj or s ij ∩ s kt =
Revised: s kj or s ij ∩ s kt =

Original: s it
Revised: s it

Original: Proof.
Revised: Proof.

Original: Let [ s ij ] ∩ [ s kt ]
Revised: Let [ s ij ] ∩ [ s kt ]

Original: = s = { v, e } .
Revised: = s = { v, e } .

Original: Let v p be the source vertex of s , v q be the target vertex of s .
Revised: Let v p be the source vertex of s , v q be the target vertex of s .

Original: If v p (cid:54) = v i and v p (cid:54)
Revised: If v p (cid:54) = v i and v p (cid:54)

Original: = v k and v q (cid:54) =
Revised: = v k and v q (cid:54) =

Original: v j and v q (cid:54) = v t , then v i , v k has path to v p and v j , v t has path from v q .
Revised: v j and v q (cid:54) = v t , then v i , v k has path to v p and v j , v t has path from v q .

Original: Therefore, v p has at least 2 immediate parents v a , v b with v a ∈
Revised: Therefore, v p has at least 2 immediate parents v a , v b with v a ∈

Original: [ s ij ] , v a (cid:54)∈ [ s kt ] , v b ∈ [ s kt ] , v b (cid:54)∈ [ s ij ] .
Revised: [ s ij ] , v a (cid:54)∈ [ s kt ] , v b ∈ [ s kt ] , v b (cid:54)∈ [ s ij ] .

Original: If so, the independence of s ij and s kt is violated.
Revised: If so, the independence of s ij and s kt is violated.

Original: Therefore, v p must be v i or v k .
Revised: Therefore, v p must be v i or v k .

Original: Same on v q , v q must be v j or v t .
Revised: Same on v q , v q must be v j or v t .

Original: If v p = v i , v q = v j , then s ij ⊂ s kt .
Revised: If v p = v i , v q = v j , then s ij ⊂ s kt .

Original: If v p = v k , v q = v t , then s kt
Revised: If v p = v k , v q = v t , then s kt

Original: ⊂ s ij .
Revised: ⊂ s ij .

Original: Therefore, v p = v i , v q = v t or v p = v k , v q = v j .
Revised: Therefore, v p = v i , v q = v t or v p = v k , v q = v j .

Original: Suppose v p = v k , v q = v j , let’s prove s is a close set.
Revised: Suppose v p = v k , v q = v j , let’s prove s is a closed set.

Original: With s ⊂ s kt , ∀ v 1 ∈ s , v 1 has no edge with v 2 (cid:54)∈ [ s kt ] .
Revised: With s ⊂ s kt , ∀ v 1 ∈ s , v 1 has no edge with v 2 (cid:54)∈ [ s kt ] .

Original: With s ⊂ s ij , ∀ v 1 ∈ s , v 1 has no edge with v 2 (cid:54)∈ [ s ij ] .
Revised: With s ⊂ s ij , ∀ v 1 ∈ s , v 1 has no edge with v 2 (cid:54)∈ [ s ij ] .

Original: Therefore, ∀ v 1 ∈ s , v 1 has no edge with v 2 (cid:54)∈ [ s ] .
Revised: Therefore, ∀ v 1 ∈ s , v 1 has no edge with v 2 (cid:54)∈ [ s ] .

Original: The independence of s is guaranteed.
Revised: The independence of s is guaranteed.

Original: In the discussion before, we can see the source vertex v p of s must be either v i or v k .
Revised: In the discussion before, we can see the source vertex v p of s must be either v i or v k .

Original: If v i and v k are both the source vertexes of s , then v
Revised: If v i and v k are both the source vertexes of s , then v

Original: i ∈ [ s kt ] and v k ∈
Revised: i ∈ [ s kt ] and v k ∈

Original: [ s ij ] , v i has path to v k and v k has path to v i , which will force v i
Revised: [ s ij ] , v i has path to v k and v k has path to v i , which will force v i

Original: = v k because the s ij , s kt is acyclic.
Revised: = v k because the s ij , s kt is acyclic.

Original: Same on v q , s can only have 1 source vertex and 1 target vertex.
Revised: Same on v q , s can only have 1 source vertex and 1 target vertex.

Original: Therefore, s is close set.
Revised: Therefore, s is closed set.

Original: Therefore, s ij ∩ s kt = s kj or s ij ∩ s kt =
Revised: Therefore, s ij ∩ s kt = s kj or s ij ∩ s kt =

Original: s it .
Revised: s it .

Original: Lemma 2.
Revised: Lemma 2.

Original: The intersection of two closets s = s i ∩ s j (cid:54) = ∅ is also a closet
Revised: The intersection of two closets s = s i ∩ s j (cid:54) = ∅ is also a closet

Original: Proof.
Revised: Proof.

Original: Given the independence of s i and s j , the independence of s is obvious.
Revised: Given the independence of s i and s j , the independence of s is obvious.

Original: The remaining thing is whether s only has 1 source vertex and 1 target vertex.
Revised: The remaining thing is whether s only has 1 source vertex and 1 target vertex.

Original: In the proof of Lemma 1, we can see any source or target vertex of s will eventually become source or target vertex of s i and s j .
Revised: In the proof of Lemma 1, we can see any source or target vertex of s will eventually become source or target vertex of s i and s j .

Original: With simple discussion, we can have this lemma.
Revised: With simple discussion, we can have this lemma.

Original: Lemma 3.
Revised: Lemma 3.

Original: If s ij ∩ s kt = s kj (cid:54)
Revised: If s ij ∩ s kt = s kj (cid:54)

Original: = ∅ , then v k is the splitting vertex of s ij and v j is the splitting vertex of s kt
Revised: = ∅ , then v k is the splitting vertex of s ij and v j is the splitting vertex of s kt

Original: Proof.
Revised: Proof.

Original: Let’s ﬁrst prove that v k is the splitting vertex of s ij .
Revised: Let’s ﬁrst prove that v k is the splitting vertex of s ij .

Original: Let s = s ij − [ s kj ) .
Revised: Let s = s ij − [ s kj ) .

Original: Obviously, s ij =
Revised: Obviously, s ij =

Original: s ∪ s kj ∪ { v k } and s ∩ s kj = ∅ .
Revised: s ∪ s kj ∪ { v k } and s ∩ s kj = ∅ .

Original: We only need to prove that s is close set.
Revised: We only need to prove that s is closed set.

Original: For convenience, let’s denote [ s ]
Revised: For convenience, let’s denote [ s ]

Original: = s ∪ { v i , v k } v i is obviously the only source vertex of [ s ] because v i is source vertex of [ s ij ] .
Revised: = s ∪ { v i , v k } v i is obviously the only source vertex of [ s ] because v i is source vertex of [ s ij ] .

Original: We discuss the target vertex here.
Revised: We discuss the target vertex here.

Original: If v k is not the target vertex of [ s ] , as v k ∈ [ s ] , v k must have path to the target vertex v of [ s ] and v also has path to v j as v ∈ s ij .
Revised: If v k is not the target vertex of [ s ] , as v k ∈ [ s ] , v k must have path to the target vertex v of [ s ] and v also has path to v j as v ∈ s ij .

Original: Because v / ∈
Revised: Because v / ∈

Original: [ s kj ] , in the path from v to v j , there exists an edge that connects a vertex v 1 ∈ s with a vertex v 2 ∈ s kt which violates the independence of s kt .
Revised: [ s kj ] , in the path from v to v j , there exists an edge that connects a vertex v 1 ∈ s with a vertex v 2 ∈ s kt which violates the independence of s kt .

Original: Therefore, the target vertex of [ s ] can only be v k .
Revised: Therefore, the target vertex of [ s ] can only be v k .

Original: As s ⊂ [ s ij ) , ∀ v 1 ∈ s , v 1 has no edge with v 2 (cid:54)∈ [ s ij ) .
Revised: As s ⊂ [ s ij ) , ∀ v 1 ∈ s , v 1 has no edge with v 2 (cid:54)∈ [ s ij ) .

Original: As s kj is close, ∀ v 1 ∈ s , v 1 has no edge with v 2 ∈ s kj .
Revised: As s kj is close, ∀ v 1 ∈ s , v 1 has no edge with v 2 ∈ s kj .

Original: ∀ v 1 ∈ s , v 1 can only have edge with v 2 ∈ [ s ] .
Revised: ∀ v 1 ∈ s , v 1 can only have edge with v 2 ∈ [ s ] .

Original: Thus the independence of s is guaranteed.
Revised: Thus the independence of s is guaranteed.

Original: Therefore, s is close set, v k is the splitting vertex of s ij .
Revised: Therefore, s is closed set, v k is the splitting vertex of s ij .

Original: Same on v j , v j is the splitting vertex of s kt
Revised: Same on v j , v j is the splitting vertex of s kt

Original: Lemma 4.
Revised: Lemma 4.

Original: If s ij has n splitting vertexes { v 1 , v 2 , ..., v n } , then s ij =
Revised: If s ij has n splitting vertexes { v 1 , v 2 , ..., v n } , then s ij =

Original: s i 1 ∪ s 12 ∪ ... ∪ s nj ∪ { v 1 , v 2 , ..., v n }
Revised: s i 1 ∪ s 12 ∪ ... ∪ s nj ∪ { v 1 , v 2 , ..., v n }

Original: Proof.
Revised: Proof.

Original: If n = 2 , the splitting vertexes are v 1 , v 2 , s ij =
Revised: If n = 2 , the splitting vertexes are v 1 , v 2 , s ij =

Original: s i 1 ∪ s 1 j ∪ { v 1 } =
Revised: s i 1 ∪ s 1 j ∪ { v 1 } =

Original: s i 2 ∪ s 2 j ∪ { v 2 } .
Revised: s i 2 ∪ s 2 j ∪ { v 2 } .

Original: Let v 1 ∈ s i 2 , v 1 (cid:54) = v 2 , then s 1 j ∩ s i 

Revised: Let v 1 ∈ s i 2 , v 1 (cid:54) = v 2 , then s 1 j ∩ s i

Original: = s 12 (cid:54) = ∅ .
Revised: = s 12 (cid:54) = ∅ .

Original: According to Lemma 3, v 1 is splitting vertex of s iand v 2 is splitting vertex of s 1 j .
Revised: According to Lemma 3, v 1 is splitting vertex of s iand v 2 is splitting vertex of s 1 j .

Original: Therefore, s ij =
Revised: Therefore, s ij =

Original: s i 1 ∪ s 12 ∪ s 2 j ∪ { v 1 , v 2 } .
Revised: s i 1 ∪ s 12 ∪ s 2 j ∪ { v 1 , v 2 } .

Original: For n > 2 , the lemma can be proved by repetitively using the conclusion in n = 2 .
Revised: For n > 2 , the lemma can be proved by repetitively using the conclusion in n = 2 .

Original: Lemma 5.
Revised: Lemma 5.

Original: If the non-branched s ij has a maximal split { [ s pq ] } , and |{ [ s pq ] }| > 2 , denote { v } as all the endpoint vertexes of [ s ] ∈ { [ s pq ] } .
Revised: If the non-branched s ij has a maximal split { [ s pq ] } , and |{ [ s pq ] }| > 2 , denote { v } as all the endpoint vertexes of [ s ] ∈ { [ s pq ] } .

Original: Then ∀ v ∈ { v } , v (cid:54) = v
Revised: Then ∀ v ∈ { v } , v (cid:54) = v

Original: i , v j , v is the endpoint vertex of at least 3 members of the maximal split.
Revised: i , v j , v is the endpoint vertex of at least 3 members of the maximal split.

Original: Proof.
Revised: Proof.

Original: If v b is the endpoint vertex of only 2 members of the maximal split, suppose the 2 members are s ab and s bc .
Revised: If v b is the endpoint vertex of only 2 members of the maximal split, suppose the 2 members are s ab and s bc .

Original: If so, s ab and s bc can be merged into s ac .
Revised: If so, s ab and s bc can be merged into s ac .

Original: If s ac (cid:54) = s ij , this violates the deﬁnition of maximal split.
Revised: If s ac (cid:54) = s ij , this violates the deﬁnition of maximal split.

Original: Otherwise, it violates the condition that s ij is non-branched and |{ [ s pq ] }| > 2 .
Revised: Otherwise, it violates the condition that s ij is non-branched and |{ [ s pq ] }| > 2 .

Original: It is impossible that the 2 members are s ab and s cb because in this way v b has no path to v j and violates the deﬁnition of close set.
Revised: It is impossible that the 2 members are s ab and s cb because in this way v b has no path to v j and violates the deﬁnition of closed set.

Original: If v b is the endpoint vertex of only 1 member of the maximal split, then v b must be either v i or v j .
Revised: If v b is the endpoint vertex of only 1 member of the maximal split, then v b must be either v i or v j .

Original: Therefore, this lemma is proved.
Revised: Therefore, this lemma is proved.

Original: Lemma 6.
Revised: Lemma 6.

Original: Any member of a maximal split can not be the subset of another close set s (cid:36) s ij .
Revised: Any member of a maximal split can not be the subset of another closed set s (cid:36) s ij .

Original: Proof.
Revised: Proof.

Original: Suppose the source vertex of s is v 1 and target vertex is v 2 , a member s xy of the maximal split is inside s .
Revised: Suppose the source vertex of s is v 1 and target vertex is v 2 , a member s xy of the maximal split is inside s .

Original: Suppose a member s ab of the maximal split has its source vertex v a inside s and target vertex v b outside s .
Revised: Suppose a member s ab of the maximal split has its source vertex v a inside s and target vertex v b outside s .

Original: Then the boundary vertex (the vertex that has edges to the non-overlapping parts of both sets) must be v 2 , otherwise the independence of s will be violated.
Revised: Then the boundary vertex (the vertex that has edges to the non-overlapping parts of both sets) must be v 2 , otherwise the independence of s will be violated.

Original: Notice that v 2 is inside s ab and the independence of s ab needs to be guaranteed, for ∀ v p ∈ s, v p / ∈ s ∩ s ab , v q ∈ s ∩ s ab , v p has no edge with v q .
Revised: Notice that v 2 is inside s ab and the independence of s ab needs to be guaranteed, for ∀ v p ∈ s, v p / ∈ s ∩ s ab , v q ∈ s ∩ s ab , v p has no edge with v q .

Original: Therefore, v a is a splitting vertex of s .
Revised: Therefore, v a is a splitting vertex of s .

Original: Similarly, if s ba has its target vertex v a inside s and source vertex v b outside s , the boundary vertex must be v 1 and v a is a splitting vertex of s .
Revised: Similarly, if s ba has its target vertex v a inside s and source vertex v b outside s , the boundary vertex must be v 1 and v a is a splitting vertex of s .

Original: For the close set s , from the discussion above, we know that there are at most 2 members of the maximal split that can overlap with s .
Revised: For the closed set s , from the discussion above, we know that there are at most 2 members of the maximal split that can overlap with s .

Original: Other members must be either completely inside s or completely outside s .
Revised: Other members must be either completely inside s or completely outside s .

Original: Let’s discuss the number of members that overlaps with s .
Revised: Let’s discuss the number of members that overlaps with s .

Original: If there are 0 member that overlaps with s , s is the union of a subset of members of the maximal split, which violates the deﬁnition of maximal split.
Revised: If there are 0 member that overlaps with s , s is the union of a subset of members of the maximal split, which violates the deﬁnition of maximal split.

Original: If there is 1 member that overlaps with s , suppose the corresponding splitting vertex is v b , and the boundary vertex is actually v 2 .
Revised: If there is 1 member that overlaps with s , suppose the corresponding splitting vertex is v b , and the boundary vertex is actually v 2 .

Original: Then s 1 b is a close set containing s xy and corresponds to the situation of 0 member overlapping.
Revised: Then s 1 b is a closed set containing s xy and corresponds to the situation of 0 member overlapping.

Original: s 1 b is the union of a subset of members of the maximal split, and violates the deﬁnition of maximal split.
Revised: s 1 b is the union of a subset of members of the maximal split, and violates the deﬁnition of maximal split.

Original: If there are 2 members that overlaps with s , suppose they generate two different splitting vertex v a and v b .
Revised: If there are 2 members that overlaps with s , suppose they generate two different splitting vertex v a and v b .

Original: Then s ab is a close set containing s xy and corresponds to the situation of 0 member overlapping.
Revised: Then s ab is a closed set containing s xy and corresponds to the situation of 0 member overlapping.

Original: s ab is the union of a subset of members of the maximal split, and violates the deﬁnition of maximal split.
Revised: s ab is the union of a subset of members of the maximal split, and violates the deﬁnition of maximal split.

Original: If they generate the same splitting vertex v b , from lemma 5, v b is also the endpoint vertex of at least 1 other member s ab which has to be inside s .
Revised: If they generate the same splitting vertex v b , from lemma 5, v b is also the endpoint vertex of at least 1 other member s ab which has to be inside s .

Original: Suppose the two overlapping members are s cb that contains v 1 , and s bd that contains v 2 .
Revised: Suppose the two overlapping members are s cb that contains v 1 , and s bd that contains v 2 .

Original: As the source vertex of s , v 1 has path to v b and v 1 has path to v a , which implies v b has path to v a .
Revised: As the source vertex of s , v 1 has path to v b and v 1 has path to v a , which implies v b has path to v a .

Original: As the target vertex of s , v 2 has path from v b and v 2 has path from v a , which implies v b has path from v a .
Revised: As the target vertex of s , v 2 has path from v b and v 2 has path from v a , which implies v b has path from v a .

Original: This conﬂicts with the fact that s is acyclic.
Revised: This conﬂicts with the fact that s is acyclic.

Original: Therefore, this case is not possible.
Revised: Therefore, this case is not possible.

Original: Therefore, this lemma is proved.
Revised: Therefore, this lemma is proved.

Original: Lemma 7.
Revised: Lemma 7.

Original: If non-branched s ij has at least 1 vertex but has 0 splitting vertex, then its maximal split has length >
Revised: If non-branched s ij has at least 1 vertex but has 0 splitting vertex, then its maximal split has length >

Original: Proof.
Revised: Proof.

Original: As s ij is not branched, the members of its maximal split cannot have the starting vertex as v i and the ending vertex as v j at the same time.
Revised: As s ij is not branched, the members of its maximal split cannot have the starting vertex as v i and the ending vertex as v j at the same time.

Original: If s ij has at least 1 vertex, and its maximal split has length 2, then its maximal split must be { [ s ik ] , [ s kj ] } , and v k will be the splitting vertex of s ij , which violates that s ij has no splitting vertex.
Revised: If s ij has at least 1 vertex, and its maximal split has length 2, then its maximal split must be { [ s ik ] , [ s kj ] } , and v k will be the splitting vertex of s ij , which violates that s ij has no splitting vertex.

Original: If s ij has at least 1 vertex without splitting vertex, it has at least 2 edges and cannot have a trivial length 1 maximal split.
Revised: If s ij has at least 1 vertex without splitting vertex, it has at least 2 edges and cannot have a trivial length 1 maximal split.

Original: Therefore, its maximal split has length >
Revised: Therefore, its maximal split has length >

Original: A. U NIQUENESS OF D IVISION T REE
Revised: A. U NIQUENESS OF D IVISION T REE

Original: To prove this uniqueness, we simply discuss the division uniqueness of close set type 1, 2 and 3.
Revised: To prove this uniqueness, we simply discuss the division uniqueness of closed set type 1, 2 and 3.

Original: A.2. U NIQUENESS OF D IVISION OF C LOSE S ET T YPE
Revised: A.2. U NIQUENESS OF D IVISION OF C LOSED S ET T YPE

Original: Proof.
Revised: Proof.

Original: By the deﬁnition of this division and Lemma 4, the uniqueness of the division is equivalent to the uniqueness of the splitting vertex set of a close set type 1.
Revised: By the deﬁnition of this division and Lemma 4, the uniqueness of the division is equivalent to the uniqueness of the splitting vertex set of a closed set type 1.

Original: The splitting vertex set is obviously unique.
Revised: The splitting vertex set is obviously unique.

Original: A.2. U NIQUENESS OF D IVISION OF C LOSE S ET T YPE
Revised: A.2. U NIQUENESS OF D IVISION OF C LOSED S ET T YPE

Original: Proof.
Revised: Proof.

Original: If there exists another division, there must be a branch member s 1 ij in division 1 and a branch member s 2 ij in division 2, where s 1 ij ∩ s 2 ij (cid:54) = ∅ and s 1 ij (cid:54) =
Revised: If there exists another division, there must be a branch member s 1 ij in division 1 and a branch member s 2 ij in division 2, where s 1 ij ∩ s 2 ij (cid:54) = ∅ and s 1 ij (cid:54) =

Original: s 2 ij .
Revised: s 2 ij .

Original: Denote s =
Revised: Denote s =

Original: s 1 ij ∩ s 2 ij .
Revised: s 1 ij ∩ s 2 ij .

Original: By Lemma 1 and 2, s = s 3 ij is also a close set.
Revised: By Lemma 1 and 2, s = s 3 ij is also a closed set.

Original: As s 1 ij and s 2 ij cannot be divided into more branches, s = s 1 ij = s 2 ij .
Revised: As s 1 ij and s 2 ij cannot be divided into more branches, s = s 1 ij = s 2 ij .

Original: Therefore, the division of close set type 2 is unique.
Revised: Therefore, the division of closed set type 2 is unique.

Original: A.2. U NIQUENESS OF D IVISION OF C LOSE S ET T YPE
Revised: A.2. U NIQUENESS OF D IVISION OF C LOSED S ET T YPE

Original: Proof.
Revised: Proof.

Original: As the close set in the division tree has at least 1 vertex, with Lemma 7, we know that the division, i.e. maximal split of a close set type 3 s ij within the division tree will have length > 2 .
Revised: As the closed set in the division tree has at least 1 vertex, with Lemma 7, we know that the division, i.e. maximal split of a closed set type 3 s ij within the division tree will have length > 2 .

Original: Denote this maximal split as { [ s pq ] } , we only need to prove this maximal split is unique.
Revised: Denote this maximal split as { [ s pq ] } , we only need to prove this maximal split is unique.

Original: Suppose there is a another different maximal split { [ s (cid:48) pq ] } , let us only check the difference between { [ s pq ] } and { [ s (cid:48) pq ] } .
Revised: Suppose there is a another different maximal split { [ s (cid:48) pq ] } , let us only check the difference between { [ s pq ] } and { [ s (cid:48) pq ] } .

Original: Denote { [ s kt ] } and { [ s (cid:48) kt ] } with { [ s pq ] } − { [ s kt ] } =
Revised: Denote { [ s kt ] } and { [ s (cid:48) kt ] } with { [ s pq ] } − { [ s kt ] } =

Original: { [ s (cid:48) pq ] } − { [ s (cid:48)
Revised: { [ s (cid:48) pq ] } − { [ s (cid:48)

Original: kt ] } and (cid:54) ∃ s ∈ { [ s kt ] } , s (cid:48) ∈ { [ s (cid:48) kt ] } , s = s (cid:48) .
Revised: kt ] } and (cid:54) ∃ s ∈ { [ s kt ] } , s (cid:48) ∈ { [ s (cid:48) kt ] } , s = s (cid:48) .

Original: As { [ s pq ] } − { [ s kt ] } =
Revised: As { [ s pq ] } − { [ s kt ] } =

Original: { [ s (cid:48) pq ] } − { [ s (cid:48)
Revised: { [ s (cid:48) pq ] } − { [ s (cid:48)

Original: kt ] } , we have ∪{ [ s kt ] } = ∪{ [ s (cid:48) kt ] }
Revised: kt ] } , we have ∪{ [ s kt ] } = ∪{ [ s (cid:48) kt ] }

Original: Obviously, |{ [ s kt ] }| ≥ 2 and |{ [ s (cid:48) kt ] }| ≥ 2 .
Revised: Obviously, |{ [ s kt ] }| ≥ 2 and |{ [ s (cid:48) kt ] }| ≥ 2 .

Original: Denote { v } as all the endpoint vertexes of [ s ] ∈ { [ s kt ] } , and { v (cid:48) } for { [ s (cid:48) kt ] } .
Revised: Denote { v } as all the endpoint vertexes of [ s ] ∈ { [ s kt ] } , and { v (cid:48) } for { [ s (cid:48) kt ] } .

Original: Obviously { v } (cid:54) = ∅ and { v (cid:48) } (cid:54) = ∅ .
Revised: Obviously { v } (cid:54) = ∅ and { v (cid:48) } (cid:54) = ∅ .

Original: As s ij is non-branched, { v } ∪ { v i , v j } − { v i , v j } (cid:54) = ∅ and { v (cid:48) } ∪ { v i , v j } − { v i , v j } (cid:54)
Revised: As s ij is non-branched, { v } ∪ { v i , v j } − { v i , v j } (cid:54) = ∅ and { v (cid:48) } ∪ { v i , v j } − { v i , v j } (cid:54)

Original: = ∅ .
Revised: = ∅ .

Original: Suppose s ab , s bc ∈ { [ s kt ] } , according to Lemma 5, there’s at least 1 other member that has v b as endpoint vertex.
Revised: Suppose s ab , s bc ∈ { [ s kt ] } , according to Lemma 5, there’s at least 1 other member that has v b as endpoint vertex.

Original: Suppose the other endpoint of this member is v d .
Revised: Suppose the other endpoint of this member is v d .

Original: Let’s discuss whether v b ∈ { v (cid:48) } and whether v d ∈ ∪{ [ s kt ] } .
Revised: Let’s discuss whether v b ∈ { v (cid:48) } and whether v d ∈ ∪{ [ s kt ] } .

Original: If v d (cid:54)∈ ∪{ [ s kt ] } , then v b must occur in { v (cid:48) } .
Revised: If v d (cid:54)∈ ∪{ [ s kt ] } , then v b must occur in { v (cid:48) } .

Original: Otherwise, v b would be inside a close set which would be violated by v d .
Revised: Otherwise, v b would be inside a closed set which would be violated by v d .

Original: Given v b ∈ { v (cid:48) } , as s ab (cid:54)∈ { [ s (cid:48) kt ] } , suppose s eb ∈ { [ s (cid:48) kt ] } and s ab ∩ s eb (cid:54) = ∅ .
Revised: Given v b ∈ { v (cid:48) } , as s ab (cid:54)∈ { [ s (cid:48) kt ] } , suppose s eb ∈ { [ s (cid:48) kt ] } and s ab ∩ s eb (cid:54) = ∅ .

Original: If v a ∈ s eb , from Lemma 1, s eb cannot be close.
Revised: If v a ∈ s eb , from Lemma 1, s eb cannot be close.

Original: If v e ∈ s ab , from Lemma 5, s ab cannot be close.
Revised: If v e ∈ s ab , from Lemma 5, s ab cannot be close.

Original: In this case, there cannot exist another different maximal split.
Revised: In this case, there cannot exist another different maximal split.

Original: If v d ∈ ∪{ [ s kt ] } , then s bd ∈ { [ s kt ] } .
Revised: If v d ∈ ∪{ [ s kt ] } , then s bd ∈ { [ s kt ] } .

Original: If v b ∈ { v (cid:48) } , we can use the same logic above to show this is impossible.
Revised: If v b ∈ { v (cid:48) } , we can use the same logic above to show this is impossible.

Original: Therefore, v b (cid:54)∈ { v (cid:48) } and s bd is included by a close set s .
Revised: Therefore, v b (cid:54)∈ { v (cid:48) } and s bd is included by a closed set s .

Original: From Lemma 6, this is impossible.
Revised: From Lemma 6, this is impossible.

Original: In this case, there cannot exist another different maximal split.
Revised: In this case, there cannot exist another different maximal split.

Original: In all the cases, there cannot exist another different maximal split.
Revised: In all the cases, there cannot exist another different maximal split.

Original: Therefore, the maximal split is unique.
Revised: Therefore, the maximal split is unique.

Original: A. C OMPLETENESS OF D IVISION T REE
Revised: A. C OMPLETENESS OF D IVISION T REE

Original: Similar with the uniqueness, the completeness of division tree is equivalent to the completeness of the division of a close set.
Revised: Similar with the uniqueness, the completeness of division tree is equivalent to the completeness of the division of a closed set.

Original: To prove this completeness, we simply discuss the division completeness of close set type 1, 2 and 3.
Revised: To prove this completeness, we simply discuss the division completeness of closed set type 1, 2 and 3.

Original: An equivalent statement of the division completeness is: there doesn’t exist a close set whose head is in one member of the division and whose tail is in another member of the division.
Revised: An equivalent statement of the division completeness is: there doesn’t exist a closed set whose head is in one member of the division and whose tail is in another member of the division.

Original: A.3. C OMPLETENESS OF D IVISION OF C LOSE S ET T YPE
Revised: A.3. C OMPLETENESS OF D IVISION OF C LOSED S ET T YPE

Original: Proof.
Revised: Proof.

Original: Suppose there exists a close set s whose head v p is in one member s 1 and whose tail v q is in another member s 2 .
Revised: Suppose there exists a closed set s whose head v p is in one member s 1 and whose tail v q is in another member s 2 .

Original: If v p is not an endpoint of s 1 , then according to Lemma 3, v p is also a splitting vertex in s 1 and can break s 1 into smaller segments, which makes v p also the splitting vertex of the whole close set.
Revised: If v p is not an endpoint of s 1 , then according to Lemma 3, v p is also a splitting vertex in s 1 and can break s 1 into smaller segments, which makes v p also the splitting vertex of the whole closed set.

Original: However, v p is not the splitting vertex of the whole close set s ij .
Revised: However, v p is not the splitting vertex of the whole closed set s ij .

Original: This also applies to v q .
Revised: This also applies to v q .

Original: Therefore, the division of close set type 1 is complete.
Revised: Therefore, the division of closed set type 1 is complete.

Original: A.3. C OMPLETENESS OF D IVISION OF C LOSE S ET T YPE
Revised: A.3. C OMPLETENESS OF D IVISION OF C LOSED S ET T YPE

Original: Proof.
Revised: Proof.

Original: Suppose there exists a close set s whose head v p is in one branch s 1 ij and whose tail v q is in another branch s 2 ij .
Revised: Suppose there exists a closed set s whose head v p is in one branch s 1 ij and whose tail v q is in another branch s 2 ij .

Original: As s crosses s 1 ij and s 2 ij , there exists a boundary vertex v in s , which belongs to [ s 1 ij ] and has direct connection with a vertex outside [ s 1 ij ] .
Revised: As s crosses s 1 ij and s 2 ij , there exists a boundary vertex v in s , which belongs to [ s 1 ij ] and has direct connection with a vertex outside [ s 1 ij ] .

Original: If v is not v i or v j , it will violate the independence of s ij .
Revised: If v is not v i or v j , it will violate the independence of s ij .

Original: If v = v i , as v i is the head of both s 1 ij and s 2 ij , it cannot be the boundary vertex, same when v = v j .
Revised: If v = v i , as v i is the head of both s 1 ij and s 2 ij , it cannot be the boundary vertex, same when v = v j .

Original: Therefore, there cannot exist such a close set s .
Revised: Therefore, there cannot exist such a closed set s .

Original: The division of close set typeis complete.
Revised: The division of closed set type 2 is complete.

Original: A.3. C OMPLETENESS OF D IVISION OF C LOSE S ET T YPE
Revised: A.3. C OMPLETENESS OF D IVISION OF C LOSED S ET T YPE

Original: Proof.
Revised: Proof.

Original: Suppose there exists a close set s whose head v p is in one member s 1 and whose tail v q is in another member s 2 .
Revised: Suppose there exists a closed set s whose head v p is in one member s 1 and whose tail v q is in another member s 2 .

Original: Same with close set type 2, the boundary vertex v has to be the endpoint vertex of s 1 or the independence of s 1 will be violated.
Revised: Same with closed set type 2, the boundary vertex v has to be the endpoint vertex of s 1 or the independence of s 1 will be violated.

Original: According to Lemma 5, v is the endpoint vertex of at least 3 members, meaning that v will at least have 1 connection with another close set s 3 .
Revised: According to Lemma 5, v is the endpoint vertex of at least 3 members, meaning that v will at least have 1 connection with another closed set s 3 .

Original: To maintain the independence of s , s has to include s 3 as well.
Revised: To maintain the independence of s , s has to include s 3 as well.

Original: However, s 3 also has its endpoints.
Revised: However, s 3 also has its endpoints.

Original: This will propagate until s becomes the whole close set.
Revised: This will propagate until s becomes the whole closed set.

Original: Therefore, there cannot exist such a close set s .
Revised: Therefore, there cannot exist such a closed set s .

Original: The division of close set type 3 is complete.
Revised: The division of closed set type 3 is complete.

Original: B C OMPLEXITY A NALYSIS B. A LGORITHM
Revised: B C OMPLEXITY A NALYSIS B. A LGORITHM

Original: Suppose there are N vertexes in the computation graph.
Revised: Suppose there are N vertexes in the computation graph.

Original: There will be O ( N 2 ) vertex pairs.
Revised: There will be O ( N 2 ) vertex pairs.

Original: For each vertex pair, the time cost is mainly on constructing accessibility graph and ﬁnding the shortest path.
Revised: For each vertex pair, the time cost is mainly on constructing accessibility graph and ﬁnding the shortest path.

Original: Suppose the source vertex of the whole computation graph is v 0 .
Revised: Denote the source vertex of the whole computation graph as v 0 .

Original: To construct an accessibility graph, ﬁrst we traverse the linear computation graph, record the accumulated sum l ( v 0 , v i ) for each vertex v i , and form a table of l ( v i , v j )
Revised: To construct an accessibility graph, ﬁrst we traverse the linear computation graph, record the accumulated sum l ( v 0 , v i ) for each vertex v i , and form a table of l ( v i , v j )

Original: =
Revised: =

Original: l ( v 0 , v j ) − l ( v 0 , v i ) − l ( v i ) .
Revised: l ( v 0 , v j ) − l ( v 0 , v i ) − l ( v i ) .

Original: These steps will cost O ( N 2 ) .
Revised: These steps will cost O ( N 2 ) .

Original: Then we traverse each ( v i , v j ) pair to form the edges of the accessibility graph, which also cost O ( N 2 ) .
Revised: Then we traverse each ( v i , v j ) pair to form the edges of the accessibility graph, which also cost O ( N 2 ) .

Original: Solving the shortest path problem in accessibility graph will also cost O ( N 2 ) as the accessibility graph has N vertexes.
Revised: Solving the shortest path problem in accessibility graph will also cost O ( N 2 ) as the accessibility graph has N vertexes.

Original: Therefore, the overall time complexity of Algorithm 1 would be O ( N 4 ) .
Revised: Therefore, the overall time complexity of Algorithm 1 would be O ( N 4 ) .

Original: The space complexity would be O ( N 2 ) for the table of l ( v i , v j ) and the accessibility graph itself.
Revised: The space complexity would be O ( N 2 ) for the table of l ( v i , v j ) and the accessibility graph itself.

Original: B. A LGORITHM
Revised: B. A LGORITHM

Original: Suppose there are N vertexes in the close set s ij .
Revised: Suppose there are N vertexes in the closed set s ij .

Original: In step 1, getting { v in } and { v out } will cost O ( N ) time for traversing the ancestors and descendents of v t .
Revised: In step 1, getting { v in } and { v out } will cost O ( N ) time for traversing the ancestors and descendents of v t .

Original: In our implementation, an array a of length N is used to represent { v in } and { v out } : a i = 1 indicates v i ∈ { v in } , a i = 2 indicates v i ∈ { v out } and a i = 0 indicates v i / ∈ { v in } ∪ { v out } .
Revised: In our implementation, an array a of length N is used to represent { v in } and { v out } : a i = 1 indicates v i ∈ { v in } , a i = 2 indicates v i ∈ { v out } and a i = 0 indicates v i / ∈ { v in } ∪ { v out } .

Original: Then the union check and intersection check in stepcan be done in O ( N ) .
Revised: Then the union check and intersection check in step 2 can be done in O ( N ) .

Original: The connection check in step 2 traverses the edges and costs O ( N 2 ) .
Revised: The connection check in step 2 traverses the edges and costs O ( N 2 ) .

Original: Other steps are O (1) .
Revised: Other steps are O (1) .

Original: Therefore, the overall time complexity of Algorithm 2 would be O ( N 2 ) .
Revised: Therefore, the overall time complexity of Algorithm 2 would be O ( N 2 ) .

Original: The space complexity would be O ( N ) for the array torepresent { v
Revised: The space complexity would be O ( N ) for the array to

Original: B. A LGORITHM
Revised: B. A LGORITHM

Original: Suppose there are N vertexes in the close set s ij .
Revised: Suppose there are N vertexes in the closed set s ij .

Original: The most time consuming part will be from step 5 to step 13.
Revised: The most time consuming part will be from step 5 to step 13.

Original: Other steps are O (1) .
Revised: Other steps are O (1) .

Original: In step 5 to step 13, every edge between two vertexes in s ij is at most visited once and there are O ( N 2 ) edges.
Revised: In step 5 to step 13, every edge between two vertexes in s ij is at most visited once and there are O ( N 2 ) edges.

Original: Therefore, the overall time complexity of Algorithm 3 would be O ( N 2 ) .
Revised: Therefore, the overall time complexity of Algorithm 3 would be O ( N 2 ) .

Original: In our implementation, an array of length N is used to represent the vertex set s = { v k } .
Revised: In our implementation, an array of length N is used to represent the vertex set s = { v k } .

Original: Therefore, the space complexity would be O ( N ) .
Revised: Therefore, the space complexity would be O ( N ) .

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Suppose there are N vertexes in the close set s ij and there are O ( N 2 ) vertex pairs.
Revised: Suppose there are N vertexes in the closed set s ij and there are O ( N 2 ) vertex pairs.

Original: For each vertex pair, the connection check in step 2-4 will cost O ( N 2 ) , similar to the connection check in
Revised: For each vertex pair, the connection check in step 2-4 will cost O ( N 2 ) , similar to the connection check in

Original: Algorithm 2.
Revised: Algorithm 2.

Original: Thus step 1-4 will cost O ( N 4 ) .
Revised: Thus step 1-4 will cost O ( N 4 ) .

Original: In our implementation, for each vertex in the close set s ij , we select the largest formed close set s kt that contains this vertex.
Revised: In our implementation, for each vertex in the closed set s ij , we select the largest formed closed set s kt that contains this vertex.

Original: The close set number is then reduced to O ( N ) and step 5-6 can be done in O ( N 3 ) .
Revised: The closed set number is then reduced to O ( N ) and step 5-6 can be done in O ( N 3 ) .

Original: Therefore, the overall time complexity of Algorithm 4 would be O ( N 4 )
Revised: Therefore, the overall time complexity of Algorithm 4 would be O ( N 4 )

Original: As O ( N 2 ) close sets can be formed in step 1-4 and each close set is a smaller DAG with O ( N ) vertexes and cost O ( N 2 ) space, the space complexity would be O ( N 4 ) for all these close sets.
Revised: As O ( N 2 ) closed sets can be formed in step 1-4 and each closed set is a smaller DAG with O ( N ) vertexes and cost O ( N 2 ) space, the space complexity would be O ( N 4 ) for all these closed sets.

Original: B. A LGORITHM
Revised: B. A LGORITHM

Original: Step 1 is similar to step 1-4 in Algorithm 4 with s ij being the whole computation graph.
Revised: Step 1 is similar to step 1-4 in Algorithm 4 with s ij being the whole computation graph.

Original: Therefore, the overall time complexity for step 1 is O ( N 4 ) .
Revised: Therefore, the overall time complexity for step 1 is O ( N 4 ) .

Original: In step 2, the complexity of building division tree is related to the complexity of getting the division of a close set.
Revised: In step 2, the complexity of building division tree is related to the complexity of getting the division of a closed set.

Original: For close set type 1, Algorithm 2 is called for each vertex to get all splitting vertexes.
Revised: For closed set type 1, Algorithm 2 is called for each vertex to get all splitting vertexes.

Original: Thus getting the division of close set type 1 cost O ( N 2 ) time.
Revised: Thus getting the division of closed set type 1 cost O ( N 3 ) time.

Original: For close set type 2, Algorithm 3 is used to solve for its division and costs O ( N 2 ) time.
Revised: For closed set type 2, Algorithm 3 is used to solve for its division and costs O ( N 2 ) time.

Original: For type 3, Algorithm 4 is called to solve for its division.
Revised: For type 3, Algorithm 4 is called to solve for its division.

Original: Notice that we have already stored all possible close sets in step 1, step 1-4 in Algorithmcan be skipped and thus the time complexity of getting the division of close set type 3 is reduced to O ( N 3 ) .
Revised: Notice that we have already stored all possible closed sets in step 1, step 1-4 in Algorithm 4 can be skipped and thus the time complexity of getting the division of closed set type 3 is reduced to O ( N 3 ) .

Original: Therefore, getting the division of an arbitrary close set costs O ( N 3 ) time.
Revised: Therefore, getting the division of an arbitrary closed set costs O ( N 3 ) time.

Original: In depth i of the division tree, suppose there are k close sets, and the number of vertexes of j th close sets is a j .
Revised: In depth i of the division tree, suppose there are k closed sets, and the number of vertexes of j th closed sets is a j .

Original: To build depth i + 1 of the division tree, we need to get the division of all these close sets, which will cost (cid:80) j O ( a 3 j ) .
Revised: To build depth i + 1 of the division tree, we need to get the division of all these closed sets, which will cost (cid:80) j O ( a 3 j ) .

Original: As (cid:80) j a j ≤ N , we have (cid:80) j O ( a 3 j ) ≤ O ( N 3 ) .
Revised: As (cid:80) j a j ≤ N , we have (cid:80) j O ( a 3 j ) ≤ O ( N 3 ) .

Original: As the depth of division tree is at most N , the overall time complexity of step 2 would be O ( N 4 ) .
Revised: As the depth of division tree is at most N , the overall time complexity of step 2 would be O ( N 4 ) .

Original: For step 3-10, if the computation graph is linear, the ACG solver will reduce to LCG solver and has complexity O ( N 4 ) .
Revised: For step 3-10, if the computation graph is linear, the ACG solver will reduce to LCG solver and has complexity O ( N 4 ) .

Original: If the computation graph is non-linear, the length of { m } would be O ( N 2 ) for there are O ( N 2 ) vertex pair.
Revised: If the computation graph is non-linear, the length of { m } would be O ( N 2 ) for there are O ( N 2 ) vertex pair.

Original: For a max term m , from step 4-10, the actual time costing part will be step 6 which calls the LCG solver, and other steps would be O (1) .
Revised: For a max term m , from step 4-10, the actual time costing part will be step 6 which calls the LCG solver, and other steps would be O (1) .

Original: Suppose the LCG solver is called k times, solving problems of a 1 , a 2 , ..., a k vertexes.
Revised: Suppose the LCG solver is called k times, solving problems of a 1 , a 2 , ..., a k vertexes.

Original: The total complexity of this would be
Revised: The total complexity of this would be

Original: O ( a 41 ) + O ( a 42 )
Revised: O ( a 41 ) + O ( a 42 )

Original: + ...
Revised: + ...

Original: + O ( a 4 k ) .
Revised: + O ( a 4 k ) .

Original: Notice that a 1 + a 2 + ... + a k ≤ N for the fact that any vertex in the computation graph would not be solved twice by LCG solver, we have a 41 + a 42 + ...
Revised: Notice that a 1 + a 2 + ... + a k ≤ N for the fact that any vertex in the computation graph would not be solved twice by LCG solver, we have a 41 + a 42 + ...

Original: + a 4 k ≤ N 4 .
Revised: + a 4 k ≤ N 4 .

Original: Therefore the time complexity of step 3-10 is O ( N 4 ) .
Revised: Therefore the time complexity of step 3-10 is O ( N 4 ) .

Original: Step 1 would cost O ( N 4 ) space to store all the possible close sets.
Revised: Step 1 would cost O ( N 4 ) space to store all the possible closed sets.

Original: Step 2 would cost O ( N 2 ) space for the division tree.
Revised: Step 2 would cost O ( N 2 ) space for the division tree.

Original: Step 3-10 would cost O ( N 2 ) space for calling LCG solver.
Revised: Step 3-10 would cost O ( N 2 ) space for calling LCG solver.

Original: In conclusion, the overall time complexity of Algorithm 5 is O ( N 4 ) and the overall space complexity of Algorithm 5 is O ( N 4 ) .
Revised: In conclusion, the overall time complexity of Algorithm 5 is O ( N 4 ) and the overall space complexity of Algorithm 5 is O ( N 4 ) .

Original: [Table]
Revised: [Table]

Original: C R UNTIME AND T HEORETICAL A NALYSIS
Revised: C R UNTIME AND T HEORETICAL A NALYSIS

Original: The number of vertexes in the computation graph and the runtime of ACG Solver (Algorithm 5) for each network is listed in Table 2.
Revised: The number of vertexes in the computation graph and the runtime of ACG Solver (Algorithm 5) for each network is listed in Table 2.

Original: All the runtime is measured on a single core of CPU i7-8700.
Revised: All the runtime is measured on a single core of CPU i7-8700.

Original: Notice that the runtime is measured on only 1 cpu core, it can be massively reduced by parallelization in multiple cpu cores.
Revised: Notice that the runtime is measured on only 1 cpu core, it can be massively reduced by parallelization in multiple cpu cores.

Original: The runtime can also be further reduced through a better implementation as our implementation is prototype.
Revised: The runtime can also be further reduced through a better implementation as our implementation is prototype

Original:  Although it might be concerning that the runtime is too much for some deep networks, it is still relatively small compared to training processes which might cost days or even weeks.
Revised: [Table] Although it might be concerning that the runtime is too much for some deep networks, it is still relatively small compared to training processes which might cost days or even weeks.

Original: More importantly, solving the optimal solution for a network is an one-time effort.
Revised: More importantly, solving the optimal solution for a network is an one-time effort.

Original: The optimal solutions for all popular networks will be released online for people to use without taking the time to run ACG solver.
Revised: The optimal solutions for all popular networks will be released online for people to use without taking the time to run ACG solver.

Original: To see how well the reality matches with the theory, we also compare the measured memory cut off and theoretical memory cut off (given by Algorithm 5) in Table 2.
Revised: To see how well the reality matches with the theory, we also compare the measured memory cut off and theoretical memory cut off (given by Algorithm 5) in Table 2.

Original: Observe that all the measured memory cut off are slightly lower than theoretical memory cut off.
Revised: Observe that all the measured memory cut off are slightly lower than theoretical memory cut off.

Original: This is because, in implementation, we assume that the whole input tensors of each operation are always stored for backward.
Revised: This is because, in implementation, we assume that the whole input tensors of each operation are always stored for backward.

Original: In reality some operations only need to store small tensors for backward.
Revised: In reality some operations only need to store small tensors for backward.

Original: For example, batchnormalization only needs a few statistics for backward and doesn’t need the whole input tensor.
Revised: For example, batchnormalization only needs a few statistics for backward and doesn’t need the whole input tensor.

Original: D V ISUALIZATION
Revised: D V ISUALIZATION

Original: We visualize the computation graph of Alexnet, vgg11, vgg13, vgg16  and vgg19 and the solution of our approach (in green) and the solution of Chen’s approach (in red).
Revised: We visualize the computation graph of Alexnet, vgg11, vgg13, vgg16 ,vgg19 and CustomNet and the solution of our approach (in green) and the solution of Chen’s approach (in red).

Original: In the computation graph, the cost of each vertex and the actual operation of each edge are also marked.
Revised: In the computation graph, the cost of each vertex and the actual operation of each edge are also marked.

Original: The cost of each vertex is the size of this tensor during forward given the input as [1 , 3 , 224 , 224] ( [1 , 3 , 300 , 300] for inception v3).
Revised: The cost of each vertex is the size of this tensor during forward given the input as [1 , 3 , 224 , 224] ( [1 , 3 , 300 , 300] for inception v3).

Original: For example, in Alexnet, the input is [1 , 3 , 224 , 224] and thus the source vertex has the cost 

Revised: For example, in Alexnet, the input is [1 , 3 , 224 , 224] and thus the source vertex has the cost

Original: = 1 × 3 × 224 × 224 .
Revised: = 1 × 3 × 224 × 224 .

Original: After 2d convolution and relu, the tensor becomes [1 , 64 , 55 , 55] and thus the second vertex has the cost 

Revised: After 2d convolution and relu, the tensor becomes [1 , 64 , 55 , 55] and thus the second vertex has the cost

Original: = 1 × 64 × 55 × 55 .
Revised: = 1 × 64 × 55 × 55 .

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]