Original: A H IERARCHICAL M ODEL FOR D EVICE P LACEMENT
Revised: A H IERARCHICAL M ODEL FOR D EVICE P LACEMENT

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We introduce a hierarchical model for efﬁcient placement of computational graphs onto hardware devices, especially in heterogeneous environments with a mixture of CPUs, GPUs, and other computational devices.
Revised: We introduce a hierarchical model for efﬁcient placement of computational graphs onto hardware devices, especially in heterogeneous environments with a mixture of CPUs, GPUs, and other computational devices.

Original: The algorithm learns to assign graph operations to groups and to allocate those groups to available devices.
Revised: Our method learns to assign graph operations to groups and to allocate those groups to available devices.

Original: The grouping and device allocations are learned jointly.
Revised: The grouping and device allocations are learned jointly.

Original: The proposed algorithm is trained by a policy gradientmethod and requires no human intervention.
Revised: The proposed method is trained with policy gradient and requires no human intervention.

Original: Experiments with widely-used computer vision and natural language models show that our algorithm can ﬁnd optimized, non-trivial placements for TensorFlow computational graphs with over 80,000 operations.
Revised: Experiments with widely-used computer vision and natural language models show that our algorithm can ﬁnd optimized, non-trivial placements for TensorFlow computational graphs with over 80,000 operations.

Original: In addition, our approach outperforms placements by human experts as well as a previous state-of-the-art placement method based on deep reinforcement learning.
Revised: In addition, our approach outperforms placements by human experts as well as a previous state-of-the-art placement method based on deep reinforcement learning.

Original: Our method achieves runtime reductions of up to 60.6% per training step when applied to models such as Neural Machine Translation.
Revised: Our method achieves runtime reductions of up to 60.6% per training step when applied to models such as Neural Machine Translation.

Original:  I NTRODUCTION & R ELATED W ORK
Revised:  I NTRODUCTION & R ELATED W ORK

Original: Deep neural networks have been successfully applied to many practical problems, such as image classiﬁcation (LeCun et al., 1998; Krizhevsky et al., 2012; Taigman et al., 2014; Szegedy et al., 2015), speech recognition (Hinton et al., 2012;
Revised: Deep neural networks have been successfully applied to many practical problems, such as image classiﬁcation (LeCun et al., 1998; Krizhevsky et al., 2012; Taigman et al., 2014; Szegedy et al., 2015), speech recognition (Hinton et al., 2012;

Original: Hannun et al., 2014), and machine translation (Sutskever et al., 2014; Bahdanau et al., 2015; Wu et al., 2016b).
Revised: Hannun et al., 2014) and machine translation (Sutskever et al., 2014; Bahdanau et al., 2015; Wu et al., 2016b).

Original: These successes have lead to a surge in demand for the computational resources needed to train and infer with neural networks.
Revised: These successes have lead to a surge in demand for the computational resources needed to train and infer with neural networks.

Original: A common approach to addressing this demand is to use a distributed environment with a combination of CPUs and GPUs.
Revised: A common approach to addressing this demand is to use a distributed environment with a combination of CPUs and GPUs.

Original: In this environment, it is typical for a machine learning practitioner to explicitly place the operations of their neural network onto particular computing devices for model parallelism and data parallelism.
Revised: In this environment, it is typical for a machine learning practitioner to explicitly place the operations of their neural network onto particular computing devices for model parallelism and data parallelism.

Original: For example, one might distribute the computation of the ﬁrst layer in a translation network onto the ﬁrst GPU and the computation of the second layer onto the second GPU (Sutskever et al., 2014; Wu et al., 2016b).
Revised: For example, one might distribute the computation of the ﬁrst layer in a translation network onto the ﬁrst GPU and the computation of the second layer onto the second GPU (Sutskever et al., 2014; Wu et al., 2016b).

Original: Although these decisions can be made by a human practitioner, such an approach does not scale well or produce optimal results, especially in the case of more complicated networks (Szegedy et al., 2016b;a).
Revised: Although these decisions can be made by a human practitioner, such an approach does not scale well or produce optimal results, especially in the case of more complicated networks (Szegedy et al., 2016b;a).

Original: Given the growing diversity of hardware devices (e.g., Google
Revised: Given the growing diversity of hardware devices (e.g., Google

Original: TPUs, Intel Nervana, etc.) and recent trends toward automated neural architecture search (Zoph & Le, 2017; Real et al., 2017; Baker et al., 2016), where new models are generated, trained and evaluated in an enitrely end-to-end fashion, it seems natural to move toward more automated solutions for efﬁciently distributing computation.
Revised: TPUs, Intel Nervana, etc.) and recent trends toward automated neural architecture search (Zoph & Le, 2017; Real et al., 2017; Baker et al., 2016), where new models are generated, trained and evaluated in an entirely end-to-end fashion, it seems natural to move toward more automated solutions for efﬁciently distributing computation.

Original: Device placement can be framed as the problem of learning to partition a graph across available devices.
Revised: Device placement can be framed as the problem of learning to partition a graph across available devices.

Original: Given that graph partitioning is a well-studied subject in computer science (Fiduccia & Mattheyses, 1988; Karypis & Kumar, 1995b; Pellegrini, 2009b),these traditional  methods represent a natural baseline for automated device placement.
Revised: Given that graph partitioning is a well-studied subject in computer science (Fiduccia & Mattheyses, 1988; Karypis & Kumar, 1995b; Pellegrini, 2009b), traditional graph partitioning methods represent a natural baseline for automated device placement.

Original: We ran experiments using Scotch (Pellegrini, 2009b), a well-established open source library for graph partitioning, which includes optimizations such as k-way Fiduccia-Mattheyses (Fiduccia & Mattheyses, 1988), Multilevel methods (Barnard & 
Revised: We ran experiments using Scotch (Pellegrini, 2009b), a well-established open source library for graph partitioning, which includes optimizations such as k-way Fiduccia-Mattheyses (Fiduccia & Mattheyses, 1988), Multilevel methods (Barnard & Simon, 1994; Hendrickson & Leland, 1993; Karypis & Kumar, 1995a), the

Original: Simon, 1994; Hendrickson & Leland, 1993; Karypis & Kumar, 1995a), the Band Method (Chevalier & Pellegrini, 2006), the Diffusion Method (Pellegrini, 2007), and DualRecursive Bipartitioning
Revised: Band Method (Chevalier & Pellegrini, 2006), the Diffusion Method (Pellegrini, 2007), and Dual

Original:  Mapping (Pellegrini & Roman, 1996).
Revised: Recursive Bipartitioning Mapping (Pellegrini & Roman, 1996).

Original: The objective was to balance the computational load across a set of connected processing nodes, while colocating neighboring nodes to minimize communication cost.
Revised: The objective was to balance the computational load across a set of connected processing nodes, while colocating neighboring nodes to minimize communication cost.

Original: Despite its promise, this approach yielded disappointing results compared toour approach. A main reason for it is that our cost which is the runtime of executing a TensorFlow graph, is non-stationary.
Revised: Despite its promise, this approach yielded disappointing results, likely due to the non-stationarity of the cost function.

Original: We target a distributed environment where we use a shared cluster of CPUs and GPUs, and our CPUs can also serve other jobs at the same time.
Revised: We target a distributed environment where we use a shared cluster of CPUs and GPUs, and our CPUs may also serve other jobs at the same time.

Original: Thus, while cost-based models such as (Matthias Boehm & Tian, 2014) provide a strong baseline for memory optimizations, since memory usage is deterministic, they cannot be directly applied to environments with dynamic costs.
Revised: Thus, while cost-based models such as (Matthias Boehm & Tian, 2014) provide a strong baseline for memory optimizations, since memory usage is deterministic, they cannot be directly applied to environments with dynamic costs.

Original: Using deep networks and reinforcement learning for combinatorial optimization has already been proposed (Vinyals et al., 2015; Bello et al., 2016; Mirhoseini et al., 2017).
Revised: Using deep networks and reinforcement learning for combinatorial optimization has already been proposed (Vinyals et al., 2015; Bello et al., 2016; Mirhoseini et al., 2017).

Original: Recent work (Mirhoseini et al., 2017) uses a recurrent neural network (RNN) policy network to predict the placement of operations in a computational graph, optimizing for speed of computation using policy gradient methods.
Revised: Recent work (Mirhoseini et al., 2017) uses a recurrent neural network (RNN) policy network to predict the placement of operations in a computational graph, optimizing for speed of computation using policy gradient methods.

Original: While this approach outperforms traditional graph partitioning heuristics and human expert placements, it is prohibitively expensive for the RNN policy to learn when the number of operations is large.
Revised: While this approach outperforms traditional graph partitioning heuristics and human expert placements, it is prohibitively expensive for the RNN policy to learn when the number of operations is large.

Original: This method is therefore limited to small graphs (with fewer than 1000 nodes) and requires human experts to manually partition the graph into collocation groups as a pre-processing step in order to scale to larger graphs.
Revised: This method is therefore limited to small graphs (with fewer than 1000 nodes) and requires human experts to manually partition the graph into collocation groups as a pre-processing step in order to scale to larger graphs.

Original: We refer to the method in (Mirhoseini et al., 2017) as ColocRL.
Revised: We refer to the method in (Mirhoseini et al., 2017) as ColocRL.

Original: In this paper, we propose a more ﬂexible approach which learns to optimize device placement for training neural networks that have tens of thousands of operations with no need for manual grouping.
Revised: In this paper, we propose a more ﬂexible approach which learns to optimize device placement for training neural networks that have tens of thousands of operations with no need for manual grouping.

Original: Our method consists of a two-level hierarchical network, in which the ﬁrst model groups the operations of the graph (the Grouper) and the second model places those groups onto devices (the
Revised: Our method consists of a two-level hierarchical network, in which the ﬁrst model groups the operations of the graph (the Grouper) and the second model places those groups onto devices (the

Original: Placer).
Revised: Placer).

Original: The Grouper is a feed forward network which reads in information about each operation and its context within the graph, in order to predict the group to which that operation should be assigned.
Revised: The Grouper is a feed forward network which reads in information about each operation and its context within the graph, in order to predict the group to which that operation should be assigned.

Original: The Placer is a sequence-to-sequence model (Sutskever et al., 2014) that reads in the embedding of the group and predicts the device placement for that group.
Revised: The Placer is a sequence-to-sequence model (Sutskever et al., 2014) that reads in the embedding of the group and predicts the device placement for that group.

Original: The entire two-level network is trained jointly using reinforcement learning to optimize for speed of computation and for feasibility (e.g., having sufﬁcient memory available on each device for the computation assigned).
Revised: The entire two-level network is trained jointly using reinforcement learning to optimize for speed of computation and for feasibility (e.g., having sufﬁcient memory available on each device for the computation assigned).

Original: Unlike the previous work, our method is end-to-end and does not require human experts to manually group operations as a pre-processing step, making it a fully automated solution to optimizing device placement.
Revised: Unlike the previous work, our method is end-to-end and does not require human experts to manually group operations as a pre-processing step, making it a fully automated solution to optimizing device placement.

Original: Our main result is that our model effectively handles very large graphs and ﬁnds non-trivial placements on multiple devices for models such as Inception-V3 (Szegedy et al., 2016b), ResNet (He et al., 2016),
Revised: Our main result is that our model effectively handles very large graphs and ﬁnds non-trivial placements on multiple devices for models such as Inception-V3 (Szegedy et al., 2016b), ResNet (He et al., 2016),

Original: Language Modeling (Jozefowicz et al., 2016), and Neural Machine Translation (Wu et al., 2016b).
Revised: Language Modeling (Jozefowicz et al., 2016), and Neural Machine Translation (Wu et al., 2016b).

Original: The placements found by our model outperform TensorFlow’s default placements (Abadi et al., 2016), the Scotch algorithm’s placements, and human expert placements, as well as those of ColocRL (Mirhoseini et al., 2017).
Revised: The placements found by our model outperform TensorFlow’s default placements (Abadi et al., 2016), the Scotch algorithm’s placements, and human expert placements, as well as those of ColocRL (Mirhoseini et al., 2017).

Original: Our results demonstrate that the proposed approach learns the properties of the environment, including the complex tradeoff between computation and communication in hardware.
Revised: Our results demonstrate that the proposed approach learns the properties of the environment, including the complex tradeoff between computation and communication in hardware.

Original: For example, on a Neural Machine Translation model, our method achieves a 60.6% reduction in training time per iteration.
Revised: For example, on a Neural Machine Translation model, our method achieves a 60.6% reduction in training time per iteration.

Original:  M ETHOD
Revised:  M ETHOD

Original: An overview of our hierarchical model for device placement is shown in Figure 1.
Revised: An overview of our hierarchical model for device placement is shown in Figure 1.

Original: Our model consists of two sub-networks: A Grouper that assigns operations to groups and a Placer that assigns groups to target devices.
Revised: Our model consists of two sub-networks: A Grouper that assigns operations to groups and a Placer that assigns groups to target devices.

Original: The two models are trained jointly.
Revised: The two models are trained jointly.

Original: The objective of the proposed approach, which we refer to as the Hierarchical Planner, is to predict a placement that speeds up the training of neural network graphs.
Revised: The objective of the proposed approach, which we refer to as the Hierarchical Planner, is to predict a placement that speeds up the training of neural network graphs.

Original: The runtime we are optimizing for is the time taken to conduct one forward pass, one back-propagation pass, and one parameter updatepass on the target neural network.
Revised: The runtime we are optimizing for is the time taken to conduct one forward pass, one back-propagation pass, and one parameter update on the target neural network.

Original: To measure the runtime, the predicted placement is run on actual hardware.
Revised: To measure the runtime, the predicted placement is run on actual hardware.

Original: Since the reward (runtime) in this problem is non-differentiable, we use policy gradients to train the Hierarchical Planner.
Revised: Since the reward (runtime) in this problem is non-differentiable, we use policy gradients to train the Hierarchical Planner.

Original: Moreover, the policy gradients ﬂow through to train both the feed forward Grouper and the recurrent Placer.
Revised: Moreover, the policy gradients ﬂow through to train both the feed forward Grouper and the recurrent Placer.

Original: The Grouper assigns each operation to a group.
Revised: The Grouper assigns each operation to a group.

Original: Once all the operations are grouped, we use information about each member operation to generate an embedding for that group.
Revised: Once all the operations are grouped, we use information about each member operation to generate an embedding for that group.

Original: We then pass these embeddings as input to the Placer, which computes device placements for each group.
Revised: We then pass these embeddings as input to the Placer, which computes device placements for each group.

Original: The
Revised: The

Original: Placer assigns zero or more groups to each available device.
Revised: Placer assigns zero or more groups to each available device.

Original: The ﬁnal placement is determined by placing operations on the device that their group was assigned to.
Revised: The ﬁnal placement is determined by placing each operation on the device that its group was assigned to.

Original: In our implementation, the Grouper is a feed forward model followed by a softmax layer with an output size equal to the number of groups.
Revised: In our implementation, the Grouper is a feed forward model followed by a softmax layer with an output size equal to the number of groups.

Original: The Placer is a sequence-to-sequence model (Sutskever et al., 2014) with Long Short-Term Memory (Hochreiter & Schmidhuber, 1997) and a content-based attention mechanism (Bahdanau et al., 2015) to predict the placements.
Revised: The Placer is a sequence-to-sequence model (Sutskever et al., 2014) with Long Short-Term Memory (Hochreiter & Schmidhuber, 1997) and a content-based attention mechanism (Bahdanau et al., 2015) to predict the placements.

Original: For creating embedding for each operation, we concatenate 3 vectors: 1) A vector that embeds operation type information (e.g., MatMul, Conv2d, Sum, etc.).
Revised: We ﬁrst generate operation embeddings to pass as input to the Grouper. Each operation embedding consists of 3 vectors: 1) A vector that embeds operation type information (e.g., MatMul, Conv2d, Sum, etc.).

Original: We learn this vector similarly to how language model embedding is learned. Our vocabulary is the set of all TF operations andwe learn an operation  embedding of size 20. 2)
Revised: We treat this as a language modeling task and learn an operation type embedding of size 20 with a vocabulary of the 200 most commonly used TF operations.

Original: A vector that contains output sizes and number of outputs for an operation.
Revised: A vector that contains output sizes and number of outputs for each operation.

Original: We set a ﬁxed threshold for maximum number of output edges for an operation, and for each output edge we set a threshold (6) for maximum dimension (4).
Revised: We limit the number of output edges to 6 and the size of each of these outputs to 4 elements.

Original: We ﬁll this vectorof size 24 by reading the outputs of an operation one by one and putting in the output operations shapes.
Revised: We populate this vector by reading the outputs of an operation one by one and inserting the output operations shapes.

Original: We ﬁll the vector with -1 for non-existing outputs edges or dimensions.
Revised: We pad the vector with -1 if we have fewer outgoing edges or smaller sizes.

Original: A vector that contains adjacency information for that operation.
Revised: A vector that contains adjacency information for that operation.

Original: We index the graph by traversing it in a BFS manner and set the maximum number of incoming and outgoing edges to 12 (6 for each direction).
Revised: We index the graph by traversing it in a BFS manner and set the maximum number of incoming and outgoing edges to 12 (6 for each direction).

Original: We then ﬁll the vector with the index of the incoming and outgoing operations.
Revised: We then ﬁll the vector with the index of the incoming and outgoing operations.

Original: We ﬁll the vector with -1, for non-existing edges.
Revised: We pad the vector with -1, in cases where the number of incoming or outgoing edges is less than 6.

Original: For creating embedding for each group, we concatenate 3 vectors: 1) A vector that counts how many of each operation typeis available in that group.
Revised: To generate input for the Placer, we take each group and create its group embedding by concatenating 3 vectors: 1) A vector containing the count of each operation type in the group.

Original: The size of this vector is the size of vocabulary of all TF most widely used operations which we limit to 200. 2)

Original: A vector that counts the overall output shapes of all the operations in that group.
Revised: A vector that counts the total number of output shapes of all the operations in that group.

Original: This vector is created by adding all the operation output shape embedding described above (not including the -1) and is of size 16.
Revised: This vector is created by concatenating all the operation output shape embeddings described above (not including the -1) and is of size 16.

Original: A vector that contains group adjacency information.
Revised: A vector that contains group adjacency information.

Original: The size of this vector is the number of groups (256 in our experiments), and its i’th value is 1 if the group has edges to the i’th group andis 0 otherwise.
Revised: The size of this vector is the number of groups (256 in our experiments), and its i-th value is 1 if the group has edges to the i-th group and 0 otherwise.

Original: The Placer’s RNN encoder reads the group embeddings one at a time and produces M hidden states.
Revised: The Placer’s RNN encoder reads the group embeddings one at a time and produces M hidden states.

Original: We treat M , which is equal to the number of groups, as a hyperparameter.
Revised: We treat M , which is equal to the number of groups, as a hyperparameter.

Original: The Placer’s decoder
Revised: The Placer’s decoder

Original: RNN predicts one device per time step.
Revised: RNN predicts one device per time step.

Original: The devices are returned in the same order as the input group embeddings, i.e., the operations in the ﬁrst group will be placed on the device returned by the ﬁrst decoder step, and so on.
Revised: The devices are returned in the same order as the input group embeddings, i.e., the operations in the ﬁrst group will be placed on the device returned by the ﬁrst decoder step, and so on.

Original: Each device has its own trainable embedding, which is then fed as input to the next decoder time step.
Revised: Each device has its own trainable embedding, which is then fed as input to the next decoder time step.

Original: At each step t (where 1 ≤ t ≤ M ), the decoder uses an attention mechanism to attend over the encoder states.
Revised: At each step t (where 1 ≤ t ≤ M ), the decoder uses an attention mechanism to attend over the encoder states.

Original: We use the attention mechanism from Vinyals et al.
Revised: We use the attention mechanism from Vinyals et al.

Original: At training time, the decoder samples one device d t per step from the Placer’s softmax.
Revised: At training time, the decoder samples one device d t per step from the Placer’s softmax.

Original: To make the activations l t less steep and to allow the model to explore, we follow Bello et al.
Revised: To make the activations l t less steep and to allow the model to explore, we follow Bello et al.

Original: and use a temperature T and apply a tanh constant C to l t .
Revised: and use a temperature T and apply a tanh constant C to l t .

Original: Thus, we sample d t as follows:
Revised: Thus, we sample d t as follows:

Original: [Equation]
Revised: [Equation]

Original: The placement decisions are then used to place the model.
Revised: The placement decisions are then used to place the model.

Original: In the following section, we describe a policy gradient method to train the planner such that it improves its decisions over time.
Revised: In the following section, we describe a policy gradient method to train the Hierarchical Planner, such that it improves its decisions over time.

Original: Training with REINFORCE:
Revised: Training with REINFORCE:

Original: The planner optimizes the training time for a target model (e.g., a TensorFlow graph) given the decisions made by the Grouper and  Placer.
Revised: The planner optimizes the training time for a target model (e.g., a TensorFlow graph) given the decisions made by the Grouper and the Placer.

Original: Let r d be the runtime per training step for a predicted device placement d .
Revised: Let r d be the runtime per training step for a predicted device placement d .

Original: We deﬁne the reward for placement d as
Revised: We deﬁne the reward for placement d as

Original: R d = − sqrt ( r ) .
Revised: R d = − sqrt ( r ) .

Original: The planner should try to maximize the expectation of R d given its decisions.
Revised: The planner should try to maximize the expectation of R d given its decisions.

Original: As such, the cost function we are optimizing for is:
Revised: As such, the cost function we are optimizing for is:

Original: [Equation]
Revised: [Equation]

Original: Let θ g
Revised: Let θ g

Original: and θ d be parameters of the Grouper and Placer, respectively.
Revised: and θ d be parameters of the Grouper and Placer, respectively.

Original: Here, p ( g ; θ g ) is the probability of a sample group assignment g drawn from the Grouper softmax distribution ∼ π g and p ( d ; θ d ) is the probability of a sample device placement d drawn from the Placer softmax distribution ∼ π d .
Revised: Here, p ( g ; θ g ) is the probability of a sample group assignment g drawn from the Grouper softmax distribution ∼ π g and p ( d ; θ d ) is the probability of a sample device placement d drawn from the Placer softmax distribution ∼ π d .

Original: We can write the derivative of the cost function deﬁned in Eq.
Revised: We can write the derivative of the cost function deﬁned in Eq.

Original: 2 w.r.t.
Revised: 2 w.r.t.

Original: θ g and θ d as follows:
Revised: θ g and θ d as follows:

Original: [Equation]
Revised: [Equation]

Original: Deriving Eqs.
Revised: Deriving Eqs.

Original: 3 and 5 from the cost function is straightforward.
Revised: 3 and 5 from the cost function is straightforward.

Original: We use the REINFORCE rule (Williams, 1992) and approximate expectation values with samples g i and d j drawn from the Grouper and Placer to arrive at Eqs.
Revised: We use the REINFORCE rule (Williams, 1992) and approximate expectation values with samples g i and d j drawn from the Grouper and Placer to arrive at Eqs.

Original: 4 and 6.
Revised: 4 and 6.

Original: In our implementation, the Grouper makes independent predictions when assigning operations to groups.
Revised: In our implementation, the Grouper makes independent predictions when assigning operations to groups.

Original: The Placer, however, conditions the placement of groups on those that have already been placed.
Revised: The Placer, however, conditions the placement of groups on those that have already been placed.

Original: To reduce the variance, we also subtract a baseline B from R .
Revised: To reduce the variance, we also subtract a baseline B from R .

Original: In our experiments, we found that the exponential moving average of the reward was an effective baseline.
Revised: In our experiments, we found that the exponential moving average of the reward was an effective baseline.

Original: Distributed Training: Our policy is trained in a distributed manner.
Revised: Distributed Training: Our policy is trained in a distributed manner.

Original: Our framework has a parameter server that is shared among several controllers.
Revised: Our framework has a parameter server that is shared among several controllers.

Original: All controllers use the same set of parameters and update the policy asynchronously.
Revised: All controllers use the same set of parameters and update the policy asynchronously.

Original: Each controller communicates with k worker nodes, where k is as shown in Eqs.
Revised: Each controller communicates with k worker nodes, where k is as shown in Eqs.

Original: 4 and 6.
Revised: 4 and 6.

Original: Each worker interacts with only one controller.
Revised: Each worker interacts with only one controller.

Original: Each worker executes the placement given by its controller and reports the runtime.
Revised: Each worker executes the placement given by its controller and reports the runtime.

Original: In our experiments, we use 4 controllers and 16 workers (4 per controller).
Revised: In our experiments, we use 4 controllers and 16 workers (4 per controller).

Original: For example, if we are optimizing a placement on 2 GPUs, each worker needs 2 GPUs to measure runtime.
Revised: For example, if we are optimizing a placement on 2 GPUs, each worker needs 2 GPUs to measure runtime.

Original: Each controller is hosted on a single GPU.
Revised: Each controller is hosted on a single GPU.

Original: Therefore, we use a total of 36 GPUs for this example.
Revised: Therefore, we use a total of 36 GPUs for this example.

Original: The workers run the placements in parallel.
Revised: The workers run the placements in parallel.

Original: Once all workers have ﬁnished running the placements, the controller computes the gradients using the measured runtimes.
Revised: Once all workers have ﬁnished running the placements, the controller computes the gradients using the measured runtimes.

Original: To reduce the variance of runtime measurements across workers (different machines), each controller maintains a separate baseline.
Revised: To reduce the variance of runtime measurements across workers (different machines), each controller maintains a separate baseline.

Original: While we get our best results by using many workers, we show in Section 3 that it is possible to train the policy and achieve comparable results with far fewer resources.
Revised: While we get our best results by using many workers, we show in Section 3 that it is possible to train the policy and achieve comparable results with far fewer resources.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we apply Hierarchical Planner to widely used machine learning models in computer vision and natural language processing.
Revised: In this section, we apply Hierarchical Planner to widely used machine learning models in computer vision and natural language processing.

Original: We compare our results to heuristic and RL-based graph optimization baselines and demonstrate our approach’s ability to ﬁnd performant placements.
Revised: We compare our results to heuristic and RL-based graph optimization baselines and demonstrate our approach’s ability to ﬁnd performant placements.

Original: We also compare our method with two simpler alternatives: 1).
Revised: We also compare our method with two simpler alternatives: 1).

Original: no grouping (a feed forward model that directly places each operation) and 2).
Revised: no grouping (a feed forward model that directly places each operation) and 2).

Original: random grouping (a random Grouper feeding into a learned
Revised: random grouping (a random Grouper feeding into a learned

Original: Placer), to demonstrate that our hierarchical architecture allows us to learn better placements.
Revised: Placer), to demonstrate that our hierarchical architecture allows us to learn better placements.

Original: Models:
Revised: Models

Original: We evaluate our approach on four widely used deep neural networks
Revised: We evaluate our approach on four widely used deep neural networks

Original: • Inception-V3(batch size=256) (Szegedy et al., 2016b) is a model used for a variety of computer vision tasks, including classiﬁcation, recognition, or generation (Khetan & Oh, 2016; Esteva et al., 2016).
Revised: • Inception-V3 (Szegedy et al., 2016b) is a model used for a variety of computer vision tasks, including classiﬁcation, recognition, or generation (Khetan & Oh, 2016; Esteva et al., 2016).

Original: The network consists of multiple blocks, each of which is made up of several convolutional and pooling layers.
Revised: The network consists of multiple blocks, each of which is made up of several convolutional and pooling layers.

Original: Within a block, the layers can be executed in parallel.
Revised: Within a block, the layers can be executed in parallel.

Original: However, since the outputs of each block are concatenated together to form the input to the next block, the blocks must be executed sequentially.
Revised: However, since the outputs of each block are concatenated together to form the input to the next block, the blocks must be executed sequentially.

Revised: We use a batch size of 1.

Original: The TensorFlow graph encoding this model contains 24,713 operations.
Revised: The TensorFlow graph encoding this model contains 24,713 operations.

Original: • ResNet (batch size=128)
Revised: • ResNet (He et al., 2016)

Original: (He et al., 2016) is a popular model for image classiﬁcation.
Revised: is a popular model for image classiﬁcation.

Original: It is a deep convolutional network that uses residual connections to avoid the vanishing gradient problem.
Revised: It is a deep convolutional network that uses residual connections to avoid the vanishing gradient problem.

Revised: We use batch size 128.

Original: The TensorFlow implementation of this model has 20,586 operations.
Revised: The TensorFlow implementation of this model has 20,586 operations.

Original: • RNNLM(batch size=64) (Zaremba et al., 2014;
Revised: • RNNLM (Zaremba et al., 2014;

Original: Jozefowicz et al., 2016)
Revised: Jozefowicz et al., 2016)

Original: Recurrent Neural Network Language Model is made of many LSTM cells organized in a grid structure.
Revised: Recurrent Neural Network Language Model is made of many LSTM cells organized in a grid structure.

Original: The processing of each LSTM cell only depends on the results of 2 other cells, which make the concurrent execution of many LSTM cells possible given enough hardware resources. 
Revised: The processing of each LSTM cell only depends on the results of 2 other cells, which make the concurrent execution of many LSTM cells possible given enough hardware resources. We use batch size 64.

Original: The corresponding TensorFlow graph contains 9,021 operations.
Revised: The corresponding TensorFlow graph contains 9,021 operations.

Original: • NMT(batch size=64) (Bahdanau et al., 2015; Wu et al., 2016a)
Revised: • NMT (Bahdanau et al., 2015; Wu et al., 2016a)

Original: Neural Machine Translation with attention mechanism has an architecture similar to that of RNNLM, but its many hidden states make it far more computationally expensive.
Revised: Neural Machine Translation with attention mechanism has an architecture similar to that of RNNLM, but its many hidden states make it far more computationally expensive.

Original: To decrease the training time, both Sutskever et al.
Revised: To decrease the training time, both Sutskever et al.

Original: (2014) and Wu et al.
Revised: (2014) and Wu et al.

Original: (2016a) propose placing each LSTM layer, as well as the attention and the softmax layer, on a separate device.
Revised: (2016a) propose placing each LSTM layer, as well as the attention and the softmax layer, on a separate device.

Original: While this strategy results in meaningful speed improvements, we show that our Hierarchical Planner can ﬁnd signiﬁcantly better placements.
Revised: While this strategy results in meaningful speed improvements, we show that our Hierarchical Planner can ﬁnd signiﬁcantly better placements.

Original: We  evaluated 3 versions of the NMT model: – The original 2-layer encoder-decoder consisting of 28,044 operations.
Revised: We use batch size 64. We evaluated 3 versions of the NMT model: – The original 2-layer encoder-decoder consisting of 28,044 operations.

Original: – An extended 4-layer version consisting of 46,600 operations.
Revised: – An extended 4-layer version consisting of 46,600 operations.

Original: – An even larger 8-layer version consisting of 83,712 operations.
Revised: – An even larger 8-layer version consisting of 83,712 operations.

Original: For a fair comparison to previous state-of-the-art deep RL methods (Mirhoseini et al., 2017), we use the same model architectures (Inception-V3, RNNLM, and 2-Layer NMT models), hyperparameters and input data.
Revised: For a fair comparison to previous state-of-the-art deep RL methods (Mirhoseini et al., 2017), we use the same model architectures (Inception-V3, RNNLM, and 2-Layer NMT models), hyperparameters and input data.

Original: In addition, we evaluate our model on a 152-layer ResNet (He et al., 2016) with
Revised: In addition, we evaluate our model on a 152-layer ResNet (He et al., 2016) with

Original: ImageNet data (Deng et al., 2009), as well as more complex NMT models with 4 and 8 layers.
Revised: ImageNet data (Deng et al., 2009), as well as more complex NMT models with 4 and 8 layers.

Original: Baselines: We compare the placements found by our approach to the following baselines:
Revised: Baselines: We compare the placements found by our approach to the following baselines:

Original: • CPU Only.
Revised: • CPU Only.

Original: Here, we execute the model on a single CPU.
Revised: Here, we execute the model on a single CPU.

Original: While this is generally slow, we ﬁnd that some large models are very hard to ﬁt on GPUs, given their memory limitations, leaving a CPU-only placement as the only naive option.
Revised: While this is generally slow, we ﬁnd that some large models are very hard to ﬁt on GPUs, given their memory limitations, leaving a CPU-only placement as the only naive option.

Original: • GPU Only.
Revised: • GPU Only.

Original: In cases where it is possible to ﬁt the entire model on a single GPU, this is a strong baseline as most graph operations run fastest on GPU and this placement incurs no cross-device communication cost.
Revised: In cases where it is possible to ﬁt the entire model on a single GPU, this is a strong baseline as most graph operations run fastest on GPU and this placement incurs no cross-device communication cost.

Original: For operations that are not implemented on GPU, TensorFlow automatically places them on CPU.
Revised: For operations that are not implemented on GPU, TensorFlow automatically places them on CPU.

Original: • Scotch.
Revised: • Scotch.

Original: We use the Scotch static mapper (Pellegrini, 2009a) which takes as input the graph, the computational cost of each operation, the volume of data ﬂowing through each edge, and the compute and communication capacities of the pertinent devices.
Revised: We use the Scotch static mapper (Pellegrini, 2009a) which takes as input the graph, the computational cost of each operation, the volume of data ﬂowing through each edge, and the compute and communication capacities of the pertinent devices.

Original: • MinCut.
Revised: • MinCut.

Original: We use the Scotch optimizer, but we only consider GPUs as our devices.
Revised: We use the Scotch optimizer, but we only consider GPUs as our devices.

Original: The objective is to balance computation across all the available devices while minimizing interdevice communication.
Revised: The objective is to balance computation across all the available devices while minimizing interdevice communication.

Original: • Human Expert.
Revised: • Human Expert.

Original: We use hand-crafted placements from previous publications.
Revised: We use hand-crafted placements from previous publications.

Original: For InceptionV3 (Szegedy et al., 2016b) and Resnet (He et al., 2016), where it is difﬁcult to exploit model parallelism, human experts place the graph on a single GPU.
Revised: For InceptionV3 (Szegedy et al., 2016b) and Resnet (He et al., 2016), where it is difﬁcult to exploit model parallelism, human experts place the graph on a single GPU.

Original: For RNNLM and NMT, existing work (Sutskever et al., 2014; Wu et al., 2016a) places each LSTM layer on a separate GPU.
Revised: For RNNLM and NMT, existing work (Sutskever et al., 2014; Wu et al., 2016a) places each LSTM layer on a separate GPU.

Original: For NMT, the attention and softmax layers are placed on the same device as the ﬁnal LSTM layer, while the embedding layer is colocated with the ﬁrst LSTM layer.
Revised: For NMT, the attention and softmax layers are placed on the same device as the ﬁnal LSTM layer, while the embedding layer is colocated with the ﬁrst LSTM layer.

Original: • ColocRL.
Revised: • ColocRL.

Original: This method (Mirhoseini et al., 2017) uses policy gradient to train a recurrent neural network that reads in hand-crafted colocation groups and then places each group on a device.
Revised: This method (Mirhoseini et al., 2017) uses policy gradient to train a recurrent neural network that reads in hand-crafted colocation groups and then places each group on a device.

Original: Measuring Reward: Our reward is the negative square root of the runtime for one training step of the target TensorFlow model (lower runtimes are better).
Revised: Measuring Reward: Our reward is the negative square root of the runtime for one training step of the target TensorFlow model (lower runtimes are better).

Original: We assign a large negative reward ofto invalid placements (e.g., due to memory limitations).
Revised: We assign a large negative reward ofto invalid placements (e.g., due to memory limitations).

Original: We deﬁne runtime as the time in seconds required to complete one training step of the target model (i.e. one forward pass, one back-propagation pass, and one parameter update).
Revised: We deﬁne runtime as the time in seconds required to complete one training step of the target model (i.e. one forward pass, one back-propagation pass, and one parameter update).

Original: To reduce measurement variance, we run each predicted placement of the model for 10 steps.
Revised: To reduce measurement variance, we run each predicted placement of the model for 10 steps.

Original: We discard thr ﬁtst 5 steps (to avoid warmup variations) and use the median value of the enxt 5 steps to calculate the reward.
Revised: We discard the ﬁrst 5 steps (to avoid warmup variations) and use the median value of the next 5 steps to calculate the reward.

Original: We found empirically that calculating the reward with square root yielded better placements than identity or logarithm.
Revised: We found empirically that calculating the reward with square root yielded better placements than identity or logarithm.

Original: Note that by altering the reward, we can use our proposed method for optimizing other metrics, such as inference speed, throughput, and network congestion.
Revised: Note that by altering the reward, we can use our proposed method for optimizing other metrics, such as inference speed, throughput, and network congestion.

Original: Devices and Software: Our experiments are run on machines with 1 Intel Haswell 2300 CPU and up to 8 Nvidia Tesla K40 GPUs.
Revised: Devices and Software: Our experiments are run on machines with 1 Intel Haswell 2300 CPU and up to 8 Nvidia Tesla K40 GPUs.

Original: We use TensorFlow r1.3 to run our experiments.
Revised: We use TensorFlow r1.3 to run our experiments.

Original: Architecture of the Policy Network: For the Hierarchical Planner, the Grouper is a feed forward network with a hidden size of 64 and the Placer is a sequence-to-sequence (Sutskever et al., 2014) model with an LSTM hidden size of 256.
Revised: Architecture of the Policy Network: In the Hierarchical Planner, the Grouper is a feed forward network with a hidden size of 64 and the Placer is a sequence-to-sequence (Sutskever et al., 2014) model with an LSTM hidden size of 256.

Original: For the encoder of the sequence-to-sequence model, we used two layers of LSTM to form a bi-LSTM similar to (Wu et al., 2016b).
Revised: For the encoder of the sequence-to-sequence model, we used two layers of LSTM to form a bi-LSTM similar to (Wu et al., 2016b).

Original: We used a uni-directional
Revised: We used a uni-directional

Original: LSTM for the decoder.
Revised: LSTM for the decoder.

Original: The Grouper’s softmax output size is equal to the number of groups, which we set to 256 in our experiments.
Revised: The Grouper’s softmax output size is equal to the number of groups, which we set to 256 in our experiments.

Original: We also experimented with a range of group sizes (64 to 1024), but got the best results with group size 256.
Revised: We also experimented with a range of group sizes (64 to 1024), but got the best results with group size 256.

Original: The number of unrolled steps in the Placer is equal to the number of groups.
Revised: The number of unrolled steps in the Placer is equal to the number of groups.

Original: The Placer’s softmax output size in both models is equal to the number of available hardware devices.
Revised: The Placer’s softmax output size in both models is equal to the number of available hardware devices.

Original: Training Details: We train both policies using Adam (Kingma & Ba, 2015) optimizer with a ﬁxed learning rate of 0.1, gradient clipping of norm 1.0, tanh constant C = 5 . 0 , and temperature T = 10 . 0 .
Revised: Training Details: We train both policies using Adam (Kingma & Ba, 2015) optimizer with a ﬁxed learning rate of 0.1, gradient clipping of norm 1.0, tanh constant C = 5 . 0 , and temperature T = 10 . 0 .

Original: The number of Grouper and Placer samples in Eqs.
Revised: The number of Grouper and Placer samples in Eqs.

Original: 4 and 6 are m = 1 and k = 4 , respectively.
Revised: 4 and 6 are m = 1 and k = 4 , respectively.

Original: To encourage more exploration, we added noise to the logits of both the Grouper and the Placer networks for the ﬁrst 500 policy training steps.
Revised: To encourage more exploration, we added noise to the logits of both the Grouper and the Placer networks for the ﬁrst 500 policy training steps.

Original: The noise was sampled from the normal distribution and modulated to have a max amplitude of 0.1.
Revised: The noise was sampled from the normal distribution and modulated to have a max amplitude of 0.1.

Original: Given that the vast majority of placements are invalid, especially forthe more complex models such as NMT
Revised: Given that the vast majority of placements are invalid, especially for more complex models such as NMT

Original: [Table]
Revised: [Table]

Original: we update the policy only for valid placements after the ﬁrst 500 steps.
Revised: we update the policy only for valid placements after the ﬁrst 500 steps.

Original: By updating the baseline and the policy only for samples that give valid placements
Revised: By updating the baseline and the policy only for samples that give valid placements

Original: [Table]
Revised: [Table]

Original: we prevent the policy from converging to the reward associated with invalid placements.
Revised: we prevent the policy from converging to the reward associated with invalid placements.

Original: [Table]
Revised: [Table]

Original: Results Compared with Graph Partitioning Heuristics: In Table 1, we report the performance of the Hierarchical Planner and compare it to the aforementioned baselines.
Revised: Results Compared with Graph Partitioning Heuristics: In Table 1, we report the performance of the Hierarchical Planner and compare it to the aforementioned baselines.

Original: The only information available to our models is the TensorFlow graph and a list of devices.
Revised: The only information available to our models is the TensorFlow graph and a list of devices.

Original: The reduction percentages are computed by taking the difference between the runtime achieved by the Hierarchical Planner and that of the best prior placement, and then dividing it by that best prior runtime.
Revised: The reduction percentages are computed by taking the difference between the runtime achieved by the Hierarchical Planner and that of the best prior placement, and then dividing it by that best prior runtime.

Original: For ResNet and RNNLM, our model learns that it is more efﬁcient to use a single GPU, as this minimizes communication cost.
Revised: For ResNet and RNNLM, our model learns that it is more efﬁcient to use a single GPU, as this minimizes communication cost.

Original: For Inception-V3 with batch size 256, the Hierarchical Planner learns to distribute the model across 2 GPUs, achieving a 16.3% reduction in runtime over placing the model on a single GPU.
Revised: For Inception-V3, the Hierarchical Planner learns to distribute the model across 2 GPUs, achieving a 16.3% reduction in runtime over placing the model on a single GPU.

Original: [Figure]
Revised: [Figure]

Original: For NMT with 2, 4, and 8 layers, we ran experiments with 2, 4, and 8 GPUs, respectively.
Revised: For NMT with 2, 4, and 8 layers, we ran experiments with 2, 4, and 8 GPUs, respectively.

Original: We outperform the best prior results by 60.6% for NMT (2-layer) and 53.7% for NMT (4-layer).
Revised: We outperform the best prior results by 60.6% for NMT (2-layer) and 53.7% for NMT (4-layer).

Original: For further insight into the model’s behavior, we visualize its placement for NMT (4-layer) in Figure 2.
Revised: For further insight into the model’s behavior, we visualize its placement for NMT (4-layer) in Figure 2.

Original: For NMT (8-layer), the Hierarchical Planner ﬁnds a placement that is 4.9% slower than that of human experts.
Revised: For NMT (8-layer), the Hierarchical Planner ﬁnds a placement that is 4.9% slower than that of human experts.

Original: Even in this one case where the method slightly underperforms, it is still useful to have an automated method of ﬁnding placements that are comparable to those of human experts.
Revised: Even in this one case where the method slightly underperforms, it is still useful to have an automated method of ﬁnding placements that are comparable to those of human experts.

Original: Results associated with both Scotch and MinCut were signiﬁcantly worse than human expert baselines, which is consistent with results reported in (Mirhoseini et al., 2017).
Revised: Results associated with both Scotch and MinCut were signiﬁcantly worse than human expert baselines, which is consistent with results reported in (Mirhoseini et al., 2017).

Original: Results Compared with ColocRL:
Revised: Results Compared with ColocRL:

Original: A fair comparison with ColocRL would require us to run the models using exactly the same software (TensorFlow version) and hardware (CPU and GPU types).
Revised: A fair comparison with ColocRL would require us to run the models using exactly the same software (TensorFlow version) and hardware (CPU and GPU types).

Original: Although our runtimes are considerably faster, they are not directly comparable to those reported in (Mirhoseini et al., 2017), because we ran with different GPU types (the slower k40 for us vs. their k80) and TensorFlow versions (our r1.3 vs. their unspeciﬁed but presumably earlier version).
Revised: Although our runtimes are considerably faster, they are not directly comparable to those reported in (Mirhoseini et al., 2017), because we ran with different GPU types (the slower k40 for us vs. their k80) and TensorFlow versions (our r1.3 vs. their unspeciﬁed but presumably earlier version).

Original: In the following section, we will discuss the relative improvements achieved by our approach.
Revised: We will discuss the relative improvements achieved by our approach.

Original: For the purpose of comparison with ColocRL, we ran experiments on Inception-V3 with a small batch size (32) and 2 GPUs or 4 GPUs.

Original: Unlike ColocRL, our policy chose to place the model on a single

Original: GPU (leaving the other GPUs unused).

Original: This is perhaps due to recent performance improvements in

Original: TensorFlow which have altered the subtle tradeoff between computation and communication.

Original: With a larger batch size (256), however, our policy distributed the computation across two GPUs, reducing the runtime by 16.3%.

Original: ColocRL, on the other hand, required 4 GPUs to achieve a comparable reduction in runtime (19.7%) for the same batch size.

Original: For NMT (2-layer), our improvement over best heuristics is 60.6%, compared to 19.0% for ColocRL.
Revised: For NMT (2-layer), our improvement over best heuristics is 60.6%, compared to 19.0% for ColocRL.

Original: For NMT (4-layer) and NMT (8-layer), no results were reported for ColocRL, which we suspect is due to the model being unable to handle the large number of operations in these graphs.
Revised: For NMT (4-layer) and NMT (8-layer), no results were reported for ColocRL, which we suspect is due to the model being unable to handle the large number of operations in these graphs.

Original: Unlike our method, ColocRL makes the strong assumption that certain operations must be colocated.
Revised: Unlike our method, ColocRL makes the strong assumption that certain operations must be colocated.

Original: Figure 2 shows the high granularity of the Hierarchical Planner’s placements, a degree of parallelism that would be infeasible for prior methods.
Revised: Figure 2 shows the high granularity of the Hierarchical Planner’s placements, a degree of parallelism that would be infeasible for prior methods.

Original: For example, the Hierarchical Planner places each step of an unrolled LSTM layer across multiple GPUs, whereas ColocRL colocates all operations in a step.
Revised: For example, the Hierarchical Planner places each step of an unrolled LSTM layer across multiple GPUs, whereas ColocRL colocates all operations in a step.

Original: Analysis: Here, we want to understand and analyze the placements generated by the RL model.
Revised: Analysis: Here, we want to understand and analyze the placements generated by the RL model.

Original: In
Revised: In

Original: Figure 2, we show a portion of the placement found by the Hierarchical Planner for NMT (4-layer).
Revised: Figure 2, we show a portion of the placement found by the Hierarchical Planner for NMT (4-layer).

Original: With this placement, the runtime per training iteration is 53 .
Revised: With this placement, the runtime per training iteration is 53 .

Original: 7% faster than that of a hand-crafted placement.
Revised: 7% faster than that of a hand-crafted placement.

Original: As shown by the ﬁgure, the generated placement is non-trivial and highly parallelized.
Revised: As shown by the ﬁgure, the generated placement is non-trivial and highly parallelized.

Original: In particular, all of the unrolled steps of the LSTM, attention, and softmax layers are distributed across multiple GPUs.
Revised: In particular, all of the unrolled steps of the LSTM, attention, and softmax layers are distributed across multiple GPUs.

Original: Note that it is impossible for an approach like ColocRL (Mirhoseini et al., 2017) to ﬁnd such a placement, as this method forces all operations within an unrolled LSTM step to be placed on the same device.
Revised: Note that it is impossible for an approach like ColocRL (Mirhoseini et al., 2017) to ﬁnd such a placement, as this method forces all operations within an unrolled LSTM step to be placed on the same device.

Original: Our method also learns to place the sparse embedding lookup operations on
Revised: Our method also learns to place the sparse embedding lookup operations on

Original: CPU.
Revised: CPU.

Original: In this placement, the policy search space is incredibly large, i.e., 5 46 , 600 (5 devices and 46, operations).
Revised: In this placement, the policy search space is incredibly large, i.e., 5 46 , 600 (5 devices and 46, operations).

Original: The automated placement enabled by jointly learned grouping not only outperforms previous methods, but unlike ColocRL, it is deployable with no human effort (e.g. manual grouping).
Revised: The automated placement enabled by jointly learned grouping not only outperforms previous methods, but unlike ColocRL, it is deployable with no human effort (e.g. manual grouping).

Original: In our experiments, we set the number of groups to 256.
Revised: In our experiments, we set the number of groups to 256.

Original: While training the policy, we observed that initially the operations are assigned nearly uniformly across all the 256 groups. However, we observed that the Grouper  converges to using only a small subset of groups across all models.
Revised: While training the policy, we observed that initially the operations are assigned nearly uniformly across all the 256 groups, but that the Grouper ultimately converges to using only a small subset of groups across all models.

Original: This indicates that the feed forward Grouper has learned to partition the computational graph such that operations that should be placed on the same device are grouped together.
Revised: This suggests that the feed forward Grouper has learned to partition the computational graph such that operations that should be placed on the same device are grouped together.

Original: We cast the device placement problem as a sequential decision-making task.
Revised: We cast the device placement problem as a sequential decision-making task.

Original: Since there is no canonical order for a TensorFlow graph, we randomized the order of the groups that we fed into the the Placer, with the Placer’s bi-LSTM architecture enabling it to look at a graph more holistically.
Revised: Since there is no canonical order for a TensorFlow graph, we randomized the order of the groups that we fed into the the Placer, with the Placer’s bi-LSTM architecture enabling it to look at a graph more holistically.

Original: We ran 10 experiments on our NMT (4-layer) baseline, shufﬂing the order of group embeddings passed to the Placer.
Revised: We ran 10 experiments on our NMT (4-layer) baseline, shufﬂing the order of group embeddings passed to the Placer.

Original: The difference between the fastest and slowest placements was less than 7%.
Revised: The difference between the fastest and slowest placements was less than 7%.

Original: Overhead of Training Hierarchical Planner: For each of the models, we train a new policy which learns to optimize placements for that particular model.
Revised: Overhead of Training Hierarchical Planner: For each of the models, we train a new policy which learns to optimize placements for that particular model.

Original: All results for the Hierarchical Planner are after 1000 iterations of updating the policy.
Revised: All results for the Hierarchical Planner are after 1000 iterations of updating the policy.

Original: In practice, this takes at most three hours for our largest benchmark.
Revised: In practice, this takes at most three hours for our largest benchmark.

Original: The runtime per policy update is dominated by the time it takes to measure the reward for a sampled placement.
Revised: The runtime per policy update is dominated by the time it takes to measure the reward for a sampled placement.

Original: To calculate the reward, we run the target model according to the predicted placement for 5 training steps and use the median runtime.
Revised: To calculate the reward, we run the target model according to the predicted placement for 5 training steps and use the median runtime.

Original: The policy itself is a lightweight network that is trained on a single GPU.
Revised: The policy itself is a lightweight network that is trained on a single GPU.

Original: For measuring the reward, however, we use actual runtime measurements for the given placements.
Revised: For measuring the reward, however, we use actual runtime measurements for the given placements.

Original: Measuring runtime is done by worker nodes.
Revised: Measuring runtime is done by worker nodes.

Original: For example, if we are optimizing a model placement on 5 devices (1 CPU and 4 GPUs), we need at least one worker with that many devices that can run the input model for the predicted placements and report the runtime.
Revised: For example, if we are optimizing a model placement on 5 devices (1 CPU and 4 GPUs), we need at least one worker with that many devices that can run the input model for the predicted placements and report the runtime.

Original: While as mentioned in Section 2, we used 16 workers to train our policy, here we want to show that we can generate good results even in limited hardware settings.
Revised: As described in Section 2, we used 16 workers to train our policy. However, in this section, we will show that we can generate good results even in limited hardware settings.

Original: We consider training the policy to optimize placement of our 4-layer NMT benchmark model on 5 devices (1 Intel Haswell 2300 and 4 Nvidia Tesla K40s).
Revised: We consider training the policy to optimize placement of our 4-layer NMT benchmark model on 5 devices (1 Intel Haswell 2300 and 4 Nvidia Tesla K40s).

Original: The goal is to show that the policy can be trained efﬁciently even when we have only one worker.
Revised: The goal is to show that the policy can be trained efﬁciently even when we have only one worker.

Original: Figure 3 demonstrates the policy loss reduction as a function of policy training time, in 2 different scenarios where we have access to 1 and 4 workers.
Revised: Figure 3 demonstrates the policy loss reduction as a function of policy training time, in 2 different scenarios where we have access to 1 and 4 workers.

Original: The policy is hosted on a single K40 GPU which sends 1 placement at a time to the worker(s) and applies a gradient step for each reward collected from worker(s).
Revised: The policy is hosted on a single K40 GPU which sends 1 placement at a time to the worker(s) and applies a gradient step for each reward collected from worker(s).

Original: While, more workers can reduce the policy training time, as seen in Figure 3, we still get reasonable training times with only one worker.
Revised: While, more workers can reduce the policy training time, as seen in Figure 3, we still get reasonable training times with only one worker.

Original: In this case, it takes less than 2.5 hours for the policy to achieve a placement with training step time of 1.94 seconds.
Revised: In this case, it takes less than 2.5 hours for the policy to achieve a placement with training step time of 1. seconds.

Original: Given that we train NMT models for hundreds of thousands of steps, the overhead of policy optimizationeven on a single worker is  justiﬁed.
Revised: Given that we train NMT models for hundreds of thousands of steps, the overhead of policy optimization is more than justiﬁed.

Original: For example, to train WMT’14 En->Fr dataset which has more than 36 million examples for one epoch (with batch-size=64), we need to run the NMT model for approximately 562500 steps.
Revised: For example, to train WMT’14 En->Fr dataset which has more than 36 million examples for one epoch (with batch-size=64), we need to run the NMT model for approximately 562500 steps.

Original: Since we reduce the runtime per step by approximately 46.7% (from 3.64 to 1.94 seconds), this saves us 265 GPU-hours, which is a signiﬁcant saving even if we consider the 12.5 GPU-hours we spent on training the policy on a single worker. 
Revised: Since we reduce the runtime per step by approximately 46.7% (from 3.64 to 1.94 seconds), this saves us 265 GPU-hours, which is a signiﬁcant savings even if we consider the 12.5 GPU-hours we spent on training the policy on a single worker. [Figure]

Original: While currently we ﬁrst train the policy and then  train the model according to optimized placements, for an even more efﬁcient design, we can utilize the steps we run themodel for measuring training  runtime (the reward for our policy) towards actual training of the model.
Revised: For simplicity, we ﬁrst train the Hierarchical Planner and then use its best placement to train the target model. For greater efﬁciency, however, we could interleave the training of the Hierarchical Planner with the training of the target model, using the runtime of actual training steps as our reward signal.

Original: Alternative Policy Architectures:
Revised: Alternative Policy Architectures:

Original: We compared the Hierarchical Planner against two alternative policy architectures described below.
Revised: We compared the Hierarchical Planner against two alternative policy architectures described below.

Original: The ﬁrst alternative we considered was a simpler model consisting of a single feed forward network.
Revised: The ﬁrst alternative we considered was a simpler model consisting of a single feed forward network.

Original: This model, which we refer to as the Simple Planner, independently predicts the placement for each operation in the input model, given information about that operation and its connectivity to others.
Revised: This model, which we refer to as the Simple Planner, independently predicts the placement for each operation in the input model, given information about that operation and its connectivity to others.

Original: This is equivalent to having only a feed forward Grouper which predicts placements rather than groups (i.e. the number of groups is equal to number of available devices).
Revised: This is equivalent to having only a feed forward Grouper which predicts placements rather than groups (i.e. the number of groups is equal to number of available devices).

Original: [Figure]

Original: The Simple Planner ﬁnds placements for Inception-V3 that are within 20% of the Hierarchical Planner, and it successfully learns to place RNNLM and ResNet benchmarks on a single GPU.
Revised: The Simple Planner ﬁnds placements for Inception-V3 that are within 20% of the Hierarchical Planner, and it successfully learns to place RNNLM and ResNet benchmarks on a single GPU.

Original: Its learned placement for NMT (2-layer), however, is more than twice as slow as that of the Hierarchical Planner.
Revised: Its learned placement for NMT (2-layer), however, is more than twice as slow as that of the Hierarchical Planner.

Original: The Simple Planner also fails to ﬁnd any valid placements for larger benchmarks, such as NMT with 4 or 8 layers.
Revised: The Simple Planner also fails to ﬁnd any valid placements for larger benchmarks, such as NMT with 4 or 8 layers.

Original: The Hierarchical Planner which breaks the problem into grouping and placing sub-tasks is able to scale to much larger models.
Revised: The Hierarchical Planner which breaks the problem into grouping and placing sub-tasks is able to scale to much larger models.

Original: The Hierarchical Planner’s sequence-to-sequence model also enables conditioning placement of an operation on those previously placed.
Revised: The Hierarchical Planner’s sequence-to-sequence model also enables conditioning placement of an operation on those previously placed.

Original: Another architecture we considered was a Hierarchical Planner with randomized grouping.
Revised: Another architecture we considered was a Hierarchical Planner with randomized grouping.

Original: To verify that the Grouper was contributing meaningfully to the performance of the Hierarchical Planner, we compared its performance against a baseline where we fed randomized groupings into the Placer.
Revised: To verify that the Grouper was contributing meaningfully to the performance of the Hierarchical Planner, we compared its performance to a baseline where we fed randomized groupings into the Placer.

Original: We ran this experiment with 10 different randomized group assignments for 1000 iterations.
Revised: We ran this experiment with 10 different randomized group assignments for 1000 iterations.

Original: As can be seen in Table 2, there is signiﬁcant variance across different trials and the best result is worse than that of the Hierarchical Planner.
Revised: As can be seen in Table 2, there is signiﬁcant variance across different trials and the best result is worse than that of the Hierarchical Planner.

Original: This suggests  end-to-end learning of grouping operations and assigning groupsto devices does indeed improve the performance.
Revised: This suggests that end-to-end learning of grouping operations and placing groups does indeed improve the performance.

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we present a hierarchical method for efﬁciently placing the operations of a computational graph onto devices.
Revised: In this paper, we present a hierarchical method for efﬁciently placing the operations of a computational graph onto devices.

Original: Our approach consists of a hierarchical model that ﬁrst assigns the operations to groups and then places those groups onto devices.
Revised: Our approach consists of a hierarchical model that ﬁrst assigns the operations to groups and then places those groups onto devices.

Original: We use a policy gradient method to optimize the parameters of the planner.
Revised: We use a policy gradient method to optimize the parameters of the planner.

Original: The proposed method enables us to scale to computational graphs containing over 80,000 operations.
Revised: The proposed method enables us to scale to computational graphs containing over 80,000 operations.

Original: Unlike previous work, our method is end-to-end and requires no manual effort.
Revised: Unlike previous work, our method is end-to-end and requires no manual effort.

Original: On a range of tasks including image classiﬁcation, language modeling, and machine translation, our method surpasses placements designed by human experts as well as those of previous state-of-the-art deep RL methods.
Revised: On a range of tasks including image classiﬁcation, language modeling, and machine translation, our method surpasses placements designed by human experts as well as those of previous state-of-the-art deep RL methods.

Original: Our approach ﬁnds highly granular parallelism within the graph, enabling us to outperform prior methods by up to 60.6%.
Revised: Our approach ﬁnds highly granular parallelism within the graph, enabling us to outperform prior methods by up to 60.6%.