Original: CoCo: Learning Strategies for Online Mixed-Integer Control
Revised: CoCo: Learning Strategies for Online Mixed-Integer Control

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original:  Introduction
Revised:  Introduction

Original: Discrete and combinatorial optimization problems arise in a wide variety of applications and are2 used as a tool to formulate and solve challenging problems in robotics [4, 8], autonomous mobilityon-demand [15], and power network design [9, 14], among others.
Revised: Discrete and combinatorial optimization problems arise in a wide variety of applications and are used as a tool to formulate and solve challenging problems in robotics [4, 8], autonomous mobilityon-demand [15], and power network design [9, 14], among others.

Original: One approach to solving such4 problems is to model them through the lens of mixed integer convex programming (MICP), whichare optimization problems with convex objective functions and constraints, but which are made6 non-convex through the inclusion of discrete decision variables.
Revised: One approach to solving such problems is to model them through the lens of mixed integer convex programming (MICP), which are optimization problems with convex objective functions and constraints, but which are made non-convex through the inclusion of discrete decision variables.

Original: While MICPs have emerged as apopular tool for solving a variety of problems, the integrality constraints make them NP -complete8 and often exceptionally difﬁcult to solve in the worst case [7].
Revised: While MICPs have emerged as a popular tool for solving a variety of problems, the integrality constraints make them NP -complete and often exceptionally difﬁcult to solve in the worst case [7].

Original: Despite great strides made in the past few decades to improve solution times by several orders of10 magnitude through algorithmic and hardware improvements, MICPs are rarely deployed in real-worldrobotics and control applications that require ﬁnding solutions in real time (i.e., 10-100Hz).
Revised: Despite great strides made in the past few decades to improve solution times by several orders of magnitude through algorithmic and hardware improvements, MICPs are rarely deployed in real-world robotics and control applications that require ﬁnding solutions in real time (i.e., 10-100Hz).

Original: Further,12 state-of-the-art commercial solvers such as Gurobi [6] and Mosek [11] are not suitable for use onembedded robotic systems as they rely on complex multithreaded implementations.
Revised: Further, state-of-the-art commercial solvers such as Gurobi [6] and Mosek [11] are not suitable for use on embedded robotic systems as they rely on complex multithreaded implementations.

Original: Thus, the ability14 to solve MICPs in real-time on limited computational resources available on robotics remains anopen and pressing challenge.
Revised: Thus, the ability to solve MICPs in real-time on limited computational resources available on robotics remains an open and pressing challenge.

Original: In this work, we build on results from [3], which presented the Combinatorial Ofﬂine, Convex Online17 (CoCo) algorithm.
Revised: In this work, we build on results from [3], which presented the Combinatorial Ofﬂine, Convex Online (CoCo) algorithm.

Original: CoCo is a two-stage, data-driven algoithm that seeks to leverage tools fromparametric convex optimization and supervised machine learning to quickly ﬁnd feasible solutions19 for MICPs.
Revised: CoCo is a two-stage, data-driven algoithm that seeks to leverage tools from parametric convex optimization and supervised machine learning to quickly ﬁnd feasible solutions for MICPs.

Original: Using the idea of task-speciﬁc strategy decompositions, we present new results onhow CoCo can be extended to yield improved (1) generalizability and (2) scalability .
Revised: Using the idea of task-speciﬁc strategy decompositions, we present new results on how CoCo can be extended to yield improved (1) generalizability and (2) scalability .

Original: We use anovel convolutional neural network architecture to allow CoCo to generalize to solve problems22 online with a different number of discrete decision variables than what it was trained for ofﬂine.
Revised: We use a novel convolutional neural network architecture to allow CoCo to generalize to solve problems online with a different number of discrete decision variables than what it was trained for ofﬂine.

Original: Wedemonstrate how this architecture can scale to problems with different horizons and discrete variables24 through numerical experiments on a spacecraft motion planning problem.
Revised: We demonstrate how this architecture can scale to problems with different horizons and discrete variables through numerical experiments on a spacecraft motion planning problem.

Original: Finally, we propose futuredirections of research to allow for online adaptation of CoCo for improved performance.
Revised: Finally, we propose future directions of research to allow for online adaptation of CoCo for improved performance.

Original:  Approach
Revised:  Approach

Original: We consider parametrized mixed-integer convex programs with general form,minimize f 0 ( x, δ ; θ ) subject to f i ( x, δ ; θ ) ≤ 0 , i = 1 , . . .
Revised: We consider parametrized mixed-integer convex programs with general form, minimize

Original: , m f δ ∈ { 0 , 1 } n δ , (1) where x ∈ R n x are continuous decision variables, δ ∈ { 0 , 1 } n δ are binary decision variables, andθ ∈ R n p are the problem parameters.
Revised: [Equation] where x ∈ R n x are continuous decision variables, δ ∈ { 0 , 1 } n δ are binary decision variables, and θ ∈ R n p are the problem parameters.

Original: The objective function f 0 and inequality constraints f i areconvex.
Revised: The objective function f 0 and inequality constraints f i are convex.

Original: A popular use for MICPs is to model logical constraints in systems using what is known31 as big-M constraints [1].
Revised: A popular use for MICPs is to model logical constraints in systems using what is known as big-M constraints [1].

Original: In the big-M formulation, binary variables δ capture high-level discreteor logical behavior of the system and enforce the resulting high-level behavior on the continuous33 variables x .
Revised: In the big-M formulation, binary variables δ capture high-level discrete or logical behavior of the system and enforce the resulting high-level behavior on the continuous variables x .

Original: An example of such a use case is, given a binary variable δ i , the imposition of the logicalconstraint:[ δ i = 1] = ⇒
Revised: An example of such a use case is, given a binary variable δ i , the imposition of the logical constraint:

Original: [ g i ( x ; θ ) ≤ 0] .
Revised: [Equation]

Original: In big-M formulation, such a logical constraint can be enforced through the constraint:
Revised: In big-M formulation, such a logical constraint can be enforced through the constraint:

Original: [Equation]
Revised: [Equation]

Original: where M i ( θ ) =
Revised: where M i ( θ ) =

Original: sup x g i ( x ; θ ) .
Revised: sup x g i ( x ; θ ) .

Original: Note that the desired logical behavior is achieved through (2): the37 constraint is active when δ i = 1 and, when δ i = 0 , we have the trivial constraint g i ( x ; θ ) ≤ M i ( θ ) .
Revised: Note that the desired logical behavior is achieved through (2): the constraint is active when δ i = 1 and, when δ i = 0 , we have the trivial constraint g i ( x ; θ ) ≤ M i ( θ ) .

Original: The linear structure of the constraint in (2) allows us to deﬁne the notion of a logical strategy.
Revised: The linear structure of the constraint in (2) allows us to deﬁne the notion of a logical strategy.

Original: Given39 parameters θ ∈ R n p , a logical strategy S ( θ ) consists of a tuple ( δ ∗ ( θ ) , T M ( θ )) , where δ ∗ ( θ ) is aparticular integer solution, and T M ( θ ) is the set of tight big-M constraints,
Revised: Given parameters θ ∈ R n p , a logical strategy S ( θ ) consists of a tuple ( δ ∗ ( θ ) , T M ( θ )) , where δ ∗ ( θ ) is a particular integer solution, and T M ( θ ) is the set of tight big-M constraints,

Original: [Equation]
Revised: [Equation]

Original: Unlike the integer strategy deﬁnition presented in [2], logical strategies account for the existence of42 multiple integer solutions satisfying some logical proposition.
Revised: Unlike the integer strategy deﬁnition presented in [2], logical strategies account for the existence of multiple integer solutions satisfying some logical proposition.

Original: As shown in the results in [3], logicalstrategies allow for the supervised learning problem used for CoCo to be well-posed, prune redundant44 integer strategies, and allow for classiﬁer performance.
Revised: As shown in the results in [3], logical strategies allow for the supervised learning problem used for CoCo to be well-posed, prune redundant integer strategies, and allow for classiﬁer performance.

Original: The utility of deﬁning a logical strategy S ( θ ) can then be made apparent by using δ ∗ ( θ ) and T M ( θ )to solve the MICP as a convex optimization problem.
Revised: The utility of deﬁning a logical strategy S ( θ ) can then be made apparent by using δ ∗ ( θ ) and T M ( θ ) to solve the MICP as a convex optimization problem.

Original: Speciﬁcally, given an optimal strategy S ( θ ) ,the optimal solution for (1) can be found by solving,minimize [Equation] subject to
Revised: Speciﬁcally, given an optimal strategy S ( θ ) , the optimal solution for (1) can be found by solving, minimize

Original: [Equation]
Revised: [Equation]

Original: which is a convex optimization problem that be solved efﬁciently.
Revised: which is a convex optimization problem that be solved efﬁciently.

Original: The CoCo algorithm leverages49 this insight to learn a mapping from problem parameters θ to a corresponding logical strategy S ( θ ) .
Revised: The CoCo algorithm leverages this insight to learn a mapping from problem parameters θ to a corresponding logical strategy S ( θ ) .

Original: Speciﬁcally, CoCo trains a neural network classiﬁer h φ ofﬂine over a dataset D = { ( θ i , S i ) } Ni =151 consisting of problem parameters θ i and the corresponding optimal strategies S i .
Revised: Speciﬁcally, CoCo trains a neural network classiﬁer h φ ofﬂine over a dataset D = { ( θ i , S i ) } Ni =1 consisting of problem parameters θ [Equation] and the corresponding optimal strategies S i .

Original: Online, presented a52 new problem parameter θ , a single forward pass of the network ranks candidate logical strategiesand a ﬁnite number of user-speciﬁed convex optimization problems are solved until either a feasible54 solution is found or the algorithm terminates with failure.
Revised: Online, presented a new problem parameter θ , a single forward pass of the network ranks candidate logical strategies and a ﬁnite number of user-speciﬁed convex optimization problems are solved until either a feasible solution is found or the algorithm terminates with failure.

Original: 2.1 Task-Speciﬁc Strategy Decomposition
Revised: 2.1 Task-Speciﬁc Strategy Decomposition

Original: In our work, we leverage the notion of task-speciﬁc strategies introduced in [3].
Revised: In our work, we leverage the notion of task-speciﬁc strategies introduced in [3].

Original: Task-speciﬁc57 strategies exploit the underlying structure and separability that commonly arise in many roboticsproblems such as piecewise-afﬁne dynamics and multi-surface contact planning.
Revised: Task-speciﬁc strategies exploit the underlying structure and separability that commonly arise in many robotics problems such as piecewise-afﬁne dynamics and multi-surface contact planning.

Original: For example, for the59 motion planning problem in [13], the logical constraints that enforce collision avoidance between therobot and an obstacle m can be individually treated by only regarding the subset of binary variables61 used in those constraints.
Revised: For example, for the motion planning problem in [13], the logical constraints that enforce collision avoidance between the robot and an obstacle m can be individually treated by only regarding the subset of binary variables used in those constraints.

Original: This separability leads to a decoupling between the logical variables in theproblem, as they do not appear together in common constraints.
Revised: This separability leads to a decoupling between the logical variables in the problem, as they do not appear together in common constraints.

Original: Our key insight is that logical constraints under the CoCo framework can be decoupled by training the64 strategy predictor to predict the logical strategy for each constraint separately.
Revised: Our key insight is that logical constraints under the CoCo framework can be decoupled by training the strategy predictor to predict the logical strategy for each constraint separately.

Original: Further, by separatingthe prediction of each constraint, we can compose strategies to solve control problems with different66 numbers of binary variables.
Revised: Further, by separating the prediction of each constraint, we can compose strategies to solve control problems with different numbers of binary variables.

Original: Indeed, this ability to solve multiple tasks with a varying number ofdiscrete decision variables is a distinguishing feature of our work as compared to [2, 3, 10, 16].
Revised: Indeed, this ability to solve multiple tasks with a varying number of discrete decision variables is a distinguishing feature of our work as compared to [2, 3, 10, 16].

Original:  Experimental Results3.1 Free-Flying Space Robots
Revised:  Experimental Results 3.1 Free-Flying Space Robots

Original: We focus our attention on a fundamental problem in robotics: motion planning in the presence of71 obstacles.
Revised: We focus our attention on a fundamental problem in robotics: motion planning in the presence of obstacles.

Original: In this scenario, we consider a free-ﬂying spacerobot in the plane that must navigatethrough a workspace with multiple obstacles.
Revised: In this scenario, we consider a free-ﬂying space robot in the plane that must navigate through a workspace with multiple obstacles.

Original: The optimal control problem for this system entails73 planning a collision free trajectory in free space X safe towards a goal state x g :minimize (cid:107)x N [Equation] x
Revised: The optimal control problem for this system entails planning a collision free trajectory in free space X safe towards a goal state x g : minimize (cid:107)

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: N
Revised: N

Original: [Equation]
Revised: [Equation]

Original: τ = (cid:107) x
Revised: τ = [Equation] −

Original: [Equation]
Revised: [Equation]

Original: − x [-g (cid:107) 
-]{+t +
+}
Revised: subject to x [-g (cid:107) 
-]{+t +
+}

Original: + (cid:107) u
Revised: =

Original: τ (cid:107)subject to x t +1 = Ax t + Bu t , t = 
Revised: Ax t + Bu t , t = 0 , ... , N − 1 || u t [Equation]

Original: [Equation]
Revised: [Equation]

Original: where p t ∈ R 2 and v t ∈ R 2 are the position and velocity, respectively, in the 2-dimensional plane.
Revised: where p t ∈ R 2 and v t ∈ R 2 are the position and velocity, respectively, in the 2-dimensional plane.

Original: The state is therefore x t = ( p t , v t ) .
Revised: The state is therefore x t = ( p t , v t ) .

Original: The input u t ∈ R 2 consists of the forces produced by the thruster.
Revised: The input u t ∈ R 2 consists of the forces produced by the thruster.

Original: The constraint in (4) that makes it highly non-convex and necessitates the inclusion of binary decision77 variables is the safety constraint x t ∈ X safe .
Revised: The constraint in (4) that makes it highly non-convex and necessitates the inclusion of binary decision variables is the safety constraint x t ∈ X safe .

Original: Indeed, the global combinatorial search required to solvethis problem is an extensive area of study and one class of methods used to solve (4) entails posing it79 as an MICP [13].
Revised: Indeed, the global combinatorial search required to solve this problem is an extensive area of study and one class of methods used to solve (4) entails posing it as an MICP [13].

Original: The procedure consists of decomposing the workspace into keep-in and keep-outpolytope regions given a set of N obs obstacles.
Revised: The procedure consists of decomposing the workspace into keep-in and keep-out polytope regions given a set of N obs obstacles.

Original: For simplicity, we consider axis-aligned rectangular81 obstacles.
Revised: For simplicity, we consider axis-aligned rectangular obstacles.

Original: For an obstacle m , we denote the coordinates of the lower-left hand corner ( x m min , y m min )and upper right-hand corner ( x m max , y m max ) .
Revised: For an obstacle m , we denote the coordinates of the lower-left hand corner ( x m min , y m min ) and upper right-hand corner ( x m max , y m max ) .

Original: The collision avoidance constraint between state x t and83 obstacle m can then be written as:
Revised: The collision avoidance constraint between state x t and obstacle m can then be written as:

Original: [Equation]
Revised: [Equation]

Original: ) where each obstacle requires four integer variables δ m,it at each time step.
Revised: where each obstacle requires four integer variables δ m,it at each time step.

Original: An additional constraint85 is introduced to enforce that the the robot lies on at least one side of the obstacle without being incollision:
Revised: An additional constraint is introduced to enforce that the the robot lies on at least one side of the obstacle without being in collision:

Original: [Equation]
Revised: [Equation]

Original: Due to the (cid:96) 2 -norm constraints imposed on the thruster forces, this problem is an MIQCQP with88 4 N obs N variables.
Revised: Due to the (cid:96) 2 -norm constraints imposed on the thruster forces, this problem is an MIQCQP with 4 N obs N variables.

Original: 3.2 Architecture
Revised: 3.2 Architecture

Original: Here, we show how the notion of task-speciﬁc strategies can be leveraged to (1) more effectively91 training the strategy classiﬁer h φ and (2) allow for the trained network to generalize to problems with92 a different number of integer strategies.
Revised: Here, we show how the notion of task-speciﬁc strategies can be leveraged to (1) more effectively training the strategy classiﬁer h φ and (2) allow for the trained network to generalize to problems with a different number of integer strategies.

Original: In [3], this was accomplished by training a single classiﬁerwith parameters θ and a one-hot vector encoding as the input.
Revised: In [3], this was accomplished by training a single classiﬁer with parameters θ and a one-hot vector encoding as the input.

Original: Thus, the network was ﬁxed to a94 particular number of obstacles m , with the one-hot vector encoding which obstacle strategy is beingqueried.
Revised: Thus, the network was ﬁxed to a particular number of obstacles m , with the one-hot vector encoding which obstacle strategy is being queried.

Original: However, this network architecture limits the use of CoCo to a ﬁxed number of obstacles.
Revised: However, this network architecture limits the use of CoCo to a ﬁxed number of obstacles.

Original: For example, if a robot traverses to a new environment with a different number of obstacles, then thetrained classiﬁer h φ is no longer useful for online control.
Revised: For example, if a robot traverses to a new environment with a different number of obstacles, then the trained classiﬁer h φ is no longer useful for online control.

Original: Instead, we propose using the convolutional neural network architecture (CNN) shown in Figure 1 for99 accomplishing this task.
Revised: Instead, we propose using the convolutional neural network architecture (CNN) shown in Figure 1 for accomplishing this task.

Original: A CNN architecture confers several advantages.
Revised: A CNN architecture confers several advantages.

Original: First, it allows for queryingstrategies for an arbitrary number of obstacles, thereby allowing it to be used for problems with101 different number of binary variables.
Revised: First, it allows for querying strategies for an arbitrary number of obstacles, thereby allowing it to be used for problems with different number of binary variables.

Original: Second, it allows for solving problems with various obstaclegeometries depicted in the scene, as the strategy classiﬁer is not ﬁxed to a particular geometry103 parameterization.
Revised: Second, it allows for solving problems with various obstacle geometries depicted in the scene, as the strategy classiﬁer is not ﬁxed to a particular geometry parameterization.

Original: Finally, the ability to solve problems with a different number of binary variablesalso paves the way for online adaptation approaches, wherein the network parameters are tuned online105 for improved performance in new environments.
Revised: Finally, the ability to solve problems with a different number of binary variables also paves the way for online adaptation approaches, wherein the network parameters are tuned online for improved performance in new environments.

Original: 3.3 Results
Revised: 3.3 Results

Original: The machine learning models were implemented in PyTorch [12] and the convex optimization108 problems solved using Mosek [11].
Revised: The machine learning models were implemented in PyTorch [12] and the convex optimization problems solved using Mosek [11].

Original: The network architecture used for the free-ﬂyer motion planning
Revised: The network architecture used for the free-ﬂyer motion planning

Original: [Figure]
Revised: [Figure]

Original: problem is depicted in Figure 1 and consists of three convolutional layers on a synthetic image110 of the obstacle.
Revised: problem is depicted in Figure 1 and consists of three convolutional layers on a synthetic image of the obstacle.

Original: The output of this convolutional block is then concatenated with the remainingproblem parameters and input to a standard ReLU feedforward network with three layers and 128 112 neurons per layer.
Revised: The output of this convolutional block is then concatenated with the remaining problem parameters and input to a standard ReLU feedforward network with three layers andneurons per layer.

Original: We further benchmark our proposed architecture against Mosek [11], the regressionframework from [10], and the k-nearest neighbor (KNN) approach from [16].
Revised: We further benchmark our proposed architecture against Mosek [11], the regression framework from [10], and the k-nearest neighbor (KNN) approach from [16].

Original: The CoCo, regressor,114 and KNN methods are trained using ninety thousand MICPs, with parameters θ drawn from a randomdistribution of initial conditions x 0 and obstacle coordinates, and evaluated on a test set of ten116 thousand feasible problems.
Revised: The CoCo, regressor, and KNN methods are trained using ninety thousand MICPs, with parameters θ drawn from a random distribution of initial conditions x 0 and obstacle coordinates, and evaluated on a test set of ten thousand feasible problems.

Original: The MICP used N = 5 and N obs = 8 , leading to a total of 160 binaryvariables.
Revised: The MICP used N = 5 and N obs = 8 , leading to a total of 160 binary variables.

Original: The code will be made publicly available.
Revised: The code will be made publicly available.

Original: [Figure]
Revised: [Figure]

Original: Figure 2 shows the results of our numerical sim- 119 ulations for Mosek and each data-driven frame- [Figure] work.
Revised: Figure 2 shows the results of our numerical simulations for Mosek and each data-driven framework.

Original: For all methods, we cap the solver at121 10ms (i.e., 100Hz) and compare the number offeasible solutions found, solution time, and the123 quality of these feasible solutions compared tothe globally optimal solution.
Revised: For all methods, we cap the solver at 10ms (i.e., 100Hz) and compare the number of feasible solutions found, solution time, and the quality of these feasible solutions compared to the globally optimal solution.

Original: We see that task- 125 speciﬁc CoCo ﬁnds feasible solutions for 87%of the test set, compared to 58%, 49%, and 29%127 for Mosek, the regressor, and the KNN, respectively.
Revised: We see that taskspeciﬁc CoCo ﬁnds feasible solutions for 87% of the test set, compared to 58%, 49%, and 29% for Mosek, the regressor, and the KNN, respectively.

Original: Figure 2c depicts the relative cost of129 ﬁrst feasible solution found by Mosek and thethree data-driven frameworks compared to the131 globally optimal solution for the problem.
Revised: Figure 2c depicts the relative cost of ﬁrst feasible solution found by Mosek and the three data-driven frameworks compared to the globally optimal solution for the problem.

Original: Crucially, we see that 80% of the feasible solutions133 solved by CoCo are indeed the globally optimal solution, compared to only 69%, 57%, and135 53% for Mosek, regression, and KNN, respectively.
Revised: Crucially, we see that 80% of the feasible solutions solved by CoCo are indeed the globally optimal solution, compared to only 69%, 57%, and 53% for Mosek, regression, and KNN, respectively.

Original: Thus, we see that CoCo outperforms the137 commerical solver and the benchmarks from [10, 16] for this problem.
Revised: Thus, we see that CoCo outperforms the commerical solver and the benchmarks from [10, 16] for this problem.

Original: Figure 3 demonstrates the ability of CoCo to generalize to a distribution of problems with a varying139 of number binary variables.
Revised: Figure 3 demonstrates the ability of CoCo to generalize to a distribution of problems with a varying of number binary variables.

Original: We trained multiple strategy classiﬁers corresponding to horizons of N = { 5 , 7 , 9 , 11 } with environments of eight obstacles.
Revised: We trained multiple strategy classiﬁers corresponding to horizons of N = { 5 , 7 , 9 , 11 } with environments of eight obstacles.

Original: We evaluated the ability of these networksto environments with an increasing number of obstacles N obs = { 6 , . . .
Revised: We evaluated the ability of these networks to environments with an increasing number of obstacles N obs = { 6 , . . .

Original: As shown, we seethat the efﬁcacy of the strategy classiﬁer diminishes with an increasing horizon length N due to143 a corresponding increase in the number of strategies.
Revised: As shown, we see that the efﬁcacy of the strategy classiﬁer diminishes with an increasing horizon length N due to a corresponding increase in the number of strategies.

Original: Intuitively, we also see that performancedecreases with an increase obstacles simply due to the increased difﬁculty of the planning problem.
Revised: Intuitively, we also see that performance decreases with an increase obstacles simply due to the increased difﬁculty of the planning problem.

Original: However, we note that the dropoff in performance remains roughly linear rather than an exponentialdecrease of performance stemming from including additional binary variables.
Revised: However, we note that the dropoff in performance remains roughly linear rather than an exponential decrease of performance stemming from including additional binary variables.

Original: Indeed, we believe that147 these results motivate improving the performance of the classiﬁer for environments with a differentnumber of obstacles as a future area of research.
Revised: Indeed, we believe that these results motivate improving the performance of the classiﬁer for environments with a different number of obstacles as a future area of research.

Original: For example, one future avenue could be to explore149 meta-learning inspired approaches such as MAML [5] to allow CoCo to adapt network parametersonline for improved performance using information gained from solving problems online.
Revised: For example, one future avenue could be to explore meta-learning inspired approaches such as MAML [5] to allow CoCo to adapt network parameters online for improved performance using information gained from solving problems online.