Paper: Sp-R-IP: A Decision-Focused Learning Strategy for Linear Programs that Avoids Overfitting (ICLR-2024)

Reviewer comment (Novelty):
I find the paper's contribution to be somewhat incremental. Although I think the technical treatment described in the paper is non-trivial, the idea of replacing a linear forecaster with a non-linear one and the deployment of the interior point method appears to be straightforward observations.

Author response:
Although the proposed method builds upon the SPO+ reformulation framework as proposed by Elmachtoub & Grigas (2022), we believe that the use of a tailored interior point method to derive intermediate results that may generalize better on unseen data is a non-trivial step. Indeed, this contrasts with the prevailing emphasis in the optimization community on optimizing for speed in finding an optimum solution. Additionally, the extension to non-linear forecasters and the mini-batch approach that was added in the revised version of the manuscript substantially enhance the versatility of this reformulation approach.

Reviewer comment (Reproducibility):
How is x^* computed?

Author response:
Do you mean in equation (9) how you should compute $x^*(c_i)$? Since $c_i$ is the ground truth, this is pre-computed before optimization: $x^*(c_i)$ constitutes the optimal decision in the case of perfect foresight.

Reviewer comment (Novelty):
**Novelty and contribution**: The method outlined in this paper is a direct extension of the dual linear programming (LP) approach put forth in Elmachtoub & Grigas (2022). The first contribution, introducing the application of an interior point method (IP) for constrained optimization, represents a straightforward but meaningful addition from an optimization standpoint. Similarly, the second contribution involving the incorporation of neural network (NN) layers into constraints is conceptually clear-cut.

Author response:
Although the proposed method builds upon the SPO+ reformulation framework as proposed by Elmachtoub & Grigas (2022), we believe that the use of a tailored interior point method to derive intermediate results that may generalize better on unseen data is a non-trivial step. Indeed, this contrasts with the prevailing emphasis in the optimization community on optimizing for speed in finding an optimum solution. Additionally, the extension to non-linear forecasters and the mini-batch approach that was added in the revised version of the manuscript substantially enhance the versatility of this reformulation approach.

Reviewer comment (Theory):
The paper refers to Appendix A to detailed explanation. However, Appendix A lacks such a description. I understand that the derivation follows from Elmachtoub & Grigas (2022) but still, it would be nice to incorporate the detailed derivation in the paper. This is also true for eq. 15.

Author response:
In the revised manuscript we have extended appendix A to include more details on the reformulation of both the linear and non-linear ERM.

Reviewer comment (Theory):
Does the method's extension to neural networks in Section 4.2 work for any general activation function or is it restricted to just the ReLU function? How does the choice of activation function (i.e. 3rd constraint in (15)) affect the optimization problem?

Author response:
In our implementation, we used the softplus activation function, which is a smooth variant of the ReLU. This approach would work for any activation function. However, we observe that some activation functions are more suitable for the interior point method in terms of required train time. For example, replacing the softplus activation with ReLU leads to an average increase in train time of around 30% in our experiment. We agree that further investigations are worthwhile to fully understand the interplay between the IP solver and the architecture of the NN, but in the context of this paper, what matters is the generic nature of the proposed framework, paving the way for future improvements.

Reviewer comment (Evaluation):
If possible, can you please add error bars to the result in Table 1.

Author response:
We agree with the relevance of such sensitivity analysis. However, given the time constraints for this revision, we were unable at this point to add such error bars.

Reviewer comment (Writing):
It is somewhat unclear what "the ERM" is in the following sentence in the introduction ``Secondly, the ERM can be re-written to a single-level optimization program by applying duality theory`` -- what ERM problem are the authors referring to? The problem of training the forecaster? The problem of minimizing the regret loss? While this is clarified later in the paper, it would be helpful to make this clear in the introduction as well.

Author response:
Q1: It is indeed the problem of training the forecaster, in this case with the regret loss function. We have re-formulated this to “Secondly, the training problem can be …” in the revised manuscript.

Reviewer comment (Evaluation):
Do the authors have any intuition on why training a neural network to minimize the SPO+ loss function $l^{SPO+}$ performs poorly?

Author response:
Q3: As a clarification, we would like to remind the reviewer that our approach (any form of Sp-R-IP) involves using the SPO+ loss function. The good performance of our model therefore contributes to the validation of the SPO+ loss function, which was already theoretically shown to be consistent with the regret in (Elmachtoub and Grigas, 2022). What we see is that training a model with their originally-proposed subgradient descent method may be less performant than our proposed interior point method. The intuition behind this is two-fold: the first reason is that training a forecaster to minimize such a decision-focused loss function is a highly non-convex problem. Whereas our method smoothens the problem by introducing the log-barrier term, (sub)gradient methods are very prone to getting stuck in local optima. This intuition is underscored by the results in Table 3, where we show that the (sub)gradient methods ID-Q and Sp-SG perform very poorly when cold started, whereas the proposed Sp-R-IP method is able to attain similar performance when cold started compared to the warm start. The second reason is that the expression of the subgradient will result in model updates even when the current forecast leads to the perfect decision, which we showcase in Appendix D.2.

Reviewer comment (Experiments):
How does the validation performance tracking procedure that the authors propose compare to just solving the original SPO+ problem (Equation 9) with regularization?

Author response:
Q4: We observe that when a regularizer is used, the difference between the best performing model along the points accessed on the central path and the final resulting model generally reduces, but there is still a noticeable difference. This can be concluded from Tables 1 and 2: Table 2 shows that for the Sp-R methods, we explore multiple values of the regularization, whereas this clearly still results in improved performance of the Sp-R-IP methods as compared to the Sp-R method in Table 1.

Reviewer comment (Writing):
Could the authors elaborate more on the following sentence: ``We argue that when the optimization program is an ERM for training a forecaster, the points on the central path should be regarded as actual intermediate solutions to be tested on the validation set"?

Author response:
Q5: See remark on ‘Interpretation of interior points’ in our comment to all reviewers.

Reviewer comment (Evaluation):
I am struggling to interpret Figure 2. Could the authors explain the difference betwen the SP-R-IP methods they evaluate in the experiments? What is the expected behavior under each of these 3 different algorithms? Are they all variants on the authors' proposed approach? What is the reason that the validation regret of SP-R-IPs and SP-R-IPd oscillates (instead of decreasing monotonically)?

Author response:
The models compared in these figures are indeed variants of our proposed approach of training the forecaster by solving the ERM that follows from using the SPO+ loss function and reformulating it to a single-level optimization program, solving it with an interior point method. The way they obtain points along the central path differs:

•	Sp-R-IP directly uses the default IPOPT algorithm for getting the values of \mu (the barrier term). This algorithm prioritizes speed of finding a solution over granularity and amount of points obtained along the central path. This makes sense in the general setting where people try to find an optimal solution as fast as possible. However, this results in fewer evaluations of the validation performance as is shown in Figure 4: only 4 values of \mu are considered. Even though this is not necessarily a problem for the performance on the train set, where you expect to find the best result for the smallest value of \mu, this may lead you to oversee manifestations of the forecaster on the central path that generalize better on the validation set, as can be seen in Figure 2b.

•	To overcome that issue, we propose to manually set the value of \mu for which IPOPT should optimize iteratively:

o	Sp-R-IPs (static) decreases $\mu$ at a steady pace, in the case of Figure 2 by a factor 1.5, i.e. $\mu_{n+1} = \mu_{n}/1.5$.
o	Sp-R-IPd (dynamic) decreases $\mu$ dynamically based on the validation performance, i.e. slowing down the rate of decreasing the value of $\mu$ based on equations (12)-(14).

There is no immediate explanation as to why the validation performance oscillates, other than that there is no perfect match between the train and validation set. This is a behaviour that can be expected in gradient descent approaches too.