Paper: Multi-Objective Multi-Solution Transport (ICLR-2024)

Reviewer comment (Experiments):
Why we do not use a Pareto set learning (PSL) model, which is now a very mature techniques, (see COSMOS[1], Lin[2]) to solve the considered problems. Since PSL is now believed can run very fast and find the whole PS/PF for fairness problems.

Author response:
> Q1. **Why not use a Pareto set learning (PSL) model?**
   - We acknowledge the existence of techniques like PSL. However, MosT was designed with a focus on handling problems with a large number of objectives (i.e., $n\ll m$, much more than the number of models), aiming for a comprehensive exploration of the Pareto front. We are open to discussing this further and would appreciate any specific objections or concerns you may have.

Reviewer comment (Theory):
Why considered OT? Can a simple JS/KL/TV divergence also work?

Author response:
> Q2. **Optimal Transport (OT) vs. Divergence Measures:**
   - The choice of OT over other divergence measures was motivated by its ability to capture structural relationships between objectives, which cannot be provided by divergence measures.

Reviewer comment (Theory):
You claim EPO can not deal non-smooth and dis-continuous problem. But I think the proposed method also have such issues. When the solution is not smooth, it is actually impossible to apply any gradient-based method (to the best of my knowledge).

Author response:
> Q3. **Issues on non-smooth solutions:**
   - The weights assigned by Optimal Transport (OT) exhibit inherent smoothness properties. The optimization process guided by OT involves finding a transport plan that minimizes the loss while satisfying the marginal constraints. This characteristic ensures that the weights smoothly transition between different solutions, contributing to the overall smoothness of the optimization landscape. As a result, MosT benefits from the inherent smoothness of the OT weights, mitigating non-smooth and discontinuous behavior in the optimization process. This contributes to the algorithm's ability to handle a wide range of objective landscapes, promoting stability and reliability across diverse problem scenarios.

Reviewer comment (Reproducibility):
As far as I have implemented, EPO works perfectly on ZDT1. The main problem of EPO is it is slow. However, the EPO you have implemented only finds a single solution. I suggest the author to re-implement EPO.

Author response:
> Q4. **EPO Implementation:**
- We compare the algorithms using their original implementations and a consistent experimental setup, including the use of reference points and the number of solutions. The performance gap may come from the fact that EPO needs to sample an extensive number of preference vectors (\~100) and SVGD needs diverse initializations (\~50). Comparing their numbers with our limited number of solutions (5 for ZDT problems), these two algorithms cannot work as reported. These results show the efficiency of MosT, which does not rely on diverse sampling to achieve well-distributed solutions over the Pareto fronts.

Reviewer comment (Evaluation):
According to my understanding, LS can work perfectly on MNIST/FMNIST problem since their PFs are almost convex. So I am skeptical to the results in Table 2.

Author response:
> Q5. **LS should perform well on MNIST/FMNIST because its PF is convex:**
- Due to data heterogeneity and the large number of clients/objectives, it is unlikely that the Pareto front of FEMNIST (using the convolutional neural network) is convex.

Reviewer comment (Theory):
The Eq.4, using a weighted version of MGDA, does not have a direct meaning. It is hard to understand its result of Eq4 since the result of MGDA is unpredictable. So a weighted version of MGDA will not lead to very meaningful solutions (according to my try several months ago).

Author response:
> Q6. **The intuition behind weighted MGDA (Eq.4):**
   - By the definition of Pareto optimality, the Pareto optimal solutions identified for weighted objectives remain Pareto optimal for the original unweighted problem if all weights are positive. That being said, the weight vector for each model per step guides the MGDA to pursue solutions at a specific region of the Pareto front, similar to a reference/preference vector. Hence, different weight vectors lead to different solutions.
   - OT-weighted MGDA leads to complementary and balanced solutions, due to the two marginal constraints in OT, which enforce the balance among the $m$ solutions and a fair coverage over all the $n$ objectives. A detailed explanation is provided in General Response #2. This controlled weighting achieves a delicate balance, uncovering solutions that uniquely contribute to a diverse and complementary Pareto front.

Reviewer comment (Evaluation):
I am curious about the distribution of final solutions (similar to 7).

Author response:
> Q7. **Distribution of Final Solutions:**
   - We have shown the distribution in Figure 1.

Reviewer comment (Evaluation):
I have to say, many of the baselines are not what the author claims. Many of the baselines (including 5 EPO) actually have much better performance than the author claims.

Author response:
Q2. It seems that other solution is dominated by the proposed method on all objectives.

In Figure 2(c), we are not displaying the performance gap for each objective separately but rather for groups of objectives. So, the figure does not suggest that MosT dominates other solutions on all objectives. What it illustrates is that MosT outperforms the baselines by a larger margin for clients with worse performance. This underscores MosT's role in promoting fairness in federated learning by delivering superior results for those clients facing greater challenges.

Reviewer comment (Reproducibility):
For many-objective problems, MGDA is too slow since calculating m gradients seems not a main-stream method. In addition, how is the OT implemented? As far as I know, Sinkhorn algorithm for solving the OT is still slow.

Author response:
> Q8. **Computational Efficiency and OT Implementation:**
  - Optimal transport (OT) only costs a tiny fraction of the total runtime in MosT. Moreover, when compared to alternatives such as linearization and EPO, MosT remains more efficient overall. For comprehensive details and specific values of the runtime, please refer to General Response #3.

Reviewer comment (Theory):
Some crucial design choices for MosT are not well motivated, and more clear discussions are needed regarding the **Weighted Objectives**. The key of MosT is to assign different weights to the objectives for different solutions. It is unclear what the relation of the Pareto sets for the problems with weighted objectives is with the Pareto set of the original unweighted problem. Additionally, it is not clear what makes the weighted problems good for diversity. On page 4, it mentions that the weighted problem can move a small gradient from the origin in Eq. (5). However, the overall picture of how the (optimal) weighted problem can lead to a set of "different but complementary and balanced" solutions that best cover all objectives is not clear.

Author response:
> Q1.1  **What is the relation of the Pareto sets for the problems with weighted objectives (1) with the Pareto set of the original unweighted problem? How the (optimal) weighted problem can lead to a set of "different but complementary and balanced" solutions that best cover all objectives?**

By the definition of Pareto optimality, the Pareto optimal solutions identified for weighted objectives remain Pareto optimal for the original unweighted problem if all weights are positive. That being said, the weight vector for each model per step guides the MGDA to pursue solutions at a specific region of the Pareto front, similar to a reference/preference vector. Hence, different weight vectors lead to different solutions.


The complementary and balanced solutions are enforced through the weights achieved by OT and they are mainly due to the two marginal constraints in OT, which enforces the balance among the $m$ solutions and a fair coverage over all the $n$ objectives. 
This controlled weighting achieves a delicate balance, uncovering solutions that uniquely contribute to a diverse and complementary Pareto front.

Reviewer comment (Theory):
Concerns about **MGDA for Weighted Problems** remain unresolved. MosT uses MGDA to find a Pareto solution for each problem with different weighted objectives. However, MGDA can find any Pareto stationary solutions and the location is not controllable. For a non-trivial problem, the Pareto set could be a large \(n-1\) dimensional manifold with a large \(n\) for each weighted problem, where MGDA can lead to any possible solution on the manifold. It is unclear why the (uncontrollable) solutions found by MGDA for each weighted problem can be guaranteed to be complementary and evenly distributed.

Author response:
> Q1.3  **Why the (uncontrollable) solutions found by MGDA for each weighted problem (1) can be guaranteed to be complementary and evenly distributed?**

MosT combines MGDA with strategic weight assignment for a balanced exploration of the Pareto front. While MGDA itself may not guarantee the location of solutions, the reweighted objective (with weights 'controlled' by OT) ensures that the algorithm converges to solutions that cover different regions of the Pareto front. The weights assigned to objectives are derived through Optimal Transport (OT), introducing a controlled and strategic weighting mechanism. This weighting process ensures that the solutions found by MGDA are not arbitrary but guided by a deliberate assignment of weights.

Reviewer comment (Theory):
For **Large Number of Objectives** (\(n \gg m\)), this work lets each objective pick the best solution out of all \(m\) obtained solutions, and then calculates the overall performance. In this case, at the end of optimization, the objectives covered by different optimal solutions should be mutually exclusive. For example, for a problem with 300 objectives and 3 solutions, solution 1 covers 95 objectives, solution 2 covers 103 objectives, and solution 3 covers 102 objectives, with each objective covered by a single solution. In this case, the optimal solution should not cover other conflicting objectives that it does not work best for, since it might lower the solution’s performance on currently covered objectives. Therefore, it is unclear why "During later stages, ..., every objective has to be covered by sufficient models (solutions)." Additionally, since the final metric of interest is the mean (std) of average performance across all objectives (e.g., Table 2), once the groups of objectives covered by different solutions are determined, it is unclear why not just use simple uniform linear scalarization to optimize all objectives for each group.

Author response:
> Q2.2  **It is unclear why "During later stages, ..., every objective has to be covered by sufficient models (solutions)."**

During later stages, the uniformity of objective marginals, coupled with performance-based solution marginals, enables each objective to select its best-performing solution(s).

We acknowledge that objectives may be covered by different optimal solutions, raising concerns about potential mutual exclusivity. However, MosT aims to harmonize these contributions rather than enforce exclusivity. MosT strategically balances marginal distributions in later stages as described. This design ensures that even if a solution excels on specific objectives, the overall objective coverage remains diverse and robust.

> Q2.3 **In addition, the final metric we care about is the mean(std) of average performance across all objectives (e.g., Table 2). Once the groups of objectives covered by different solutions are determined, why not just use simple uniform linear scalarization to optimize all objectives for each group?**


Our main interest is in achieving diverse and complementary assignments from solutions to objectives, not just in terms of average accuracy. We want to make sure our solutions cover various objectives effectively. 

Note that, even within the same group you mentioned, objectives may exhibit varying characteristics and trade-offs. Consequently, a one-size-fits-all approach, such as simple uniform linear scalarization, might not capture the intricacies of each objective grouping. In addition, such alternative approaches require additional design, such as stopping criteria, which can increase the complexity of the algorithm. In contrast, MGDA tailors the optimization process to the specific characteristics of each group. This fine-grained optimization approach ensures that each objective's contribution is considered with the attention it deserves.

Reviewer comment (Theory):
The time complexity of MosT is unclear. MosT formulates the original multi-objective optimization as iterative bi-level optimization, involving solving an OT problem and \(m\) MGDA problems at each iteration. It is important to know what the time complexity of MosT is, and whether the OT solver has high computational overhead.

Author response:
### 3. Time Complexity

We acknowledge the importance of this aspect and have provided a detailed analysis in General Response #3. In summary, optimal transport constitutes only a small fraction of the total runtime in MosT. Moreover, when compared to alternatives such as linearization and EPO, MosT remains more efficient overall. For comprehensive details and specific values, please refer to General Response #3.

Reviewer comment (Experiments):
For the above few-objective case, the weights of interpolation are uniformly drawn from a Dirichlet distribution over the simplex. However, uniform weights usually do not represent uniform solutions on the Pareto front. If \(n'\) is small, will it hurt the performance?

Author response:
### 4. Extension to Few Objective Case (n<<m)

We empirically show that a relatively small $n'$ for interpolations of a few (i.e., $n$) original objectives is sufficient to achieve adequate performance. In the accuracy-fairness tradeoff ($n=2$), we conducted experiments sweeping over $n'\in \{10, 20, 30, 40, 50\}$. Our results indicate that utilizing $n' = 10$ yields comparable performance to those larger $n'$, as clarified in Appendix D.3. Consequently, we adopted the smaller $n' = 10$ in our experiments, obtaining the final results reported in Table 3.

Reviewer comment (Experiments):
There is a lack of experiments on more problems with a large number of objectives. While the main motivation of this work is to tackle the *small solution set for a large number of objectives* problem, the experiments only include a single real-world application with many objectives in the federated learning setting with FEMNIST. Other experiments are on synthetic problems or problems with few objectives. It is important to test MosT’s performance on more realistic application problems with a large number of objectives, especially those beyond federated learning.

Author response:
> Q5.2 **Realistic Application Problems with a Large Number of Objectives:**
   - **Expansion to Multi-Task Learning Scenarios:** We appreciate your suggestion to include more realistic application scenarios with a higher number of objectives. In response, we have expanded our experiments to include multi-task learning scenarios with two real datasets (n = 4 and 6, resp) and showed promising results. This addition aims to provide a more diverse and comprehensive evaluation of MosT's performance across different application domains. Detailed results and analysis are available in Appendix E and General Response #4.1.

Reviewer comment (Evaluation):
A **Runtime Comparison** is missing. The paper should report the runtime for different algorithms on all experiments.

Author response:
> Q5.3 **Runtime Comparison:**
   
   - In response to your feedback, we have included detailed runtime metrics for MosT and other algorithms over two real datasets, detailed in General Response #3. The results show that MosT is more efficient compared to linearization and EPO, and comparable to MGDA.

Reviewer comment (Theory):
Why is $\epsilon$ needed in Algorithm 1?

Author response:
Setting $\epsilon$ to be a small non-zero constant guarantees that any Pareto solutions for the re-weighted objective in Eq.(5) are also Pareto solutions for the original unweighted problem. Empirically, we found that it also provides numerical stability during the optimization process. Its effectiveness is also supported by our convergence, which incorporates a non-zero $\epsilon$. We will add more details in the revision.

Reviewer comment (Experiments):
I would expect the hypervolume of the solutions to be higher if one runs MGDA with diverse weight vectors $m$ times instead of random seeds. Could you compare MosT with such a version of MGDA?

Author response:
Following your suggestion, we conduct an ablation study in the context of federated learning with $n\gg m$, in order to compare MosT and MGDA with $m$ different weight vectors for $n$ objectives. Our ablation study shows MosT with OT-generated weights achieves higher accuracy averaged across all objectives than the MGDA alternative. This indicates the importance of optimal transport in simultaneously balancing multiple solutions and multiple objectives. Comprehensive details and results are available in Appendix C.3 and General Response #4.2.