From: NeurIPS.cc/2022/Workshop/TEA/Program_Chairs
title: Paper Decision
decision: Accept
comment: Overall, the reviewers find the paper interesting. Optimizing multi-objectives in learning is important, and the idea of using user feedback for this purpose is interesting and sound. However, as mentioned by the reviewers, the main issue is that it is not clear wether the method is realistic to implement and what specific embodied systems the proposed method can be applied to. Please consider clarify these issues as well as the other review comments in the final version.

From: NeurIPS.cc/2022/Workshop/TEA/Paper7/Reviewer_wdW2
title: Interesting work with a great poential 
review: I believe this paper studies a significant problem: How to optimize multi-objectives/metrics in the presence of user feedback? The paper states, "rather than committing to a static or pre-defined tradeoff for the multiple objectives, our model is guided by the feedback received, which is used to update its internal state." Despite the promise of the proposed theoretical modeling, the current applications do not seem quite interesting. I wonder if the proposed approach has the potential to be used to optimize multiple competing objectives (e.g., robustness vs. fairness) in trustworthy learning. Or it can be used for human-in-the-loop learning, which involves competing objectives such as accuracy vs. interpretability. I would suggest having such discussions in the paper to benefit the workshop's targeted audience. 





rating: 7: Good paper, accept
confidence: 3: The reviewer is fairly confident that the evaluation is correct

From: NeurIPS.cc/2022/Workshop/TEA/Paper7/Reviewer_ENGK
title: Sound idea for using user feedback to optimize potentially contradictory objectives, but may be difficult to scale and obtain all of the assumed data 
review: **Summary:** 
The authors propose a method for online optimization of many and possibly conflicting metrics using user feedback. They provide regret bounds for certain cases using the proposed method and demonstrate the technique on a simulated and real-world data example. 

**Quality / Clarity:** The intuition for using user feedback to optimize multiple metrics is clear and the examples in the introduction about the failings of existing methods is helpful in understanding the potential power of the proposed approach. As far as I can tell, the work is technically sound and the level of detail in the appendix for the main regret bound proofs is helpful in understanding why the overall approach will work.

**Originality:** 
To my knowledge, the proposed techniques and bounds are novel. The paper addresses the important problem of conflicting optimization objects in a new and interesting way. It would be helpful if the authors could discuss their intuition for where the method might breakdown or may not improve over existing methods in the absence of testing on more complicated settings. 

**Significance:** The results are on fairly simple simulated and real-world datasets; however, it is clear the proposed method is able to compete with the best-in-hindsight solution. The approach has the potential for high impact if it is implementable in larger-scale settings. I would appreciate some kind of baseline to compare the proposed algorithms. For example, even comparing with randomly searching would help the reader understand the significance of the approach if there is a large improvement over the baseline. 

**Issues:** 
- I am unsure about how realistic the method is to implement. The authors assume access to a small m correlation set, and precomputed costs associated with the complaints. These assumptions seem restrictive for scaled settings.
- The authors provide an illustrative example on L40-L42 about a biased tool which has twice the false-positive rate for black defendants as compared with white defendants. However, I’m not sure how the proposed method is able to address this issue. On L353-L355, the authors mention that they receive a batch of complaints in a real-world dataset example. If the population is skewed, then more complaints could come from the larger subgroups (even if individuals in the smaller subgroups are more likely to have complains) and continue to skew the updates. The proposed method seems sound but the associated regret bound does not seem to prevent a similar bias to the one the authors mention in L40-L42. Could the authors address this potential “unfairness” point directly for the presented method?
rating: 6: Marginally above acceptance threshold
confidence: 2: The reviewer is willing to defend the evaluation, but it is quite likely that the reviewer did not understand central parts of the paper

From: NeurIPS.cc/2022/Workshop/TEA/Paper7/Reviewer_C8Ni
title: Dificult to Follow
review: When machine learning models (or really, any system that is based on some optimization) are deployed, there is usually more than one criterion that it should be evaluated. These criteria are often competing / conflicting, and sometimes incompatible. This paper proposes a formulation for this and proposes an online learning algorithm that tries to find the lowest loss due to these conflicting objectives. Instead of considering a simple linear combination of different criteria, a graph structure is leveraged to model criteria that are impossible to satisfy together, and the proposed formulation is flexible enough to account for losses that stem from not only individual criteria, but also combinations of them.

The early parts of the paper are well-written and easy to follow (until around line 246). However, the proposed algorithm and the supporting theory become too difficult to follow afterwards. I have some major concerns, and it is possible that some of them are already addressed in the paper but I missed. Below I elaborate on these concerns.

1- At the end of page 4, the cost function for a state transition is defined. It is basically the cost for the action and the losses due to unsatisfied criteria in the next state. With such a formulation, the goal should be to find the state that has the lowest loss (which is as defined in Equation 3) as soon as possible and stay there. However, the problem is formulated as a regret minimization problem, which is not the same. In multi-armed bandit problems (or online learning in general), regret minimization and best arm identification can lead to different solutions. Similarly, here, the goal should be to find the best state as quickly as possible rather than minimizing the sum of the costs along the trajectory. Because, when horizon is finite (which seems to be the assumption in line 208), this will mean the optimal policy prefers a trajectory that has the lowest cumulative cost and this will prevent it from finding the optimal state if the optimal state requires passing through states that incur high cost. So overall, I am skeptical about how this problem is formulated. Assuming the horizon is infinite could alleviate some of these concerns (although it is still a different problem from best arm identification).

2- In Figure 3, it is assumed that it is possible to move from the current state to another state s in at most $k$ time steps. I do not understand how this is guaranteed. State transitions are stochastic, and it can be sometimes the chase that the target state could not be reached in $k$ time steps. What am I missing? Also, are there any assumptions about the stochasticity / distribution of transitions?

3- A non-technical major concern is about the relevance of the paper to the workshop. I agree that this formulation is general and can be applied to embodied systems, but the paper is not specifically on embodied systems. I would recommend polishing the paper/presentation towards that theme as much as possible, if the paper gets accepted.
rating: 6: Marginally above acceptance threshold
confidence: 3: The reviewer is fairly confident that the evaluation is correct