Paper: Zero-shot Cross-task Preference Alignment for Offline RL via Optimal Transport (ICLR-2024)

Reviewer comment (Theory):
I don't feel the simple addition of labels can work. Let me provide a toy example. If the authors pointed out my error, I would be glad to change my score. Assume the pair-wise labels of source $\{x_1, x_2, x_3\}$ is $Z_s = \[\[/, 0, 0\], \[1, /, 1\], \[1, 0, /\]\]$. Let's assume the target is exactly the same as the source, i.e., $\{y_1 = x_1, y_2 = x_2, y_3 = x_3\}$. Then the transportation matrix is $\boldsymbol T = \[\[1/3, 0, 0\], \[0, 1/3, 0\], \[0, 0, 1/3\]\]$. Let's say we want to compute the label of $(y_2, y_3)$: $\boldsymbol A^{23} = \[\[0, 0, 0\], \[0, 0, 1/9\], \[0, 0, 0\]\]$. $z(y_2, y_3)=\frac{1}{9}z(x_2, x_3)=1/9$, while we know the ground truth is $z(y_2, y_3)=z(x_2, x_3)=1$.

Author response:
Your result is right and we apologize for the confusion. Actually, we have noticed that most of the computed label values are very close to $0$. And we should pay attention to the relative label values instead of absolute values. Therefore, we adopt a min-max normalization to the matrix $Z_t$, with each element $z(y_j, y_j^\prime)$ representing the POT label of $(y_j, y_j^\prime)$. Also, since the labels are binary in practical implementation, we let the values greater than $0.5$ be label $1$ and make the other values be label $0$.
In this toy example, after min-max normalization and binarization, $Z_t = [[/, 0, 0], [1, /, 1], [1, 0, /]]$, which is exactly equal to $Z_s$. We will update the draft to make it clear.

Reviewer comment (Writing):
Some notations are ambiguous, e.g., $\mathcal S$ for state space and source task at the same time.

Author response:
Thanks for pointing out this. We will carefully revise the notations in the updated draft.

Reviewer comment (Evaluation):
In Figure 3b, why is RPT+POT even better than Oracle PT? If it is due to the variance, error bars should be provided in all figures.

Author response:
This is an insteresting question. In preference-based RL, the oracle labels are computed based on the ground-truth reward function of each task. One reason may be the reward function is not defined properly. And sometimes the oracle labels cannot reflect the real human intentions, which is pointed out by [1]. So, directly using oracle labels may obtain unsatisfied results. We will update the draft to include the error bars.