Paper: Dual-Balancing for Multi-Task Learning (ICLR-2024)

Reviewer comment (Evaluation):
It would be better to compare the training time efficiency of each method.

Author response:
**A1**: As suggested, we provided the comparison of training efficiency for all methods in Figure 7 of the updated paper (Appendix C.4). We can see that the proposed DB-MTL is as efficient as existing gradient balancing methods since all task gradients are computed in every iteration (Line 6 in Algorithm 1 of the paper).

Reviewer comment (Experiments):
More discussions can be made on the proposed method, such that the stability, e.g. will the balancing method cause the model to collapse during training?

Author response:
**A2**: As suggested, we conducted an empirical analysis on the training stability of DB-MTL. Figure 8 (in Appendix C.5 of the updated paper) shows the curves of training loss and gradient norm for EW and DB-MTL on tasks of the *Office-31* dataset. As can be seen, DB-MTL converges as smoothly as EW, suggesting the proposed balancing methods do not affect the training stability.

Reviewer comment (Evaluation):
The authors could provide the training dynamics (both loss and gradient) such that readers can better understand how the balancing strategy affects the model learning.

Author response:
**A3**: We provided the training dynamics, including the training loss and gradient norm during the training process in Appendix C.5 of the updated paper. We can see that DB-MTL, which balances tasks across both loss and gradient levels, converges faster than EW.

Reviewer comment (Presentation):
Since each design component shows to be effective and useful for existing multi-task optimisation strategy, I think it’s important to highlight DB could be considered as a drop-in enhancement to existing multi-task methods to further improve performance.

Author response:
**A1**: Thanks for your suggestions. We revised the paper accordingly to highlight the logarithm transformation is beneficial to existing gradient balancing methods in the contribution and conclusion parts.

Reviewer comment (Experiments):
From the algorithm 1, it looks like the method should be able to run very fast, as similar to standard weight-based methods. Maybe it’s also useful to highlight its running speed, comparing to other gradient-based methods such as CAGrad, Nash-MTL which take considerably longer training time.

Author response:
**A2**: As suggested, we added the running time for all methods in Figure 7 of the update paper (Appendix C.4). We can see that the proposed DB-MTL runs as fast as previous gradient balancing methods since all task gradients are required to compute at every iteration (Line 6 in Algorithm 1 of the paper).

Reviewer comment (Experiments):
It might be also interesting to see whether DB could also enhance auxiliary learning methods, such as AuxiLearn (Navon et al 2021), GCS (Du et al 2018) and Auto-Lambda (Liu et al 2022). This could further emphasise the design benefits and may approach even wider range of audiences.

Author response:
**A3**: The goals of multi-task learning and auxiliary learning are different: the former expects all tasks to perform well but the latter only focuses on the main task(s). Hence, it is challenging to apply our balancing methods to auxiliary learning directly. We added discussions on the mentioned papers in Appendix D.

Reviewer comment (Novelty):
The novelty of this paper is limited. Loss and gradient balance have been extensively explored, and this paper does not have any new technical insights. It is only a combination of the existing works.

Author response:
**A1:** This paper aims to alleviate the task balancing problem from loss and gradient perspectives. The proposed method is simple yet effective and largely outperforms baselines on five datasets.

(i) As discussed in Section 3.1 of the submission, 
logarithm transformation is only used as a baseline studied in Nash-MTL and its advantages for MTL methods are **NOT** researched there. In our paper, we conducted extensive experiments to show that logarithm transformation is beneficial to various MTL methods. As shown in Figure 1 of the submission, MTL methods integrated with logarithm transformation can boost performance by a large margin. The effectiveness of combining logarithm transformation with MTL methods, **one of our major contributions**, is useful for the MTL community.

(ii) As discussed in Section 3.2 of the submission, the proposed gradient balancing method is different from GradNorm: firstly, GradNorm cannot guarantee all task gradients to have the same magnitude in each iteration since it updates the model parameters and task weights alternately; secondly, GradNorm does not consider the effect of the update magnitude (i.e., $\alpha_k$ in our paper), which significantly affects the performance (as shown in Figures 4 and 6 in our paper).

We are the **first** to propose normalizing task gradients to the same magnitude as the maximum gradient norm, and extensively experimental results (like Figure 3, Tables 1 and 6 in the submission) show it achieves better performance than GradNorm and other gradient balancing methods, which is **one of our major contributions**.

(iii) Logarithm transformation and the proposed gradient normalization are two **simple but effective** techniques, which are complementary and can be simply combined together as the proposed DB-MTL. Compared with IMTL, which is complex in learning weights for balancing both losses and gradients, DB-MTL is simpler and more effective (Tables 1, 2, 3, 4, and 5 in the submission). Furthermore, by a simple combination, DB-MTL achieves state-of-the-art performance on various datasets, which is our primary contribution.

(iv) Recent works [1, 2] have conducted extensive experiments to demonstrate the most simple method, equal weighting (EW in our paper), can perform comparably or even better than many complex multi-task/multi-objective methods via tuning some hyperparameters (like the learning rate) in [1] or using some regularization and stabilization techniques in [2]. Hence, we believe proposing simple but effective methods for the MTL field is useful.

Reviewer comment (Theory):
The theoretical analysis is not sufficient to suppor the advantage of the proposed method. Proving convergence guarantees or other theoretical properties of DB-MTL could reveal more advantages over existing methods.

Author response:
**A2**: As mentioned in our reply to Q1, recent works [1, 2] have conducted extensive experiments to demonstrate the performance of the existing MTL methods are unsatisfactory, although some of them have convergence guarantees. Besides, as discussed in MoCo [3], many MTL methods (like MGDA, PCGrad, CAGrad, etc.) provide the convergence analysis in the deterministic case but they are stochastic algorithms in practice. Although MoCo has a convergence guarantee in the practice stochastic setting, its performance is worse than the proposed DB-MTL (results are reported in Tables 1-5 in the submission). Hence, we believe that it is more important to propose an effective method for the current MTL field. In our paper, extensive experiments on five datasets consistently show that the proposed DB-MTL achieves better performance than existing methods. We will conduct the challenging theoretical analysis in the future.

Reviewer comment (Evaluation):
Detailed experimental analyzing on why DB-MTL outperforms certain baselines is missing.

Author response:
**A3**: In the submission, we have conducted extensive experiments on various datasets and provided analysis to demonstrate the superiority of DB-MTL over a large number of baselines. As shown in Tables 1-5, DB-MTL achieves state-of-the-art performance on all five datasets. This is attributed to 

(i) Logarithm transformation is a simple but effective loss balancing method. As shown in Figure 1 in the submission, integrating logarithm transformation into various gradient balancing methods boosts the performance largely. Moreover, logarithm transformation is more effective than IMTL-L (Figure 2 in the submission).

(ii) Our gradient-magnitude balancing method is more effective than the existing gradient normalization method (i.e., GradNorm), as shown in Figure 3 of the submission.

(iii) The ablation study in Section 4.4 demonstrates the effectiveness of using loss-scale balancing, gradient-magnitude balancing, and combining them together.

Reviewer comment (Writing):
The discription of the intuition is not clear.

Author response:
**A4**: We suppose the reviewer asks clarification for the intuition of the choice of $\alpha_k$. We further clarify it as follows.

(i) When some tasks have large gradient norms and others have small gradient norms, the model $\theta_k$ is close to a point where the former tasks have not yet converged while the latter tasks have converged. This point is unsatisfactory in MTL, as we hope all tasks achieve convergence. If we use the min-norm strategy, the model will be caught by such a point. Hence, a large $\alpha_k$ is more suitable to escape the unsatisfactory point, and the max-norm strategy is adopted.

(ii) When all task gradient norms are small, the model $\theta_k$ is close to a stationary point of all tasks (which is a satisfactory solution in MTL). Even using the max-norm strategy, the $\alpha_k$ is small as all task gradient norms are small. Hence, the model does NOT escape the stationary point.

Reviewer comment (Theory):
Can you provide any theoretical analysis for DB-MTL? Convergence guarantees or other properties could reveal advantages over existing methods.

Author response:
**A2**: As mentioned in our reply to Q1, recent works [1, 2] have conducted extensive experiments to demonstrate the performance of the existing MTL methods are unsatisfactory, although some of them have convergence guarantees. Besides, as discussed in MoCo [3], many MTL methods (like MGDA, PCGrad, CAGrad, etc.) provide the convergence analysis in the deterministic case but they are stochastic algorithms in practice. Although MoCo has a convergence guarantee in the practice stochastic setting, its performance is worse than the proposed DB-MTL (results are reported in Tables 1-5 in the submission). Hence, we believe that it is more important to propose an effective method for the current MTL field. In our paper, extensive experiments on five datasets consistently show that the proposed DB-MTL achieves better performance than existing methods. We will conduct the challenging theoretical analysis in the future.

Reviewer comment (Experiments):
For the loss transformation, is there an adaptive way to estimate the scale? Using the maximum seems to work well, but a learned loss scale could be interesting.

Author response:
**A5**: As discussed in Section 3.1 of the submission, previous works (e.g., IMTL-L) attempt to learn loss scales by **approximately** solving an optimization problem by one-step gradient descent at every iteration. Our Proposition A.1 in Appendix shows the logarithm transformation is equivalent to IMTL-L when the optimization problem is solved **exactly**. Hence, the loss transformation is better than IMTL-L in solving an optimization problem of learning loss scales. Experimental results (Figure 2 in the submission) show that logarithm transformation is more effective than IMTL-L.

Reviewer comment (Theory):
**Imprecise Overview of the MTL Objective:** The MTL objective (Equation 1) in Section 2 is imprecise. It is applicable primarily to existing loss balancing and certain gradient balancing methods. For example, in some gradient balancing methods like PCGrad and CAGrad, the weights of task-specific parameters are all 1, and in all hybrid balancing methods, the weights of task-specific and task-shared parameters are different.

Author response:
**A1**: We have revised Section 2 for clarification:

(i) In the "Loss Balancing Methods" paragraph: For loss balancing methods, the task weight $\gamma_t$ affects both the update of both task-sharing parameter $\theta$ and task-specific parameter $\psi$.

(ii) In the "Gradient Balancing Methods" paragraph: For most gradient balancing methods (e.g., PCGrad, CAGrad, MoCo, GradDrop, and IMTL), the task weight $\gamma_t$ affects the update of task-sharing parameter $\theta$ only. In some gradient balancing methods (e.g., GradNorm, MGDA, and Nash-MTL), task weight $\gamma_t$ plays the same role as in loss balancing methods.

(iii) In the "Hybrid Balancing Methods" paragraph: For hybrid balancing methods, the task weight $\gamma_t$ is the product of loss and gradient balancing weights.

Reviewer comment (Novelty):
**Omission of Relevant Literature (Part 1):** The DB-MTL method, albeit not defined as such within the text, is one of hybrid balancing methods. However, the treatment of hybrid balancing in Section 2 is overly simplistic and lacks a formal definition, along with an inadequate reference to relevant literature. For instance, approaches like RLW, Auto-λ[1] and IGB[2] have also conducted exploratory work combining loss balancing and gradient balancing.

Author response:
**A2**: (i) Combining loss balancing and gradient balancing is first proposed in IMTL (as discussed in Related Work in the submission), while the mentioned three follow-up works (RLW, Auto-$\lambda$ [1], and IGB [2]) combine their balancing methods with some existing loss/gradient balancing methods. We have added the discussion in Section 2 of the updated paper.

(ii) We have revised Section 3.1 accordingly by adding a discussion with IGB. (1) IGB combines the logarithm transformation with **loss** balancing methods to achieve better performance. In our paper, we demonstrate that logarithm transformation benefits existing **gradient** balancing methods, which is a contribution complementary to the exploration in the IGB paper. (2) Furthermore, IGB does not clearly explain why the logarithm transformation can work well. We show that it can address the loss scale problem in multi-task learning.

(iii) We have added IGBv2 as a baseline and compared it with our DB-MTL on five benchmark datasets. As shown in Tables 1-5 of the updated paper, DB-MTL consistently outperforms IGBv2.

Reviewer comment (Novelty):
**Omission of Relevant Literature (Part 2):** In the discussion part of Section 3.1, IGB[2] has also studied the logarithm transformation and has further extended it into a new loss balancing paradigm.

Author response:
**A2**: (i) Combining loss balancing and gradient balancing is first proposed in IMTL (as discussed in Related Work in the submission), while the mentioned three follow-up works (RLW, Auto-$\lambda$ [1], and IGB [2]) combine their balancing methods with some existing loss/gradient balancing methods. We have added the discussion in Section 2 of the updated paper.

(ii) We have revised Section 3.1 accordingly by adding a discussion with IGB. (1) IGB combines the logarithm transformation with **loss** balancing methods to achieve better performance. In our paper, we demonstrate that logarithm transformation benefits existing **gradient** balancing methods, which is a contribution complementary to the exploration in the IGB paper. (2) Furthermore, IGB does not clearly explain why the logarithm transformation can work well. We show that it can address the loss scale problem in multi-task learning.

(iii) We have added IGBv2 as a baseline and compared it with our DB-MTL on five benchmark datasets. As shown in Tables 1-5 of the updated paper, DB-MTL consistently outperforms IGBv2.

Reviewer comment (Novelty):
**Lack of Novelty:** The proposed loss balancing method, as well as the combination of loss and gradient balancing, have been previously discussed and experimented in prior works.

Author response:
**A3**: (i) The logarithm transformation is one of the components of our proposed DB-MTL, but we do not claim it is proposed in this paper. As discussed in Section 3.1 of the submission, logarithm transformation is only used as a baseline studied in Nash-MTL. Although IGB combines the logarithm transformation with loss balancing methods to achieve better performance, IGB also does not clearly explain why the logarithm transformation can work well. In our paper, we show that the logarithm transformation can address the loss scale problem in multi-task learning, which has not been studied in previous works. Besides, we conduct extensive experiments to show that logarithm transformation is beneficial to existing gradient balancing methods, which is a contribution complementary to the exploration in the IGB paper. As shown in Figure 1 of the submission, existing gradient balancing methods integrated with logarithm transformation can boost performance by a large margin. The effectiveness of combining logarithm transformation with gradient balancing methods, **one of our major contributions**, is useful for the MTL community.

(ii) Logarithm transformation and the proposed gradient normalization are two **simple but effective** techniques, which are complementary and can be simply combined together as the proposed DB-MTL. Compared with IMTL, which is complex in learning weights for balancing both losses and gradients, DB-MTL is simpler and more effective (Tables 1, 2, 3, 4, and 5 in the submission). Furthermore, by a simple combination, DB-MTL achieves state-of-the-art performance on various datasets, which is our primary contribution.

Reviewer comment (Theory):
**Lack of Theoretical Analysis:** Though the proposed method is easy to understand intuitively, it lacks theoretical understanding about why the proposed gradient balancing is beneficial for the overall MTL objective. In terms of gradients, the primary factor affecting the MTL performance is gradient conflicts, specifically the conflicts in gradient directions. The discrepancy in the magnitude of gradients does not lead to performance degradation if their directions are aligned. How does the proposed gradient balancing method directly or indirectly mitigate the conflicts in gradient directions? Are there any theoretical analyses, empirical experiments (maybe toy examples), or even intuitive explanations?

Author response:
**A4**: (i) Recent works [3, 4] have conducted extensive experiments to demonstrate the performance of the existing MTL methods are unsatisfactory. Hence, we believe that it is more important to propose an effective method for the current MTL field. In our paper, extensive experiments on five datasets consistently show that the proposed DB-MTL achieves better performance than existing methods. We will conduct the challenging theoretical analysis in the future. 

(ii) Intuitively, both gradient-magnitude discrepancies and gradient-direction conflicts can affect the MTL performance, but which one is more crucial is open. A very recent work [5] observes that the negative transfer and gradient-direction conflicts are not strongly correlated. Our experimental results show that the proposed gradient balancing method (i.e., simply normalizing all gradients as the maximum-norm one) is more effective than previous complex gradient manipulating methods like eliminating gradient-direction conflicts in PCGrad. Thus, it seems that gradient-direction conflicts are not crucial to mitigating negative transfer in MTL. This problem is still open.

\begin{array}{lccccc}\hline& \text{\textit{NYUv2}} & \text{\textit{Cityscapes}} & \text{\textit{Office-31}} & \text{\textit{Office-Home}} & \text{\textit{QM9}} \newline\hline\text{PCGrad} & -1.57_{\pm0.44} & -2.36_{\pm1.17} & -0.68_{\pm0.57} & -1.04_{\pm0.32} & -117.8_{\pm3.97} \newline\text{GradVac} & -1.75_{\pm0.39} & -2.45_{\pm0.54} & -0.58_{\pm0.78} & -1.49_{\pm0.28} & -150.7_{\pm7.41} \newline\text{gradient-magnitude balancing (\textbf{ours})} & \mathbf{+0.76}\_{\pm0.25} & \mathbf{+0.12}\_{\pm0.70} & \mathbf{+0.01}\_{\pm0.39} & \mathbf{-0.78}\_{\pm0.49} & \mathbf{-65.73}\_{\pm2.86} \newline\hline\end{array}

Reviewer comment (Experiments):
**Fairness of Comparative Experiments:** Combining loss balancing and gradient balancing is beneficial. Thus, directly comparing MB-MTL with existing gradient balancing methods might introduce unfairness. In the image classification scenario, when only the proposed gradient-magnitude balancing is employed, the performance is not optimal within gradient balancing methods (NashMTL yields better results). Would combining logarithmic transformation with NashMTL surpass MB-MTL?

Author response:
**A5**: As suggested, we conducted additional experiments on image classification scenarios (*Office-31* and *Office-Home*) to compare "Nash-MTL + logarithm transformation" with DB-MTL. The table below shows the results. As can be seen, DB-MTL performs better. Moreover, integrating logarithm transformation into Nash-MTL is better than  Nash-MTL. Furthermore, on *NYUv2*, *Cityscapes*, and *QM9* datasets, the proposed gradient-magnitude balancing method (the 3rd row of Table 6 in the submission) outperforms existing gradient balancing methods (results are reported in Tables 1, 2, and 5 of the submission).

\begin{array}{lcc}\hline &\text{\textit{Office-31}} & \text{\textit{Office-Home}} \newline\hline\text{Nash-MTL} & +0.24_{\pm0.89} & -0.08_{\pm0.69} \newline\text{Nash-MTL+Log} & +0.71_{\pm0.44} & +0.06_{\pm0.38} \newline\text{DB-MTL (\textbf{ours})} & \mathbf{+1.05}\_{\pm0.20} & \mathbf{+0.17}\_{\pm0.44} \newline\hline\end{array}