Paper: Musketeer: Joint Training/Inference for Multi-task Vision-Language Model with Task Explanation Prompts (ICLR-2024)

Reviewer comment (Novelty):
Replacing simple tasks descriptions with complex descriptions is a simple idea and has been successfully explored in language model literature [1]; thus the novelty is limited.

Author response:
A1: Thanks for your comments. Here we hope to clarify several points that may be misunderstandings:

### 1. Joint training on multiple heterogeneous vision-language tasks while preserving single-task performance is quite challenging.  
Unlike NLP tasks which are homogeneous and suitable for large-scale concurrent training,  the computer vision tasks are generally heterogeneous, (e.g., object detection outputs a set of bounding boxes, which is fairly different from visual grounding where both a caption and the foreground object are described). Unified LLMs like GPTs have achieved a tremendous success due to the nature of sequence-to-sequence property in languages, yet a unified foundational generalist model (not with multiple heavily designed specific task heads) to perform a large variety of computer vision tasks such as object detection, image segmentation, image generation, 3D estimation, pose recognition, etcs. is not yet available.  Unifying heterogeneous tasks   in single shared models may result in  obvious performance drops (according to previous works such as Unified-IO and UniTAB) because of the task interference. When the number of heterogeneous tasks increases, the task interference will also become more significant. Therefore, single-task finetuning or task-specific adapters are necessary to achieve satisfying downstream task performance.  For example, the OFA model which is pre-trained on eight heterogeneous can not generate usable results in many pretraining tasks (e.g., pretrained OFA_base achieves ~70 CIDEr score on image caption, v.s. ~140 CIDEr for single task fine-tuned model ) without single task finetuning. 

### 2. TEP is not a simple “complex” prompt. 
The main idea behind TEP is to use structural prompts to explicitly identify heterogeneous tasks in subprompt spaces.  In fact, as shown in Table 7 of the paper,  we have demonstrated that without structural form, even the more complex task description generated by ChatGPT with the richest information can not effectively mitigate the task interference problem. 


### 3. The performance improvement of Musketeer and TEP is not marginal. 
We’d like to clarify that the performance improvements are not marginal from several perspectives:  
- First, as shown in Table 4 of the main text, Musketeer outperforms all the joint-trained models on heterogeneous tasks and even demonstrates competitive results to ~20 times larger models.  
- In addition, as shown in Table 2 and Table 3 of the main text, TEP-powered Musketeer outperforms BaseP consistently for all tasks and demonstrates comparable and even better results to single-task finetuned SOTA methods.  Please kindly note that fully eliminating the effects of task interference in heterogeneous tasks has proven to be challenging, as indicated by previous works such as Unified-IO and Pixel2SeqV2. However, to the best of our knowledge, Musketeer stands as the first endeavor to successfully address this challenge by unifying not only detection, visual grounding, and VQA tasks but also other tasks involving pure text or pure image inputs. 
- TEP’s improvements can be more substantial in few-shot and zero-shot scenarios.  As demonstrated in Appendix A.2 and General Response, TEP outperforms BaseP significantly in few-shot and zero-shot learning settings.

Reviewer comment (Experiments):
One big contributor in description is the inclusion of “data source”. It could be because out of the 7 tasks, many data are from COCO so the model learns to utilize this information. What if the pre-training datasets all come from different image sources?

Author response:
A2: Thanks. This might be a misunderstanding.   First, the main contribution of TEP is the structured form to provide explicit guidelines for heterogeneous tasks, not each subpromp.  For example,  as shown in Table 1 below, keeping the subprompt in a structured form while removing “data description” in TEP can already give a good performance boost, whereas using a complex prompt without structured subprompts performs poorly in our experiments (See Table 7 of the paper).

**Table 1 Ablations on specific TEP subprompts. VG: Visual Grounding. VE: Visual Entailment. B@4: BLEU@4. Y/N: with / without data description.**
| **Prompt Type** | **Data Description** | **VG-val** | **VG-test-A** | **VG-test-B** | **VE-dev** | **VE-test** | **Caption-B@4** | **Caption-CIDEr** |
|-----------------|----------------------|-------------|---------------|---------------|------------|-------------|-----------------|:-----------------:|
| Base            | N                    | 85.8        | 88.8          | 81.7          | 83.4       | 83.5        | 37.3            | 125.5             |
| TEP             | N                    | 87.2        | 90.1          | 83.0          | 84.4       | 84.2        | 38.4            | 130.3             |
| TEP             | Y                    | 87.5        | 90.3          | 83.1          | 84.9       | 84.5        | 38.3            | 130.3     

Additionally, it is worth noting that our current experiments have already incorporated images from various sources. Specifically, our six vision-related tasks encompass images sourced from four distinct data sources. In our final version, we can further expand our experiments to include only one task from each data source, providing a more comprehensive analysis.

Reviewer comment (Reproducibility):
How many "soft task vectors" does each task have for the learnable task vector baseline?

Author response:
A4: We employ a  576-d soft task vector for each task. To investigate the potential impact of increasing the number of task vectors, we conducted experiments using 2 and 4 soft task vectors for each task. The results in Table 3 demonstrate that increasing the number of soft task vectors has a limited impact on Vision-Language multi-task learning.

**Table 3. Ablations on number of soft task vectors. VG: Visual Grounding. VE: Visual Entailment. B@4: BLEU@4.**  

| **Prompt Type** |  **VG-val** | **VG-test-A** | **VG-test-B** | **VE-dev** | **VE-test** | **Caption-B@4** | **Caption-CIDEr** |
|-----------------|-------------|---------------|---------------|------------|-------------|-----------------|:-----------------:|
| 1 Soft vector     	| 84.9    	| 87.8      	| 80.7      	| 83.7   	| 83.6    	| 37.3        	| 125.5         	|
| 2 Soft vector     	|  85.1  	|  87.7    	|   80.9    	|   	83.6 |  	83.3   |	37.4     	|  	125.4    	|
| 4 Soft vector     	| 	84.8 	|	87.5   	|	80.1   	|	83.7	| 	83.5	| 	37.7    	|  	125.7

Reviewer comment (Experiments):
TEP still relies on pretrained weights for initialization which can be expensive, the discussion regarding the additional cost might be good to provide.

Author response:
Thanks for your comments. We agree that the usage of pretrained weights may involve extra cost for pretraining. We’ll add corresponding discussion in our final version.  However, we hope to clarify that even without pretrained weights, TEP can still performs well on joint-training scenario for heterogeneous tasks.  In Table 1 we have provided results of Musketeer trained from scratch (without pretrained weights). As shown, Musketeer trained with TEP significantly outperforms BaseP and demonstrate usable performance even without pre-training. In addition, TEP is flexible and utilizes other pretrained weights (e.g.,  VL-T5 and OFA) for better overall performance. 

Table 1. Visual entailment performance of Musketeer trained with TEP and BaseP, both from scratch
| **Prompt Type** | **dev** | **test** 
|----------------|---------|------------|
| BaseP          | 56.7 |  56.9 |
| TEP            | 73.1 | 73.0 |

Reviewer comment (Reproducibility):
The hyper-parameter setting as well as the Needs carefully designed TEPs for new tasks which may require some expertise.

Author response:
Thanks. For the hyper-parameter settings, we hope to highlight that unlike many previous tasks which carefully design hyper-parameters and training metrics for each downstream task (e.g., OFA), we use fully-shared training settings (e.g., learning rates, epochs, and dropout ratio) for each task to simplify the hyper-parameter tuning efforts.  Besides, most hyper-parameter settings of TEP are following common practice of Transformer models of similar size (e.g., VL-T5).  Therefore, we hope to clarify that the  hyper-parameters settings of TEP are not carefully-designed and we believe its performance on each specific task can be further improved if we adopt extra careful hyper-parameter tuning. 
For the TEP design,  since the data description subprompt are generated by ChatGPT and verified by humans, we agree that it  may require some expertise. However, as shown in Table 2 below, the TEP without data description can already significantly outperform BaseP and the improvement by including data description is marginal, which means TEP can also be used without data description. Since other structural subprompts (instance prompt, I/O format and I/O description) are designed intuitively, we hope to point that TEP with no data description can also be used without much expertise. 









**Table 2. Ablations on specific TEP subprompts. VG: Visual Grounding. VE: Visual Entailment. B@4: BLEU@4. Y/N: with / without data description.**
| **Prompt Type** | **Data Description** | **VG-val** | **VG-test-A** | **VG-test-B** | **VE-dev** | **VE-test** | **Caption-B@4** | **Caption-CIDEr** |
|-----------------|----------------------|-------------|---------------|---------------|------------|-------------|-----------------|:-----------------:|
| Base            | N                    | 85.8        | 88.8          | 81.7          | 83.4       | 83.5        | 37.3            | 125.5             |
| TEP             | N                    | 87.2        | 90.1          | 83.0          | 84.4       | 84.2        | 38.4            | 130.3             |
| TEP             | Y                    | 87.5        | 90.3          | 83.1          | 84.9       | 84.5        | 38.3            | 130.3

Reviewer comment (Experiments):
The study on how well TEPs could transfer to unseen tasks is unknown.

Author response:
Thanks for your thoughtful review that will help us strengthen the manuscript. We have provided discussions of “how TEPs could transfer to unseen tasks” in [General Response to All Reviewers](https://openreview.net/forum?id=jb37oaGZXy&noteId=GV9Py9R2R3). Below we discuss other identified weaknesses.

Reviewer comment (Evaluation):
Could the author explain more on the varied performance on VQA in table 4, will using full VQAv2 training data mitigate the problems?

Author response:
Thanks for advice. Due to the limitation on our computing resources, in order to complete the training within the specified rebuttal time window, we increase the VQAv2 training data utilization only from 44% to 60%. The outcomes reveal a 0.4% enhancement in performance on the VQAv2 test set. Table 3 further illustrates the correlation between increased VQAv2 data usage and improved model performance.  Based on these findings, we are confident that increasing the usage of VQAv2 data will mitigate the problem. Additionally, it's noteworthy that the Musketeer Large model, with only 472 M parameters, is significantly smaller than the parameter count of the Unified-IO_XLarge model, which stands at 2925 M.

Reviewer comment (Evaluation):
Could the author provide additional training cost including the pretrainining cost for the proposed methods in Table 3 and 4 for a comprehensive evaluations?

Author response:
Thanks for advice. We have provided pretraining costs comparison of Musketeer and other jointly-trained models without task-specific fine-tuning in the Table 3 below.  We’ll update the Table 3 and Table 4 considering pretraining cost and add more corresponding discussion in our final version. 


**Table 3 Pretraining costs comparison of Musketeer and other jointly-trained models without task-specific fine-tuning. Pretrain-LM: Dataset size of Pretrained language models. #Pretrain-others: numbers of samples for other pretrainining tasks.**

| **Method**             | **#Params** | **Pretrain-LM** | **LM-source** | **#Pretrain-others** |
|------------------------|-------------|-----------------|---------------|----------------------|
| Flamingo               | -           | 182G            | M3W           | 339M                 |
| UniTAB                 | -           | 160G            | ReBERTa       | 200K                 |
| Unified-IO_Base        | 241M        | 750G            | C4            | 170M                 |
| Unified-IO_Large       | 776M        | 750G            | C4            | 170M                 |
| Unified-IO_XLarge      | 2925M       | 750G            | C4            | 170M                 |
| Uni-Perceiver-v2_Base  | 308M        | 160G            | ReBERTa       | 30M                  |
| Uni-Perceiver-v2_Large | 446M        | 160G            | ReBERTa       | 30M                  |
| Musketeer_Base         | 182M        | 140G            | Pile          | 24M                  |
| Musketeer_Large        | 472M        | 140G            | Pile          | 24M                  |