Original: L EARNING TO J OINTLY S HARE AND P RUNE W EIGHTS FOR G ROUNDING B ASED V ISION AND L ANGUAGE M ODELS
Revised: L EARNING TO J OINTLY S HARE AND P RUNE W EIGHTS FOR G ROUNDING B ASED V ISION AND L ANGUAGE M ODELS

Original: A BSTRACT
Revised: A BSTRACT

Original: Transformers have been successful in processing different data modalities, such as language and image data, which could use transformers with similar architectures to achieve good performance.
Revised: Transformers have been successful in processing different data modalities, such as language and image data, which could use transformers with similar architectures to achieve good performance.

Original: Leveraging this observation, we propose weight sharing across two transformer backbones and within the same transformer backbone and pruning across two backbones in a unified framework.
Revised: Leveraging this observation, we propose weight sharing across two transformer backbones and within the same transformer backbone and pruning across two backbones in a unified framework.

Original: More specifically, we investigate weight sharing and pruning for two components of the transformers: (1) Multi-Head Attention (MSA) and (2) Feed-Forward Network (FFN) layers.
Revised: More specifically, we investigate weight sharing and pruning for two components of the transformers: (1) Multi-Head Attention (MSA) and (2) Feed-Forward Network (FFN) layers.

Original: To jointly perform weight sharing and pruning, we propose to use a regularization term to align model weights and the desired structure during the multimodal pre-training step.
Revised: To jointly perform weight sharing and pruning, we propose to use a regularization term to align model weights and the desired structure during the multimodal pre-training step.

Original: The structure vectors of sharing and pruning are generated by using a hypernetwork, which can capture complex interactions between pruning and sharing across layers and modalities.
Revised: The structure vectors of sharing and pruning are generated by using a hypernetwork, which can capture complex interactions between pruning and sharing across layers and modalities.

Original: We train the hypernetwork and model weights iteratively so that the learned structure evolves along with model weights.
Revised: We train the hypernetwork and model weights iteratively so that the learned structure evolves along with model weights.

Original: After minimizing the proposed objective in pre-training step, we perform weight sharing and pruning and fine-tune the compressed model on downstream tasks.
Revised: After minimizing the proposed objective in pre-training step, we perform weight sharing and pruning and fine-tune the compressed model on downstream tasks.

Original: Finally, we perform experiments on vision and language tasks, including Referring Expression Comprehension (REC), Visual Question Answering (VQA), and
Revised: Finally, we perform experiments on vision and language tasks, including Referring Expression Comprehension (REC), Visual Question Answering (VQA), and

Original: Object Detection using the state-of-the-art grounding based models: MDETR and
Revised: Object Detection using the state-of-the-art grounding based models: MDETR and

Original: GLIP.
Revised: GLIP.

Original: Our experiments show that we can compress these models by 35 − 40% by sharing and pruning MSA and FFN weights without almost any loss in accuracy.
Revised: Our experiments show that we can compress these models by 35 − 40% by sharing and pruning MSA and FFN weights without almost any loss in accuracy.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The dominant architecture in natural language processing (NLP) is Transformer (Vaswani et al., 2017).
Revised: The dominant architecture in natural language processing (NLP) is Transformer (Vaswani et al., 2017).

Original: Besides NLP, recent advance in computer vision shows that transformer based model, like
Revised: Besides NLP, recent advance in computer vision shows that transformer based model, like

Original: ViT (Dosovitskiy et al., 2021) or DeiT (Touvron et al., 2020), can achieve similar or even better performance than convolutional neural networks (CNNs) on various tasks.
Revised: ViT (Dosovitskiy et al., 2021) or DeiT (Touvron et al., 2020), can achieve similar or even better performance than convolutional neural networks (CNNs) on various tasks.

Original: As a result, it allows us to use architecturally similar models on cross-modal tasks with vision and language data.
Revised: As a result, it allows us to use architecturally similar models on cross-modal tasks with vision and language data.

Original: This setting naturally provides foundations to structurally share weights across different modalities.
Revised: This setting naturally provides foundations to structurally share weights across different modalities.

Original: The advantage of weight sharing is that it encourages weight reuse and thus reduces the number of parameters while maintaining the model capacity to some extent.
Revised: The advantage of weight sharing is that it encourages weight reuse and thus reduces the number of parameters while maintaining the model capacity to some extent.

Original: On the other hand, existing weight sharing techniques have some limitations.
Revised: On the other hand, existing weight sharing techniques have some limitations.

Original: Most of them (Lee et al., 2021; You et al., 2022; Lan et al., 2019; Reid et al., 2021) use manually designed sharing rules to share a whole layer or block, largely restricting the flexibility of weight sharing.
Revised: Most of them (Lee et al., 2021; You et al., 2022; Lan et al., 2019; Reid et al., 2021) use manually designed sharing rules to share a whole layer or block, largely restricting the flexibility of weight sharing.

Original: This reduced flexibility can lead to drastic performance drops.
Revised: This reduced flexibility can lead to drastic performance drops.

Original: To maximally utilize model parameters, we propose to unify cross-modal sharing, layerwise sharing, and pruning, all in a single unified framework.
Revised: To maximally utilize model parameters, we propose to unify cross-modal sharing, layerwise sharing, and pruning, all in a single unified framework.

Original: Unlike previous works, the minimal structure of these operations is a weight vector instead of a whole layer or block, which drastically increases the flexibility of sharing and pruning.
Revised: Unlike previous works, the minimal structure of these operations is a weight vector instead of a whole layer or block, which drastically increases the flexibility of sharing and pruning.

Original: Also, to avoid using manually designed strategies, the positions of sharing and pruning are learned in an end-to-end differentiable manner.
Revised: Also, to avoid using manually designed strategies, the positions of sharing and pruning are learned in an end-to-end differentiable manner.

Original: To pursue a better trade-off between the model performance and the parameter efficiency, we aim to maximize flexibility by utilizing the structure of transformer backbones.
Revised: To pursue a better trade-off between the model performance and the parameter efficiency, we aim to maximize flexibility by utilizing the structure of transformer backbones.

Original: If only cross-modal sharing is considered, there will be an upper bound for the compression rate ( ∼ 50% ) when sharing all layers of one backbone for another one.
Revised: If only cross-modal sharing is considered, there will be an upper bound for the compression rate ( ∼ 50% ) when sharing all layers of one backbone for another one.

Original: Another direction is to share layers within a single backbone similar to Albert (Lan et al., 2019), however, the downside is that there is no prior success of cross-layer sharing for the vision transformer.
Revised: Another direction is to share layers within a single backbone similar to Albert (Lan et al., 2019), however, the downside is that there is no prior success of cross-layer sharing for the vision transformer.

Original: In addition to weight sharing, pruning is also an option; however, Li et al.
Revised: In addition to weight sharing, pruning is also an option; however, Li et al.

Original: Yu et al.
Revised: Yu et al.

Original: (2022) show that it limits the capacity of the model when the compression rate is high.
Revised: (2022) show that it limits the capacity of the model when the compression rate is high.

Original: We argue that, especially in multimodal tasks, it can be hard to achieve a high compression rate while maintaining high accuracy by using only pruning or sharing.
Revised: We argue that, especially in multimodal tasks, it can be hard to achieve a high compression rate while maintaining high accuracy by using only pruning or sharing.

Original: In this direction, we unify cross-modal sharing, layer-wise sharing, and pruning into a single framework to maximize the flexibility for sharing and pruning.
Revised: In this direction, we unify cross-modal sharing, layer-wise sharing, and pruning into a single framework to maximize the flexibility for sharing and pruning.

Original: With the increased flexibility, finding which weights to prune or share can be very difficult.
Revised: With the increased flexibility, finding which weights to prune or share can be very difficult.

Original: To solve this problem, we design a hypernetwork for learning the binary structure vectors for sharing and pruning efficiently.
Revised: To solve this problem, we design a hypernetwork for learning the binary structure vectors for sharing and pruning efficiently.

Original: The hypernetwork can effectively capture the complex interactions between pruning and sharing across different layers and modalities.
Revised: The hypernetwork can effectively capture the complex interactions between pruning and sharing across different layers and modalities.

Original: Since we can not share and prune a weight vector at the same time, we apply constraints on different parts of structure vectors.
Revised: Since we can not share and prune a weight vector at the same time, we apply constraints on different parts of structure vectors.

Original: We use a regularization term to softly align the learned structures and backbone weights instead of directly sharing and pruning by applying structure vectors.
Revised: We use a regularization term to softly align the learned structures and backbone weights instead of directly sharing and pruning by applying structure vectors.

Original: The benefit of such a regularization term is that we do not need to add a separate fine-tuning process to the whole pre-training step.
Revised: The benefit of such a regularization term is that we do not need to add a separate fine-tuning process to the whole pre-training step.

Original: Finally, we iteratively train the hypernetwork and the model so that the learned structures can adapt to the changes in the model during training.
Revised: Finally, we iteratively train the hypernetwork and the model so that the learned structures can adapt to the changes in the model during training.

Original: With these novel designs, our method learns suitable structures for sharing and pruning with small extra costs.
Revised: With these novel designs, our method learns suitable structures for sharing and pruning with small extra costs.

Original: We evaluate the effectiveness of our method with two state-of-the-art vision and language grounding models: MDETR (Kamath et al., 2021), GLIP (Li et al., 2022).
Revised: We evaluate the effectiveness of our method with two state-of-the-art vision and language grounding models: MDETR (Kamath et al., 2021), GLIP (Li et al., 2022).

Original: We use the CLIP text transformer (Radford et al., 2021) and DeiT (Touvron et al., 2020) as the text backbone and the vision backbone, and thus two backbones have similar architectures.
Revised: We use the CLIP text transformer (Radford et al., 2021) and DeiT (Touvron et al., 2020) as the text backbone and the vision backbone, and thus two backbones have similar architectures.

Original: Our method is evaluated on different downstream tasks: RefCOCO/RefCOCO+ (Yu et al.
Revised: Our method is evaluated on different downstream tasks: RefCOCO/RefCOCO+ (Yu et al.

Original: , 2016)/RefCOCOg (Mao et al., 2016) for referring expression comprehension (REC), GQA (Hudson & Manning, 2019) for visual question answering (VQA), MS-COCO object detection (Lin et al., 2014) and
Revised: , 2016)/RefCOCOg (Mao et al., 2016) for referring expression comprehension (REC), GQA (Hudson & Manning, 2019) for visual question answering (VQA), MS-COCO object detection (Lin et al., 2014) and

Original: Flickr-30k
Revised: Flickr-30k

Original: (Plummer et al., 2015) for phrase grounding.
Revised: (Plummer et al., 2015) for phrase grounding.

Original: On these benchmarks, our method can remove around 40% ∼ 45% parameters of the backbones and 35% ∼ 40% of all the parameters with almost no accuracy drop.
Revised: On these benchmarks, our method can remove around 40% ∼ 45% parameters of the backbones and 35% ∼ 40% of all the parameters with almost no accuracy drop.

Original: In some tasks, our method even outperforms the original model.
Revised: In some tasks, our method even outperforms the original model.

Original: These results show that the proposed framework achieves a prominent trade-off between the number of parameters and accuracy.
Revised: These results show that the proposed framework achieves a prominent trade-off between the number of parameters and accuracy.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Vision and Language Models.
Revised: Vision and Language Models.

Original: Existing vision and language models can be broadly grouped into two categories: (1) two-stage and (2) single-stage.
Revised: Existing vision and language models can be broadly grouped into two categories: (1) two-stage and (2) single-stage.

Original: Two-stage methods (Yu et al., 2018;
Revised: Two-stage methods (Yu et al., 2018;

Original: Chen et al., 2020; Lu et al., 2019) utilize off-the-shelf object detectors, e.g., Faster-RCNN, to detect objects and represent them with convolutional features.
Revised: Chen et al., 2020; Lu et al., 2019) utilize off-the-shelf object detectors, e.g., Faster-RCNN, to detect objects and represent them with convolutional features.

Original: For referring expression comprehension, the referring expression is matched to its appropriate region (Yu et al., 2018).
Revised: For referring expression comprehension, the referring expression is matched to its appropriate region (Yu et al., 2018).

Original: For other vision and language tasks, region descriptions are passed through two transformers to get image and language representations (Lu et al., 2019; Zhang et al., 2021a; Gan et al., 2020).
Revised: For other vision and language tasks, region descriptions are passed through two transformers to get image and language representations (Lu et al., 2019; Zhang et al., 2021a; Gan et al., 2020).

Original: Single-stage methods (Kamath et al., 2021;
Revised: Single-stage methods (Kamath et al., 2021;

Original: Deng et al., 2021;
Revised: Deng et al., 2021;

Original: Yang et al., 2020; Chen et al., 2018; Li & Sigal, 2021) avoid using a detached off-the-shelf object detector and perform end-to-end training and inference, reducing the computational complexity of the two-stage methods.
Revised: Yang et al., 2020; Chen et al., 2018; Li & Sigal, 2021) avoid using a detached off-the-shelf object detector and perform end-to-end training and inference, reducing the computational complexity of the two-stage methods.

Original: Another advantage of single-stage methods is that the full model can be pre-trained for text-conditioned object detection as in MDETR (Kamath et al., 2021), GLIP (Li et al., 2022) GLIP-v

Revised: Another advantage of single-stage methods is that the full model can be pre-trained for text-conditioned object detection as in MDETR (Kamath et al., 2021), GLIP (Li et al., 2022) GLIP-v

Original: (Zhang et al., 2022), etc.
Revised: (Zhang et al., 2022), etc.

Original: Pruning with Transformers.
Revised: Pruning with Transformers.

Original: With the increasing popularity of vision transformers, there are many works that perform pruning on different components, such as pruning tokens (Kong et al., 2021; Xu et al., 2021;
Revised: With the increasing popularity of vision transformers, there are many works that perform pruning on different components, such as pruning tokens (Kong et al., 2021; Xu et al., 2021;

Original: Rao et al., 2021), pruning structures or weights (Chen et al., 2021;
Revised: Rao et al., 2021), pruning structures or weights (Chen et al., 2021;

Original: Yin et al., 2023;
Revised: Yin et al., 2023;

Original: Lou et al., 2022b), or all of the above (Yang et al., 2021).
Revised: Lou et al., 2022b), or all of the above (Yang et al., 2021).

Original: The pruning for language transformers (Li et al., 2020; Gale et al., 2019) can be traced back even earlier.
Revised: The pruning for language transformers (Li et al., 2020; Gale et al., 2019) can be traced back even earlier.

Original: Unlike these works, in the cross-modal setting, we only prune weights that are not useful in both modalities.
Revised: Unlike these works, in the cross-modal setting, we only prune weights that are not useful in both modalities.

Original: Transformer-based Unified Backbones.
Revised: Transformer-based Unified Backbones.

Original: Transformers recently enjoyed tremendous popularity in processing different modalities, including images, videos (Girdhar et al., 2019), speech (Dong et al., , audio (Jaegle et al., 2021), and language (Jaegle et al., 2021; Hu & Singh, 2021; Zhang et al., 2021b).
Revised: Transformers recently enjoyed tremendous popularity in processing different modalities, including images, videos (Girdhar et al., 2019), speech (Dong et al., 2018), audio (Jaegle et al., 2021), and language (Jaegle et al., 2021; Hu & Singh, 2021; Zhang et al., 2021b).

Original: The universality of transformers has been exploited for multi-tasking using the same vision backbone for different vision-only tasks (Girdhar et al., 2022), the same vision and language backbone for different vision and language tasks (Hu & Singh, 2021).
Revised: The universality of transformers has been exploited for multi-tasking using the same vision backbone for different vision-only tasks (Girdhar et al., 2022), the same vision and language backbone for different vision and language tasks (Hu & Singh, 2021).

Original: Likhosherstov et al.
Revised: Likhosherstov et al.

Original: (2021), on the other hand, performs multitasking with the same transformer backbone on different modalities, and Wang et al.
Revised: (2021), on the other hand, performs multitasking with the same transformer backbone on different modalities, and Wang et al.

Original: pre-trains the model on vision-only, text-only, and vision-language tasks and fine-tune the model on vision or text-only or vision-language tasks.
Revised: pre-trains the model on vision-only, text-only, and vision-language tasks and fine-tune the model on vision or text-only or vision-language tasks.

Original: Similarly, Li et al.
Revised: Similarly, Li et al.

Original: ; You et al.
Revised: ; You et al.

Original: pre-train a unified transformer backbone with images and their captions and fine-tune the backbone for the single modality downstream tasks.
Revised: pre-train a unified transformer backbone with images and their captions and fine-tune the backbone for the single modality downstream tasks.

Original: Different from these studies, we propose structural cross-modal, layer-wise sharing and pruning across two transformer backbones for grounding based vision and language downstream tasks with almost no loss in accuracy.
Revised: Different from these studies, we propose structural cross-modal, layer-wise sharing and pruning across two transformer backbones for grounding based vision and language downstream tasks with almost no loss in accuracy.

Original: Weight Sharing with Transformers.
Revised: Weight Sharing with Transformers.

Original: There has been a number of studies that focus on sharing weights in a single transformer on language-only tasks (Lan et al., 2019; Reid et al., 2021; Takase &
Revised: There has been a number of studies that focus on sharing weights in a single transformer on language-only tasks (Lan et al., 2019; Reid et al., 2021; Takase &

Original: Kiyono, 2021; Lou et al., 2022a).
Revised: Kiyono, 2021; Lou et al., 2022a).

Original: These studies mostly re-use a language transformer’s full encoder (MSA+FFN) at different depths using some manually defined strategy.
Revised: These studies mostly re-use a language transformer’s full encoder (MSA+FFN) at different depths using some manually defined strategy.

Original: Parameter efficient crossmodal transformers (Lee et al., 2021) reduces parameters with manually designed weight sharing strategies.
Revised: Parameter efficient crossmodal transformers (Lee et al., 2021) reduces parameters with manually designed weight sharing strategies.

Original: Another recent study, Kim et al.
Revised: Another recent study, Kim et al.

Original: (2022), performs weight sharing in the multimodal fusion network, which can be orthogonal to our study as we explore weight sharing across the transformer backbones.
Revised: (2022), performs weight sharing in the multimodal fusion network, which can be orthogonal to our study as we explore weight sharing across the transformer backbones.

Original: Finally, a recent work, MS-CLIP (You et al., 2022), shares weights across modalities by rigorously examining the architecture choices and adding early specialization layers.
Revised: Finally, a recent work, MS-CLIP (You et al., 2022), shares weights across modalities by rigorously examining the architecture choices and adding early specialization layers.

Original: On the other hand, our method is general and flexible as we learn where to share and prune weight vectors in a vision and language model.
Revised: On the other hand, our method is general and flexible as we learn where to share and prune weight vectors in a vision and language model.

Original: Additionally, unlike MS-CLIP, we do not change the architecture of the original model.
Revised: Additionally, unlike MS-CLIP, we do not change the architecture of the original model.

Original: As a result, our method can be applied to any vision and language model with transformer backbones.
Revised: As a result, our method can be applied to any vision and language model with transformer backbones.

Original: Finally, different from these studies, we focus on grounding based vision and language models, which have shown impressive performance on many vision and language tasks (Kamath et al., 2021;
Revised: Finally, different from these studies, we focus on grounding based vision and language models, which have shown impressive performance on many vision and language tasks (Kamath et al., 2021;

Original: Li et al., 2022; Zhang et al., 2022).
Revised: Li et al., 2022; Zhang et al., 2022).

Original:  L EARNING TO S HARE AND P RUNE S TRUCTURES FOR V ISION AND L ANGUAGE T RANSFORMERS 3. O VERVIEW
Revised:  L EARNING TO S HARE AND P RUNE S TRUCTURES FOR V ISION AND L ANGUAGE T RANSFORMERS 3. O VERVIEW

Original: In a typical vision and language model, we have two backbones: a vision backbone ( v ) and a text backbone ( t ).
Revised: In a typical vision and language model, we have two backbones: a vision backbone ( v ) and a text backbone ( t ).

Original: We use W ∗ l ∗ ∈ { v, t } to denote the weight matrix of l ’th layer from the vision ( W v l ) or the text backbone ( W t l ), and l = 1 , . . .
Revised: We use W ∗ l ∗ ∈ { v, t } to denote the weight matrix of l ’th layer from the vision ( W v l ) or the text backbone ( W t l ), and l = 1 , . . .

Original: , L .
Revised: , L .

Original: L is the total number of layers.
Revised: L is the total number of layers.

Original: This definition also applies to sharing and pruning vectors: s ∗ and m ∗ .
Revised: This definition also applies to sharing and pruning vectors: s ∗ and m ∗ .

Original: Since we consider architecturally similar backbones, the vision and text backbone have the same L .
Revised: Since we consider architecturally similar backbones, the vision and text backbone have the same L .

Original: Usually, transformers are used for processing text and vision data.
Revised: Usually, transformers are used for processing text and vision data.

Original: A classic transformer has two core components: (1) Multi-Head Attention (MSA) and (2) Feed-Forward Network (FFN) layers.
Revised: A classic transformer has two core components: (1) Multi-Head Attention (MSA) and (2) Feed-Forward Network (FFN) layers.

Original: They are the targets for structural weight sharing and pruning since these components contain most of the weights in backbones.
Revised: They are the targets for structural weight sharing and pruning since these components contain most of the weights in backbones.

Original: To find which structures to prune or share, we first need to optimize the sharing and pruning vectors produced by the hypernetwork during the pre-training process of
Revised: To find which structures to prune or share, we first need to optimize the sharing and pruning vectors produced by the hypernetwork during the pre-training process of

Original: MDETR (Kamath et al., 2021) and GLIP (Li et al., 2022).
Revised: MDETR (Kamath et al., 2021) and GLIP (Li et al., 2022).

Original: To further increase the flexibility of the pruning and sharing process, layer-wise sharing (inspired by Albert (Lan et al., 2019) but at a fine-grained level) across the text backbone is also performed.
Revised: To further increase the flexibility of the pruning and sharing process, layer-wise sharing (inspired by Albert (Lan et al., 2019) but at a fine-grained level) across the text backbone is also performed.

Original: A detailed comparison of different sharing and pruning options is given in Fig.
Revised: A detailed comparison of different sharing and pruning options is given in Fig.

Original: Since we learn discrete structure vectors, directly applying them in the fine-tuning step can result in severe accuracy drops.
Revised: Since we learn discrete structure vectors, directly applying them in the fine-tuning step can result in severe accuracy drops.

Original: We instead softly align model weights and learned structures via a regularization term in the pre-training step.
Revised: We instead softly align model weights and learned structures via a regularization term in the pre-training step.

Original: After the pretraining step, we conduct structural pruning and sharing based on learned structure vectors.
Revised: After the pretraining step, we conduct structural pruning and sharing based on learned structure vectors.

Original: Finally, we fine-tune the compressed model for the downstream task.
Revised: Finally, we fine-tune the compressed model for the downstream task.

Original: An overview of how our method shares and prunes weights is provided in Fig. 1.
Revised: An overview of how our method shares and prunes weights is provided in Fig. 1.

Original: 3. S HARING AND P RUNING ACROSS T WO B ACKBONES
Revised: 3. S HARING AND P RUNING ACROSS T WO B ACKBONES

Original: For MSA layers, we have three weight matrices for query W ∗ q ∈ R d × d , key W ∗ k ∈ R d × d and value
Revised: For MSA layers, we have three weight matrices for query W ∗ q ∈ R d × d , key W ∗ k ∈ R d × d and value

Original: W ∗ v ∈ R d × d and again ∗ ∈ { v, t } .
Revised: W ∗ v ∈ R d × d and again ∗ ∈ { v, t } .

Original: With input tokens X ∗ ∈ R N ∗ × d , the final Q ∗ , K ∗ and V ∗ for self-attention is obtained by:
Revised: With input tokens X ∗ ∈ R N ∗ × d , the final Q ∗ , K ∗ and V ∗ for self-attention is obtained by:

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: where d is the embedding dimension, N ∗ represents the number of tokens given the modality.
Revised: where d is the embedding dimension, N ∗ represents the number of tokens given the modality.

Original: To make minimal changes to the original model, we can perform pruning on W ∗ q and W ∗ k , and structural sharing are applied to all three weight matrix between backbones and across layers.
Revised: To make minimal changes to the original model, we can perform pruning on W ∗ q and W ∗ k , and structural sharing are applied to all three weight matrix between backbones and across layers.

Revised: [Figure]

Original: For pruning, we use a binary vector m ∗ ∈ { 0 , 1 } d on W ∗ q and W ∗ k .
Revised: For pruning, we use a binary vector m ∗ ∈ { 0 , 1 } d on W ∗ q and W ∗ k .

Original: Let’s use the vision backbone as an example for pruning.
Revised: Let’s use the vision backbone as an example for pruning.

Original: The production of the query and key when calculating MSA then becomes:
Revised: The production of the query and key when calculating MSA then becomes:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ) where m v l is first resized to have the same dimension of W v l q and ⊙ is the element-wise production.
Revised: where m v l is first resized to have the same dimension of W v l q and ⊙ is the element-wise production.

Original: X v l is the feature map from the previous layer.
Revised: X v l is the feature map from the previous layer.

Original: We perform cross-modal sharing between two backbones, and cross-layer sharing for the text backbone.
Revised: We perform cross-modal sharing between two backbones, and cross-layer sharing for the text backbone.

Original: To achieve sharing, we use a binary vector s ∗ ∈ { 0 , 1 } d to share weights between two modalities and different layers.
Revised: To achieve sharing, we use a binary vector s ∗ ∈ { 0 , 1 } d to share weights between two modalities and different layers.

Original: Weights of the text backbone are used for cross-modal sharing.
Revised: Weights of the text backbone are used for cross-modal sharing.

Original: As a result, a weight vector could be implicitly used in different layers across text and vision backbones.
Revised: As a result, a weight vector could be implicitly used in different layers across text and vision backbones.

Original: Let’s use W v l q as an example for cross-modal sharing:
Revised: Let’s use W v l q as an example for cross-modal sharing:

Original: [Equation]
Revised: [Equation]

Original: where s l is also first expanded to have the same size of W v l q .
Revised: where s l is also first expanded to have the same size of W v l q .

Original: In short, weight vectors of index i with s li = 0 are shared.
Revised: In short, weight vectors of index i with s li = 0 are shared.

Original: Without loss of generality, we omitted the modal notation ( v or t ) for cross-modal sharing s l , since the sharing direction is fixed.
Revised: Without loss of generality, we omitted the modal notation ( v or t ) for cross-modal sharing s l , since the sharing direction is fixed.

Original: Similarly, cross-layer sharing has the following setup:
Revised: Similarly, cross-layer sharing has the following setup:

Original: [Equation]
Revised: [Equation]

Original: where W t l q represents the weights of l th layer in the text backbone, W t b q is the assigned base weights, and s t l is the vector used for cross-layer sharing.
Revised: where W t l q represents the weights of l th layer in the text backbone, W t b q is the assigned base weights, and s t l is the vector used for cross-layer sharing.

Original: As a result, the final weights for the text backbone is divided into two parts: layer-specific weights and shared weights from base layers.
Revised: As a result, the final weights for the text backbone is divided into two parts: layer-specific weights and shared weights from base layers.

Original: A naive way for cross-layer sharing uses weights from a certain layer as base weights for all other layers (Lan et al., 2019).
Revised: A naive way for cross-layer sharing uses weights from a certain layer as base weights for all other layers (Lan et al., 2019).

Original: However, it can restrict the diversity of weights across layers.
Revised: However, it can restrict the diversity of weights across layers.

Original: To enhance the flexibility of cross-layer sharing, we divide the text backbone into several blocks, and each block has its own base weights.
Revised: To enhance the flexibility of cross-layer sharing, we divide the text backbone into several blocks, and each block has its own base weights.

Original: Specifically, we establish a set of base layers (in sorted order)
Revised: Specifically, we establish a set of base layers (in sorted order)

Original: B = { b 1 , b 2 , · · · , b | B | } .
Revised: B = { b 1 , b 2 , · · · , b | B | } .

Original: All layers are naturally split into several blocks based on base layers, e.g., the first block contains layers with b 1 ≤ l < b 2 .
Revised: All layers are naturally split into several blocks based on base layers, e.g., the first block contains layers with b 1 ≤ l < b 2 .

Original: In our experiments, the base weight is set to the first layer of each block.
Revised: In our experiments, the base weight is set to the first layer of each block.

Original: When l is a base layer ( l ∈ B ), we force s t l = 1 since we assign the base layer its own weights and potentially reuse it for the other layers in the same block.
Revised: When l is a base layer ( l ∈ B ), we force s t l = 1 since we assign the base layer its own weights and potentially reuse it for the other layers in the same block.

Original: Note that after cross-layer sharing, the base weights for cross-modal sharing is also changed.
Revised: Note that after cross-layer sharing, the base weights for cross-modal sharing is also changed.

Original: By putting cross-layer sharing and cross-modal sharing together, we have:
Revised: By putting cross-layer sharing and cross-modal sharing together, we have:

Original: [Equation]
Revised: [Equation]

Revised: As a result, the final W v l q consists weights from W

Original: [Figure]
Revised: [Figure]

Original: As a result, the final W v l q consists weights from W t l q (weights from the text backbone of the same layer), W t b q (weights from the text backbone of the base layer) and vision modality specific weights.
Revised: , we can

Original: For FFN layers, we have a weight matrix
Revised: For FFN layers, we have a weight matrix

Original: W ∗ E ∈ R d × d ′ expanding the feature dimension, and another weight matrix W ∗ C ∈ R d ′ ×
Revised: W ∗ E ∈ R d × d ′ expanding the feature dimension, and another weight matrix W ∗ C ∈ R d ′ ×

Original: d compressing the feature dimension.
Revised: d compressing the feature dimension.

Original: To keep the feature map size d unchanged, we conduct sharing and pruning along the dimension with size d ′ and the process is similar to MSA layers.
Revised: To keep the feature map size d unchanged, we conduct sharing and pruning along the dimension with size d ′ and the process is similar to MSA layers.

Original: The detailed formulation is provided in the supplementary materials.
Revised: The detailed formulation is provided in the supplementary materials.

Original: We do not apply sharing and pruning for other layers including Layer Norm (Ba et al., 2016) since they contain a very small amount of parameters.
Revised: We do not apply sharing and pruning for other layers including Layer Norm (Ba et al., 2016) since they contain a very small amount of parameters.

Original: 3. C ONFLICTS OF S TRUCTURE V ECTORS
Revised: 3. C ONFLICTS OF S TRUCTURE V ECTORS

Original: Without any constraint, m and s could have conflicts.
Revised: Without any constraint, m and s could have conflicts.

Original: For example, if we prune weights, it becomes meaningless to share the same weights.
Revised: For example, if we prune weights, it becomes meaningless to share the same weights.

Original: To resolve such conflicts, we impose the following constraint for cross-modal sharing and pruning:
Revised: To resolve such conflicts, we impose the following constraint for cross-modal sharing and pruning:

Original: [Equation]
Revised: [Equation]

Original: where i is the index of sharing or pruning for a certain weight vector.
Revised: where i is the index of sharing or pruning for a certain weight vector.

Original: If ( m i , s i ) ∈ C , then the structure vector does not share and prune the same weight vector which resolves the conflicts.
Revised: If ( m i , s i ) ∈ C , then the structure vector does not share and prune the same weight vector which resolves the conflicts.

Original: This constraint can also be applied for cross-layer sharing and pruning for the text backbone, but in a more sophisticated way:(cid:40) ( m t l i , s t l i ) ∈ C, where C = { ( x, y ) | ( x, y ) ̸
Revised: This constraint can also be applied for cross-layer sharing and pruning for the text backbone, but in a more sophisticated way:

Original: = (0 , 0) } , l / ∈ B, ( m t l i , s t l
Revised: [Equation] l / ∈ B, ( m t l i , s t l

Original: i s t
Revised: i s t

Original: l +1 i · · · s t b ′ i ) ∈ C, where C = { ( x, y ) | ( x, y ) ̸
Revised: l +1 i · · · s t b ′ i ) ∈ C, where C = { ( x, y ) | ( x, y ) ̸

Original: = (0 , 0) } , l ∈ B, (7) where b ′ represents the last element of the current block.
Revised: = (0 , 0) } , l ∈ B, (7) where b ′ represents the last element of the current block.

Original: For example, if l = b 1 , the current block consists layers of b 1 ≤ l < b 2 , and b ′ =
Revised: For example, if l = b 1 , the current block consists layers of b 1 ≤ l < b 2 , and b ′ =

Original: b 2 − 1 .
Revised: b 2 − 1 .

Original: With s t l i s t l +1 i · · · s t b
Revised: With s t l i s t l +1 i · · · s t b

Original: i
Revised: i

Original: we represent all the shared elements from different layers in the current block, and these elements should be kept in the base layer.
Revised: we represent all the shared elements from different layers in the current block, and these elements should be kept in the base layer.

Original: We do not add this constraint between s li and s t l i , since there is no conflict between sharing.
Revised: We do not add this constraint between s li and s t l i , since there is no conflict between sharing.

Original: To directly apply this constraint, we can prioritize m i or s i and set the other one manually.
Revised: To directly apply this constraint, we can prioritize m i or s i and set the other one manually.

Original: For example, let m t l i = s li
Revised: For example, let m t l i = s li

Original: = 0 , we can directly set m t l i = 1 to comply with the constraint in Eq.
Revised: = 0 , we can directly set m t l i = 1 to comply with the constraint in Eq.

Original: 6 if s i is prioritized (shared weights will not be pruned).
Revised: 6 if s i is prioritized (shared weights will not be pruned).

Original: In practice, we prefer sharing weights to pruning weights, since sharing preserves the model capacity to some extent.
Revised: In practice, we prefer sharing weights to pruning weights, since sharing preserves the model capacity to some extent.

Original: 3. L EARNING S TRUCTURE V ECTORS
Revised: 3. L EARNING S TRUCTURE V ECTORS

Original: To generate m, s , we use a hypernetwork (HN) parameterized by θ and Gumbel-Sigmoid (Jang et al., 2016) technique:[Equation]
Revised: To generate m, s , we use a hypernetwork (HN) parameterized by θ and Gumbel-Sigmoid (Jang et al., 2016) technique:

Original: [Equation]
Revised: [Equation]

Original: where z is the input to the HN, and it is a predefined vector.
Revised: where z is the input to the HN, and it is a predefined vector.

Original: Basically, the HN is composed of
Revised: Basically, the HN is composed of

Original: GRUs (Chung et al., 2014) and multilayer perceptrons (MLPs).
Revised: GRUs (Chung et al., 2014) and multilayer perceptrons (MLPs).

Original: GRU can capture inter-layer interactions, and MLP is used for intra-layer interactions.
Revised: GRU can capture inter-layer interactions, and MLP is used for intra-layer interactions.

Original: More details of the HN are presented in the supplementary materials.
Revised: More details of the HN are presented in the supplementary materials.

Original: To learn the HN, we want to solve the following optimization problem:
Revised: To learn the HN, we want to solve the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: where x is the input sample of the image and text pair, y is its label, L pre-training is the original pretraining loss from MDETR (Kamath et al., 2021) or GLIP (Li et al., 2022), and its model structure is  decided by m, s .
Revised: where x is the input sample of the image and text pair, y is its label, L pre-training is the original pretraining loss from MDETR (Kamath et al., 2021) or GLIP (Li et al., 2022), and its model structure is Get by pruning and based on m s decided by m, s .

Original: On the other hand, λ controls the strength of R ( P ( m, s ) , pP total ) .
Revised: On the other hand, λ controls the strength of R ( P ( m, s ) , pP total ) .

Original: The regularization loss, R ( P ( m, s ) , pP total ) , controls how much parameters we should keep given by p ∈ (0 , 1] .
Revised: The regularization loss, R ( P ( m, s ) , pP total ) , controls how much parameters we should keep given by p ∈ (0 , 1] .

Original: P ( m, s ) in the regularization loss represents the remaining number of parameters decided by m, s and P total is the total number of parameters of MSA+FFN layers in backbones.
Revised: P ( m, s ) in the regularization loss represents the remaining number of parameters decided by m, s and P total is the total number of parameters of MSA+FFN layers in backbones.

Original: In practice, we let
Revised: In practice, we let

Original: R ( P ( m, s ) , pP total ) =
Revised: R ( P ( m, s ) , pP total ) =

Original: log(max( P ( m, s ) , pP total ) /pP total ) (Gao et al., 2021).
Revised: log(max( P ( m, s ) , pP total ) /pP total ) (Gao et al., 2021).

Original: On the other hand, R could be any regression loss functions, including MSE and MAE; however, it might be harder to push R small enough, especially when the number of parameters is large.
Revised: On the other hand, R could be any regression loss functions, including MSE and MAE; however, it might be harder to push R small enough, especially when the number of parameters is large.

Original: During the optimization of the HN, we keep the weights of the whole model including backbones and the fusion part frozen.
Revised: During the optimization of the HN, we keep the weights of the whole model including backbones and the fusion part frozen.

Original: Since the time cost for pre-training is quite large, we use a small portion of the whole dataset to train HN.
Revised: Since the time cost for pre-training is quite large, we use a small portion of the whole dataset to train HN.

Original: Next, we train model weights on the whole pre-training dataset while controlling its architecture with m, s by training the hypernetwork on a subset iteratively.
Revised: Next, we train model weights on the whole pre-training dataset while controlling its architecture with m, s by training the hypernetwork on a subset iteratively.

Original: 3. L EARNING TO J OINTLY S HARE AND P RUNE W EIGHTS
Revised: 3. L EARNING TO J OINTLY S HARE AND P RUNE W EIGHTS

Original: To share and prune the weights, we can directly apply m and s (from the hypernetwork) to vision and text transformers.
Revised: To share and prune the weights, we can directly apply m and s (from the hypernetwork) to vision and text transformers.

Original: However, it can drop the accuracy since the outputs of the vision and text backbones are drastically changed.
Revised: However, it can drop the accuracy since the outputs of the vision and text backbones are drastically changed.

Original: Thus, to alleviate this problem, we use a selection based regularization mechanism to softly push selected weights closer (for sharing) or to zeros (for pruning):
Revised: Thus, to alleviate this problem, we use a selection based regularization mechanism to softly push selected weights closer (for sharing) or to zeros (for pruning):

Original: [Equation]
Revised: [Equation]

Original: (10) where the first two terms push selected weight vectors closer and the final term uses Group Lasso to push selected weight vectors to 0 .
Revised: where the first two terms push selected weight vectors closer and the final term uses Group Lasso to push selected weight vectors to 0 .

Original: With this regularization term, it gradually aligns model weights to the learned structure vectors which creates a smooth process for reducing the number of model parameters.
Revised: With this regularization term, it gradually aligns model weights to the learned structure vectors which creates a smooth process for reducing the number of model parameters.

Original: Given the regularization loss, we learn model weights W by optimizing the following objective function:
Revised: Given the regularization loss, we learn model weights W by optimizing the following objective function:

Original: [Equation]
Revised: [Equation]

Original: where γ controls the strength of R w ( W, m, s )
Revised: where γ controls the strength of R w ( W, m, s )

Original: After the pre-training process, the corresponding weights are pruned and shared to compress the model.
Revised: After the pre-training process, the corresponding weights are pruned and shared to compress the model.

Original: The compressed model can then be used for fine-tuning on downstream tasks.[Table] [Table] 119M
Revised: The compressed model can then be used for fine-tuning on downstream tasks.

Original: We present the training process of the proposed method Algorithm
Revised: We present the training process of the proposed method in Algorithm.

Original: Note  when optimizing the model (Obj. 11) and HN (Obj. 9)
Revised: Note that (1) the forward calculation is different when optimizing the model (Obj. 11) and HN (Obj. 9).

Original: When optimizing HN, m, s are applied in the forward calculation, and when optimizing the model, the remains intact
Revised: When optimizing HN, m, s are applied in the forward calculation, and when optimizing the model, the forward calculation [Table] [Table] 119M intact

Original:  To reduce the training time of the HN, weuse a small subsetD subof the pre- dataset, D sub≪ D.
Revised: (2) To reduce the training time of the , we a small subset sub the pre- dataset D sub D

Original:  The goal of using HN is to accelerate the learning of m, s , and better capture the complicated interaction between pruning and sharing across modalities and layers.
Revised: (3) The goal of using HN is to accelerate the learning of m, s , and better capture the complicated interaction between pruning and sharing across modalities and layers.

Original: We note that we can also directly set m, s as learnable parameters, however, it can slow down the learning process and thus lower the final performance.
Revised: We note that we can also directly set m, s as learnable parameters, however, it can slow down the learning process and thus lower the final performance.

Original:  E XPERIMENTS 4. S ETTINGS
Revised:  E XPERIMENTS 4. S ETTINGS

Original: GLIP and MDETR.
Revised: GLIP and MDETR.

Original: We apply our method to two recently published grounding based vision and language models: (1) MDETR (Kamath et al., 2021) and (2) GLIP (Li et al., 2022).
Revised: We apply our method to two recently published grounding based vision and language models: (1) MDETR (Kamath et al., 2021) and (2) GLIP (Li et al., 2022).

Original: MDETR performs end-to-end pre-training on dataset of 1.3M image-text pairs for the task of text-conditioned object detection.
Revised: MDETR performs end-to-end pre-training on dataset of 1.3M image-text pairs for the task of text-conditioned object detection.

Original: On the other hand, GLIP scales up the pre-training dataset for text-conditioned object detection to 27M image-text pairs by training a teacher network on 1.3M image-text pairs with ground truth bounding boxes and using the teacher on 27M image-text pairs to get pseudo bounding boxes.
Revised: On the other hand, GLIP scales up the pre-training dataset for text-conditioned object detection to 27M image-text pairs by training a teacher network on 1.3M image-text pairs with ground truth bounding boxes and using the teacher on 27M image-text pairs to get pseudo bounding boxes.

Original: Pre-trained GLIP and MDETR are then used for vision and language tasks.
Revised: Pre-trained GLIP and MDETR are then used for vision and language tasks.

Original: Baselines.
Revised: Baselines.

Original: We constructed several alternative settings to verify the design choice of our method.
Revised: We constructed several alternative settings to verify the design choice of our method.

Original: The first setting is F ully cross-modal S haring ( FS ).
Revised: The first setting is F ully cross-modal S haring ( FS ).

Original: For FS, we share all MSA+FFN layers across two backbones during both the pre-training stage and the fine-tuning stage without any regularization term.
Revised: For FS, we share all MSA+FFN layers across two backbones during both the pre-training stage and the fine-tuning stage without any regularization term.

Original: We note that FS can be considered as a manually designed weight sharing method similar to You et al.
Revised: We note that FS can be considered as a manually designed weight sharing method similar to You et al.

Original: ; Lee et al.
Revised: ; Lee et al.

Original: The second setting is cross-modal S haring and P runing ( SP ).
Revised: The second setting is cross-modal S haring and P runing ( SP ).

Original: For SP, we only apply cross-modal sharing and pruning for the vision and text backbones.
Revised: For SP, we only apply cross-modal sharing and pruning for the vision and text backbones.

Original: The third setting is cross-modal, L ayer- W ise S haring and P runing ( LWSP ).
Revised: The third setting is cross-modal, L ayer- W ise S haring and P runing ( LWSP ).

Original: On top of SP, LWSP also shares weights across layers in the text backbone.
Revised: On top of SP, LWSP also shares weights across layers in the text backbone.

Original: The third setting is cross-modal, B lock- W ise
Revised: The third setting is cross-modal, B lock- W ise

Original: S haring and P runing ( BWSP ).
Revised: S haring and P runing ( BWSP ).

Original: For BWSP, the text backbone is split into three blocks, and each block contains 4 consecutive MSA and MLP layers.
Revised: For BWSP, the text backbone is split into three blocks, and each block contains 4 consecutive MSA and MLP layers.

Original: Fig.
Revised: Fig.

Original: 2 shows the concept of these settings.
Revised: 2 shows the concept of these settings.

Original: By default, layer-wise sharing and block-wise sharing are applied for the text backbone, and the weights of the text backbone are reused for the vision backbone for the cross-modal sharing since Lu et al.
Revised: By default, layer-wise sharing and block-wise sharing are applied for the text backbone, and the weights of the text backbone are reused for the vision backbone for the cross-modal sharing since Lu et al.

Original: (2021) showed that language transformers can generalize to vision tasks.
Revised: (2021) showed that language transformers can generalize to vision tasks.

Original: Implementation Details.
Revised: Implementation Details.

Original: In all experiments, we replace the original backbones used in MDETR and GLIP with the CLIP text transformer and DeiT to have architecturally similar backbones.
Revised: In all experiments, we replace the original backbones used in MDETR and GLIP with the CLIP text transformer and DeiT to have architecturally similar backbones.

Original: Due to the resource limitation, we change the GLIP pre-training dataset to be the same as MDETR, and the[Figure] [Table] pre-training dataset for MDETR is not changed.
Revised: Due to the resource limitation, we change the GLIP pre-training dataset to be the same as MDETR, and the pre-training dataset for MDETR is not changed.

Original: We note that MDETR and GLIP train the models on different size images and test the models on images with smaller size resized to 800 pixels.
Revised: We note that MDETR and GLIP train the models on different size images and test the models on images with smaller size resized to 800 pixels.

Original: On the other hand, we resize the images to 224x224 pixels in training and test time as the motivation of  our study is to maintain the accuracy of the full model after compression.
Revised: On the other hand, we resize the images to 224x224 pixels in training and test time as the motivation of [Figure] [Table] our study is to maintain the accuracy of the full model after compression.

Original: Due to space limitation, other implementation details are given in supplementary materials.
Revised: Due to space limitation, other implementation details are given in supplementary materials.

Original: 4. R ESULTS MDETRresults.
Revised: 4. R ESULTS MDETR

Revised: results.

Original: For MDETR, we construct two models, MDETR-Small and MDETRLarge, with different model settings.
Revised: For MDETR, we construct two models, MDETR-Small and MDETRLarge, with different model settings.

Original: We report results on the validation dataset of RefCOCO/RefCOCO+/RefOCOCg and GQA.
Revised: We report results on the validation dataset of RefCOCO/RefCOCO+/RefOCOCg and GQA.

Original: The result of MDETR-Small is shown in Tab. 1.
Revised: The result of MDETR-Small is shown in Tab. 1.

Original: From the table, we can see that the performance of BWSP is on par with the baseline model (relative changes: +0.1/+0.0/-0.5/+0.1 for four tasks).
Revised: From the table, we can see that the performance of BWSP is on par with the baseline model (relative changes: +0.1/+0.0/-0.5/+0.1 for four tasks).

Original: At the same time, BWSP reduces 24M parameters (around 38% of two backbones) compared to the baseline model.
Revised: At the same time, BWSP reduces 24M parameters (around 38% of two backbones) compared to the baseline model.

Original: LWSP and SP perform worse than
Revised: LWSP and SP perform worse than

Original: BWSP, indicating that increasing flexibility is helpful when reducing the number of parameters.
Revised: BWSP, indicating that increasing flexibility is helpful when reducing the number of parameters.

Original: The results of MDETR-Large is shown in Tab. 2.
Revised: The results of MDETR-Large is shown in Tab. 2.

Original: On MDETR-Large, BWSP also performs similarly to the baseline model, but the parameter reduction rate is larger (40% of the baseline model).
Revised: On MDETR-Large, BWSP also performs similarly to the baseline model, but the parameter reduction rate is larger (40% of the baseline model).

Original: For both
Revised: For both

Original: MDETR-Large and MDETR-Small, the manually designed sharing rule FS causes significant performance drops.
Revised: MDETR-Large and MDETR-Small, the manually designed sharing rule FS causes significant performance drops.

Original: In addition, given a similar number of parameters, BWSP outperforms other full models with different backbones on both the large and small model setups.
Revised: In addition, given a similar number of parameters, BWSP outperforms other full models with different backbones on both the large and small model setups.

Original: Finally, we remove R w during pre-training, and we directly train the hypernetwork and fine-tune the model.
Revised: Finally, we remove R w during pre-training, and we directly train the hypernetwork and fine-tune the model.

Original: We call this setting BWSP w/o R w .
Revised: We call this setting BWSP w/o R w .

Original: As expected, BWSP w/o R w results in a much larger performance drop, which justifies the design choice of using a R w to align model weights and structures during pre-training.
Revised: As expected, BWSP w/o R w results in a much larger performance drop, which justifies the design choice of using a R w to align model weights and structures during pre-training.

Original: GLIP results.
Revised: GLIP results.

Original: We evaluate our method with GLIP on MS-COCO object detection (COCO) and
Revised: We evaluate our method with GLIP on MS-COCO object detection (COCO) and

Original: Flickr-30k for phrase grounding.
Revised: Flickr-30k for phrase grounding.

Original: The results on COCO are shown in Tab. 3.
Revised: The results on COCO are shown in Tab. 3.

Original: From the table, we can see that BWSP still outperforms SP and LWSP.
Revised: From the table, we can see that BWSP still outperforms SP and LWSP.

Original: Besides the performance, BWSP also removes more parameters.
Revised: Besides the performance, BWSP also removes more parameters.

Original: In Tab. 4, we can have a similar observation.
Revised: In Tab. 4, we can have a similar observation.

Original: In summary, BWSP provides more flexibility than SP and LWSP, which results in a stronger performance/parameter trade-off.
Revised: In summary, BWSP provides more flexibility than SP and LWSP, which results in a stronger performance/parameter trade-off.

Original: 4. D ETAILED A NALYSIS
Revised: 4. D ETAILED A NALYSIS

Original: Effects of R w .
Revised: Effects of R w .

Original: In Fig.
Revised: In Fig.

Original: 4, we plot the weight difference with or without R w .
Revised: 4, we plot the weight difference with or without R w .

Original: Let’s consider crossmodal sharing as an example, for l th layer, we plot 1 n (cid:80) ni ∥ (1 − s i ) ⊙ W t l − (1 − s i ) ⊙ W v l ∥ 1 for [Figure] Settings Val
Revised: Let’s consider crossmodal sharing as an example, for l th layer, we plot 1 n (cid:80) ni ∥ (1 − s i ) ⊙ W t l − (1 − s i ) ⊙ W v l ∥ 1 for regularized weights, and we plot 1 n (cid:80) ni

Original: Test

Original: [Table]

Original: PT

Original: [Table]

Original: R@

Original: [Table] [Table] [Table] [Table]
Revised: ∥ s i ⊙ W t l − s i ⊙ W v l ∥ 1 for weights without regularization.

Original: Backbones: CLIP+DeiT-Small

Original: Baseline 72. [Table] [Table] 83M 63M
Revised: The figure shows that the weight difference is much smaller with regularization for both pruning and cross-modal sharing.

Original: SP [Table] [Table] [Table]
Revised: For layer-wise sharing, the difference is smaller since the ratio of layer-shared

Original: [Table]
Revised: [Figure]

Original: 63M 43M

Original: LWSP 72. [Table] [Table] [Table]
Revised: Settings

Original: [Table]

Original: [Table]

Original: 60M 40M

Original: [Table]

Original: [Table]
Revised: [Table]

Original: 57M 37M regularized weights, and we plot 1 n (cid:80) ni ∥ s i ⊙ W t l − s i ⊙ W v l ∥ 1 for weights without regularization. The figure shows that the weight difference is much smaller with regularization for both pruning and cross-modal sharing. For layer-wise sharing, the difference is smaller since the ratio of layer-shared weights is larger.
Revised: weights is larger.

Original: In conclusion, Fig. 4 shows that the proposed regularization R w can effectively encourage sharing and pruning of desired weights, and learning other weights is almost intact.
Revised: In conclusion, Fig. 4 shows that the proposed regularization R w can effectively encourage sharing and pruning of desired weights, and learning other weights is almost intact.

Original: Analysis of Pruned and Shared Weights.
Revised: Analysis of Pruned and Shared Weights.

Original: In Fig.
Revised: In Fig.

Original: 3, we plot the details of pruned and shared weights of BWSP for MDETR-Small.
Revised: 3, we plot the details of pruned and shared weights of BWSP for MDETR-Small.

Original: We observe that the layer-wise sharing rate is large for both
Revised: We observe that the layer-wise sharing rate is large for both

Original: FFN and MSA layers of the text backbone.
Revised: FFN and MSA layers of the text backbone.

Original: For the vision backbone, earlier layers are almost preserved for both FFN and MSA layers.
Revised: For the vision backbone, earlier layers are almost preserved for both FFN and MSA layers.

Original: We also observe that FFN layers from the vision backbone are shared more aggressively whereas MSA layers are only shared for later layers.
Revised: We also observe that FFN layers from the vision backbone are shared more aggressively whereas MSA layers are only shared for later layers.

Original: On the other hand, the ratio of pruning is not obvious except for the vision FFN layers.
Revised: On the other hand, the ratio of pruning is not obvious except for the vision FFN layers.

Original: In summary, the text backbone contributes more to the reduction of weights, which is reasonable since vision tasks are more complex than language tasks in general for the tasks required for MDETR and GLIP models.
Revised: In summary, the text backbone contributes more to the reduction of weights, which is reasonable since vision tasks are more complex than language tasks in general for the tasks required for MDETR and GLIP models.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we investigate how to perform structural sharing and pruning for grounding based vision and language models.
Revised: In this paper, we investigate how to perform structural sharing and pruning for grounding based vision and language models.

Original: Specifically, we perform cross-modal sharing and pruning across the vision and text backbones and layer-wise sharing in the text backbone.
Revised: Specifically, we perform cross-modal sharing and pruning across the vision and text backbones and layer-wise sharing in the text backbone.

Original: With larger flexibility, it becomes harder to select the right set of weight vectors to share and prune.
Revised: With larger flexibility, it becomes harder to select the right set of weight vectors to share and prune.

Original: To overcome this challenge, we propose an hypernetwork to capture the interactions of pruning and sharing across layers and modalities.
Revised: To overcome this challenge, we propose an hypernetwork to capture the interactions of pruning and sharing across layers and modalities.

Original: Next, in the pre-training stage, we gradually align the backbones’ weights with the desired architecture learned by the hypernetwork.
Revised: Next, in the pre-training stage, we gradually align the backbones’ weights with the desired architecture learned by the hypernetwork.

Original: Finally, the model is fine-tuned for downstream tasks.
Revised: Finally, the model is fine-tuned for downstream tasks.

Original: Our experiments on MDETR and GLIP show that we can reduce 35% ∼ 40% of model weights without a significant loss in accuracy.
Revised: Our experiments on MDETR and GLIP show that we can reduce 35% ∼ 40% of model weights without a significant loss in accuracy.

Original: A S UPPLEMENTARY M ATERIALS A. P RUNING AND S HARING FOR FFN L AYERS
Revised: A S UPPLEMENTARY M ATERIALS A. P RUNING AND S HARING FOR FFN L AYERS

Original: We will provide more details for the pruning and sharing of FFN Layers in this section.
Revised: We will provide more details for the pruning and sharing of FFN Layers in this section.

Original: Recall that we have a weight matrix W ∗ E ∈ R d × d
Revised: Recall that we have a weight matrix W ∗ E ∈ R d × d

Original: expanding the feature dimension, and another weight matrix
Revised: expanding the feature dimension, and another weight matrix

Original: W ∗ C ∈ R d ′ ×
Revised: W ∗ C ∈ R d ′ ×

Original: d compressing the feature dimension for FFN layers.
Revised: d compressing the feature dimension for FFN layers.

Original: To keep the model dimension d unchanged, we conduct sharing and pruning along the dimension with size d ′ .
Revised: To keep the model dimension d unchanged, we conduct sharing and pruning along the dimension with size d ′ .

Original: For the cross-modal sharing, we then have:
Revised: For the cross-modal sharing, we then have:

Original: [Equation]
Revised: [Equation]

Original: like MSA layers, s l is first resized to have the same size of W v l E .
Revised: like MSA layers, s l is first resized to have the same size of W v l E .

Original: For layer-wise sharing, we have the following equations:
Revised: For layer-wise sharing, we have the following equations:

Original: [Equation]
Revised: [Equation]

Original: 13) s t l is resized to have the same size of W t l E .
Revised: 13) s t l is resized to have the same size of W t l E .

Original: Let’s take the vision backbone as an example, the pruning for W v l E and W v l C is done by inserting the mask m v l to the intermediate feature map between W v l E and W v l C : ˆ X v l
Revised: Let’s take the vision backbone as an example, the pruning for W v l E and W v l C is done by inserting the mask m v l to the intermediate feature map between W v l E and W v l C : ˆ X v l

Original: = m v l ⊙ X v l .
Revised: = m v l ⊙ X v l .

Original: To simplify notations, we reuse the notations s l , s t l and m v l for
Revised: To simplify notations, we reuse the notations s l , s t l and m v l for

Original: FFN layers.
Revised: FFN layers.

Original: Similar to MSA layers, sharing and pruning for FFN layers also satisfy the constraints defined in Eq.
Revised: Similar to MSA layers, sharing and pruning for FFN layers also satisfy the constraints defined in Eq.

Original: 7 and Eq. 6.
Revised: 7 and Eq. 6.

Original: A. D ETAILED S TRUCTURE OF H YPERNETWORK
Revised: A. D ETAILED S TRUCTURE OF H YPERNETWORK

Original: The detailed architecture is summarized in Tab. S1.
Revised: The detailed architecture is summarized in Tab. S1.

Original: The input of the hypernetwork z ∈ R 2 L × 32 ,
Revised: The input of the hypernetwork z ∈ R 2 L × 32 ,

Original: [Table]
Revised: [Table]

Original: (32,64) → LayerNorm → GeLU
Revised: (32,64) → LayerNorm → GeLU

Original: MLP
Revised: MLP

Original: [Equation]
Revised: [Equation]

Original: (64, d ls )
Revised: (64, d ls )

Original: [Equation]
Revised: [Equation]

Original: Outputs
Revised: Outputs

Original: [Equation]
Revised: [Equation]

Original: ls , l = 1 , · ·
Revised: ls , l = 1 , · ·

Original: [Equation]
Revised: [Equation]

Original: MLP l (64, d v l m )
Revised: MLP l (64, d v l m )

Original: [Equation]
Revised: [Equation]

Original: Outputs o v l m , l = 1 , · ·
Revised: Outputs o v l m , l = 1 , · ·

Original: [Equation]
Revised: [Equation]

Original: L
Revised: L

Original: MLP l (64, d t l s )
Revised: MLP l (64, d t l s )

Original: → Outputs o t l s , l = 1 , · · ·
Revised: → Outputs o t l s , l = 1 , · · ·

Original: [Equation]
Revised: [Equation]

Original: MLP l (64, d t l m )
Revised: MLP l (64, d t l m )

Original: → Outputs o t l m , l = 1 , · · · , L and is sampled from a uniform distribution and it is kept unchanged after the training starts.
Revised: → Outputs o t l m , l = 1 , · · · , L and is sampled from a uniform distribution and it is kept unchanged after the training starts.

Original: The outputs of GRUs are parallelly fed into MLP layers to get the outputs before the final pruning and sharing vectors.
Revised: The outputs of GRUs are parallelly fed into MLP layers to get the outputs before the final pruning and sharing vectors.

Original: The final s l , m v l , s t l , m t l is obtained by using the Gumbel-Sigmoid (Jang et al., 2016) technique:
Revised: The final s l , m v l , s t l , m t l is obtained by using the Gumbel-Sigmoid (Jang et al., 2016) technique:

Original: [Equation]
Revised: [Equation]

Original: 14) where sigmoid ( · ) is the sigmoid function, round ( · ) is the function that rounds its input to its nearest integer, g follows Gumbel distribution: g ∼ Gumbel (0 , 1) , b is a constant and τ is the temperature hyper-parameter.
Revised: 14) where sigmoid ( · ) is the sigmoid function, round ( · ) is the function that rounds its input to its nearest integer, g follows Gumbel distribution: g ∼ Gumbel (0 , 1) , b is a constant and τ is the temperature hyper-parameter.

Original: We set b = 3 . 

Revised: We set b = 3 .

Original: so that model structures are kept intact at the beginning.
Revised: so that model structures are kept intact at the beginning.

Original: The τ is set to 0 .
Revised: The τ is set to 0 .

Original: 4 for all experiments.
Revised: 4 for all experiments.

Original: Since round ( · ) is not differentiable, we use the straight-through gradient estimator (Bengio et al., 2013) to circumvent this problem.
Revised: Since round ( · ) is not differentiable, we use the straight-through gradient estimator (Bengio et al., 2013) to circumvent this problem.

Original: L is the total number of MSA and FFN layers, and the order of MSA and FFN layers is followed by their natural order in the transformer.
Revised: L is the total number of MSA and FFN layers, and the order of MSA and FFN layers is followed by their natural order in the transformer.

Original: Recall that the model embedding dimension is d , and d ′ is the dimension after expansion for FFN layers.
Revised: Recall that the model embedding dimension is d , and d ′ is the dimension after expansion for FFN layers.

Original: The dimension of each MLP is decided by whether it is MSA or FFN layers.
Revised: The dimension of each MLP is decided by whether it is MSA or FFN layers.

Original: For sharing operations, d ls and d t l s equal to 3 d when they are MSA layers.
Revised: For sharing operations, d ls and d t l s equal to 3 d when they are MSA layers.

Original: For pruning operations, d v l m and d t l m are equal to d , and they are shared for query weights and key weights.
Revised: For pruning operations, d v l m and d t l m are equal to d , and they are shared for query weights and key weights.

Original: For FFN layers, d ls , d v l m , d t l s and d t l m equal to d ′ .
Revised: For FFN layers, d ls , d v l m , d t l s and d t l m equal to d ′ .

Original: A. A DDITIONAL D ETAILS FOR E XPERIMENTS
Revised: A. A DDITIONAL D ETAILS FOR E XPERIMENTS

Original: [Table]
Revised: [Table]

Original: The hyper-parameters λ and γ for R and R w are set to 6 . 0 and 0 .
Revised: The hyper-parameters λ and γ for R and R w are set to 6 . 0 and 0 .

Original: 0005 for all datasets and tasks.
Revised: 0005 for all datasets and tasks.

Original: We train the hypernetwork with ADAM and a constant learning rate of 0 .
Revised: We train the hypernetwork with ADAM and a constant learning rate of 0 .

Original: 001 and weight decay 0 . 0001 .
Revised: 001 and weight decay 0 . 0001 .

Original: For D sub , we random sample 10% of samples from D .
Revised: For D sub , we random sample 10% of samples from D .

Original: We found that this hyper-parameter setting performs well for both MDETR and GLIP.
Revised: We found that this hyper-parameter setting performs well for both MDETR and GLIP.

Original: Other pre-training and fine-tuning settings are identical to the original setting.
Revised: Other pre-training and fine-tuning settings are identical to the original setting.

Original: In our method, p controls how much compressible backbone parameters should be removed.
Revised: In our method, p controls how much compressible backbone parameters should be removed.

Original: We list the detailed choice of p in Tab. S2.
Revised: We list the detailed choice of p in Tab. S2.

Original: We only need to modify p if we want to change how much weight we want to compress.
Revised: We only need to modify p if we want to change how much weight we want to compress.

Original: The results of all MDETR tasks are obtained by fine-tuning on downstream tasks.
Revised: The results of all MDETR tasks are obtained by fine-tuning on downstream tasks.

Original: Results of GLIP on COCO detection task
Revised: Results of GLIP on COCO detection task

Original: Tab. 3 are also obtained by fine-tuning.
Revised: Tab. 3 are also obtained by fine-tuning.

Original: We follow the fine-tuning setting in their original papers and codes 1 .
Revised: We follow the fine-tuning setting in their original papers and codes 1 .

Original: Let’s take GLIP pre-taining as an example.
Revised: Let’s take GLIP pre-taining as an example.

Original: We use the hyper-parameters given in https://github.com/microsoft/GLIP/blob/main/configs/pretrain/glip
Revised: We use the hyper-parameters given in https://github.com/microsoft/GLIP/blob/main/configs/pretrain/glip

Original: Swin T O365.yaml.
Revised: Swin T O365.yaml.

Original: We replaced BACKBONE, LANGUAGE BACKBONE, and DATASETS in the configuration file, and we inserted hyper-parameters for training the hypernetwork.
Revised: We replaced BACKBONE, LANGUAGE BACKBONE, and DATASETS in the configuration file, and we inserted hyper-parameters for training the hypernetwork.

Original: For Flickr-30k results in Tab. 4, we add a very small fine-tuning period ( ∼ 3 epochs) to obtain them (including the baseline).
Revised: For Flickr-30k results in Tab. 4, we add a very small fine-tuning period ( ∼ 3 epochs) to obtain them (including the baseline).

Original: For more information on the pre-training dataset, we refer the readers to the MDETR (Kamath et al., 2021) and GLIP (Li et al., 2022) papers.
Revised: For more information on the pre-training dataset, we refer the readers to the MDETR (Kamath et al., 2021) and GLIP (Li et al., 2022) papers.

Original: In addition, the implementation of Eq.
Revised: In addition, the implementation of Eq.

Original: 6 and Eq.
Revised: 6 and Eq.

Original: 7 may not be straightforward at first glance.
Revised: 7 may not be straightforward at first glance.

Original: We provide a PyTorch style pseudo code in this section to give an example for Eq. 6.
Revised: We provide a PyTorch style pseudo code in this section to give an example for Eq. 6.

Original: The implementation of Eq.
Revised: The implementation of Eq.

Original: 7 can be done similarly.
Revised: 7 can be done similarly.

Original: for i in range(num_layers): # cross_modal_shared_index_list contains indices of shared weights for all layers if cross_modal_shared_index_list[i].nelement() !
Revised: for i in range(num_layers): # cross_modal_shared_index_list contains indices of shared weights for all layers if cross_modal_shared_index_list[i].nelement() !

Original: = 

Revised: =

Original: : # shared_index contains indices of shared weights for the current layer shared_index
Revised: : # shared_index contains indices of shared weights for the current layer shared_index

Original: = cross_modal_shared_index_list[i] # apply the constraint for vision pruning vectors # vision_pruning_vector_list contains all pruning vectors.
Revised: = cross_modal_shared_index_list[i] # apply the constraint for vision pruning vectors # vision_pruning_vector_list contains all pruning vectors.

Original: vision_pruning_
Revised: vision_pruning_

Original: [Table]
Revised: [Table]

Original: = vision_
Revised: = vision_

Original: [Table]
Revised: [Table]

Original: _vector_list[i].clone() # apply the constraint defined in Eq.
Revised: _vector_list[i].clone() # apply the constraint defined in Eq.

Original: vision
Revised: vision

Original: [Table]
Revised: [Table]

Original: pruning
Revised: pruning

Original: [Table]
Revised: [Table]

Original: clone[shared
Revised: clone[shared

Original: [Table]
Revised: [Table]

Original: [i]] =
Revised: [i]] =

Original: [Table]
Revised: [Table]

Original: vector
Revised: vector

Original: [Table]
Revised: [Table]

Original: list[i] =
Revised: list[i] =

Original: vision_pruning
Revised: vision_pruning

Original: [Table]
Revised: [Table]

Original: vector
Revised: vector

Original: [Table]
Revised: [Table]

Original: clone.clone() # the same process is applied for text
Revised: clone.clone() # the same process is applied for text

Original: [Table]
Revised: [Table]

Original: A. S TUDY OF D IFFERENT S ETTINGS
Revised: A. S TUDY OF D IFFERENT S ETTINGS

Original: [Table]
Revised: [Table]

Original: To see how different choices of hyper-parameters affect our method, we adjust λ , γ , | D sub | (the number of samples of D sub ) and | B | (the number of blocks) for our method.
Revised: To see how different choices of hyper-parameters affect our method, we adjust λ , γ , | D sub | (the number of samples of D sub ) and | B | (the number of blocks) for our method.

Original: The overall pretraining cost is quite high for GLIP and MDETR.
Revised: The overall pretraining cost is quite high for GLIP and MDETR.

Original: As a result, we only use Flickr-30k (Plummer et al., 2015) as the pre-training dataset.
Revised: As a result, we only use Flickr-30k (Plummer et al., 2015) as the pre-training dataset.

Original: The experiments are conducted on GLIP with COCO object detection as the evaluation task.
Revised: The experiments are conducted on GLIP with COCO object detection as the evaluation task.

Original: All other settings are the same as Tab. 3.
Revised: All other settings are the same as Tab. 3.

Original: γ is used to control the regularization strength of R w in Obj.
Revised: γ is used to control the regularization strength of R w in Obj.

Original: From Tab.
Revised: From Tab.

Original: S3, we can see that a smaller γ will negatively affect the final results, and this is probably because weights from different layers and different modalities are not well aligned, which creates difficulty for fine-tuning.
Revised: S3, we can see that a smaller γ will negatively affect the final results, and this is probably because weights from different layers and different modalities are not well aligned, which creates difficulty for fine-tuning.

Original: This observation is consistent with the results of ‘BWSP w/o R w ’ ( γ = 0 ) in Tab. 1 and Tab. 2, which all indicates that poor alignment will be harmful to the final performance.
Revised: This observation is consistent with the results of ‘BWSP w/o R w ’ ( γ = 0 ) in Tab. 1 and Tab. 2, which all indicates that poor alignment will be harmful to the final performance.

Original: λ is used to control the regularization strength of the parameter constraint R ( P ( m, s ) , pP total ) in
Revised: λ is used to control the regularization strength of the parameter constraint R ( P ( m, s ) , pP total ) in

Original: Obj.
Revised: Obj.

Original: From Tab.
Revised: From Tab.

Original: S3, we can see that a too large λ ( λ = 10 ) is harmful to our method.
Revised: S3, we can see that a too large λ ( λ = 10 ) is harmful to our method.

Original: This is because a too large λ will ignore the pre-training loss L pre-training and only focus on reducing the number parameters.
Revised: This is because a too large λ will ignore the pre-training loss L pre-training and only focus on reducing the number parameters.

Original: As a result, it hinders the hypernetwork from finding the ideal sharing and pruning structures.
Revised: As a result, it hinders the hypernetwork from finding the ideal sharing and pruning structures.

Original: | D sub | controls the number of samples in | D sub | .
Revised: | D sub | controls the number of samples in | D sub | .

Original: From Tab.
Revised: From Tab.

Original: S3, we can see that increasing the size of | D sub | is beneficial to the final performance.
Revised: S3, we can see that increasing the size of | D sub | is beneficial to the final performance.

Original: At the same time, it increases the pre-training costs.
Revised: At the same time, it increases the pre-training costs.

Original: If | D sub | is too small, it will definitely negatively affect the results, as shown in the table.
Revised: If | D sub | is too small, it will definitely negatively affect the results, as shown in the table.

Original: | D sub | = 0 .
Revised: | D sub | = 0 .

Original: 10 | D | creates a good trade-off between pre-training costs and the final performance.
Revised: 10 | D | creates a good trade-off between pre-training costs and the final performance.

Original: | B | controls the number of blocks for cross-layer sharing.
Revised: | B | controls the number of blocks for cross-layer sharing.

Original: We tested three settings, and the results are shown in Tab. S3.
Revised: We tested three settings, and the results are shown in Tab. S3.

Original: When | B | = 2 , the final model has fewer unique base weights, which restricts the weight diversity.
Revised: When | B | = 2 , the final model has fewer unique base weights, which restricts the weight diversity.

Original: When | B | = 6 , the potential compression rate is limited.
Revised: When | B | = 6 , the potential compression rate is limited.

Original: Clearly, when | B | = 3 , we can have a good trade-off between the weight diversity and the potential compression rate for the text transformer.
Revised: Clearly, when | B | = 3 , we can have a good trade-off between the weight diversity and the potential compression rate for the text transformer.