Original: Patching open-vocabulary models by interpolating weights
Revised: Patching open-vocabulary models by interpolating weights

Original: Abstract
Revised: Abstract

Original: Open-vocabulary models like CLIP achieve high accuracy across many image classification tasks.
Revised: Open-vocabulary models like CLIP achieve high accuracy across many image classification tasks.

Original: However, there are still settings where their zero-shot performance is far from optimal.
Revised: However, there are still settings where their zero-shot performance is far from optimal.

Original: We study model patching , where the goal is to improve accuracy on specific tasks without degrading accuracy on tasks where performance is already adequate.
Revised: We study model patching , where the goal is to improve accuracy on specific tasks without degrading accuracy on tasks where performance is already adequate.

Original: Towards this goal, we introduce PAINT , a patching method that uses interpolations between the weights of a model before fine-tuning and the weights after fine-tuning on a task to be patched.
Revised: Towards this goal, we introduce PAINT , a patching method that uses interpolations between the weights of a model before fine-tuning and the weights after fine-tuning on a task to be patched.

Original: On nine tasks where zeroshot CLIP performs poorly, PAINT increases accuracy by 15 to 60 percentage points while preserving accuracy on ImageNet within one percentage point of the zero-shot model.
Revised: On nine tasks where zeroshot CLIP performs poorly, PAINT increases accuracy by 15 to 60 percentage points while preserving accuracy on ImageNet within one percentage point of the zero-shot model.

Original: PAINT also allows a single model to be patched on multiple tasks and improves with model scale.
Revised: PAINT also allows a single model to be patched on multiple tasks and improves with model scale.

Original: Furthermore, we identify cases of broad transfer , where patching on one task increases accuracy on other tasks even when the tasks have disjoint classes.
Revised: Furthermore, we identify cases of broad transfer , where patching on one task increases accuracy on other tasks even when the tasks have disjoint classes.

Original: Finally, we investigate applications beyond common benchmarks such as counting or reducing the impact of typographic attacks on
Revised: Finally, we investigate applications beyond common benchmarks such as counting or reducing the impact of typographic attacks on

Original: CLIP.
Revised: CLIP.

Original: Our findings demonstrate that it is possible to expand the set of tasks on which open-vocabulary models achieve high accuracy without re-training them from scratch.
Revised: Our findings demonstrate that it is possible to expand the set of tasks on which open-vocabulary models achieve high accuracy without re-training them from scratch.

Original:  Introduction
Revised:  Introduction

Original: Open-vocabulary models are characterized by their ability to perform any image classification task based on text descriptions of the classes [56].
Revised: Open-vocabulary models are characterized by their ability to perform any image classification task based on text descriptions of the classes [56].

Original: Thanks to advances in large-scale pre-training, recent examples of open-vocabulary models such as CLIP and BASIC have reached parity with or surpassed important task-specific baselines, even when the open-vocabulary models are not fine-tuned on task-specific data (i.e., in a zero-shot setting)
Revised: Thanks to advances in large-scale pre-training, recent examples of open-vocabulary models such as CLIP and BASIC have reached parity with or surpassed important task-specific baselines, even when the open-vocabulary models are not fine-tuned on task-specific data (i e., in a zero-shot setting)

Original: For instance, the largest CLIP model from Radford et al.
Revised: For instance, the largest CLIP model from Radford et al.

Original: [57] used in a zero-shot setting matches the ImageNet accuracy of a
Revised: [57] used in a zero-shot setting matches the ImageNet accuracy of a

Original: ResNet-50 trained on 1.2 million ImageNet images [14, 24].
Revised: ResNet-50 trained on 1.2 million ImageNet images [14, 24].

Original: Nevertheless, current open-vocabulary models still face challenges.
Revised: Nevertheless, current open-vocabulary models still face challenges.

Original: The same CLIP model that matches a ResNet-50 on ImageNet has lower MNIST accuracy than simple logistic regression in pixel space [57].
Revised: The same CLIP model that matches a ResNet-50 on ImageNet has lower MNIST accuracy than simple logistic regression in pixel space [57].

Original: Moreover, even when zero-shot models achieve good performance, they are usually still worse than models trained or fine-tuned on specific downstream tasks.
Revised: Moreover, even when zero-shot models achieve good performance, they are usually still worse than models trained or fine-tuned on specific downstream tasks.

Original: [Table] To address these issues several authors have proposed methods for adapting zero-shot models to a task of interest using labeled data [82, 91, 21, 89, 37, 73].
Revised: To address these issues, several authors have proposed methods for adapting zero-shot models to a task of interest using labeled data [82, 91, 21, 89, 37, 73].

Original: A common practice is to fine-tune the zero-shot model on the task of interest [82, 56].
Revised: A common practice is to fine-tune the zero-shot model on the task of interest [82, 56].

Original: However, fine-tuned models can suffer from catastrophic forgetting [48, 76, 20, 33] performing poorly on tasks where the zero-shot model initially performed well [2, 82, 56].
Revised: However, fine-tuned models can suffer from catastrophic forgetting [48, 76, 20, 33], performing poorly on tasks where the zero-shot model initially performed well [2, 82, 56].

Original: Additionally, fine-tuning typically produces a task-specific classification head, sacrificing the flexible text-based API that makes open-vocabulary models so appealing.
Revised: Additionally, fine-tuning typically produces a task-specific classification head, sacrificing the flexible text-based API that makes open-vocabulary models so appealing.

Original: Whereas an open-vocabulary model can perform any classification task in a zero-shot fashion, a fine-tuned model with a task-specific head can only process the specific task that it was fine-tuned on.
Revised: Whereas an open-vocabulary model can perform any classification task in a zero-shot fashion, a fine-tuned model with a task-specific head can only process the specific task that it was fine-tuned on.

Original: This specialization can prevent knowledge obtained by fine-tuning on one task from transferring to other related tasks with different classes.
Revised: This specialization can prevent knowledge obtained by fine-tuning on one task from transferring to other related tasks with different classes.

Original: Another approach to adapting zero-shot models would be to add data from the downstream task to the pre-training dataset and train a new open-vocabulary model from scratch.
Revised: Another approach to adapting zero-shot models would be to add data from the downstream task to the pre-training dataset and train a new open-vocabulary model from scratch.

Original: The resulting model could still perform any classification task, and zero-shot performance may improve on related tasks.
Revised: The resulting model could still perform any classification task and zero-shot performance may improve on related tasks.

Original: However, training large image-text models from scratch can require hundreds of thousands of GPU hours [57, 56, 86], which makes this approach practically infeasible in most settings.
Revised: However, training large image-text models from scratch can require hundreds of thousands of GPU hours [57, 56, 86] which makes this approach practically infeasible in most settings.

Original: In this paper, we study patching open-vocabulary models, where the goal is to increase accuracy on new target tasks while maintaining the flexibility of the model and its accuracy on other tasks.
Revised: In this paper we study patching open-vocabulary models, where the goal is to increase accuracy on new target tasks while maintaining the flexibility of the model and its accuracy on other tasks.

Original: Patching aims to combine the benefits of fine-tuning and re-training from scratch: improved performance on the task of interest, maintaining the flexibility of an open vocabulary, transfer between tasks, and fast adaptation time.
Revised: Patching aims to combine the benefits of fine-tuning and re-training from scratch: improved performance on the task of interest, maintaining the flexibility of an open vocabulary, transfer between tasks and fast adaptation time.

Original: Motivated by these goals, we extend existing fine-tuning techniques [82] to open-vocabulary settings, where the class space is not fixed.
Revised: Motivated by these goals, we extend existing fine-tuning techniques [82] to open-vocabulary settings, where the class space is not fixed.

Original: We introduce Patching with
Revised: We introduce Patching with

Original: Interpolation ( PAINT ), a simple, two-step procedure for patching models: first, fine-tune the model on the patching task without introducing any task-specific parameters; then, linearly interpolate between the weights of the model before and after fine-tuning.
Revised: Interpolation ( PAINT ), a simple, two-step procedure for patching models: first, fine-tune the model on the patching task without introducing any task-specific parameters; then, linearly interpolate between the weights of the model before and after fine-tuning.

Original: Linearly interpolating neural network weights [52, 19, 54] has been previously used to improve accuracy on a single task [28, 81] or robustness to distribution shift [82].
Revised: Linearly interpolating neural network weights [52, 19, 54] has been previously used to improve accuracy on a single task [28, 81] or robustness to distribution shift [82].

Original: Indeed, averaging network weights has been explored in continual learning contexts, although for closed-vocabulary models [40].
Revised: Indeed, averaging network weights has been explored in continual learning contexts, although for closed-vocabulary models [40].

Original: With PAINT , accuracy can improve on new tasks without degrading accuracy on unrelated tasks, as illustrated in Figure 1.
Revised: With PAINT , accuracy can improve on new tasks without degrading accuracy on unrelated tasks, as illustrated in Figure 1.

Original: For instance, applying PAINT to a CLIP ViT-L/14 [57] independently on nine image classification tasks [35, 11, 25, 71, 22, 39, 7, 84, 53] improves accuracy by 15 to 60 percentage points compared to the unpatched model, while accuracy on ImageNet [14] decreases by less than one percentage point.
Revised: For instance, applying PAINT to a CLIP ViT-L/14 [57] independently on nine image classification tasks [35, 11, 25, 71, 22, 39, 7, 84, 53] improves accuracy by 15 to 60 percentage points compared to the unpatched model, while accuracy on ImageNet [14] decreases by less than one percentage point.

Original: We also observe a promising trend: patching becomes more effective with model scale (Section 4.1).
Revised: We also observe a promising trend: patching becomes more effective with model scale (Section 4.1).

Original: Beyond single tasks, we show that models can be patched on multiple tasks (Section 5).
Revised: Beyond single tasks, we show that models can be patched on multiple tasks (Section 5).

Original: When patching on nine image classification tasks simultaneously, a single CLIP ViT-L/14 model is competitive with using one specialized model for each task—the average accuracy difference is less than 0.
Revised: When patching on nine image classification tasks simultaneously, a single CLIP ViT-L/14 model is competitive with using one specialized model for each task—the average accuracy difference is less than 0.

Original: percentage points.
Revised: percentage points.

Original: Moreover, PAINT enables broad transfer (Section 6): accuracy on related tasks can increase, even when the class space changes.
Revised: Moreover, PAINT enables broad transfer (Section 6): accuracy on related tasks can increase, even when the class space changes.

Original: For instance, we partition EuroSAT [25], a satellite image dataset, into two halves with disjoint labels.
Revised: For instance, we partition EuroSAT [25], a satellite image dataset, into two halves with disjoint labels.

Original: Patching a ViT-L/14 model on the first half improves accuracy on the second half by 7.3 percentage points, even though the classes are unseen during patching.
Revised: Patching a ViT-L/14 model on the first half improves accuracy on the second half by 7.3 percentage points, even though the classes are unseen during patching.

Original: Finally, we investigate PAINT on case studies including typographic attacks [23], counting [32], and visual question answering [4] (Section 7).
Revised: Finally, we investigate PAINT on case studies including typographic attacks [23], counting [32], and visual question answering [4] (Section 7).

Original: For instance, applying PAINT using synthetic typographic attacks leads to a model that is less susceptible to typographic attacks in the real world, improving its accuracy by 41 percentage points.
Revised: For instance, applying PAINT using synthetic typographic attacks leads to a model that is less susceptible to typographic attacks in the real world, improving its accuracy by 41 percentage points.

Original: In summary:
Revised: In summary:

Original: • Even the best pre-trained models are not perfect.
Revised: • Even the best pre-trained models are not perfect.

Original: We introduce PAINT , a method designed to improve accuracy on new tasks without harming accuracy elsewhere.
Revised: We introduce PAINT , a method designed to improve accuracy on new tasks without harming accuracy elsewhere.

Original: • PAINT incurs no extra computational cost compared to standard fine-tuning, neither during fine-tuning itself nor at inference time.
Revised: • PAINT incurs no extra computational cost compared to standard fine-tuning, neither during fine-tuning itself nor at inference time.

Original: • PAINT can also be applied with multiple tasks, providing a single model that is competitive with many specialized models.
Revised: • PAINT can also be applied with multiple tasks, providing a single model that is competitive with many specialized models.

Original: • Applying PAINT with one task can improve accuracy on a related task, even when they do not share the same classes.
Revised: • Applying PAINT with one task can improve accuracy on a related task, even when they do not share the same classes.

Original: • PAINT improves with model scale, indicating a promising trend for future models.
Revised: • PAINT improves with model scale, indicating a promising trend for future models.

Original:  Patching with interpolation ( PAINT )
Revised:  Patching with interpolation ( PAINT )

Original: This section details our method for patching models on a single and multiple tasks.
Revised: This section details our method for patching models on a single and multiple tasks.

Original: Patching on a single task.
Revised: Patching on a single task.

Original: Given an open-vocabulary model with weights θ zs and a patching task
Revised: Given an open-vocabulary model with weights θ zs and a patching task

Original: D patch , our goal is to produce a new model θ patch which achieves high accuracy on D patch without decreasing model performance on tasks where accuracy is already acceptable.
Revised: D patch , our goal is to produce a new model θ patch which achieves high accuracy on D patch without decreasing model performance on tasks where accuracy is already acceptable.

Original: We let D supp denote a representative supported task where model performance is adequate, and later show that the method is stable under different choices of D supp (Section 4.2).
Revised: We let D supp denote a representative supported task where model performance is adequate, and later show that the method is stable under different choices of D supp (Section 4.2).

Original: The two-step procedure we explore for producing θ patch is given below.
Revised: The two-step procedure we explore for producing θ patch is given below.

Original: Step 1.
Revised: Step 1.

Original: Fine-tune θ zs on training data from D patch to produce a model with weights θ ft .
Revised: Fine-tune θ zs on training data from D patch to produce a model with weights θ ft .

Original: Step 2.
Revised: Step 2.

Original: For mixing coefficient α ∈ [0 , 1] , linearly interpolate between θ zs and θ ft to produce θ patch =
Revised: For mixing coefficient α ∈ [0 , 1] , linearly interpolate between θ zs and θ ft to produce θ patch =

Original: 1 − α ) · θ zs + α · θ ft .
Revised: (1 − α ) · θ zs + α · θ ft .

Original: The mixing coefficient is determined via held-out validation sets for D supp and D patch .
Revised: The mixing coefficient is determined via held-out validation sets for D supp and D patch .

Original: We refer to the resulting model as θ patch .
Revised: We refer to the resulting model as θ patch .

Original: In our experiments, we do not introduce any additional task-specific parameters when fine-tuning, as discussed in Section 3 and Appendices B and C.
Revised: In our experiments, we do not introduce any additional task-specific parameters when fine-tuning, as discussed in Section 3 and Appendices B and C.

Original: Patching on a multiple tasks.
Revised: Patching on a multiple tasks.

Original: In practice, we often want to improve model accuracy on multiple patching tasks D (1) patch , ..., D ( k ) patch , which can be accomplished with straightforward modifications to the procedure above.
Revised: In practice, we often want to improve model accuracy on multiple patching tasks D (1) patch , ..., D ( k ) patch , which can be accomplished with straightforward modifications to the procedure above.

Original: We explore three alternatives and examine their relative trade-offs in Section 5:
Revised: We explore three alternatives and examine their relative trade-offs in Section 5:

Original: • Joint patching , where we merge all the patching tasks D ( i ) patch into a single task D patch before running the patching procedure; • Sequential patching , where we iteratively repeat the patching procedure above on each new task
Revised: • Joint patching , where we merge all the patching tasks D ( i ) patch into a single task D patch before running the patching procedure; • Sequential patching , where we iteratively repeat the patching procedure above on each new task

Original: D ( i ) patch and let θ zs ← θ patch after each completed iteration; • Parallel patching , where we apply the first step on each task in parallel to produce fine-tuned models with weights θ (1) ft , ..., θ ( k ) ft .
Revised: D ( i ) patch and let θ zs ← θ patch after each completed iteration; • Parallel patching , where we apply the first step on each task in parallel to produce fine-tuned models with weights θ (1) ft , ..., θ ( k ) ft .

Original: Then, we search for mixing coefficients α i to produce 
Revised: Then, we search for mixing coefficients α i to produce θ patch =

Original: [Equation]{+(1 − (cid:80) ki =1 α i ) · θ zs + (cid:80) ki =
 α i · θ ( i ) ft .+}
Revised: {+(1 − (cid:80) ki =1 α i ) · θ zs + (cid:80) ki =
 α i · θ ( i ) ft .+}

Original: For joint and parallel patching we assume access to held-out validation sets for all tasks, while in sequential patching we only assume access to held-out validation sets from the tasks seen so far.
Revised: For joint and parallel patching we assume access to held-out validation sets for all tasks, while in sequential patching we only assume access to held-out validation sets from the tasks seen so far.

Original: Unless mentioned otherwise, we pick the mixing coefficient α that optimizes average accuracy on the held-out validation sets from the supported and patching tasks.
Revised: Unless mentioned otherwise, we pick the mixing coefficient α that optimizes average accuracy on the held-out validation sets from the supported and patching tasks.

Original:  Experimental setup
Revised:  Experimental setup

Original: Tasks.
Revised: Tasks.

Original: We consider a diverse set of image classification tasks from Radford et al.
Revised: We consider a diverse set of image classification tasks from Radford et al.

Original: In most experiments, we use ImageNet [14] as a representative supported task, although we explore other supported tasks in Section 4.2.
Revised: In most experiments, we use ImageNet [14] as a representative supported task, although we explore other supported tasks in Section 4.2.

Original: We categorize tasks into patching tasks or supported tasks based on the accuracy difference between the zero-shot model and a model specialized to the task.
Revised: We categorize tasks into patching tasks or supported tasks based on the accuracy difference between the zero-shot model and a model specialized to the task.

Original: A large accuracy difference indicates that the task is a relevant target for patching because the zero-shot model is still far from optimal.
Revised: A large accuracy difference indicates that the task is a relevant target for patching because the zero-shot model is still far from optimal.

Original: Specifically, we consider a subset tasks from Radford et al.
Revised: Specifically, we consider a subset tasks from Radford et al.

Original: [57], categorizing tasks where the linear probes outperform the zero-shot model by over 10 percentage points as patching tasks: Cars [35], DTD [11], EuroSAT [25], GTSRB [71], KITTI [22], MNIST [39], RESISC45 [7], SUN397 [84], and SVHN [53].
Revised: [57], categorizing tasks where the linear probes outperform the zero-shot model by over 10 percentage points as patching tasks: Cars [35], DTD [11], EuroSAT [25], GTSRB [71], KITTI [22], MNIST [39], RESISC45 [7], SUN397 [84], and SVHN [53].

Original: We use the remaining tasks as supported tasks:
Revised: We use the remaining tasks as supported tasks:

Original: CIFAR10 [36], CIFAR100 [36], Food101 [5], ImageNet [14], and STL10 [12].
Revised: CIFAR10 [36], CIFAR100 [36], Food101 [5], ImageNet [14], and STL10 [12].

Original: We investigate additional patching tasks as case studies in Section 7 and provide further details in Appendix A.
Revised: We investigate additional patching tasks as case studies in Section 7 and provide further details in Appendix A.

Original: Models.
Revised: Models.

Original: We primarily use CLIP [57] pre-trained vision transformer (ViT) models [15].
Revised: We primarily use CLIP [57] pre-trained vision transformer (ViT) models [15].

Original: Unless otherwise mentioned our experiments are with the ViT-L/14 model, while Section 4.2 studies ResNets [24].
Revised: Unless otherwise mentioned our experiments are with the ViT-L/14 model, while Section 4.2 studies ResNets [24].

Original: Fine-tuning on patching tasks.
Revised: Fine-tuning on patching tasks.

Original: Unless otherwise mentioned, we fine-tune with a batch size offor 2000 iterations using learning rate 1e-5 with 200 warm-up steps with a cosine annealing learning rate schedule and the AdamW optimizer [43, 55] (weight decay 0.1).
Revised: Unless otherwise mentioned, we fine-tune with a batch size offor 2000 iterations using learning rate 1e-5 with 200 warm-up steps with a cosine annealing learning rate schedule and the AdamW optimizer [43, 55] (weight decay 0.1).

Original: When fine-tuning, we use the frozen final classification layer output by CLIP’s text tower so that we do not introduce additional learnable parameters.
Revised: When fine-tuning, we use the frozen final classification layer output by CLIP’s text tower so that we do not introduce additional learnable parameters.

Original: This design decision keeps the model open-vocabulary and does not harm accuracy, as discussed in in Appendices B and C.
Revised: This design decision keeps the model open-vocabulary and does not harm accuracy, as discussed in in Appendices B and C.

Original: Evaluation.
Revised: Evaluation.

Original: We use accuracy as the evaluation metric unless otherwise stated.
Revised: We use accuracy as the evaluation metric unless otherwise stated.

Original: We refer to the average of the mean accuracy on the patching tasks and the mean accuracy on the supported tasks as combined accuracy .
Revised: We refer to the average of the mean accuracy on the patching tasks and the mean accuracy on the supported tasks as combined accuracy .

Original:  Patching models on a single new task
Revised:  Patching models on a single new task

Original: As shown in Figure 1, when patching a model on a single task, we interpolate the weights of the zero-shot and fine-tuned model, producing a model that achieves high accuracy on both the patching task and the supported task.
Revised: As shown in Figure 1, when patching a model on a single task, we interpolate the weights of the zero-shot and fine-tuned model, producing a model that achieves high accuracy on both the patching task and the supported task.

Original: On the nine tasks, PAINT improves the accuracy of ViT-L/14 by 15 topercentage points, while accuracy on ImageNet decreases by less than one percentage point.
Revised: On the nine tasks, PAINT improves the accuracy of ViT-L/14 by 15 topercentage points, while accuracy on ImageNet decreases by less than one percentage point.

Original: PAINT also allows practitioners to control the accuracy trade-off on the patching and supported tasks without re-training a new model, by varying the mixing coefficient α .
Revised: PAINT also allows practitioners to control the accuracy trade-off on the patching and supported tasks without re-training a new model, by varying the mixing coefficient α .

Original: 4.1 The effect of scale
Revised: 4.1 The effect of scale

Original: We consistently observe that PAINT is more effective for larger models.
Revised: We consistently observe that PAINT is more effective for larger models.

Original: Our findings are aligned with those of Ramasesh et al.
Revised: Our findings are aligned with those of Ramasesh et al.

Original: [59], who observed that larger models are less susceptible to catastrophic forgetting.
Revised: [59], who observed that larger models are less susceptible to catastrophic forgetting.

Original: This section formalizes and provides insights for these observations.
Revised: This section formalizes and provides insights for these observations.

Original: Measuring the effectiveness of patching.
Revised: Measuring the effectiveness of patching.

Original: We measure the effectiveness of patching via the accuracy difference between the single patched model and two specialized models with the same architecture and initialization.
Revised: We measure the effectiveness of patching via the accuracy difference between the single patched model and two specialized models with the same architecture and initialization.

Original: For both the supported task and patching task, we take specialized models that maximize performance on the task, considering the set of all interpolations between the zero-shot and fine-tuned models.
Revised: For both the supported task and patching task, we take specialized models that maximize performance on the task, considering the set of all interpolations between the zero-shot and fine-tuned models.

Original: We refer to this measure as accuracy distance to optimal .
Revised: We refer to this measure as accuracy distance to optimal .

Original: Formally, accuracy distance to optimal is given by
Revised: Formally, accuracy distance to optimal is given by

Original: [Equation]
Revised: [Equation]

Original: where Acc ( θ, D ) represents the accuracy of model θ on task D .
Revised: where Acc ( θ, D ) represents the accuracy of model θ on task D .

Original: In Figure 2 (left), we show that accuracy distance to optimal decreases with scale, indicating that patching becomes more effective for larger models.
Revised: In Figure 2 (left), we show that accuracy distance to optimal decreases with scale, indicating that patching becomes more effective for larger models.

Original: Model similarity.
Revised: Model similarity.

Original: Fine-tuning modifies overparameterized models less [9], which provides insights on why larger models are easier to patch: less movement is required to fit new data.
Revised: Fine-tuning modifies overparameterized models less [9], which provides insights on why larger models are easier to patch: less movement is required to fit new data.

Original: We demonstrate this by evaluating representational similarity using Centered Kernel Alignment (CKA) [34] (see
Revised: We demonstrate this by evaluating representational similarity using Centered Kernel Alignment (CKA) [34] (see

Original: Appendix D for details).
Revised: Appendix D for details).

Original: As shown in Figure 2 (center), the representations of the unpatched and fine-tuned models become more similar as models grow larger, indicated by larger CKA values.
Revised: As shown in Figure 2 (center), the representations of the unpatched and fine-tuned models become more similar as models grow larger, indicated by larger CKA values.

Original: Moreover, Figure 2 (right) shows that the cosine similarity between the weights of the unpatched and fine-tuned models, cos( θ zs , θ ft ) = ⟨ θ zs , θ ft ⟩ / ( ||
Revised: Moreover, Figure 2 (right) shows that the cosine similarity between the weights of the unpatched and fine-tuned models, cos( θ zs , θ [Equation]

Original: θ zs
Revised: ) =

Original: ||
Revised: [Equation] [Equation]

Original: || θ ft || ) , increases with scale.
Revised: , increases with scale.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: accuracy trade-offs can be recovered by linearly interpolating weights.
Revised: accuracy trade-offs can be recovered by linearly interpolating weights.

Original: Interpolating the unpatched and fine-tuned models recovers the accuracy trade-off of early stopping, regularization towards the initialization, and changes in hyperparameters.
Revised: Interpolating the unpatched and fine-tuned models recovers the accuracy trade-off of early stopping, regularization towards the initialization, and changes in hyperparameters.

Original: Additional details and comparisons can be found in Appendix E.
Revised: Additional details and comparisons can be found in Appendix E.

Original: 4.2 Baselines and ablations
Revised: 4.2 Baselines and ablations

Original: Baselines.
Revised: Baselines.

Original: There are many alternatives which enable a trade-off between accuracy on the supported and patching tasks.
Revised: There are many alternatives which enable a trade-off between accuracy on the supported and patching tasks.

Original: These methods include early stopping during fine-tuning, applying a regularization term which penalizes movement from initialization, or training with different hyperparameters including a smaller learning rate.
Revised: These methods include early stopping during fine-tuning, applying a regularization term which penalizes movement from initialization, or training with different hyperparameters including a smaller learning rate.

Original: Unlike interpolation, these methods do not enable navigating the accuracy trade-off without fine-tuning the model again many times.
Revised: Unlike interpolation, these methods do not enable navigating the accuracy trade-off without fine-tuning the model again many times.

Original: Moreover, Figure 3 demonstrates that the accuracy trade-off frontier for early stopping, regularization, or varying hyperparameters can be recovered by interpolating weights with different mixing coefficients.
Revised: Moreover, Figure 3 demonstrates that the accuracy trade-off frontier for early stopping, regularization, or varying hyperparameters can be recovered by interpolating weights with different mixing coefficients.

Original: Appendix E provides additional baselines and discussion, including EMA [74], EWC [33], LwF [41], re-training a model with data from the patching task, and mixing the pre-training and fine-tuning objectives.
Revised: Appendix E provides additional baselines and discussion, including EMA [74], EWC [33], LwF [41], re-training a model with data from the patching task, and mixing the pre-training and fine-tuning objectives.

Original: Additional supported tasks.
Revised: Additional supported tasks.

Original: In Figure 1, we use ImageNet as a representative supported task.
Revised: In Figure 1, we use ImageNet as a representative supported task.

Original: This section demonstrates that PAINT is stable under different choices of the supported task.
Revised: This section demonstrates that PAINT is stable under different choices of the supported task.

Original: Instead of ImageNet, we use CIFAR10, CIFAR100, Food101 and STL10.
Revised: Instead of ImageNet, we use CIFAR10, CIFAR100, Food101 and STL10.

Original: Figure 4 displays representative results, where performance is averaged over the nine patching tasks (see Appendix F for additional results).
Revised: Figure 4 displays representative results, where performance is averaged over the nine patching tasks (see Appendix F for additional results).

Original: We observe consistent results across supported tasks, and that the optimal mixing coefficients are stable across different choices of supported tasks (Figure 4, right).
Revised: We observe consistent results across supported tasks, and that the optimal mixing coefficients are stable across different choices of supported tasks (Figure 4, right).

Original: Additional models.
Revised: Additional models.

Original: In addition to the CLIP ViTs used in the majority of our experiments, we study four ResNet models [24] from Radford et al.
Revised: In addition to the CLIP ViTs used in the majority of our experiments, we study four ResNet models [24] from Radford et al.

Original: [57] in Appendix G.
Revised: [57] in Appendix G.

Original: We find that patching is less effective for ResNets compared to ViTs of similar size, which corroborates the findings of Ramasesh et al.
Revised: We find that patching is less effective for ResNets compared to ViTs of similar size, which corroborates the findings of Ramasesh et al.

Original: [59] that ResNets are generally more susceptible to catastrophic forgetting.
Revised: [59] that ResNets are generally more susceptible to catastrophic forgetting.

Original: However, similarly to ViTs, we still observe improvements with scale.
Revised: However, similarly to ViTs, we still observe improvements with scale.

Original: Finally, we show that patching is also effective for closed-vocabulary models in Appendix H. 
Revised: Finally, we show that patching is also effective for closed-vocabulary models in Appendix H. [Figure] cc

Original: [Figure]
Revised: [Figure]

Original: c y
Revised: c y

Original: [Figure]
Revised: [Figure]

Original:  Patching models on multiple tasks
Revised:  Patching models on multiple tasks

Original: This section details experimental results for patching on multiple datasets.
Revised: This section details experimental results for patching on multiple datasets.

Original: Recall from Sectionthat there are various strategies for extending PAINT to multiple datasets, which we briefly revisit.
Revised: Recall from Sectionthat there are various strategies for extending PAINT to multiple datasets, which we briefly revisit.

Original: For joint patching we merge all the datasets into a single fine-tuning task and apply our patching procedure as before.
Revised: For joint patching we merge all the datasets into a single fine-tuning task and apply our patching procedure as before.

Original: For sequential patching we iteratively perform our procedure once per task, using the patched model at each step as the initialization for the next step.
Revised: For sequential patching we iteratively perform our procedure once per task, using the patched model at each step as the initialization for the next step.

Original: 4 We also explore parallel patching, for which we have an unpatched model θ zs and independently fine-tune on each of the tasks in parallel.
Revised: 3 We also explore parallel patching, for which we have an unpatched model θ zs and independently fine-tune on each of the tasks in parallel.

Original: We then search for mixing coefficients to combine the resulting models.
Revised: We then search for mixing coefficients to combine the resulting models.

Original: For tasks 1 , ..., k , let θ (1) ft , ..., θ ( k ) ft denote the fine-tuned models for each task.
Revised: For tasks 1 , ..., k , let θ (1) ft , ..., θ ( k ) ft denote the fine-tuned models for each task.

Original: Since it is impractical to exhaustively search over each α
Revised: Since it is impractical to exhaustively search over each α

Original: i , we instead search over a one-dimensional scalar α ∈ [0 , 1] , which interpolates between θ zs and the average of all fine-tuned solutions 1 k (cid:80) ki
Revised: i , we instead search over a one-dimensional scalar α ∈ [0 , 1] , which interpolates between θ zs and the average of all fine-tuned solutions 1 k (cid:80) ki

Original: =1 θ ( i ) ft . 5 Appendix J provides further experimental details.
Revised: =1 θ ( i ) ft . 4 Appendix J provides further experimental details.

Original: These methods have various trade-offs and may be applicable for different scenarios.
Revised: These methods have various trade-offs and may be applicable for different scenarios.

Original: Joint patching is only possible when data from all tasks you wish to patch is available.
Revised: Joint patching is only possible when data from all tasks you wish to patch is available.

Original: On the other hand, sequential patching is appropriate when the tasks are observed one after another.
Revised: On the other hand, sequential patching is appropriate when the tasks are observed one after another.

Original: Finally, parallel patching can leverage distributed hardware.
Revised: Finally, parallel patching can leverage distributed hardware.

Original: Figure 5 displays experimental results when patching on all nine tasks from Section 4.
Revised: Figure 5 displays experimental results when patching on all nine tasks from Section 4.

Original: We observe that joint patching is the best-performing method on average.
Revised: We observe that joint patching is the best-performing method on average.

Original: This is perhaps unsurprising since joint patching has simultaneous access to all patching datasets, unlike other patching strategies.
Revised: This is perhaps unsurprising since joint patching has simultaneous access to all patching datasets, unlike other patching strategies.

Original: Nevertheless, it is still interesting that for ViT-L/14, joint patching yields a single model with only 0.
Revised: Nevertheless, it is still interesting that for ViT-L/14, joint patching yields a single model with only 0.

Original: percentage points worse combined accuracy than using multiple specialized models.
Revised: percentage points worse combined accuracy than using multiple specialized models.

Original: 6 Joint patching also achieves a 15.8 percentage points improvement over the unpatched model.
Revised: 5 Joint patching also achieves a 15.8 percentage points improvement over the unpatched model.

Original: Moreover, patching a
Revised: Moreover, patching a

Original: ViT-B/32 model with the joint strategy achieves a combined accuracy 6.1 percentage points higher than a ViT-L/14 unpatched model, which requires 12x more GMACs.
Revised: ViT-B/32 model with the joint strategy achieves a combined accuracy 6.1 percentage points higher than a ViT-L/14 unpatched model, which requires 12x more GMACs.

Original: The accuracy of sequential patching approaches that of joint patching, especially for larger models.
Revised: The accuracy of sequential patching approaches that of joint patching, especially for larger models.

Original: Note that, unlike in joint patching, forgetting can compound since the patching procedure is applied multiple times in sequence.
Revised: Note that, unlike in joint patching, forgetting can compound since the patching procedure is applied multiple times in sequence.

Original: In sequential patching, weight interpolations do not completely eradicate forgetting, but greatly mitigate it.
Revised: In sequential patching, weight interpolations do not completely eradicate forgetting, but greatly mitigate it.

Original: This is most noticeable for smaller models: sequentially fine-tuning a ViT-B/32 without interpolation reduces the combined accuracy by 4.6 percentage points compared to the unpatched model, as shown in Appendix J. This is compared to a combined accuracy increase of 11 percentage points when using sequential patching.
Revised: This is most noticeable for smaller models: sequentially fine-tuning a ViT-B/32 without interpolation reduces the combined accuracy by 4.6 percentage points compared to the unpatched model, as shown in Appendix J. This is compared to a combined accuracy increase of 11 percentage points when using sequential patching.

Original: Additional results, including experiments on
Revised: Additional results, including experiments on

Original: SplitCIFAR [61], can be found in Appendix J.
Revised: SplitCIFAR [61], can be found in Appendix J.

Original: Finally, parallel patching underperforms other patching strategies.
Revised: Finally, parallel patching underperforms other patching strategies.

Original: Like sequential patching, parallel patching is in the challenging setting where data from all patching tasks is not available simultaneously.
Revised: Like sequential patching, parallel patching is in the challenging setting where data from all patching tasks is not available simultaneously.

Original: ImageNet
Revised: ImageNet

Original: Cars
Revised: Cars

Original: DTD
Revised: DTD

Original: EuroSAT
Revised: EuroSAT

Original: GTSRB
Revised: GTSRB

Original: KITTI
Revised: KITTI

Original: MNIST
Revised: MNIST

Original: RESISC
Revised: RESISC

Original: SUN
Revised: SUN

Original: SVHN 0.2 0.
Revised: SVHN 0.2 0.

Original: Accuracies for ViT-B/
Revised: Accuracies for ViT-B/

Original: ImageNet
Revised: ImageNet

Original: Cars
Revised: Cars

Original: DTD
Revised: DTD

Original: EuroSAT
Revised: EuroSAT

Original: GTSRB
Revised: GTSRB

Original: KITTI
Revised: KITTI

Original: MNIST
Revised: MNIST

Original: RESISC
Revised: RESISC

Original: SUN
Revised: SUN

Original: SVHN 0.2 0.
Revised: SVHN 0.2 0.

Original: Accuracies for ViT-L/
Revised: Accuracies for ViT-L/

Original: Sequential patching
Revised: Sequential patching

Original: [Figure]
Revised: [Figure]

Original: Parallel patching
Revised: Parallel patching

Original: Unpatched model
Revised: Unpatched model

Original: [Figure]
Revised: [Figure]

Original: models
Revised: models

Original: Cars
Revised: Cars

Original: DTD EuroSAT GTSRB KITTI MNIST RESISC45 SUN397 SVHN
Revised: DTD EuroSAT GTSRB KITTI MNIST RESISC45 SUN397 SVHN

Original: Unpatched accuracy 86.
Revised: Unpatched accuracy 86.

Original: Patched accuracy 87.
Revised: Patched accuracy 87.

Original: 88.9 (+0.8) (+1.2) (+7.3) (+19.4) (+17.0) (+8.7) (+0.8) (+2.4) (+16.1)
Revised: 88.9 (+0.8) (+1.2) (+7.3) (+19.4) (+17.0) (+8.7) (+0.8) (+2.4) (+16.1)

Original: Moreover, unlike in joint or sequential patching, no model is optimized on data from all patching tasks.
Revised: Moreover, unlike in joint or sequential patching, no model is optimized on data from all patching tasks.

Original: Using a black box optimization algorithm for finding the mixing coefficients did not yield large improvements over using the same mixing coefficient for all models.
Revised: Using a black box optimization algorithm for finding the mixing coefficients did not yield large improvements over using the same mixing coefficient for all models.

Original: However, it is possible that more sophisticated search methods could yield better results.
Revised: However, it is possible that more sophisticated search methods could yield better results.

Original: In Appendix J, we present additional experiments for a subset of the tasks where exhaustively searching the space of mixing coefficients is tractable, finding headroom for improvement in most cases.
Revised: In Appendix J, we present additional experiments for a subset of the tasks where exhaustively searching the space of mixing coefficients is tractable, finding headroom for improvement in most cases.

Original:  Broad transfer
Revised:  Broad transfer

Original: An alternative to our patching approach is to introduce parameters which are specific to each new task.
Revised: An alternative to our patching approach is to introduce parameters which are specific to each new task.

Original: By contrast, PAINT always maintains a single model.
Revised: By contrast, PAINT always maintains a single model.

Original: This section describes an additional advantage of the single model approach: patching the model on task A can improve accuracy on task
Revised: This section describes an additional advantage of the single model approach: patching the model on task A can improve accuracy on task

Original: B , even when task A and B do not share the same classes.
Revised: B , even when task A and B do not share the same classes.

Original: We refer to this phenomenon as broad transfer .
Revised: We refer to this phenomenon as broad transfer .

Original: Note that we are able to study this phenomenon because the single patched model remains open-vocabulary throughout the patching procedure.
Revised: Note that we are able to study this phenomenon because the single patched model remains open-vocabulary throughout the patching procedure.

Original: This is a key advantage of PAINT compared to maintaining a collection of task-specific models.
Revised: This is a key advantage of PAINT compared to maintaining a collection of task-specific models.

Original: We now describe two experiments to measure the effects on a task B when patching the model on a task A .
Revised: We now describe two experiments to measure the effects on a task B when patching the model on a task A .

Original: First, we explore broad transfer by randomly partitioning datasets into disjoint sets with no class overlap.
Revised: First, we explore broad transfer by randomly partitioning datasets into disjoint sets with no class overlap.

Original: For a dataset D we partition the class space Y into two disjoint sets of roughly equal size Y A and Y B .
Revised: For a dataset D we partition the class space Y into two disjoint sets of roughly equal size Y A and Y B .

Original: We build task A with the examples ( x, y ) ∈ D where y belongs to Y A , and task B with examples ( x, y ) where y belongs to Y B .
Revised: We build task A with the examples ( x, y ) ∈ D where y belongs to Y A , and task B with examples ( x, y ) where y belongs to Y B .

Original: Table 1 shows how patching a model on task A affects the accuracy on task B for nine datasets D .
Revised: Table 1 shows how patching a model on task A affects the accuracy on task B for nine datasets D .

Original: The accuracy improvements on task B range from 0.8 to 19.4 percentage points, even though the classes from task B are not seen during patching.
Revised: The accuracy improvements on task B range from 0.8 to 19.4 percentage points, even though the classes from task B are not seen during patching.

Original: To further understand transfer, we consider additional task pairs A and B , which are now different datasets.
Revised: To further understand transfer, we consider additional task pairs A and B , which are now different datasets.

Original: While some pairs A , B share classes, there are still instances of broad transfer.
Revised: While some pairs A , B share classes, there are still instances of broad transfer.

Original: Concretely,
Revised: Concretely,

Original: [Table]
Revised: [Table]

Original: Table 2 examines i) MNIST and SVHN, two digit recognition tasks with shared classes; ii) EuroSAT and RESISC45, two satellite imagery recognition tasks where there are unshared classes but some overlap; iii) GTSRB and MTSD [17], two traffic sign recognition datasets where there are unshared classes but some overlap; and iv) MNIST and FashionMNIST [83], which do not share any classes but appear visually similar.
Revised: Table 2 examines i) MNIST and SVHN, two digit recognition tasks with shared classes; ii) EuroSAT and RESISC45, two satellite imagery recognition tasks where there are unshared classes but some overlap; iii) GTSRB and MTSD [17], two traffic sign recognition datasets where there are unshared classes but some overlap; and iv) MNIST and FashionMNIST [83], which do not share any classes but appear visually similar.

Original: In seven out of eight experiments, patching on task A improves accuracy by 1.1 to 19.2 percentage points on task B .
Revised: In seven out of eight experiments, patching on task A improves accuracy by 1.1 to 19.2 percentage points on task B .

Original: The exception is when A is EuroSAT and B is RESISC45, where accuracy decreases by 1.3 percentage points.
Revised: The exception is when A is EuroSAT and B is RESISC45, where accuracy decreases by 1.3 percentage points.

Original: In all experiments, when patching on task A we choose the mixing coefficient α by optimizing the held-out validation accuracy on task A and a supported task (in this experiment we use ImageNet).
Revised: In all experiments, when patching on task A we choose the mixing coefficient α by optimizing the held-out validation accuracy on task A and a supported task (in this experiment we use ImageNet).

Original: While it is possible for a method that introduces new parameters for each task to exhibit broad transfer to new data, this also requires knowing which parameters to apply for the new data.
Revised: While it is possible for a method that introduces new parameters for each task to exhibit broad transfer to new data, this also requires knowing which parameters to apply for the new data.

Original: This is not necessary in the single model approach.
Revised: This is not necessary in the single model approach.

Original:  Case studies
Revised:  Case studies

Original: We further examine the performance of PAINT in three additional settings, which highlight weaknesses of the zero-shot CLIP model and showcase broad transfer (Section 6).
Revised: We further examine the performance of PAINT in three additional settings, which highlight weaknesses of the zero-shot CLIP model and showcase broad transfer (Section 6).

Original: Typographic attacks.
Revised: Typographic attacks.

Original: Goh et al.
Revised: Goh et al.

Original: [23] find that CLIP models are susceptible to typographic attacks , where text superimposed on an image leads to misclassification.
Revised: [23] find that CLIP models are susceptible to typographic attacks , where text superimposed on an image leads to misclassification.

Original: For example, in Figure 6 (a) , the text on the pink note saying “dog” leads a CLIP to misclassify the image of a cat as a dog.
Revised: For example, in Figure 6 (a) , the text on the pink note saying “dog” leads a CLIP to misclassify the image of a cat as a dog.

Original: To fix this vulnerability, we procedurally generate typographic attack data by adding text with incorrect class names to SUN397 [84], as seen in Figure 6 (b) .
Revised: To fix this vulnerability, we procedurally generate typographic attack data by adding text with incorrect class names to SUN397 [84], as seen in Figure 6 (b) .

Original: We then collect a test set of 110 real world images by placing notes on objects and taking photos.
Revised: We then collect a test set of 110 real world images by placing notes on objects and taking photos.

Original:  After applying PAINT using the synthetic data, we evaluate on the real-world images (Figure 6 (c) ) and synthetic test set (Figure 6 (d) ).
Revised: 6 After applying PAINT using the synthetic data, we evaluate on the real-world images (Figure 6 (c) ) and synthetic test set (Figure 6 (d) ).

Original: We observe that while larger models are more susceptible to typographic attacks, they are also more amenable to patching.
Revised: We observe that while larger models are more susceptible to typographic attacks, they are also more amenable to patching.

Original: Furthermore, we see an example of broad transfer between the synthetic and real-world data: when patching ViT-L/14 on synthetic data, its accuracy on real-world typographic attacks improves 41 percentage points even though the real-world classes are unseen.
Revised: Furthermore, we see an example of broad transfer between the synthetic and real-world data: when patching ViT-L/14 on synthetic data, its accuracy on real-world typographic attacks improves 41 percentage points even though the real-world classes are unseen.

Original: The cost is a reduction of less than 1 percentage point on ImageNet.
Revised: The cost is a reduction of less than 1 percentage point on ImageNet.

Original: We present details on the task and data collection in Appendix K.
Revised: We present details on the task and data collection in Appendix K.

Original: Counting.
Revised: Counting.

Original: Radford et al.
Revised: Radford et al.

Original: [57] find that CLIP models struggle to count the number of objects in
Revised: [57] find that CLIP models struggle to count the number of objects in

Original: CLEVR [32].
Revised: CLEVR [32].

Original: Here, the task is to choose an integer between 3 and 10 for each image, corresponding to the number of visible objects.
Revised: Here, the task is to choose an integer between 3 and 10 for each image, corresponding to the number of visible objects.

Original: While a straightforward way to patch such a task is to fine-tune on it directly, we investigate if applying PAINT using a subset of the classes allows the patched model to generalize to other numbers.
Revised: While a straightforward way to patch such a task is to fine-tune on it directly, we investigate if applying PAINT using a subset of the classes allows the patched model to generalize to other numbers.

Original: Specifically, we patch on images with 4, 5, 6, 8, or 9 objects.
Revised: Specifically, we patch on images with 4, 5, 6, 8, or 9 objects.

Original: To evaluate broad transfer, we test on images with 3, 7, and 10 objects (7 for understanding interpolation and 3 and 10 for extrapolation).
Revised: To evaluate broad transfer, we test on images with 3, 7, and 10 objects (7 for understanding interpolation and 3 and 10 for extrapolation).

Original: We find that PAINT improves accuracy from 59% to over 99% on unseen classes with less than half a percentage point decrease in ImageNet accuracy.
Revised: We find that PAINT improves accuracy from 59% to over 99% on unseen classes with less than half a percentage point decrease in ImageNet accuracy.

Original: For more details see Appendix L.
Revised: For more details see Appendix L.

Original: Visual question answering.
Revised: Visual question answering.

Original: As shown by Shen et al.
Revised: As shown by Shen et al.

Original: [68], zero-shot CLIP models perform poorly on visual question answering [4].
Revised: [68], zero-shot CLIP models perform poorly on visual question answering [4].

Original: Using CLIP for VQA typically involves additional parameters—for instance, Shen et al.
Revised: Using CLIP for VQA typically involves additional parameters—for instance, Shen et al.

Original: [68] trains a transformer [77] on CLIP features.
Revised: [68] trains a transformer [77] on CLIP features.

Original: In contrast, our procedure for patching CLIP on VQA does not introduce new parameters.
Revised: In contrast, our procedure for patching CLIP on VQA does not introduce new parameters.

Original: Following Shen et al.
Revised: Following Shen et al.

Original: [68], we contrast images with a series of text prompts, where each prompt corresponds to an option in multiple-choice
Revised: [68], we contrast images with a series of text prompts, where each prompt corresponds to an option in multiple-choice

Original: VQA, formed by both the question and a candidate answer using the following template: “Question: [question text]
Revised: VQA, formed by both the question and a candidate answer using the following template: “Question: [question text]

Original: Answer: [answer text]”.
Revised: Answer: [answer text]”.

Original: We evaluate on multiple-choice VQA v1 [4], where each question is associated with 18 candidate answers.
Revised: We evaluate on multiple-choice VQA v1 [4], where each question is associated with 18 candidate answers.

Original: Our results, further detailed in Appendix M, show that patching is effective for visual question answering: PAINT improves the accuracy of a ViT-L/ model by 18 percentage points, while accuracy drops by less than one percentage point on ImageNet.
Revised: Our results, further detailed in Appendix M, show that patching is effective for visual question answering: PAINT improves the accuracy of a ViT-L/ model by 18 percentage points, while accuracy drops by less than one percentage point on ImageNet.

Original:  Related work
Revised:  Related work

Original: Continual learning and catastrophic forgetting.
Revised: Continual learning and catastrophic forgetting.

Original: Learning tasks sequentially remains a challenge for neural networks.
Revised: Learning tasks sequentially remains a challenge for neural networks.

Original: When a neural network learns a new task, the accuracy on other tasks often decreases, a phenomenon known as catastrophic forgetting [48, 76, 20, 33].
Revised: When a neural network learns a new task, the accuracy on other tasks often decreases, a phenomenon known as catastrophic forgetting [48, 76, 20, 33].

Original: While forgetting in neural networks may actually aid learning [90], researchers have proposed various approaches for alleviating catastrophic forgetting, including: i) Regularization-based approaches such as elastic weight consolidation (EWC) [33] and synaptic intelligence (SI) [87] which penalize the movement of parameters and are related to weight-interpolation by Lubana et al.
Revised: While forgetting in neural networks may actually aid learning [90], researchers have proposed various approaches for alleviating catastrophic forgetting, including: i) Regularization-based approaches such as elastic weight consolidation (EWC) [33] and synaptic intelligence (SI) [87] which penalize the movement of parameters and are related to weight-interpolation by Lubana et al.

Original: [44]; ii) Replay methods [61, 69, 42, 6, 64, 50], which incorporate data or gradient information from previous tasks when learning a new task; and iii) Introducing task-specific parameters [65, 85, 46, 8, 78, 80].
Revised: [44]; ii) Replay methods [61, 69, 42, 6, 64, 50], which incorporate data or gradient information from previous tasks when learning a new task; and iii) Introducing task-specific parameters [65, 85, 46, 8, 78, 80].

Original: In contrast to these approaches, PAINT requires no modification to the standard fine-tuning process besides the later weight interpolation step.
Revised: In contrast to these approaches, PAINT requires no modification to the standard fine-tuning process besides the later weight interpolation step.

Original: Moreover, unlike regularization or replay based methods,
Revised: Moreover, unlike regularization or replay based methods,

Original: PAINT requires no extra computational cost during training.
Revised: PAINT requires no extra computational cost during training.

Original: In contrast to methods with task specific parameters, we maintain a single model.
Revised: In contrast to methods with task specific parameters, we maintain a single model.

Original: Having a single model is beneficial when there is new data which is similar to one of the tasks which have already been patched.
Revised: Having a single model is beneficial when there is new data which is similar to one of the tasks which have already been patched.

Original: Even without explicitly knowing which task the new data is similar to, we can observe accuracy improvements (see Section 6).
Revised: Even without explicitly knowing which task the new data is similar to, we can observe accuracy improvements (see Section 6).

Original: Similar to our work is that of Mirzadeh et al.
Revised: Similar to our work is that of Mirzadeh et al.

Original: [50], who observe high accuracy on task A on the linear path between a model which achieves high accuracy on task A and a model which is fine-tuned jointly on task A and
Revised: [50], who observe high accuracy on task A on the linear path between a model which achieves high accuracy on task A and a model which is fine-tuned jointly on task A and

Original: B. Moreover, they observe high accuracy on task B on the linear path between a model fine-tuned on task B, and the jointly fine-tuned model.
Revised: B. Moreover, they observe high accuracy on task B on the linear path between a model fine-tuned on task B, and the jointly fine-tuned model.

Original: Therefore, there exists a path between a model which achieves good performance on task A and a model fine-tuned on task B along which accuracy is high on both tasks.
Revised: Therefore, there exists a path between a model which achieves good performance on task A and a model fine-tuned on task B along which accuracy is high on both tasks.

Original: However, in Mirzadeh et al.
Revised: However, in Mirzadeh et al.

Original: [50] this combined path can be non-linear, leading them to propose a regularization and replay based method.
Revised: [50] this combined path can be non-linear, leading them to propose a regularization and replay based method.

Original: In our work, we find that examining models on a linear path between the unpatched model (which has high accuracy on task A) and the model fine-tuned on task B is often sufficient for obtaining a model which achieves high accuracy on both tasks (Figure 1).
Revised: In our work, we find that examining models on a linear path between the unpatched model (which has high accuracy on task A) and the model fine-tuned on task B is often sufficient for obtaining a model which achieves high accuracy on both tasks (Figure 1).

Original: We speculate that this is due to scale and model architecture: in contrast to Mirzadeh et al.
Revised: We speculate that this is due to scale and model architecture: in contrast to Mirzadeh et al.

Original: [50], we initialize with a model pre-trained on a large dataset consisting ofmillion images [57], and primarily use vision transformers [15].
Revised: [50], we initialize with a model pre-trained on a large dataset consisting ofmillion images [57], and primarily use vision transformers [15].

Original: As shown in Section 4.2, our method performs substantially worse with ResNets [24], which are used by Mirzadeh et al.
Revised: As shown in Section 4.2, our method performs substantially worse with ResNets [24], which are used by Mirzadeh et al.

Original: Finally, Ramasesh et al.
Revised: Finally, Ramasesh et al.

Original: [59] and Mehta et al.
Revised: [59] and Mehta et al.

Original: [49] also observed that catastrophic forgetting is less problematic for large and pre-trained models.
Revised: [49] also observed that catastrophic forgetting is less problematic for large and pre-trained models.

Original: In addition, Ramasesh et al.
Revised: In addition, Ramasesh et al.

Original: [59] found—similar to our results—that vision transformers are less susceptible to forgetting than ResNets of the same size.
Revised: [59] found—similar to our results—that vision transformers are less susceptible to forgetting than ResNets of the same size.

Original: Linear mode connectivity and robust fine-tuning.
Revised: Linear mode connectivity and robust fine-tuning.

Original: Linearly interpolating neural network weights is a key step in PAINT.
Revised: Linearly interpolating neural network weights is a key step in PAINT.

Original: Because of the many nonlinear activations in a neural network, it is not clear a priori that linearly interpolating between two sets of weights can result in a high accuracy solution.
Revised: Because of the many nonlinear activations in a neural network, it is not clear a priori that linearly interpolating between two sets of weights can result in a high accuracy solution.

Original: However, researchers have observed that interpolating neural network weights can achieve high accuracy when training on MNIST from a common initialization [52] or when part of the optimization trajectory is shared [19, 28, 54, 18, 82, 47, 16, 81, 10].
Revised: However, researchers have observed that interpolating neural network weights can achieve high accuracy when training on MNIST from a common initialization [52] or when part of the optimization trajectory is shared [19, 28, 54, 18, 82, 47, 16, 81, 10].

Original: The term linear mode connectivity was coined by Frankle et al.
Revised: The term linear mode connectivity was coined by Frankle et al.

Original: [19]: two networks exhibit linearly mode connectivity if the accuracy does not decrease when using weights on the linear path between them [52, 19].
Revised: [19]: two networks exhibit linearly mode connectivity if the accuracy does not decrease when using weights on the linear path between them [52, 19].

Original: Weight averaging for continual learning has also been studied by Lee et al.
Revised: Weight averaging for continual learning has also been studied by Lee et al.

Original: [40] for closed-vocabulary models.
Revised: [40] for closed-vocabulary models.

Original: While Nagarajan and Kolter [52] and Frankle et al.
Revised: While Nagarajan and Kolter [52] and Frankle et al.

Original: [19] focused on accuracy on a single task, Wortsman et al.
Revised: [19] focused on accuracy on a single task, Wortsman et al.

Original: [82] use linear mode connectivity to fine-tune models while preserving their robustness to natural distribution shifts.
Revised: [82] use linear mode connectivity to fine-tune models while preserving their robustness to natural distribution shifts.

Original: By interpolating the weights of a zero-shot and fine-tuned model, they find a solution which performs well both on the fine-tuning task and under distribution shift.
Revised: By interpolating the weights of a zero-shot and fine-tuned model, they find a solution which performs well both on the fine-tuning task and under distribution shift.

Original: In contrast to
Revised: In contrast to

Original: Wortsman et al.
Revised: Wortsman et al.

Original: [82], we do not modify any task-specific parameters when fine-tuning, preserving the open-vocabulary nature of the models we patch.
Revised: [82], we do not modify any task-specific parameters when fine-tuning, preserving the open-vocabulary nature of the models we patch.

Original: Unlike Wortsman et al.
Revised: Unlike Wortsman et al.

Original: [82], we examine accuracy trade-offs across different tasks with little or no class overlap and adapt a model to multiple tasks.
Revised: [82], we examine accuracy trade-offs across different tasks with little or no class overlap and adapt a model to multiple tasks.

Original: In addition, closely related to our work is that of Matena and Raffel [47], who use Fisher-weighted averaging of language models before and after fine-tuning on downstream tasks.
Revised: In addition, closely related to our work is that of Matena and Raffel [47], who use Fisher-weighted averaging of language models before and after fine-tuning on downstream tasks.

Original: Unlike Fisherweighted averaging of Matena and Raffel [47], we do not use different mixing coefficients for each parameter, and thus require no extra compute when patching.
Revised: Unlike Fisherweighted averaging of Matena and Raffel [47], we do not use different mixing coefficients for each parameter, and thus require no extra compute when patching.

Original: Moreover, we explore new strategies for patching on multiple tasks (see Section 5), and focus on open-vocabulary image classifiers.
Revised: Moreover, we explore new strategies for patching on multiple tasks (see Section 5), and focus on open-vocabulary image classifiers.

Original: Interventions to change the behavior of a trained model.
Revised: Interventions to change the behavior of a trained model.

Original: Several authors have studied the problem of updating a model to locally alter its behavior on certain inputs without external disruptions on other inputs [70, 13, 51, 66, 63, 62].
Revised: Several authors have studied the problem of updating a model to locally alter its behavior on certain inputs without external disruptions on other inputs [70, 13, 51, 66, 63, 62].

Original: Previous literature uses various terms to refer to this process, including model editing, patching or debugging.
Revised: Previous literature uses various terms to refer to this process, including model editing, patching or debugging.

Original: A popular use case is to update trained language models to reflect changes in the world (for instance, facts like who is the current president of Brazil) [29, 45, 38, 30].
Revised: A popular use case is to update trained language models to reflect changes in the world (for instance, facts like who is the current president of Brazil) [29, 45, 38, 30].

Original: Moreover, inspired by software engineering practice, previous work explored “debugging” language models through user interaction [63, 62], including providing corrective feedback to the models via natural language [3].
Revised: Moreover, inspired by software engineering practice, previous work explored “debugging” language models through user interaction [63, 62], including providing corrective feedback to the models via natural language [3].

Original: Mitchell et al.
Revised: Mitchell et al.

Original: [51], De Cao et al.
Revised: [51], De Cao et al.

Original: [13] propose training auxiliary networks to perform local edits on pre-trained models.
Revised: [13] propose training auxiliary networks to perform local edits on pre-trained models.

Original: Santurkar et al.
Revised: Santurkar et al.

Original: [66] introduce a method for rewriting the prediction rules of a classifier, focusing on specific failure modes such as reliance on spurious correlations.
Revised: [66] introduce a method for rewriting the prediction rules of a classifier, focusing on specific failure modes such as reliance on spurious correlations.

Original: In contrast with previous literature, our work explores patching models at the task level, aiming to systemically improve accuracy on a dataset—for instance, enabling a model to recognize dozens of satellite imagery classes with a single patch.
Revised: In contrast with previous literature, our work explores patching models at the task level, aiming to systemically improve accuracy on a dataset—for instance, enabling a model to recognize dozens of satellite imagery classes with a single patch.

Original:  Limitations and conclusion
Revised:  Limitations and conclusion

Original: Limitations.
Revised: Limitations.

Original: When applying PAINT , accuracy on supported tasks can still decrease, especially for smaller models.
Revised: When applying PAINT , accuracy on supported tasks can still decrease, especially for smaller models.

Original: This limitation is perhaps best reflected in the case of sequential patching: patched models underperform using multiple specialized models when many tasks are added sequentially.
Revised: This limitation is perhaps best reflected in the case of sequential patching: patched models underperform using multiple specialized models when many tasks are added sequentially.

Original: Using larger models and weight interpolations can alleviate this issue, but do not completely resolve it.
Revised: Using larger models and weight interpolations can alleviate this issue, but do not completely resolve it.

Original: Finally, better understanding on which datasets patching is more effective is an exciting direction for future research.
Revised: Finally, better understanding on which datasets patching is more effective is an exciting direction for future research.

Original: Conclusion.
Revised: Conclusion.

Original: In this work, we explore several techniques for patching open-vocabulary models with the goal of improving accuracy on new tasks without decreasing accuracy elsewhere.
Revised: In this work, we explore several techniques for patching open-vocabulary models with the goal of improving accuracy on new tasks without decreasing accuracy elsewhere.

Original: PAINT is effective in several scenarios, ranging from classifying digits to defending against typographic attacks.
Revised: PAINT is effective in several scenarios, ranging from classifying digits to defending against typographic attacks.

Original: PAINT becomes more effective with scale, and can be applied on multiple tasks sequentially or simultaneously.
Revised: PAINT becomes more effective with scale, and can be applied on multiple tasks sequentially or simultaneously.

Original: Our findings demonstrate that in many circumstances it is possible to expand the set of tasks on which models achieve high accuracy, without introducing new parameters, without re-training them from scratch, and without catastrophic forgetting.
Revised: Our findings demonstrate that in many circumstances it is possible to expand the set of tasks on which models achieve high accuracy, without introducing new parameters, without re-training them from scratch, and without catastrophic forgetting.

Original: Acknowledgments
Revised: Acknowledgments

Original: We thank Akari Asai, Alex Fang, David Fleet, Huy Ha, Ari Holtzman, Pieter-Jan Kindermans, Marco
Revised: We thank Akari Asai, Alex Fang, David Fleet, Huy Ha, Ari Holtzman, Pieter-Jan Kindermans, Marco

Original: Tulio Ribeiro, Ofir Press, Sarah Pratt, Sewon Min, Thao Nguyen and Tim Dettmers for helpful discussions and feedback, and Hyak at UW for computing support.
Revised: Tulio Ribeiro, Ofir Press, Sarah Pratt, Sewon Min, Thao Nguyen and Tim Dettmers for helpful discussions and feedback, and Hyak at UW for computing support.

Original: This work is in part supported by the NSF AI Institute for Foundations of Machine Learning (IFML), Open Philanthropy, NSF
Revised: This work is in part supported by the NSF AI Institute for Foundations of Machine Learning (IFML), Open Philanthropy, NSF

Original: IIS 1652052, NSF IIS 17303166, NSF IIS 2044660, NSF IIS 2132519, ONR N00014-18-1-2826,
Revised: IIS 1652052, NSF IIS 17303166, NSF IIS 2044660, NSF IIS 2132519, ONR N00014-18-1-2826,

Original: DARPA N66001-19-2-4031, DARPA W911NF-15-1-0543, the Sloan Fellowship and gifts from AllenInstitute for AI. A Dataset details In Table
Revised: DARPA N66001-19-2-4031, DARPA W911NF-15-1-0543, the Sloan Fellowship and gifts from Allen

Original: [Table]

Original: we present the number of classes and the size of the training, validation and test sets we use for the each patching and supported tasks: Stanford Cars [35], Describable Textures (DTD) [11],

Original: EuroSAT [25], German Traffic Sign Recognition Benchmark (GTSRB) [71], KITTI distance [22],

Original: MNIST [39], RESISC45 [7], SUN397 [84], SVHN [53] ImageNet [14], FashionMNIST [83], MTSD [17], CIFAR10 [36], CIFAR100 [36], Food101 [5], STL10 [12] and ImageNet [14].

Original: For datasets that did not have publicly available, labeled test sets, we use the validation set as the test set, and split the training set into training and validation sets in a stratified fashion.

Original: For all these datasets, we use accuracy as our evaluation metric.

Original: B Background on open-vocabulary models

Original: In contrast to typical image classifiers, open-vocabulary models are not constrained to a fixed classification space.

Original: Instead, they are able to perform any image classification task, by using textual descriptions of the class names.

Original: Recently, many open-vocabulary models have been proposed [57, 31, 56, 88, 86, 1].

Original: A popular class of open-vocabulary models are contrastive image-text models like CLIP, BASIC and

Original: ALIGN.

Original: Following Radford et al.

Original: [57] we use the term CLIP to refer to any contrastive image-text model.

Original: This class of models are the focus of this work, although PAINT can naturally be extended to other open vocabulary models like Flamingo [1].

Original: CLIP models are trained to contrast images and textual descriptions, and are usually pre-trained on large, heterogeneous data collected from the web, ranging from hundreds of millions to billions of pairs of images and captions.

Original: This class of models consists of a vision encoder f which processes images and a text encoder g which processes text.

Original: Given a set of image-caption pairs { ( x 1 , y 1 ) , ..., ( x k , y k ) } , the model is optimized to maximize the similarity of aligned pairs ⟨ f ( x i ) , g ( y i ) ⟩ relative to unaligned pairs.

Original: When performing a classification task, a set of captions { y 1 , ..., y k } are procedurally generated based on text descriptions of the classes and some template.

Original: For instance, for distinguishing between cats and dogs, the set of candidate captions could be { “an image of a dog”, “an image of a cat” } .

Original: Then, given an image x , the chosen class is selected based on the caption that maximizes feature similarity with the image ( arg max i ⟨ f ( x ) , g ( y i ) ⟩ ).

Original: A common practice is to generate multiple candidate captions for each class (for instance, “an image of a dog” and “a photo of a dog”) and average their representations before computing the similarities.

Original: This practice, introduced by Radford et al.

Original: [57], is known as prompt ensembling.

Original: [Figure]

Original: Adapting open-vocabulary models to downstream tasks is typically done by introducing task-specific parameters, mapping visual representations to a fixed class space determined by the downstream task.

Original: The model can then be fine-tuned end-to-end, where both the visual encoder and the new classification head are updated, or adapted via training linear probe on top of the visual representations.

Original: By contrast, we do not introduce or modify any task-specific parameters.

Original: To do so, we adapt only the weights of the visual encoder, maintaining the text encoder without updating its weights.

Original: Freezing the text encoder allows for faster fine-tuning, since the text features can be pre-computed once per task.

Original: Moreover, in Appendix C, we show that the decision of freezing the text encoder does not substantially harm performance.

Original: Importantly, this fine-tuning procedure allows us to adapt the model without loosing its ability to perform any image classification task.

Original: C Frozen CLIP heads for fine-tuning

Original: Instead of introducing a learnable classification layer, we use the frozen output of CLIP’s text encoder to map image features to the space of classes.

Original: Figure 7 shows that this modification has a negligible effect on downstream accuracy for MNIST, EuroSAT, and SUN397.

Original: D The effect of scale on patching

Original: This section introduces additional metrics for the effectiveness of patching and how similar the unpatched and fine-tuned models are.

Original: Results are shown in Figure 8.

Original: D.1 Measuring the effectiveness of patching

Original: In addition to accuracy distance to optimal presented in Section 4.1, we present two other metrics to measure the effectiveness of patching, accuracy distance to endpoints and path correction cost , which are summarized in Table 4.

Original: For simplicity, let x α denote the accuracy of model θ α on the supported task and y α on the patching task.

Original: For all metrics, lower values indicate that patching is more effective.

Original: Accuracy distance to endpoints contrasts the accuracy of a single model with using two specialized models, the fine-tuned model for the patching task and the unpatched model for the supported task.

Original: Recall that x α denotes the accuracy of model θ α on the supported task and y α on the patching task.

Original: Accuracy distance to endpoints is given by ( x 0 + y 1 ) / 2 − max α ( x α + y α ) / 2 .

Original: Path correction cost measures how the curve traced by varying the mixing coefficient compares to a curve without accuracy trade-offs.

Original: Its value is the average cost needed to move points in the traced curve to the ideal curve I , where

Original: I =

Original: { x, y ∈ [0 , 1] ×

Original: [0 , 1] | x = x zs ∨ y

Original: =

Original: y ft } is the set of points on the scatter plots where either accuracy on the supported task is equal to that of the unpatched model or accuracy on the patching task is equal to that of the fine-tuned model.

Original: More precisely, patch correction cost is given by, E α

Original: [-[ δ (( x α , y α ) , I ) 
-]
Revised: [-[ δ (( x α , y α ) , I ) 
-]

Original: [ x α <x 0 ∧ y α <y 1 ]] , where δ (( x, y ) , I ) =

Original: min x ′ ,y ′ ∈I

Original: || ( x − x ′ , y − y ′ )

Original: || 2 is the distance between ( x, y ) and I , and 1 [ ] is the indicator function.

Original: The indicator term 1 [ ] ensures costs are only computed for points where both accuracy on the supported task is smaller than that of the unpatched model and accuracy on the patching task is smaller than that of the fine-tuned model.

Original: [Figure]

Original: their representations

Original: [Table]

Original: D.2 Model similarity

Original: In addition to the angle between the unpatched and fine-tuned models, we also measure the average absolute difference || θ ft − θ zs ||

Original: 1 /n between the weights θ zs , θ ft ∈ R n .

Original: Representational similarity is measured through CKA [34], which is shown in Equation 2.

Original: In short,

Original: CKA can be used to compare the similarity between two models by inspecting their representations on a given set of inputs.

Original: Formally, let Θ zs , Θ ft ∈ R n × d denote the d -dimensional features of the unpatched and fine-tuned model on a dataset with n samples, pre-processed to center the columns.

Original: CKA is then given by:

Original: [Equation]

Original: where ||

Original: A || F indicates the Frobenious norm of a matrix A .

Original: Larger values of CKA indicate more similar representations.

Original: When reporting CKA values, we compute the last layer features using samples from the supported task.

Original: E Additional baselines and comparisons

Original: This section details additional baselines for our patching procedure in the context of patching a single task.

Original: We discuss exponential moving averages (Section E.1), elastic weight consolidation (Section E.2), learning without forgetting (Section E.3, re-training (Section E.4), and finally mixing the pre-training and fine-tuning objective (Section E.5).

Original: E.

Original: Exponential moving averages (EMA)

Original: In Figure 3 we show that interpolating the weights of the unpatched and fine-tuned model can recover the “forgetting frontier” [59].

Original: The forgetting frontier is formed by fine-tuning with various

Original: [Figure]

Original: [Figure]

Original: set it is possible to augment standard fine-tuning with EWC [33].

Original: EWC provides a solution with a good accuracy trade-off when patching on MNIST and EuroSAT, but not SUN397.

Original: We also show interpolations between the weights of the unpatched model and a model fine-tuned with EWC.

Original: As pre-training data is required to compute the fisher information matrix for

Original: EWC, these experiments use a ViT-B/16 model from a CLIP reproduction OpenCLIP [27] pre-trained on LAION 400M [67].

Original: hyperparameters.

Original: In particular, we show that interpolating the unpatched and fine-tuned models can recover a solution with similar or better accuracies than early termination.

Original: However, these early termination solutions can potentially be suboptimal because learning rate has not yet decayed to zero.

Original: As such, we recreate this comparison in Figure 9 where we fine-tune with a constant learning rate and EMA [74].

Original: E.2 Elastic weight consolidation (EWC)

Original: Elastic weight consolidation [33] (EWC) is a method which penalizes the movement of parameters which are deemed important for solving previous tasks.

Original: However, access to the pre-training data is required to investigate which parameters are important.

Original: Therefore, we could not use this method for

Original: Figure 3 since the pre-training data for the CLIP models of Radford et al.

Original: [57] is private.

Original: To examine the performance of EWC, we use a reproduction of CLIP from the OpenCLIP repository [27], which is pre-trained on the open source LAION 400M dataset [67].

Original: We use 2,000 iterations of pre-training to compute the fisher information matrix required for EWC.

Original: The results are illustrated in Figure 10 which show EWC solutions corresponding to different coefficients from the EWC loss ( { 0 . 001 , 0 . 01 , 0 . 1 , 1 , 10 , 100 , 1000 , 10000 , 100000 , 1000000 } ).

Original: We also interpolate the weights of the unpatched model and the EWC solution fine-tuned with coefficient 1000000.

Original: On MNIST and EuroSAT, an EWC solution exhibits slightly better accuracy trade-offs than any solution on the interpolation between the unpatched model and model fine-tuned without EWC.

Original: However, on

Original: SUN397, interpolating the unpatched and fine-tuned model provides a better trade-off than EWC.

Original: [Figure]

Original: Patching on SUN

Original: [Figure]

Original: E.3 Learning without forgetting (LwF)

Original: Learning without forgetting [41](LwF) adds an additional regularization term when fine-tuning based on knowledge distillation [26].

Original: We contrast PAINT with LwF in Figure 11.

Original: When fine-tuning with LwF, we use multiple loss balance weights (0 . 1 , 0 . 2 , ..., 0 . 9) , and leave the remaining hyperparameters unchanged.

Original: As in Section E.2, we use SUN397, MNIST and EuroSAT as our patching tasks, ImageNet as our supported task, and a patch CLIP ViT-B/16 model.

Original: As shown in Figure 11,

Original: PAINT is competitive or better than LwF on all tasks.

Original: Moreover, weight interpolations can further improve on LwF, showing that PAINT and LwF are complementary, rather than mutually exclusive alternatives.

Original: E.4 Re-training with data from the patching task

Original: We further contrast patching with re-training a model from scratch, adding data from the patching task to the pre-training dataset.

Original: For such, we use a ViT-B/32 model, training for 32 epochs with cosine learning rate schedule with lienar warmup of 5000 steps and learning rate of 0.001, AdamW optimizer with weight decay of 0.1 and global batch size of 1024, using the open-source library

Original: OpenCLIP [27].

Original: We train both on data from YFCC-15M [57] alone, and with both YFCC-15M and

Original: MNIST data, without upsampling.

Original: Results are shown in Figure 12.

Original: We find that re-training with data from the patching task is highly effective at improving accuracy on that task, and only slightly decreases zero-shot accuracy on

Original: ImageNet.

Original: However, we note that pre-training is substantially more expensive than patching.

Original: For instance, considering the ViT-L/14 models trained by Radford et al.

Original: [57], pre-training takes around 10,000 times more compute than our patching procedure, which makes re-training impractical in most scenarios.

Original: E.5 Objective mixing

Original: When data is available from the pre-training dataset it becomes possible mix the pre-training and fine-tuning objectives.

Original: This baseline, which we refer to as objective mixing , is similar to replay methods in continual learning [61, 69, 42, 6, 64, 50].

Original: However, objective mixing is only possible when the pre-training data is available, which is not the case for the official CLIP models of Radford et al.

Original: Therefore, we use models from a CLIP reproduction, OpenCLIP [27].

Original: First, we use an OpenCLIP ViT-B/32 model which is trained on the publicly available LAION 400M dataset [67] and showcase results in Figure 13 (left).

Original: Like CLIP models from Radford et al.

Original: [57], this model is trained with a batch size of approximately 32,000.

Original: We use a single machine and try batch sizes 128, 256, 512, and 1024 for the pre-training objective.

Original: Overall the loss is given by (1 − β ) · ℓ pre-training + β · ℓ fine-tuning for β ∈ { 0 , 0 .

Original: 001 , 0 . 01 , 0 . 1 , 0 . 2 , ..., 0 . 9 , 1 . 0 } where ℓ pre-training and ℓ fine-tuning are the pre-training and fine-tuning loss, respectively.

Original: We notice and interesting phenomenon: even when β = 0 , the zero-shot accuracy on ImageNet still drops from the pre-trained

Original: [Figure]

Original: model (Figure 13).

Original: We hypothesize that the drop in zero-shot accuracy is likely due to a reduction in batch size and note that the importance of large batches for CLIP objectives has been studied by Pham et al.

Original: [56], Radford et al.

Original: Because of this drop in accuracy, the patching procedure we study matches or provides better accuracy trade-offs in this setting.

Original: Next, we pre-train our own ViT-B/32 model on a single machine using OpenCLIP [27].

Original: using a batch size of 512 for the pre-training objective.

Original: We also use the smaller pre-training dataset

Original: YFCC-15M [75, 57].

Original: In Figure 13 (right) we perform objective mixing with coefficients β ∈ { 0 , 0 .

Original: 001 , 0 . 01 , 0 . 1 , 0 . 2 , ..., 0 . 9 , 1 . 0 } and batch size 512 for the pre-training objective.

Original: In this setting, zero-shot accuracy does not drop for β = 0 and objective mixing performs better than the patching procedure we propose.

Original: We believe this is because scale helps our method (see Section 4.1), and this is the smallest scale model we examine in terms of pre-training set size.

Original: In terms of absolute accuracy, applying PAINT to the LAION pre-trained ViT-B/32 model is the best option.

Original: In conclusion, the findings of these experiments are that:

Original: Objective mixing is difficult because it requires a large batch size.

Original: Objective mixing with a small batch size performs well when applied to CLIP models trained with small batch sizes, but these models are worse overall.

Original: F Additional plots for patching on a single task

Original: Breakdowns for each supported task and patching task are shown in Figures 16 to 19.

Original: [Figure]

Original: Patching a LAION ViT-B/

Original: [Figure]

Original: [Figure]

Original: n

Original: [Figure]

Original: w

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: ViT-L/

Original: [Figure]

Original: B

Original: [Figure]

Original: ViT-B/

Original: ViT-L/

Original: [Figure]

Original: ViT

Original: [Figure]

Original: [Figure]

Original: ViT

Original: [Figure]

Original: [Figure]

Original: G Additional models

Original: In addition to ViT models, we measure the effectiveness of patching for four ResNet models [24].

Original: Specifically, we examine ResNet-50, ResNet-101, and two wider networks, ResNet-50x4, ResNet50x16 [24, 57].

Original: Results are shown in Figure in 20.

Original: H Patching closed-vocabulary models

Original: Beyond open-vocabulary models, we show that PAINT is also effective for closed-vocabulary image classifiers.

Original: Our experimental setting is as follows: we start with a (closed-vocabulary) model trained on ImageNet from scratch, from the Pytorch ImageNet Models library [79].

Original: Our goal is to expand the set of categories known by the model to improve its performance on MNIST, without hurting accuracy on ImageNet.

Original: In other words, we wish to build a model that is competent at classifying an image both amongst the 1000 categories from ImageNet, and amongst 10 digit categories from

Original: MNIST.

Original: For such, we expand the classification head from the original model by adding 10 new classes, and initialize the corresponding weights and biases to zero.

Original: We then fine-tune the model on

Original: MNIST without any frozen weights, and interpolate with the model before fine-tuning.

Original: Figure 21 shows results for patching closed-vocabulary ViT-B/32, ViT-B/16 and ViT-L/16 models that are trained from scratch on ImageNet.

Original: For all models, accuracy on MNIST improves to over 99%, while accuracy on ImageNet decreases by less than one percentage point.

Original: These experiments show that patching is effective beyond open-vocabulary models.

Original: [Figure]

Original: [Figure]

Original: Patching closed-vocabulary models on MNIST

Original: [Figure]

Original: I Broad Transfer

Original: This section provides additional results for the broad transfer experiments in Section 6, as well as new experiments where we fine-tune on ImageNet.

Original: In Section 6, Tables 1 and 2 only show results for the single mixing coefficient that out procedure chooses.

Original: To supplement these tables, we show how accuracy changes for various mixing coefficients α ∈ { 0 , 0 . 05 , 0 .

Original: For Table 1 the corresponding figure with all mixing coefficient information is Figure 22.

Original: Similarly, Table 2 is expanded in Figures 23 and 24.

Original: Finally, we measure broad transfer on 13 datasets when patching on ImageNet.

Original: In Figure 25 we fine-tune on ImageNet then interpolate with the unpatched model.

Original: Surprisingly, for the ViT-B/ model, fine-tuning on ImageNet improves accuracy on KITTI accuracy by more than 10 percentage points, and MNIST accuracy by more than 20 percentage points, even without patching.

Original: More investigation into broad transfer is required to understand when and how it applies.

Original: [Figure]

Original: [Figure]

Original: ss

Original: [Figure]

Original: SUN

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: N

Original: [Figure]

Original: cc

Original: [Figure]

Original: m

Original: [Figure]

Original: g

Original: [Figure]

Original: N e t a cc u r a c y

Original: [Figure]

Original: J Patching models on multiple tasks

Original: This section expands the experiments and results on patching models on multiple tasks from Section 5.

Original: Figure 26 illustrates the different patching strategies.

Original: We first show an exhaustive search over the mixing coefficients for parallel patching on pairs of datasets.

Original: Next, we provide additional results for sequential patching, including additional orders in which the datasets are seen and how the number of datasets affects the quality of patching.

Original: Finally, we present results on SplitCIFAR [61].

Original: J.1 Exhaustive parallel search

Original: Recall from Section 5 that exhaustively searching the mixing coefficients is prohibitively expensive when many patching tasks are used.

Original: Here, we study patching on two patching tasks, for which an exhaustive search is still feasible, and contrast it with other search strategies.

Original: More specifically, we examine three pairs of datasets: i) MNIST and EuroSAT; ii) MNIST and DTD; and iii) Cars and DTD.

Original: The results are shown in Figures 27 to 29.

Original: For two patching tasks D 1 and D 2 , let θ (1) ft

Original: and θ (2) ft be the models fine-tuned on them.

Original: We then measure accuracy of models θ =

Original: (1 − α 1 − α 2 ) · θ zs + α 1 · θ (1) ft + α 2 · θ (2) ft for α 1 , α 2 ∈ [0 , 1] .

Original: In most cases, there exists some values of the mixing coefficients such that accuracy is high on all the three tasks.

Original: For instance, when patching a ViT-L/14 on MNIST and EuroSAT, when α 1 = 0 .

Original: 35 and α 2 = 0 .

Original: 45 , accuracy on MNIST and EuroSAT is 39 andpercentage points higher compared to the unpatched model, while accuracy on ImageNet decreases by less than 1 percentage point.

Original: Note that the area of high average accuracy typically increases with scale, supporting findings of Section 4.1.

Original: In Table 5 we contrast the average accuracy obtained via exhaustive search with using other search strategies, uniform search and black-box optimization .

Original: Recall that these methods optimize for average accuracy on the validation sets.

Original: The uniform search strategy, also described in Section 5, consists of searching over a single scalar β ∈ [0 , 1] , inspecting the models θ = (1 − β ) · θ zs + α/ 2 · θ (1) ft + α/ 2 · θ (2) ft .

Original: For black-box optimization, we explore an adaptive black box optimization algorithm, Nevergrad [60] 8 over coefficients α (1) , α (2) ∈ [0 , 1] for the model θ =

Original: (1 − α 1 − α 2 ) · θ zs + α 1 · θ (1) ft + α 2 · θ (2) ft .

Original: We initialize all α (1) and α (2) with 1 / 2 , and run for 50 optimization steps.

Original: As shown in Table 5, uniform search and black-box optimization are comparable in performance, and substantially outperform the unpatched model.

Original: However, both strategies still lag behind exhaustive search, indicating headroom for more sophisticated search strategies.

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: N I S T

Original: m i x

Original: i n g c o e f .

Original: [Table]

Original: on

Original: [Table]

Original: ViT

Original: [Table]

Original: M N I S T

Original: m i x i n g

Original: [Table]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: M N I S T

Original: m i x i n g c o e f . Acc. on MNIST
Revised: Institute for AI.

Original: (ViT-L/

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: a r s m i x i n g c o e f .

Original: [Figure]

Original: on

Original: [Figure]

Original: ViT

Original: [Figure]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: C a r s m i x i n g c o e f .

Original: [Figure]

Original: m i x i n g

Original: [Figure]

Original: o e

Original: [Figure]

Original: [Figure]

Original: J.2 Sequential patching

Original: In Figure

Original: [Figure]

Original: we show the evolution of sequential patching as more tasks are added.

Original: The accuracy distance of using a single, patched model to using multiple specialized models increases with with the number of patched tasks, leaving headroom for future work on more sophisticated sequential strategies for patching.

Original: Interestingly, sequential patching outperforms sequential fine-tuning (where no interpolation is used) by a large margin

Original: [Figure]

Original: J.3 SplitCIFAR

Original: Figure 31 compares the patching methods described in Section 2 when patching on ten tasks from

Original: SplitCIFAR100 [61].

Original: We split CIFAR100 randomly into ten different 10-way classification problems which are either learned jointly (as in joint fine-tuning or joint patching) or independently.

Original: In Figurewe show how the number of tasks learned affects accuracy on i) ImageNet, the supported task used for this experiment (first row), ii) the patching tasks (second row) and iii) average accuracy on the patching tasks averaged with ImageNet accuracy (third row).

Original: We also display accuracy for two additional tasks, Food101 and STL10, in rows four and five, respectively.

Original: While this experiment explores PAINT in a more conventional continual learning setting, there is a key difference: we also examine model accuracy on other tasks like ImageNet and Food101.

Original: To choose the mixing coefficients we optimize for performance on the held-out validation set of all patching tasks seen so far (i.e, the SplitCIFAR100 tasks) and the supported task ImageNet with equal weight.

Original: This is the reason for also examining accuracy on Food101 and STL10—we want to make sure that we are not overfitting to the representative supported task, ImageNet.

Original: As intended, PAINT show less catastrophic forgetting than the alternative approaches on ImageNet,

Original: Food101, and STL10.

Original: All methods are fine-tuned with a total of 2,000 iterations.

Original: As such, we only use 200 iterations per-task when fine-tuning independently.

Original: [Figure]

Original: m a

Original: [Figure]

Original: e

Original: N

Original: [Figure]

Original: A v e r a g e a cc u r a c y o n t h e p a t c h i n

Original: [Figure]

Original: s

Original: [Figure]

Original: g e a cc u r a c y o n p a t c h i n

Original: [Figure]

Original: n d

Original: [Figure]

Original: m

Original: [Figure]

Original: e

Original: N

Original: [Figure]

Original: K Typographic attacks

Original: In this section we present more details on our typographic attacks experiments presented in Section 7.

Original: We first discuss our procedure for creating synthetic typographic attacks.

Original: We then outline our real world data collection scheme.

Original: Finally we present additional experimental details.

Original: Synthetic dataset details.

Original: Starting with the SUN397 [84] dataset, we procedurally add text to images as seen in Figure 32.

Original: We resize the shorter dimension to 224 pixels using bicubic interpolation and take a 224 pixel by 224 pixel center crop, which is the standard CLIP resize and crop augmentation [57].

Original: We randomize over three fonts: Courier, Helvetica and Times.

Original: For font size, we randomly sample over the range 20 to 40 points.

Original: We consider eight colors, and sample uniformly from them: red, green, blue, cyan, magenta, yellow, white, and black.

Original: To make sure font is visible, we outline text with apoint shadow that is a different color than the main font color.

Original: The text is randomly placed in the image with checks to ensure that the text fits completely on the image.

Original: The text specifies an incorrect class label, chosen at random.

Original: For instance, if the ground truth label from an image is “raceway”, the text content will be uniformly sampled from the 397 − 1 = 396 other options.

Original: We apply this typographic attack procedure on the originally SUN397 train and test sets to create attacked train and test sets.

Original: Real dataset details.

Original: To test if patching on synthetic typographic attacks transfers to real-world usecases, we generate a real-world test set of typographic attacks.

Original: We provide the following instructions to in-house annotators: 1) Write the name of an object on a sticky note, then place the sticky note on an object that is not the one that you have written.

Original: For instance, you can write “desk” on the sticky note then put it on a mug.

Original: The objects should be reasonable categories (e.g., common objects, animals, etc.).

Original: 2) Take a picture—make sure that the underlying object is centered and the text is visible.

Original: We acquire consent from each annotator to use their images for research purposes and to release data publicly.

Original: No faces or other identifying human characteristics are present in the dataset.

Original: We show images from our dataset in Figure 33, highlighting the category diversity.

Original: As highlights: calipers are attacked with text saying “ruler,” an artichoke is attacked with text saying “pineapple,” and a GPU is attacked with text saying "CPU."

Original: Critically, none of the object categories—either in the image or in the text attack—overlap with those in SUN397.

Original: This allows us to study typographic attacks in the broad transfer setting (see Section 6 for more details).

Original: Experimental details.

Original: We first fine-tune on the attacked version of SUN397, using our standard fine-tuning setup discussed in Section 2.

Original: We then evaluate interpolated models on attacked SUN and real-world test sets.

Original: For SUN397, this amounts to standard multi-class classification as the class space is fixed.

Original: For real typographic attacks, the task is binary classification between the image category and the text category.

Original: The classification head for each image is generated using the frozen

Original: CLIP text encoder.

Original: L Counting

Original: We discuss specifics about the dataset and experimental setup for evaluating broad transfer for object counting.

Original: Dataset details.

Original: We consider the CLEVR dataset [32] with annotations for the number of objects per image.

Original: We split the dataset based on its classes into train and test sets as seen in Figure 34.

Original: By training on one split and testing on the other with unseen classes, we are able to evaluate broad transfer for this counting task.

Original: Experimental details.

Original: The patching task is a 5-way classification task.

Original: The evaluation task is a 3-way classification task, with a different set of classes.

Original: Again the head for the test-time task is created zero-shot using the CLIP text encoder.

Original: Full interpolation curves for transfer and the original task are presented in Figure 35.

Original: M Visual Question Answering

Original: For multiple-choice visual question answering, we use CLIP to contrast images with each candidate answer.

Original: For each candidate answer, we construct a prompt that also includes the question following the template “Question: [question text]

Original: Answer: [answer text]”.

Original: This text is fed to CLIP’s text encoder, which remains frozen during the patching process.

Original: During fine-tuning, the weights of the vision encoder are updated using a contrastive loss: the feature similarity with the text features of the correct prompt are maximized with respect to the other candidates.

Original: We patch and evaluate on

Original: multiple-choice

Original: VQA v1 [4], where each question is associated with 18 candidate answers.

Original: Interestingly, 87% of the prompts used for evaluation are not present in the training data.

Original: The results are shown in Figure 36.

Original: Patching improves VQA performance by 13 to 18 percentage points, while reducing ImageNet accuracy by less than one percentage point.

Original: N Computational resources

Original: For all of our experiments, we used NVIDIA A-40 GPUs with 46GB of RAM from an internal cluster.

Original: We estimate the total amount of compute used for the experiments in this paper is around 10 thousand

Original: GPU hours.

Original: found unfreezing the text encoder improved accuracy, but we present results with an frozen text encoder for experimental consistency.

Original: [Figure]

Original: CLEVR Counting images and splits.

Original: Each image has a number of objects betweenand 10.

Original: For a patching (train) task, models are fine-tuned on images with 4, 5, 6, 8 or 9 objects.

Original: We test on images with 3, 7 or 10 objects to evaluate the model’s interpolation and extrapolation of the class space.

Original: [Figure]

Original: [Figure]

Original: O Tabular results

Original: We now present numerical results for the main experiments presented in this paper.

Original: Tables 6 topresent results for patching on a single task, for various supported tasks.

Original: Table 11 details results for various strategies for patching on multiple tasks

Original: [Figure]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: e

Original: N e t

Original: C a r s

Original: D T D

Original: E u r o S

Original: A T

Original: G T S R

Original: B

Original: K I TT

Original: I M N I S T R E S

Original: I S C

Original: S U N

Original: S V H N

Original: A v e r a

Original: [Table]

Original: e

Original: [Table]