Original: Feed-Forward Source-Free Latent Domain Adaptation via Cross-Attention
Revised: Feed-Forward Source-Free Latent Domain Adaptation via Cross-Attention

Original: Abstract
Revised: Abstract

Original: We study the highly practical but comparatively under-studied problem of latent-domain adaptation, where a source model should be adapted to a target dataset that contains a mixture of unlabelled domain-relevant and domain-irrelevant examples.
Revised: We study the highly practical but comparatively under-studied problem of latent-domain adaptation, where a source model should be adapted to a target dataset that contains a mixture of unlabelled domain-relevant and domain-irrelevant examples.

Original: Motivated by the requirements for data privacy and the need for embedded and resourceconstrained devices of all kinds to adapt to local data distributions, we further focus on the setting of feed-forward source-free domain adaptation, where adaptation should not require access to the source dataset, and also be back propagation-free.
Revised: Motivated by the requirements for data privacy and the need for embedded and resourceconstrained devices of all kinds to adapt to local data distributions, we further focus on the setting of feed-forward source-free domain adaptation, where adaptation should not require access to the source dataset, and also be back propagation-free.

Original: Our solution is to meta-learn a network capable of embedding the mixed-relevance target dataset and dynamically adapting inference for target examples using cross-attention.
Revised: Our solution is to meta-learn a network capable of embedding the mixed-relevance target dataset and dynamically adapting inference for target examples using cross-attention.

Original: The resulting framework leads to consistent strong improvements.
Revised: The resulting framework leads to consistent strong improvements.

Original: 1. Introduction
Revised: 1. Introduction

Original: Domain shift presents a real-world challenge for applying pre-trained models because performance degrades when deployment data are not from the training data distribution.
Revised: Domain shift presents a real-world challenge for applying pre-trained models because performance degrades when deployment data are not from the training data distribution.

Original: For example, a model that has been only trained on day-time images will perform poorly when presented with night-time images.
Revised: For example, a model that has been only trained on day-time images will perform poorly when presented with night-time images.

Original: This issue is ubiquitious as it is often impossible or prohibitively costly to pre-collect and annotate training data that is sufficiently representative of test data statistics.
Revised: This issue is ubiquitious as it is often impossible or prohibitively costly to pre-collect and annotate training data that is sufficiently representative of test data statistics.

Original: The field of domain adaptation  (Kouw & Loog, 2021; Csurka et al., 2022) has therefore attracted a lot of attention with the promise of adapting models during deployment to perform well using only unlabeled deployment data.
Revised: The field of domain adaptation (DA) (Kouw & Loog, 2021; Csurka et al., 2022) has therefore attracted a lot of attention with the promise of adapting models during deployment to perform well using only unlabeled deployment data.

Original: We make two main contributions:
Revised: We make two main contributions:

Original: A conceptual contribution, framing domain adaptation in a new highly practical way; and an algorithm for effective domain adaptation in these conditions., Anonymous City, Anonymous Region, Anonymous Country. Correspondence to:
Revised: A conceptual contribution, framing domain adaptation in a new highly practical way; and an algorithm for effective domain adaptation in these conditions.

Original: Author anon

Original: Preliminary work.

Original: Under review by the First Workshop of Pre- training: Perspectives, Pitfalls, and Paths Forward at ICML 2022.

Original: Do not distribute.

Original: Latent domain adaptation
Revised: Latent domain adaptation

Original: While domain adaptation is now very well studied (Kouw & Loog, 2021; Csurka et al., 2022), the vast majority of work assumes that suitable meta-data is available in order to correctly group instances into one or more subsets (domains) that differ statistically across groups, while being similar within groups.
Revised: While domain adaptation is now very well studied (Kouw & Loog, 2021; Csurka et al., 2022), the vast majority of work assumes that suitable meta-data is available in order to correctly group instances into one or more subsets (domains) that differ statistically across groups, while being similar within groups.

Original: We join a growing minority (Mancini et al., 2021;
Revised: We join a growing minority (Mancini et al., 2021;

Original: Deecke et al., 2022;
Revised: Deecke et al., 2022;

Original: Hoffman et al., 2014; Wang et al., 2022) in arguing that this is an overly restrictive assumption that does not hold in most real applications of interest.
Revised: Hoffman et al., 2014; Wang et al., 2022) in arguing that this is an overly restrictive assumption that does not hold in most real applications of interest.

Original: On one hand some datasets or collection processes may not provide meta-data suitable for defining domain groupings.
Revised: On one hand some datasets or collection processes may not provide meta-data suitable for defining domain groupings.

Original: Alternatively, for other data sources that occur with rich meta-data there may be no obviously correct grouping and existing domain definitions may be sub-optimal (Deecke et al., 2022).
Revised: Alternatively, for other data sources that occur with rich meta-data there may be no obviously correct grouping and existing domain definitions may be sub-optimal (Deecke et al., 2022).

Original: Consider the popular iWildCam (Beery et al., 2020) benchmark for animal detection within the WILDS (Koh et al., 2021) suite.
Revised: Consider the popular iWildCam (Beery et al., 2020) benchmark for animal detection within the WILDS (Koh et al., 2021) suite.

Original: The default setup within WILDS defines domains by camera ID.
Revised: The default setup within WILDS defines domains by camera ID.

Original: But given that images span different weather conditions and day/night cycles, such domains may neither be internally homogenous, nor similarly distinct.
Revised: But given that images span different weather conditions and day/night cycles, such domains may neither be internally homogenous, nor similarly distinct.

Original: For example there may be more transferability between images from nearby cameras at similar times of day than between images from the same camera taken on a sunny day vs a snowy night.
Revised: For example there may be more transferability between images from nearby cameras at similar times of day than between images from the same camera taken on a sunny day vs a snowy night.

Original: As remarked by some (Hoffman et al., 2014; Wang et al., 2022), domains may more naturally define a continuum, rather than discrete groups.
Revised: As remarked by some (Hoffman et al., 2014; Wang et al., 2022), domains may more naturally define a continuum, rather than discrete groups.

Original: And that continuum may even be multi-dimensional – such as timestamp of image and spatial proximity of cameras.
Revised: And that continuum may even be multi-dimensional – such as timestamp of image and spatial proximity of cameras.

Original: In this paper, we propose a flexible formulation of the domain adaptation problem that can span all these situations where domains are hard to define, while aligning with the requirements of real use cases.
Revised: In this paper, we propose a flexible formulation of the domain adaptation problem that can span all these situations where domains are hard to define, while aligning with the requirements of real use cases.

Original: Feed-forward and source-free conditions
Revised: Feed-forward and source-free conditions

Original: Unsupervised domain adaptation aims to adapt models from source datasets (e.g. ImageNet) to the peculiarities of specific data distributions in the wild.
Revised: Unsupervised domain adaptation aims to adapt models from source datasets (e.g. ImageNet) to the peculiarities of specific data distributions in the wild.

Original: The mainstream line of work here uses labelled source domain data alongside unlabelled target domain data and updates the model so it performs well on the target domain using back-propagation (Kouw & Loog, 2021; Csurka et al., 2022).
Revised: The mainstream line of work here uses labelled source domain data alongside unlabelled target domain data and updates the model so it performs well on the target domain using back-propagation (Kouw & Loog, 2021; Csurka et al., 2022).

Original: However, the key use cases motivating domain adaptation are edge devices such as autonomous vehicles, smartphones and hospital scanners.
Revised: However, the key use cases motivating domain adaptation are edge devices such as autonomous vehicles, smartphones and hospital scanners.

Original: Storing and processing large source datasets on such devices is usually infeasible.
Revised: Storing and processing large source datasets on such devices is usually infeasible.

Original: This has led a growing number of studies to investigate the source-free condition (Liang et al., 2020), where a pre-trained model is distributed and adapted using solely unlabeled target data.
Revised: This has led a growing number of studies to investigate the source-free condition (Liang et al., 2020), where a pre-trained model is distributed and adapted using solely unlabeled target data.

Original: In this paper, we go further in considering the practical requirements of an edge device, namely that most edge devices are not designed in either hardware or software stack to support back-propagation.
Revised: In this paper, we go further in considering the practical requirements of an edge device, namely that most edge devices are not designed in either hardware or software stack to support back-propagation.

Original: This leads us to focus on the feed-forward condition where adaptation algorithms should proceed using only feed-forward operations.
Revised: This leads us to focus on the feed-forward condition where adaptation algorithms should proceed using only feed-forward operations.

Original: For example, simply updating batch normalisation statistics, which can be done without back-propagation, provides a strong baseline for adaptation (Schneider et al., 2020; Zhang et al., 2021).
Revised: For example, simply updating batch normalisation statistics, which can be done without back-propagation, provides a strong baseline for adaptation (Schneider et al., 2020; Zhang et al., 2021).

Original: Feed-forward source-free latent domain adaptation
Revised: Feed-forward source-free latent domain adaptation

Original: Bringing these ideas together, we envisage a setup where edge devices maintain an unlabeled target dataset that need not be a cleanly meta-data induced domain in the conventional sense, but which may contain examples relevant to the inference of test instances.
Revised: Bringing these ideas together, we envisage a setup where edge devices maintain an unlabeled target dataset that need not be a cleanly meta-data induced domain in the conventional sense, but which may contain examples relevant to the inference of test instances.

Original: Instances in the target set may be of varied relevance to a given test instance.
Revised: Instances in the target set may be of varied relevance to a given test instance.

Original: For example, if true instance relevance is a function of timestamp similarity.
Revised: For example, if true instance relevance is a function of timestamp similarity.

Original: These target examples should then drive model adaptation on the fly, leveraging neither source data, nor back-propagation.
Revised: These target examples should then drive model adaptation on the fly, leveraging neither source data, nor back-propagation.

Original: To solve the challenge posed above, we propose a feedforward adaptation framework based on cross-attention between test instances and the target set.
Revised: To solve the challenge posed above, we propose a feedforward adaptation framework based on cross-attention between test instances and the target set.

Original: The cross-attention module is meta-learned based on a set of training domains, inspired by (Zhang et al., 2021).
Revised: The cross-attention module is meta-learned based on a set of training domains, inspired by Zhang et al.

Original: During deployment it flexibly enables each inference operation to draw upon any part of the target set, exploiting each target instance to a continuous degree.
Revised: During deployment it flexibly enables each inference operation to draw upon any part of the target set, exploiting each target instance to a continuous degree.

Original: For example, this could potentially exclude transfer from target instances that would be conventionally in-domain (e.g., same camera/opposite time of day example earlier), include transfer from target instances that would conventionally be out-of-domain (e.g., similar images/different camera example earlier), and continuously weight similarity to each target image (e.g., temporal distance of images taken in sequence).
Revised: For example, this could potentially exclude transfer from target instances that would be conventionally in-domain (e.g., same camera/opposite time of day example earlier), include transfer from target instances that would conventionally be out-of-domain (e.g., similar images/different camera example earlier), and continuously weight similarity to each target image (e.g., temporal distance of images taken in sequence).

Original: Our experiments show that our cross-attention approach provides useful adaptation in this highly practical setting across a variety of synthetic and real benchmarks.
Revised: Our experiments show that our cross-attention approach provides useful adaptation in this highly practical setting across a variety of synthetic and real benchmarks.

Original: 2. Methods 2.1. Set-up Preliminaries
Revised: 2. Methods 2.1. Set-up Preliminaries

Original: During deployment the high-level assumption made by many source-free domain adaptation frameworks is that we are provided with a predictive model f ψ and an unlabeled target dataset x s whose label-space is the same as that of the pre-trained model (Liang et al., 2020).
Revised: During deployment the high-level assumption made by many source-free domain adaptation frameworks is that we are provided with a predictive model f ψ and an unlabeled target dataset x s whose label-space is the same as that of the pre-trained model (Liang et al., 2020).

Original: Given these, source-free DA approaches define some algorithm A that ultimately leads to classifying a test instance x q as y q ≈ ˆ y q
Revised: Given these, source-free DA approaches define some algorithm A that ultimately leads to classifying a test instance x q as y q ≈ ˆ y q

Original: = A ( x q , x s , ψ ) .
Revised: = A ( x q , x s , ψ ) .

Original: There are numerous existing algorithms for this.
Revised: There are numerous existing algorithms for this.

Original: For example, pseudo-label strategies (Liang et al., 2020; Li et al., 2020; Yang et al., 2021) proceed by estimating labels ˆ y s for the target set x s , treating these as ground-truth, back-propagating to update the model ψ ′ such that it predicts ˆ y s , and then classifying the test point as f ψ ′ ( x q ) .
Revised: For example, pseudo-label strategies (Liang et al., 2020; Li et al., 2020; Yang et al., 2021) proceed by estimating labels ˆ y s for the target set x s , treating these as ground-truth, back-propagating to update the model ψ ′ such that it predicts ˆ y s , and then classifying the test point as f ψ ′ ( x q ) .

Original: We address the feed-forward setting where algorithm A should not use back-propagation.
Revised: We address the feed-forward setting where algorithm A should not use back-propagation.

Original: For example,
Revised: For example,

Original: BN-based approaches (Schneider et al., 2020; Zhang et al., 2021) use the target set x
Revised: BN-based approaches (Schneider et al., 2020; Zhang et al., 2021) use the target set x

Original: s to update the BN statistics in ψ as ψ ′ and then classify the test point as f ψ ′ ( x q ) .
Revised: s to update the BN statistics in ψ as ψ ′ and then classify the test point as f ψ ′ ( x q ) .

Original: While the conventional domain adaptation setting assumes that x q and x s are all drawn from a common distribution, the latent domain assumption has no such requirement.
Revised: While the conventional domain adaptation setting assumes that x q and x s are all drawn from a common distribution, the latent domain assumption has no such requirement.

Original: For example, x s may be drawn from a mixture distribution and x q may be drawn from only one component of that mixture.
Revised: For example, x s may be drawn from a mixture distribution and x q may be drawn from only one component of that mixture.

Original: In this case only a subset of elements in x s may be relevant to adapting the inference for x q .
Revised: In this case only a subset of elements in x s may be relevant to adapting the inference for x q .

Original: Deployment phase
Revised: Deployment phase

Original: Rather than explicitly updating model parameters, we aim to define a flexible inference routine f ψ that processes both x q and x s to produce ˆ y q in a feedforward manner, i.e., ˆ y q
Revised: Rather than explicitly updating model parameters, we aim to define a flexible inference routine f ψ that processes both x q and x s to produce ˆ y q in a feedforward manner, i.e., ˆ y q

Original: =
Revised: =

Original: A ( x q , x s , ψ ) =
Revised: A ( x q , x s , ψ ) =

Original: f ψ ( x q , x s ) .
Revised: f ψ ( x q , x s ) .

Original: In this regard our inference procedure follows a similar flow to variants of ARM (Zhang et al., 2021), with the following key differences: (1) ARM is transductive: it processes a batch of instances at once without distinguishing test instances and target adaptation set, so elements x q are members of x s .
Revised: In this regard our inference procedure follows a similar flow to variants of Adaptive Risk Minimization (ARM) (Zhang et al., 2021), with the following key differences: (1) ARM is transductive: it processes a batch of instances at once without distinguishing test instances and target adaptation set, so elements x q are members of x s .

Original:  ARM makes the conventional domain-observed assumption that domains have been defined by an external process that ensures all x q and x s are drawn from the same distribution.
Revised: (2) ARM makes the conventional domain-observed assumption that domains have been defined by an external process that ensures all x q and x s are drawn from the same distribution.

Original: We do not make this assumption and require robustness to irrelevant elements in x s .
Revised: We do not make this assumption and require robustness to irrelevant elements in x s .

Original: Training phase
Revised: Training phase

Original: To train a model than can be used as described above, we follow an episodic meta-learning paradigm (Zhang et al., 2021; Hospedales et al., 2021).
Revised: To train a model than can be used as described above, we follow an episodic meta-learning paradigm (Zhang et al., 2021; Hospedales et al., 2021).

Original: This refers to training f ψ using a set of simulated domain adaptation tasks.
Revised: This refers to training f ψ using a set of simulated domain adaptation tasks.

Original: At each iteration, we generate a task with a unique pair of query and support instances ( x s , ( y q , x q )) keeping label space the same across all tasks.
Revised: At each iteration, we generate a task with a unique pair of query and support instances ( x s , ( y q , x q )) keeping label space the same across all tasks.

Original: We simulate training episodes where x s contains instances with varying relevance to x q .
Revised: We simulate training episodes where x s contains instances with varying relevance to x q .

Original: The goal is for f ψ to learn how to select and exploit instances from x s in order to adapt inference for x q to better predict y q .
Revised: The goal is for f ψ to learn how to select and exploit instances from x s in order to adapt inference for x q to better predict y q .

Original: In particular, our task sampler defines each task as having support examples uniformly sampled across a random set of N D domains, with the query example being from one of these domains.
Revised: In particular, our task sampler defines each task as having support examples uniformly sampled across a random set of N D domains, with the query example being from one of these domains.

Original: More formally, each task can be defined as T = {{ x s, 1 , x s, 2 , . . .
Revised: More formally, each task can be defined as T = {{ x s, 1 , x s, 2 , . . .

Original: x s,N s } , x q , y q } for N s unlabelled support examples x s,.
Revised: x s,N s } , x q , y q } for N s unlabelled support examples x s,.

Original: and query example x q with label y q .
Revised: and query example x q with label y q .

Original: We give an example of a task in Figure 1 for K = 3 domains with N s = 3 support examples and N q = 1 query example.
Revised: We give an example of a task in Figure 1 for K = 3 domains with N s = 3 support examples and N q = 1 query example.

Original: 2.2. Architecture
Revised: 2.2. Architecture

Original: The key to solving the proposed problem is defining an architecture f ψ that can identify and exploit relevant support instances within x s .
Revised: The key to solving the proposed problem is defining an architecture f ψ that can identify and exploit relevant support instances within x s .

Original: Our solution relies on cross-attention between query and support images, as illustrated in Figure 2.
Revised: Our solution relies on cross-attention between query and support images, as illustrated in Figure 2.

Original: We first embed the support and query examples using the same feature extractor, after which we pass the embeddings through the cross-attention module.
Revised: We first embed the support and query examples using the same feature extractor, after which we pass the embeddings through the cross-attention module.

Original: The cross-attention module gives us transformed query examples that are then added to the embeddings of the query examples as a residual connection, after which the classifier makes predictions.
Revised: The cross-attention module gives us transformed query examples that are then added to the embeddings of the query examples as a residual connection, after which the classifier makes predictions.

Original: Cross-attention module
Revised: Cross-attention module

Original: Given a set of support examples x s and query examples x q , we use the feature extractor f θ to extract features f θ ( x s ) , f θ ( x q ) .
Revised: Given a set of support examples x s and query examples x q , we use the feature extractor f θ to extract features f θ ( x s ) , f θ ( x q ) .

Original: Cross-attention module CA ω
Revised: Cross-attention module CA ω

Original: ( f θ ( x s ); f θ ( x q )) parameterized by ω then transforms query embeddings f θ ( x q ) , using support embeddings f θ ( x s ) as keys.
Revised: ( f θ ( x s ); f θ ( x q )) parameterized by ω then transforms query embeddings f θ ( x q ) , using support embeddings f θ ( x s ) as keys.

Original: The output of the cross-attention module is added to the query example features as a residual connection, which is then used by the classifier f ϕ to predict labels of the query examples ˆ y q = f ϕ
Revised: The output of the cross-attention module is added to the query example features as a residual connection, which is then used by the classifier f ϕ to predict labels of the query examples ˆ y q = f ϕ

Original: ( f θ ( x q ) + CA ω
Revised: ( f θ ( x q ) + CA ω

Original: ( f θ ( x s ); f θ ( x q ))) .
Revised: ( f θ ( x s ); f θ ( x q ))) .

Original: In our notation ψ = { θ , ϕ , ω } .
Revised: In our notation ψ = { θ , ϕ , ω } .

Original: The cross-attention module performs image-to-image crossattention, rather than patch-to-patch.
Revised: The cross-attention module performs image-to-image crossattention, rather than patch-to-patch.

Original: After extracting the features we flatten all spatial dimensions and channels into one vector, which represents the whole image.
Revised: After extracting the features we flatten all spatial dimensions and channels into one vector, which represents the whole image.

Original: Image-toimage attention is more suitable for domain adaptation than patch-based option because the overall representation should better capture the nature of the domain rather than a patch.
Revised: Image-toimage attention is more suitable for domain adaptation than patch-based option because the overall representation should better capture the nature of the domain rather than a patch.

Original: Another benefit of image-to-image attention is efficiency –
Revised: Another benefit of image-to-image attention is efficiency –

Original: Require:
Revised: Require:

Original: # training steps T , # latent domains in a task N D , # support examples N s , # query examples N q , learning rate η 1: Initialize: θ , ϕ , ω 2: for t = 1 , . . .
Revised: # training steps T , # latent domains in a task N D , # support examples N s , # query examples N q , learning rate η 1: Initialize: θ , ϕ , ω 2: for t = 1 , . . .

Original: , T do 3:
Revised: , T do 3:

Original: Sample N D support domains { D s } N D4:
Revised: Sample N D support domains { D s } N D4:

Original: Sample query domain D q from support domains { D s } N D5:
Revised: Sample query domain D q from support domains { D s } N D5:

Original: Sample N s unlabelled support images x s uniformly from the selected support domains { D s } N D6:
Revised: Sample N s unlabelled support images x s uniformly from the selected support domains { D s } N D6:

Original: Sample N q labelled query images x q , y q from domain D q 7:
Revised: Sample N q labelled query images x q , y q from domain D q 7:

Original: Predict ˆ y q = f ϕ
Revised: Predict ˆ y q = f ϕ

Original: ( f θ ( x q ) + CA ω
Revised: ( f θ ( x q ) + CA ω

Original: ( f θ ( x s ); f θ ( x q ))) 8: ( θ , ϕ , ω ) ← ( θ , ϕ , ω ) − η ∇ ( θ , ϕ , ω ) (cid:80) N q k =1 ℓ (ˆ y q,k , y q,k ) 9: end for //
Revised: ( f θ ( x s ); f θ ( x q ))) 8: ( θ , ϕ , ω ) ← ( θ , ϕ , ω ) − η ∇ ( θ , ϕ , ω ) (cid:80) N q k =1 ℓ (ˆ y q,k , y q,k ) 9: end for //

Original: Inference on a new 
Revised: Inference on a new task

Original: Require: θ , ϕ , ω , support x s and query x q examples 10: ˆ y q = f ϕ
Revised: Require: θ , ϕ , ω , support x s and query x q examples 10: ˆ y q = f ϕ

Original: ( f θ ( x q ) + CA ω
Revised: ( f θ ( x q ) + CA ω

Original: ( f θ ( x s ); f θ ( x q )))
Revised: ( f θ ( x s ); f θ ( x q )))

Original: we attend to the whole image rather than patches, making the overall computations manageable even with more images.
Revised: we attend to the whole image rather than patches, making the overall computations manageable even with more images.

Original: Our cross-attention module is parameterized by a set of learnable projection matrices W q , W k , W v (all of size R C × ( C/R ) ) with additional projection matrix W ∈
Revised: Our cross-attention module is parameterized by a set of learnable projection matrices W q , W k , W v (all of size R C × ( C/R ) ) with additional projection matrix W ∈

Original: R ( C/R ) × C to transform the queried outputs (we refer to all of these parameters collectively as ω ).
Revised: R ( C/R ) × C to transform the queried outputs (we refer to all of these parameters collectively as ω ).

Original: The output of the feature extractor f θ is flattened into one vector, giving C channels, so f θ ( x q ) ∈
Revised: The output of the feature extractor f θ is flattened into one vector, giving C channels, so f θ ( x q ) ∈

Original: R N q × C , f θ ( x s ) ∈ R N s × C .
Revised: R N q × C , f θ ( x s ) ∈ R N s × C .

Original: We also specify ratio R that leads to rectangular projection matrices with fewer parameters, improving efficiency and providing regularization.
Revised: We also specify ratio R that leads to rectangular projection matrices with fewer parameters, improving efficiency and providing regularization.

Original: Formally we express CA ω as:
Revised: Formally we express CA ω as:

Original: [Equation]
Revised: [Equation]

Original: Similarly as CrossViT (Chen et al., 2021) and self-attention more broadly, we use multiple heads h and layer norm.
Revised: Similarly as CrossViT (Chen et al., 2021) and self-attention more broadly, we use multiple heads h and layer norm.

Original: 2.3. Meta-Learning
Revised: 2.3. Meta-Learning

Original: We train the model by meta-learning across many tasks.
Revised: We train the model by meta-learning across many tasks.

Original: Meta-learning is efficient here as the inner loop does not include back-propagation based optimization – the adaptation to the support examples is done purely feed-forward.
Revised: Meta-learning is first-order as the inner loop does not include back-propagation based optimization – the adaptation to the support examples is done purely feed-forward.

Original: Both meta-training and inference are summarized in Algorithm 1.
Revised: Both meta-training and inference are summarized in Algorithm 1.

Original: 3. Experiments 3.1. Benchmarks
Revised: 3. Experiments 3.1. Benchmarks

Original: We evaluate our approach on a variety of synthetic and real-world benchmarks, namely FEMNIST (Caldas et al.,[Table]
Revised: We evaluate our approach on a variety of synthetic and real-world benchmarks, namely FEMNIST (Caldas et al.,

Original: [Table]
Revised: [Table]

Original: 2018), CIFAR-10-C (Hendrycks & Dietterich, 2019),
Revised: 2018), CIFAR-10-C (Hendrycks & Dietterich, 2019),

Original: TinyImageNet-C (Hendrycks & Dietterich, 2019) and iWildCam (Beery et al., 2020).
Revised: TinyImageNet-C (Hendrycks & Dietterich, 2019) and iWildCam (Beery et al., 2020).

Original: We follow the splits into metatraining, meta-validation and meta-testing sets as selected in (Zhang et al., 2021) and (Koh et al., 2021).
Revised: We follow the splits into metatraining, meta-validation and meta-testing sets as selected by Zhang et al. (2021) and Koh et al.

Original: 3.2. Baselines
Revised: 3.2. Baselines

Original: We consider three baselines: 1) Empirical risk minimization (ERM) that trains on all training domains and performs no domain adaptation.
Revised: We consider three baselines: 1) Empirical risk minimization (ERM) that trains on all training domains and performs no domain adaptation.

Original: 2) Batch normalization (BN) statistics update that uses support examples to update the statistics.
Revised: 2) Batch normalization (BN) statistics update that uses support examples to update the statistics.

Original: Contextual meta-learning (CML) (Zhang et al., 2021) that extracts information from the support examples using a context network and uses it as additional channels for adaptation.
Revised: Contextual meta-learning (CML) (Zhang et al., 2021) that extracts information from the support examples using a context network and uses it as additional channels for adaptation.

Original: BN and CML assume examples from one domain, rather than mixture of both relevant and irrelevant domains, so it is unclear how successful they will be in LDA.
Revised: BN and CML assume examples from one domain, rather than mixture of both relevant and irrelevant domains, so it is unclear how successful they will be in LDA.

Original: 3.3. Implementation Details
Revised: 3.3. Implementation Details

Original: Our solution - CXDA
Revised: Our solution - CXDA

Original: In line with existing literature (Vaswani et al., 2017; Chen et al., 2021) we use 8 heads and layer normalization on the flattened features of support and query images.
Revised: In line with existing literature (Vaswani et al., 2017; Chen et al., 2021) we use 8 heads and layer normalization on the flattened features of support and query images.

Original: The use of layer normalisation means our approach does not rely on a minibatch of query examples i.e. it natively supports streaming mode and does not need mutiple query examples to obtain strong results, unlike existing test-time domain adaptation approaches (Zhang et al., 2021;
Revised: The use of layer normalisation means our approach does not rely on a minibatch of query examples i.e. it natively supports streaming mode and does not need mutiple query examples to obtain strong results, unlike existing test-time domain adaptation approaches (Zhang et al., 2021;

Original: Wang et al., 2021).
Revised: Wang et al., 2021).

Original: Support images are projected into keys and values, while query images act as queries for cross-attention after transformation by a projection matrix.
Revised: Support images are projected into keys and values, while query images act as queries for cross-attention after transformation by a projection matrix.

Original: After calculating the attention map and applying it to the values, we multiply the output by a further projection matrix.
Revised: After calculating the attention map and applying it to the values, we multiply the output by a further projection matrix.

Original: We use only one cross-attention layer and our projection matrices have rectangular shape of C ×
Revised: We use only one cross-attention layer and our projection matrices have rectangular shape of C ×

Original: C/ 2 where C is the dimensionality of the flattened features.
Revised: C/ 2 where C is the dimensionality of the flattened features.

Original: No dropout is used.
Revised: No dropout is used.

Original: Data augmentation
Revised: Data augmentation

Original: We use weak data augmentation during meta-training – cropping, horizontal flipping, small rotations (up to 30 degrees).
Revised: We use weak data augmentation during meta-training – cropping, horizontal flipping, small rotations (up to 30 degrees).

Original: These are different from the corruptions tested in some of the benchmarks and are applied independently with probability 0.
Revised: These are different from the corruptions tested in some of the benchmarks and are applied independently with probability 0.5.

Original: 5. Task sampling
Revised: Task sampling

Original: Our tasks have 5 support domains, with 20 examples in each, overall 100 support examples.
Revised: Our tasks have 5 support domains, with 20 examples in each, overall 100 support examples.

Original: Query examples come from one randomly selected support set domain and there are 20 of them.
Revised: Query examples come from one randomly selected support set domain and there are 20 of them.

Original: The method fully supports streaming mode, so no statistics are calculated across the minibatch and it works independently for each.
Revised: The method fully supports streaming mode, so no statistics are calculated across the minibatch and it works independently for each.

Original: There are 420, 11000, 11000 and 2125 test tasks for FEMNIST,
Revised: There are 420, 11000, 11000 and 2125 test tasks for FEMNIST,

Original: CIFAR-10-C, TinyImageNet-C and iWildCam respectively.
Revised: CIFAR-10-C, TinyImageNet-C and iWildCam respectively

Original:  Training
Revised: . Training

Original: We follow the hyperparameters used in (Zhang et al., 2021) for FEMNIST, CIFAR-10-C and TinyImageNetC, and we also train the cross-attention parameters with the same optimizer.
Revised: We follow the hyperparameters used by Zhang et al. (2021) for FEMNIST, CIFAR-10-C and TinyImageNetC, and we also train the cross-attention parameters with the same optimizer.

Original: For FEMNIST and CIFAR-10-C a small
Revised: For FEMNIST and CIFAR-10-C a small

Original: CNN model is used, while for TinyImageNet-C a pre-trained
Revised: CNN model is used, while for TinyImageNet-C a pre-trained

Original: ResNet-50 (He et al., 2015) is fine-tuned.
Revised: ResNet-50 (He et al., 2015) is fine-tuned.

Original: For iWildCam we follow the hyperparameters selected in (Koh et al., 2021), but with images resized to 112 × 112 , training for 50 epochs and with mini-batch size resulting from our task design ( support and 20 query examples).
Revised: For iWildCam we follow the hyperparameters selected by Koh et al. , but with images resized to 112 × 112 , training for 50 epochs and with mini-batch size resulting from our task design ( support and 20 query examples).

Original: All our experiments are repeated across three random seeds.
Revised: All our experiments are repeated across three random seeds.

Original: 3.4. Results
Revised: 3.4. Results

Original: We report our results in Table 1 and include both average performance as well as reliability via the worst case performance on the most challenging 10% tasks.
Revised: We report our results in Table 1 and include both average performance as well as reliability via the worst case performance on the most challenging 10% tasks.

Original: From the results we can see our cross-attention approach results in consistent improvements over the strong ERM baseline across all benchmarks, as well as the CML and BN baselines.
Revised: From the results we can see our cross-attention approach results in consistent improvements over the strong ERM baseline across all benchmarks, as well as the CML and BN baselines.

Original: Overall we see CML and BN strategies that naively combine information from all support examples have limited success when the support set has both domain relevant and domain irrelevant examples.
Revised: Overall we see CML and BN strategies that naively combine information from all support examples have limited success when the support set has both domain relevant and domain irrelevant examples.

Original: The results confirm our proposed mechanism based on cross-attention can successfully select useful information from the set of examples with both relevant and irrelevant examples and achieve superior performance.
Revised: The results confirm our proposed mechanism based on cross-attention can successfully select useful information from the set of examples with both relevant and irrelevant examples and achieve superior performance.

Original: 4. Conclusion
Revised: 4. Conclusion

Original: We have introduced a new highly practical setting where we adapt a model using examples that come from a mixture of domains and are without domain or class labels.
Revised: We have introduced a new highly practical setting where we adapt a model using examples that come from a mixture of domains and are without domain or class labels.

Original: To answer this new highly challenging adaptation problem, we have developed a novel solution based on cross-attention that is able to automatically select relevant examples and use them for adaptation on the fly.
Revised: To answer this new highly challenging adaptation problem, we have developed a novel solution based on cross-attention that is able to automatically select relevant examples and use them for adaptation on the fly.