Original: Subsidiary Prototype Alignment for Universal Domain Adaptation
Revised: Subsidiary Prototype Alignment for Universal Domain Adaptation

Original: Anonymous Author s)

Original: Abstract
Revised: Abstract

Original: Universal Domain Adaptation (UniDA) deals with the problem of knowledgetransfer between two datasets with domain-shift as well as category-shift.
Revised: Universal Domain Adaptation (UniDA) deals with the problem of knowledge transfer between two datasets with domain-shift as well as category-shift.

Original: The goalis to categorize unlabeled target samples, either into one of the “known” categoriesor into a single “unknown” category.
Revised: The goal is to categorize unlabeled target samples, either into one of the “known” categories or into a single “unknown” category.

Original: A major problem in UniDA is negativetransfer, i.e . misalignment of “known” and “unknown” classes.
Revised: A major problem in UniDA is negative transfer, i.e . misalignment of “known” and “unknown” classes.

Original: To this end, wefirst uncover an intriguing tradeoff between negative-transfer-risk and domaininvariance exhibited at different layers of a deep network.
Revised: To this end, we first uncover an intriguing tradeoff between negative-transfer-risk and domaininvariance exhibited at different layers of a deep network.

Original: It turns out we can strikea balance between these two metrics at a mid-level layer.
Revised: It turns out we can strike a balance between these two metrics at a mid-level layer.

Original: Towards designing aneffective framework based on this insight, we draw motivation from Bag-of-visualWords (BoW).
Revised: Towards designing an effective framework based on this insight, we draw motivation from Bag-of-visualWords (BoW).

Original: Word-prototypes in a BoW-like representation of a mid-level layerwould represent lower-level visual primitives that are likely to be unaffected by thecategory-shift in the high-level features.
Revised: Word-prototypes in a BoW-like representation of a mid-level layer would represent lower-level visual primitives that are likely to be unaffected by the category-shift in the high-level features.

Original: We develop modifications that encouragelearning of word-prototypes followed by word-histogram based classification.
Revised: We develop modifications that encourage learning of word-prototypes followed by word-histogram based classification.

Original: Following this, subsidiary prototype-space alignment (SPA) can be seen as a closedset alignment problem, thereby avoiding negative transfer.
Revised: Following this, subsidiary prototype-space alignment (SPA) can be seen as a closedset alignment problem, thereby avoiding negative transfer.

Original: We realize this with anovel word-histogram-related pretext task to enable closed-set SPA, operating inconjunction with goal task UniDA.
Revised: We realize this with a novel word-histogram-related pretext task to enable closed-set SPA, operating in conjunction with goal task UniDA.

Original: We demonstrate the efficacy of our approachon top of existing UniDA techniques, yielding state-of-the-art performance acrossthree standard UniDA and Open-Set DA object recognition benchmarks.
Revised: We demonstrate the efficacy of our approach on top of existing UniDA techniques 1 , yielding state-of-the-art performance across three standard UniDA and Open-Set DA object recognition benchmarks.

Original:  Introduction
Revised:  Introduction

Original: Despite the success of deep networks trained on large-scale datasets, they are found to be brittleunder input distribution shift i e .
Revised: Despite the success of deep networks trained on large-scale datasets, they are found to be brittle under input distribution shift i e .

Original: domain-shift [6].
Revised: domain-shift [8].

Original: Thus, adapting a trained model for a new targetenvironment becomes challenging as data annotation is too expensive or time-consuming [7] forevery new target dataset.
Revised: Thus, adapting a trained model for a new target environment becomes challenging as data annotation is too expensive or time-consuming [9] for every new target dataset.

Original: Unsupervised Domain Adaptation (DA) [11] is one of the solutions to thisproblem where knowledge is transferred from a labeled source domain to an unlabeled target domain.
Revised: Unsupervised Domain Adaptation (DA) [13] is one of the solutions to this problem where knowledge is transferred from a labeled source domain to an unlabeled target domain.

Original: While initial works [12, 22] focused on Closed-Set DA, where source and target label sets are shared( C s = C t ), follow-up works introduced disjoint label set scenarios like Partial DA [47, 4] ( C t ⊂ C s )and Open-Set DA [33, 18] ( C s ⊂ C t ).
Revised: While most works [14, 30, 23] focused on Closed-Set DA, where source and target label sets are shared ( C s = C t ), recent works introduced disjoint label set scenarios like Partial DA [58, 6] ( C t ⊂ C s ) and Open-Set DA [43, 21] ( C s ⊂ C t ).

Original: However, the most practical setting is Universal DA (UniDA)[44] where the relation between the source and target label sets is unknown i.e .
Revised: However, the most practical setting is Universal DA (UniDA) [55] where the relation between the source and target label sets is unknown i.e .

Original: with any number ofshared, source-private and target-private classes.
Revised: with any number of shared, source-private and target-private classes.

Original: In UniDA, a model is trained to categorize unlabeledtarget samples into one of the shared classes (“known” classes) or into a single “unknown” class.
Revised: In UniDA, a model is trained to categorize unlabeled target samples into one of the shared classes (“known” classes) or into a single “unknown” class.

Original: The major problem in UniDA is negative-transfer [34] where misalignment between the shared andprivate classes degrades the adaptation performance.
Revised: The major problem in UniDA is negative-transfer [44] where misalignment between the shared and private classes degrades the adaptation performance.

Original: On the other hand, we have the domain-shiftproblem, which is usually handled by learning domain-invariant features [44].
Revised: On the other hand, we have the domain-shift problem, which is usually handled by learning domain-invariant features [55, 22, 25]. So, we first Pretext task to aid SPA [Figure]

Revised: Realized via BoW

Original: So, we first perform acontrol experiment to analyze the negative-transfer-risk (NTR) and domain-invariance-score (DIS)at different layers of the deep network (Fig. 2).
Revised: [Figure] inspired framework [Figure] perform a control experiment to analyze the negative-transfer-risk (NTR) and domain-invariance-score (DIS) at different layers of the deep network (Fig. 2).

Original: NTR is measured through the class-specificity ofthe features via an entropy-based shared-vs-unknown binary classifier while DIS is measured as theinverse of the standard A -distance [1] that quantifies domain discrepancy.
Revised: NTR is measured through the class-specificity of the features via an entropy-based shared-vs-unknown binary classifier while DIS is measured as the inverse of the standard A -distance [1] that quantifies domain discrepancy.

Original: We observe that NTRand DIS are at odds with each other i.e .
Revised: We observe that NTR and DIS are at odds with each other i.e .

Original: NTR increases while DIS decreases as we move fromlower to higher-level (deeper) layers.
Revised: NTR increases while DIS decreases as we move from lower to higher-level (deeper) layers.

Original: Thus, we arrive at contradicting solutions where avoiding

Original: Pretext task to aid SPA

Original: [Figure]

Original: SPA Realized via BoW-inspired framework [Figure] negative-transfer requires adaptation at a lower-level layer while effective domain-invariance requiresadaptation at a deeper layer.
Revised: Thus, we arrive at contradicting solutions where avoiding negative-transfer requires adaptation at a lower-level layer while effective domain-invariance requires adaptation at a deeper layer.

Original: While a balance can be naively struck at a mid-level layer, we ask, canwe further develop and constrain the mid-level representation space to avoid negative transfer?
Revised: While a balance can be naively struck at a mid-level layer, we ask, can we further develop and constrain the mid-level representation space to avoid negative transfer?

Original: Motivated by Bag-of-visual-Words (BoW) [45], we hypothesize that a BoW-like representation at amid-level layer would represent lower-level visual primitives that are unaffected by the category-shiftin the higher-level features.
Revised: Motivated by Bag-of-visual-Words (BoW) [56], we hypothesize that a BoW-like representation at a mid-level layer would represent lower-level visual primitives that are unaffected by the category-shift in the higher-level features.

Original: We illustrate this idea in Fig.
Revised: We illustrate this idea in Fig.

Original: 1 with a UniDA scenario where source andtarget have private and shared classes.
Revised: 1 with a UniDA scenario where source and target have private and shared classes.

Original: Note that, in most practical scenarios, the private categoriesare usually related to the shared categories ( e.g . a tractor and an excavator may be private classes inrural and urban scenes respectively, but they share some common visual attributes like their chassis).
Revised: Note that, in most practical scenarios, the private categories are usually related to the shared categories ( e.g . a tractor and an excavator may be private classes in rural and urban scenes respectively, but they share some common visual attributes like their chassis).

Original: Thus, in a word-prototype-space, different visual primitives can be shared across domains and classes(including unknown classes) and a closed-set alignment of target features with the primitives can beperformed.
Revised: Thus, in a word-prototype-space, different visual primitives can be shared across domains and classes (including unknown classes) and a closed-set alignment of target features with the primitives can be performed.

Original: Next, we explain how to realize this subsidiary prototype-space alignment (SPA).
Revised: Next, we explain how to realize this subsidiary prototype-space alignment (SPA).

Original: First, we propose architecture modifications to introduce explicit word-prototypes and extract a wordhistogram output.
Revised: First, we propose architecture modifications to introduce explicit word-prototypes and extract a wordhistogram output.

Original: We analyze the alignment of feature vectors with different word-prototypes.
Revised: We analyze the alignment of feature vectors with different word-prototypes.

Original: Here,better prototype-space alignment would imply sparser word-histograms as a specific word-prototypewould have a significant contribution to the word-histogram w.r.t.
Revised: Here, better prototype-space alignment would imply sparser word-histograms as a specific word-prototype would have a significant contribution to the word-histogram w.r.t.

Original: other prototypes.
Revised: other prototypes.

Original: Intuitively, withhigher sparsity, the word-histogram space can better represent the intrinsic class structure (includingunknown classes).
Revised: Intuitively, with higher sparsity, the word-histogram space can better represent the intrinsic class structure (including unknown classes).

Original: To enforce this sparsity i.e . the subsidiary prototype-space alignment (SPA),we minimize a self-entropy loss at the word-histogram level.
Revised: To enforce this sparsity i.e . the subsidiary prototype-space alignment (SPA), we minimize a self-entropy loss at the word-histogram level.

Original: However, this remains susceptible toword-level misalignment due to a lack of word-related priors.
Revised: However, this remains susceptible to word-level misalignment due to a lack of word-related priors.

Original: Thus, we seek a word-related prior that can be cast into a self-supervised pretext task so that bothlabeled source and unlabeled target can be used.
Revised: Thus, we seek a word-related prior that can be cast into a self-supervised pretext task so that both labeled source and unlabeled target can be used.

Original: Given that each instance can be represented as aword-histogram, we find a simple property based on grid-shuffling of image crops (Fig. 4).
Revised: Given that each instance can be represented as a word-histogram, we find a simple property based on grid-shuffling of image crops (Fig. 4).

Original: Here, theword-histogram entropy of a grid-shuffled image increases with the number of distinct instances thatcontribute crops.
Revised: Here, the word-histogram entropy of a grid-shuffled image increases with the number of distinct instances that contribute crops.

Original: Based on this, we create a novel pretext task to classify the number of instances usedin an input grid-shuffled image.
Revised: Based on this, we create a novel pretext task to classify the number of instances used in an input grid-shuffled image.

Original: Intuitively, it encourages prototype alignment (SPA) as distinguishingdifferent classes in this pretext task becomes easier with higher sparsity of word-histograms.
Revised: Intuitively, it encourages prototype alignment (SPA) as distinguishing different classes in this pretext task becomes easier with higher sparsity of word-histograms.

Original: To summarize, our contributions are as follows,
Revised: To summarize, our contributions are as follows,

Original: • We are the first to uncover and analyze the tradeoff between negative-transfer and domain-invariance in UniDA.
Revised: • We are the first to uncover and analyze the tradeoff between negative-transfer and domain-invariance in UniDA.

Original: While a naive balance can be struck, we introduce BoW-inspired word-prototypes and a subsidiary prototype-space alignment (SPA) objective to further alleviate negative-transfer.
Revised: While a naive balance can be struck, we introduce BoW-inspired word-prototypes and a subsidiary prototype-space alignment (SPA) objective to further alleviate negative-transfer.

Original: • We devise a word-related prior and cast it as a self-supervised pretext task to further improve SPAby avoiding word-level misalignment.
Revised: • We devise a word-related prior and cast it as a self-supervised pretext task to further improve SPA by avoiding word-level misalignment.

Original: • Our approach, coupled with existing UniDA approaches, yields state-of-the-art performance acrossthree standard Open-Set DA and UniDA benchmarks for object recognition.
Revised: • Our approach, coupled with existing UniDA approaches, yields state-of-the-art performance across three standard Open-Set DA and UniDA benchmarks for object recognition.

Original:  Related work
Revised:  Related work

Original: Open-Set DA (OSDA) has been studied in several scenarios [33, 26, 3, 8].
Revised: Open-Set DA (OSDA) has been studied in several scenarios [43, 34, 3, 10].

Original: We focus on the casegiven by [33], where target domain contains private classes, unknown to source.
Revised: We focus on the case given by [43], where target domain contains private classes, unknown to source.

Original: [33] presentedan adversarial learning method where feature generator learns known-unknown separation.
Revised: [43] presented an adversarial learning method where feature generator learns known-unknown separation.

Original: Other works focus on anomaly measurement [27] or learn to discriminate known and unknown samples[43, 30, 25].
Revised: Other works focus on anomaly measurement [35] or learn to discriminate known and unknown samples [54, 39, 33].

Original: ROS [3] uses rotation prediction to separate known and unknown samples whereas ourword-based pretext task regularizes UniDA and implicitly improves known-unknown separation.
Revised: ROS [3] uses rotation prediction to separate known and unknown samples whereas our word-based pretext task regularizes UniDA and implicitly improves known-unknown separation.

Original: Universal DA (UniDA) [44] is a complex DA scenario that assumes no prior knowledge of therelationship between the source and target label spaces.
Revised: Universal DA (UniDA) [55] is a complex DA scenario that assumes no prior knowledge of the relationship between the source and target label spaces.

Original: Similar to Open-Set DA, UniDA also requiresidentification of target-private classes.
Revised: Similar to Open-Set DA, UniDA also requires identification of target-private classes.

Original: Prior works [10, 44,34, 3] computed a confidence score forknown classes, and data with a score below a threshold were considered unknown.
Revised: Prior works [12, 55, 44, 3] computed a confidence score for known classes, and data with a score below a threshold were considered unknown.

Original: [34] proposedneighbourhood clustering to understand the target domain structure and an entropy separation loss forfeature alignment.
Revised: [44] proposed neighbourhood clustering to understand the target domain structure and an entropy separation loss for feature alignment.

Original: OVANet [32] used binary classifiers in a one-vs-all manner to identify unknownsamples, and DCC [20] used domain consensus knowledge to find discriminative clusters in bothshared and private data.
Revised: OVANet [42] used binary classifiers in a one-vs-all manner to identify unknown samples, and DCC [27] used domain consensus knowledge to find discriminative clusters in both shared and private data.

Original: In contrast to prior arts, we draw motivation from Bag-of-visual-Words andconstruct a pretext task to complement these works by enhancing their intrinsic domain structure.
Revised: In contrast to prior arts, we draw motivation from Bag-of-visual-Words and construct a pretext task to complement these works by enhancing their intrinsic domain structure.

Original: BoW-related works.
Revised: BoW-related works.

Original: Early works utilized Bag-of-visual-Words (BoW) representations for downstream applications like object recognition [40], object detection [2], image retrieval [35], etc .
Revised: Early works utilized Bag-of-visual-Words (BoW) representations for downstream applications like object recognition [51], object detection [2], image retrieval [46], etc .

Original: Morerecent work [13, 14] leveraged BoW prediction for self-supervised learning of representations fordownstream tasks.
Revised: More recent work [15, 16] leveraged BoW prediction for self-supervised learning of representations for downstream tasks.

Original: To the best of our knowledge, we are the first to utilize BoW concepts in UniDA.
Revised: To the best of our knowledge, we are the first to utilize BoW concepts in UniDA.

Original:  Approach3.1 Preliminaries
Revised:  Approach 3.1 Preliminaries

Original: Consider a labeled source dataset D s = { ( x s , y s ) : x s ∈ X , y s ∈ C s } where C s is the source label set,
Revised: Consider a labeled source dataset D s = { ( x s , y s ) : x s ∈ X , y s ∈ C s } where C s is the source label set,

Original: X is the input space, and x s is drawn from the marginal distribution p s .
Revised: X is the input space, and x s is drawn from the marginal distribution p s .

Original: Also consider an unlabeleddataset D t = { x t : x t ∈ X} where x t is drawn from the marginal distribution p t .
Revised: Also consider an unlabeled dataset D t = { x t : x t ∈ X} where x t is drawn from the marginal distribution p t .

Original: Let C t denote thetarget label set.
Revised: Let C t denote the target label set.

Original: In Universal DA [44], the relationship between C s and C t is unknown.
Revised: In Universal DA [55], the relationship between C s and C t is unknown.

Original: Without lossof generality, the shared label set is C =
Revised: Without loss of generality, the shared label set is C =

Original: C s ∩ C t and the private label sets for source and target are
Revised: C s ∩ C t and the private label sets for source and target are

Original: C s = C s \ C t and C t =
Revised: C s = C s \ C t and C t =

Original: C t \ C s respectively.
Revised: C t \ C s respectively.

Original: Next, we define two measures for our insights.
Revised: Next, we define two measures for our insights.

Original: Negative-Transfer-Risk (NTR).
Revised: Negative-Transfer-Risk (NTR).

Original: Negative transfer [39] is a major problem in DA where class-levelmisalignment occurs ( e.g . source class “dog" may get wrongly aligned with target class “cat" due tosome similarities between the two classes).
Revised: Negative transfer [50] is a major problem in DA where class-level misalignment occurs ( e.g . source class “dog" may get wrongly aligned with target class “cat" due to some similarities between the two classes).

Original: The problem is aggravated in UniDA as both source andtarget may have private classes which may be wrongly aligned with the shared classes [44].
Revised: The problem is aggravated in UniDA as both source and target may have private classes which may be wrongly aligned with the shared classes [55].

Original: Thus, weintroduce a negative-transfer-risk (NTR) γ NTR ( h ) for a given feature extractor h : X →Z , where Z isan intermediate feature-space.
Revised: Thus, we introduce a negative-transfer-risk (NTR) γ NTR ( h ) for a given feature extractor h : X →Z , where Z is an intermediate feature-space.

Original: NTR is computed as the target shared-vs-private classification accuracyvia a self-entropy threshold on a source-trained linear task-classifier (see Suppl.
Revised: NTR is computed as the target shared-vs-private classification accuracy via a self-entropy threshold on a source-trained linear task-classifier (see Suppl.

Original: for complete details),
Revised: for complete details),

Original: [Equation]
Revised: [Equation]

Original: Here, ˆ y ( sp ) t is the prediction (shared or private) using a fixed self-entropy threshold and y ( sp ) trepresents shared-private label (0 for shared, 1 for private).
Revised: Here, ˆ y ( sp ) t is the prediction (shared or private) using a fixed self-entropy threshold and y ( sp ) t represents shared-private label (0 for shared, 1 for private).

Original: We access the shared-private labelsfor a subset of target data only for analysis (not for training).
Revised: We access the shared-private labels for a subset of target data only for analysis (not for training).

Original: Intuitively, if the features h ( x ) arehighly class-specific, then target-private samples would yield highly uncertain predictions (as they areunseen by the source-trained linear classifier) compared to shared-label samples.
Revised: Intuitively, if the features h ( x ) are highly class-specific, then target-private samples would yield highly uncertain predictions (as they are unseen by the source-trained linear classifier) compared to shared-label samples.

Original: Thus, target-privatesamples would be easily separable with the self-entropy threshold, leading to a high NTR.
Revised: Thus, target-private samples would be easily separable with the self-entropy threshold, leading to a high NTR.

Original: Domain-Invariance-Score (DIS).
Revised: Domain-Invariance-Score (DIS).

Original: The standard A -distance [1] measures the discrepancy betweentwo domains.
Revised: The standard A -distance [1] measures the discrepancy between two domains.

Original: It is computed using the accuracy of a linear domain classifier (source vs target).
Revised: It is computed using the accuracy of a linear domain classifier (source vs target).

Original: Sincewe aim to measure domain invariance, i.e . the inverse of domain discrepancy, we  the inverse of  source  feature  h 
Revised: Since we aim to measure domain invariance, i.e . the inverse of domain discrepancy, we use the inverse of A -distance between the source and target datasets for the given feature extractor h ,

Original: [Equation]
Revised: [Equation]

Revised: Here, d A ( ., . ) denotes A -distance computed using feature outputs of h .

Revised: Note that 0 ≤ d A ( ., . )

Original:  3.2 Balancing negative-transfer-risk and domain-invariance
Revised: ≤ 2 . 3.2 Balancing negative-transfer-risk and domain-invariance

Original: Given that UniDA is highly susceptible to negative-transfer due to the category-shift problem, weanalyze the negative-transfer-risk at different layers of the deep network.
Revised: Given that UniDA is highly susceptible to negative-transfer due to the category-shift problem, we analyze the negative-transfer-risk at different layers of the deep network.

Original: Here, we consider that shallow layers encode low-level visual features like edges, corners, etc .
Revised: Here, we consider that shallow layers encode low-level visual features like edges, corners, etc .

Original: while deeper layers encodemore abstract, class-specific features [46].
Revised: while deeper layers encode more abstract, class-specific features [57].

Original: In the context of UniDA, the feature space of the deeperlayers for source and target would be more difficult to align due to the disjointness of the source andtarget label sets.
Revised: In the context of UniDA, the feature space of the deeper layers for source and target would be more difficult to align due to the disjointness of the source and target label sets.

Original: We empirically observe the same
Revised: We empirically observe the same

Original: , i.e . NTR increases as we go deeper in the model(solid blue curve in Fig. 2).
Revised: , i.e . NTR increases as we go deeper in the model (solid blue curve in Fig. 2).

Original: This suggests that adaptation should be performed at a shallower layer.
Revised: This suggests that adaptation should be performed at a shallower layer.

Original: [Figure]
Revised: [Figure]

Original:  While we have considered the category-shiftproblem to understand where to perform theadaptation, we cannot overlook the domain-shiftproblem.
Revised: decreases While we have considered the category-shift problem to understand where to perform the adaptation, we cannot overlook the domain-shift problem.

Original: Most DA works [12] perform adaptation at deeper layers, which tend to learn increasingly more domain-specific features [16, 36].
Revised: Most DA works [14] perform adaptation at deeper layers, which tend to learn increasingly more domain-specific features [47, 24].

Original: The higher capacity of the deeper layers leadsto unregularized domain-specific learning.
Revised: The higher capacity of the deeper layers leads to unregularized domain-specific learning.

Original: Wealso empirically observe that DIS (inverse ofdomain-specificity) decreases for the deeper layers (solid pink curve in Fig. 2).
Revised: We also empirically observe that DIS (inverse of domain-specificity) decreases for the deeper layers (solid pink curve in Fig. 2).

Original: This suggeststhat adaptation should be performed at a deeperlayer to encourage domain-invariance.
Revised: This suggests that adaptation should be performed at a deeper layer to encourage domain-invariance.

Original: Thus, the problems of domain-shift andcategory-shift are contradictory in suggestingadaptation at a deeper and shallower layer respectively.
Revised: Thus, the problems of domain-shift and category-shift are contradictory in suggesting adaptation at a deeper and shallower layer respectively.

Original: In other words, there is a tradeoffbetween DIS and NTR (Fig. 2) where reducingthe negative-transfer-risk negatively affects the domain-invariance and vice versa.
Revised: In other words, there is a tradeoff between DIS and NTR (Fig. 2) where reducing the negative-transfer-risk negatively affects the domain-invariance and vice versa.

Original: Thus, we define acriterion for minimal negative-transfer and maximal domain-invariance.
Revised: Thus, we define a criterion for minimal negative-transfer and maximal domain-invariance.

Original: Definition 1.
Revised: Definition 1.

Original: (Optimal tradeoff between negative-transfer and domain-invariance) Considerthat adaptation is performed at the L th layer of the backbone h (let h L denote the backbone upto
Revised: (Optimal tradeoff between negative-transfer and domain-invariance) Consider that adaptation is performed at the L th layer of the backbone h (let h L denote the backbone upto

Original: L th layer).
Revised: L th layer).

Original: Then adaptation at h L will encounter minimal negative-transfer and exhibit maximumdomain-invariance if with at least (1 − δ ) probability, γ NTR ( h L ) does not exceed ζ n by more than ε n ,and γ DIS ( h L ) exceeds ζ d by no less than ε d , i.e .,
Revised: Then adaptation at h L will encounter minimal negative-transfer and exhibit maximum domain-invariance if with at least (1 − δ ) probability, γ NTR ( h L ) does not exceed ζ n by more than ε n , and γ DIS ( h L ) exceeds ζ d by no less than ε d , i.e .,

Original: [Equation]
Revised: [Equation]

Original: Thus, an optimal tradeoff requires NTR to be less than the threshold ζ n and DIS to be greater than thethreshold ζ d simultaneously.
Revised: Thus, an optimal tradeoff requires NTR to be less than the threshold ζ n and DIS to be greater than the threshold ζ d simultaneously.

Original: Empirically, we find a good tradeoff at a mid-level layer (green verticalline in Fig.
Revised: Empirically, we find a good tradeoff at a mid-level layer (green vertical line in Fig.

Original: 2, e.g . Res3 block in ResNet), i.e . a compromise between the contradicting suggestions.
Revised: 2, e.g . Res3 block in ResNet), i.e . a compromise between the contradicting suggestions.

Original: Note that we re-use h to represent the backbone upto an optimal layer L instead of h L for simplicity. 
Revised: Note that we re-use h to represent the backbone upto an optimal layer L instead of h L for simplicity. Why is low NTR desirable?

Revised: High NTR implies known and unknown samples are well-separated.

Revised: However, unsupervised adaptation at a higher NTR layer is more susceptible to misalignment between shared and private (unknown) classes because target-private classes get grouped into a single unknown cluster.

Revised: In contrast, lower NTR feature space can better represent all the different classes without grouping the target-private classes into a single unknown cluster ( i.e . better intrinsic structure).

Original: 3.3 Conceptualizing Bag-of-visual-Words (BoW) for UniDA
Revised: Hence, alignment in this space would better respect the separations of private classes than at a higher NTR feature space, which is necessary to avoid misalignment in UniDA. For example, consider “hatchback" (compact car) and “SUV" (large-sized car) as a shared and target-private class, respectively, in an object recognition task. Before adaptation, at a high NTR layer, hatchback and SUV features would be well-separated as SUV is yet unseen to the source-trained model. However, during unsupervised adaptation, the similarities between hatchbacks and SUVs may align the single target-private cluster (containing SUV features) with the hatchback cluster.

Revised: Due to this, other target-private classes also become closer to this hatchback class which increases the misalignment.

Revised: In contrast, at a lower NTR layer, the target-private classes (including SUV) would not be grouped together.

Revised: Hence, misalignment of hatchback and SUV clusters would not disturb other clusters unlike the higher NTR scenario.

Revised: 3.3 Conceptualizing Bag-of-visual-Words (BoW) for UniDA

Original: Now we ask whether a meaningful representation space can be developed at this mid-level layer toeffectively mitigate negative-transfer.
Revised: Now we ask whether a meaningful representation space can be developed at this mid-level layer to effectively mitigate negative-transfer.

Original: To this end, we draw motivation from traditional Bag-of-visualWords (BoW) concepts.
Revised: To this end, we draw motivation from traditional Bag-of-visualWords (BoW) concepts.

Original: Consider a vocabulary V
Revised: Consider a vocabulary V

Original: =
Revised: =

Original: [v 1 , v 2 , . .
Revised: [v 1 , v 2 , . .

Original: . , v K ] where each word-prototypev k ∈ R N d ∀ k ∈ { 1 , 2 , . .
Revised: . , v K ] where each word-prototype v k ∈ R N d ∀ k ∈ { 1 , 2 , . .

Original: , K } ,
Revised: , K } ,

Original: i.e . K word-prototypes where each v k is a N d -dimensional vector.
Revised: i.e . K word-prototypes where each v k is a N d -dimensional vector.

Original: Here, the word-prototypes are representative of lower-level visual primitives [45] ( e.g . SIFT-likefeatures).
Revised: Here, the word-prototypes are representative of lower-level visual primitives [56] ( e.g . SIFT-like features).

Original: Under the BoW idea [41], a histogram of word-prototypes ( word-histogram ) can be used asa representation of an image for downstream applications.
Revised: Under the BoW idea [52], a histogram of word-prototypes ( word-histogram ) can be used as a representation of an image for downstream applications.

Original: Through the following insight, we arguethat BoW concepts can be leveraged in UniDA, for the problems arising from disjoint label sets.
Revised: Through the following insight, we argue that BoW concepts can be leveraged in UniDA, for the problems arising from disjoint label sets.

Original: Insight 1.
Revised: Insight 1.

Original: (Suitability of BoW concepts for UniDA)
Revised: (Suitability of BoW concepts for UniDA)

Original: Word-prototypes represent lower-level visualprimitives which are largely unaffected by category-shift in the high-level features.
Revised: Word-prototypes represent lower-level visual primitives which are largely unaffected by category-shift in the high-level features.

Original: Thus, subsidiaryclosed-set word-prototype-space alignment assists UniDA with minimal negative-transfer-risk (NTR).
Revised: Thus, subsidiary closed-set word-prototype-space alignment assists UniDA with minimal negative-transfer-risk (NTR).

Original: Remarks.
Revised: Remarks.

Original: We conceptually illustrate this in Fig.
Revised: We conceptually illustrate this in Fig.

Original: Usually, the private categories are somewhatrelated to the shared categories.
Revised: Usually, the private categories are somewhat related to the shared categories.

Original: Thus, we hypothesize that a set of lower-level visual primitives (orword-prototypes) are capable of representing all categories (even unknown) in the word-prototypespace shared across domains and classes.
Revised: Thus, we hypothesize that a set of lower-level visual primitives (or word-prototypes) are capable of representing all categories (even unknown) in the word-prototypespace shared across domains and classes.

Original: In Fig.
Revised: In Fig.

Original: 1, before adaptation, different target features arescattered around the word-prototypes.
Revised: 1, before adaptation, different target features are scattered around the word-prototypes.

Original: Then, performing closed-set alignment between the featuresand word-prototypes can better capture the intrinsic class-structure to support UniDA.
Revised: Then, performing closed-set alignment between the features and word-prototypes can better capture the intrinsic class-structure to support UniDA.

Original: 3.4 Subsidiary Prototype-space Alignment (SPA)
Revised: 3.4 Subsidiary Prototype-space Alignment (SPA)

Original: An obvious question remains: How to realize the closed-set word-prototype-space alignment described in Insight 1?
Revised: An obvious question remains: How to realize the closed-set word-prototype-space alignment described in Insight 1?

Original: This is crucial because word-prototypes are abstract concepts which need to beexplicitly realized.
Revised: This is crucial because word-prototypes are abstract concepts which need to be explicitly realized.

Original: To this end, we propose minor architectural modifications in the backbone.
Revised: To this end, we propose minor architectural modifications in the backbone.

Original: We insert a block ψ after the backbone h (see Fig. 3).
Revised: We insert a block ψ after the backbone h (see Fig. 3).

Original: First, a 1 × 1 -conv.
Revised: First, a 1 × 1 -conv.

Original: layer converts the N d -dimensional spatial features to K -dimensional spatial features (same spatial size as a 1 × 1 filteris used with stride 1).
Revised: layer converts the N ddimensional spatial features to K -dimensional spatial features (same spatial size as a 1 × 1 filter is used with stride 1).

Original: We interpret its weight matrix ( V ∈ R N d × K ) as a vocabulary containing Knumber of N d -dimensional word-prototypes.
Revised: We interpret its weight matrix ( V ∈ R N d × K ) as a vocabulary containing K number of N d -dimensional word-prototypes.

Original: The softmax activation performs soft-quantization ofthe input features h ( x ) w.r.t.
Revised: The softmax activation performs soft-quantization of the input features h ( x ) w.r.t.

Original: the word-prototypes in V .
Revised: the word-prototypes in V .

Original: [Figure]
Revised: [Figure]

Original: Consider h u ( x ) ∈ R N d , the feature vector at aspatial location u ∈ { 1 , 2 , . .
Revised: Consider h u ( x ) ∈ R N d , the feature vector at a spatial location u ∈ { 1 , 2 , . .

Original: , U } (where U isthe number of spatial locations in the features) and the vocabulary V =[ v 1 , v 2 , . .
Revised: , U } (where U is the number of spatial locations in the features) and the vocabulary V =[v 1 , v 2 , . .

Original: . , v K ] wherev k represents the k th word-prototype.
Revised: . , v K ] where v k represents the k th word-prototype.

Original: Then, thesoft-quantization (soft equivalent of number ofoccurrences in a word-histogram) at a spatiallocation u for the k th word-prototype v k is,
Revised: Then, the soft-quantization (soft equivalent of number of occurrences in a word-histogram) at a spatial location u for the k th word-prototype v k is,

Original: [Equation]
Revised: [Equation]

Original:  Following this, we employ the remaining Res4 block (as Res3 was chosen for adaptation in Fig. 2)and global average pooling (GAP) to obtain a feature vector ψ ( x ) ∈ R K from the spatially densefeatures ϕ ( x ) .
Revised: ϕ [Equation] Following this, we employ the remaining Res4-like conv-layers block (as Res3 was chosen for adaptation in Fig. 2) and global average pooling (GAP) to obtain a feature vector ψ ( x ) ∈ R K from the spatially dense features ϕ ( x ) .

Original: Thus, we repurpose a simple 1 × 1 conv.
Revised: Thus, we repurpose a simple 1 × 1 conv.

Original: layer to implement soft word-quantizationwith the layer weights as the word-prototypes.
Revised: layer to implement soft word-quantization with the layer weights as the word-prototypes.

Original: Now, we introduce a prototype-alignment-score (PAS)to further motivate the effectiveness of BoW concepts.
Revised: Now, we introduce a prototypealignment-score (PAS) to further motivate the effectiveness of BoW concepts.

Original: Prototype-Alignment-Score (PAS).
Revised: Prototype-Alignment-Score (PAS).

Original: Consider the backbone feature vector h u ( x ) at spatial locationu .
Revised: Consider the backbone feature vector h u ( x ) at spatial location u .

Original: We compute PAS, γ PAS , as k -means loss i.e . distance of h u ( x ) to the closest word-prototype in V , 
Revised: We compute PAS, γ PAS , as k -means loss i.e . distance of h u ( x ) to the closest word-prototype in V , γ PAS ( [Equation] ( x ) , [Equation]

Revised: [Equation]

Revised: −

Original: [Equation]
Revised: [Equation]

Original: 204 Here, ℓ cos denotes cosine-distance.
Revised: ∈ [Equation] Here, ℓ cos denotes cosine-distance.

Original: Intuitively, γ PAS indicates how close the feature vector at u is toone of the word-prototypes in V .
Revised: Intuitively, γ PAS indicates how close the feature vector at u is to one of the word-prototypes in V .

Original: Since ϕ u ( x ) is a word-histogram (softmax probabilities), a higherγ PAS indicates that the closest prototype would have a much higher contribution in the word-histogramthan the others i.e . a sparser word-histogram.
Revised: Since ϕ u ( x ) is a word-histogram (softmax probabilities), a higher γ PAS indicates that the closest prototype would have a much higher contribution in the word-histogram than the others i.e . a sparser word-histogram.

Original: Based on PAS, we arrive at the following insight. 
Revised: Based on PAS, we arrive at the following insight. Crop-level

Revised: Word-hist.

Revised: ClassEach crop yields a similarword-hist.

Original: Insight 2.
Revised: Imagesamples ClassClass- Grid-shuffled images Entropy of word-hist increases with #inst.

Revised: #inst.

Revised: =

Revised: #inst.

Revised: =

Revised: #inst.

Revised: =

Revised: ImagelevelWord- hist.

Revised: [Figure]

Revised: [Figure]

Revised: task use no. of instances for grid-shufﬂing (at least one crop from each of the instances)

Revised: Dataset

Revised: C l a ssC l a ssC l a ssC l a ssGrid-Size (4 here)

Revised: [Figure]

Revised: Insight 2.

Original: (Encouraging Prototype-Alignment for UniDA)
Revised: (Encouraging Prototype-Alignment for UniDA)

Original: Since better prototype-alignmentimplies sparser word-histograms, the word-histogram-space ϕ
Revised: Since better prototype-alignment implies sparser word-histograms, the word-histogram-space ϕ

Original: ( x ) would better represent the intrinsicclass structure (including private classes).
Revised: ( x ) would better represent the intrinsic class structure (including private classes).

Original: Thus, by encouraging higher PAS, DA at word-histogramlevel would exhibit lower negative-transfer-risk as word-prototypes are common across domains. 
Revised: Thus, by encouraging higher PAS, DA at word-histogram level would exhibit lower negative-transfer-risk as word-prototypes are common across domains. Remarks. Consider Fig. 1 as an example.

Revised: The shallower layers would extract generic shapes like rectangles, circles, lines, etc .

Revised: while deeper layers would extract semantic shapes like windows, arms, chassis, etc.

Original: Insight 2 encourages sparser word-histograms and a naive way to enforce this would be a self-entropyminimization objective on the  word-histogram vectors ϕ u ( x ) ∀ u .
Revised: Intuitively, the word-histogram space at generic-shape-level cannot be sparse for the object recogition task while sparsity is desirable at deeper layers. Hence, we seek objectives that ensure word-histogram sparsity at a sufficiently high semantic level, catering to the UniDA problem. With this, the pre-classifier features better capture the class-level intrinsic structure that improves UniDA performance.

Revised: Note that a good intrinsic structure refers to a scenario where individual classes, including private classes, are well clustered in the feature space.

Revised: Insight 2 encourages sparser word-histograms and a naive way to enforce this would be a self-entropy minimization objective on the word-histogram vectors ϕ u ( x ) ∀ u .

Original: Intuitively, this objective wouldincrease the contributions of the closest word-prototype in ϕ u ( x ) i.e . increase the PAS.
Revised: Intuitively, this objective would increase the contributions of the closest word-prototype in ϕ u ( x ) i.e . increase the PAS.

Original: Formally,min h,ψ
Revised: Formally,

Original: [Equation] 
Revised: [Equation] [Equation]

Original: L uem ]; where 
Revised: ]; where L uem

Original: [Equation]
Revised: [Equation]

Original: However, the self-entropy objective is susceptible to word-level misalignment due to a lack ofconstraints.
Revised: However, the self-entropy objective is susceptible to word-level misalignment due to a lack of constraints.

Original: For example, different classes may be mapped to the same word-prototype, or differentsame-class samples may be mapped to distinct word-prototypes.
Revised: For example, different classes may be mapped to the same word-prototype, or different same-class samples may be mapped to distinct word-prototypes.

Original: As the target domain is unsupervisedwith unseen private classes and the word-prototype concepts are abstract, it is very difficult to developexplicit constraints.
Revised: As the target domain is unsupervised with unseen private classes and the word-prototype concepts are abstract, it is very difficult to develop explicit constraints.

Original: Thus, we look for implicit constraints via self-supervision to enforce Insight 2.
Revised: Thus, we look for implicit constraints via self-supervision to enforce Insight 2.

Original: 3.5 BoW-based pretext task
Revised: 3.5 BoW-based pretext task

Original: We seek a word-related prior or property that can be used to encourage prototype-alignment and betterword-prototypes V through a self-supervised pretext task.
Revised: We seek a word-related prior or property that can be used to encourage prototype-alignment and better word-prototypes V through a self-supervised pretext task.

Original: We hypothesize that different crops fromthe same image would yield similar word-histograms (Fig. 4A) assuming that all crops are extractedwith some part of the object inside the crop.
Revised: We hypothesize that different crops from the same image would yield similar word-histograms (Fig. 4A) assuming that all crops are extracted with some part of the object inside the crop.

Original: This is reasonable as long as partially out-of-frame cropsare not used.
Revised: This is reasonable as long as partially out-of-frame crops are not used.

Original: Based on this, we provide the following insight to formulate our pretext task.
Revised: Based on this, we provide the following insight to formulate our pretext task.

Original: Crop-level

Original: Word-hist.

Original: ClassEach crop yields a similarword-hist.

Original: Imagesamples

Original: ClassClass-

Original: Grid-shuffled images

Original: Entropy of word-hist increases with #inst.

Original: #inst.

Original: =

Original: #inst.

Original: =

Original: #inst.

Original: =

Original: ImagelevelWord- hist.

Original: [Figure]

Original: [Figure]

Original: task use no. of instances for grid-shufﬂing (at least one crop from each of the instances)

Original: Dataset

Original: C l a ssC l a ssC l a ssC l a ssGrid-Size (4 here)

Original: [Figure]

Original: Insight 3.
Revised: Insight 3.

Original: (Word-histogram self-entropy prior in grid-shuffled crops) Consider a grid-shuffledimage where crops from y ins distinct instances are assembled (Fig. 4B).
Revised: (Word-histogram self-entropy prior in grid-shuffled crops) Consider a grid-shuffled image where crops from y ins distinct instances are assembled (Fig. 4B).

Original: The word-histogram selfentropy of the grid-shuffled image, say H e
Revised: The word-histogram selfentropy of the grid-shuffled image, say H e

Original: ( y ins ) , would increase with the number of distinct instancesy ins involved in the construction of the grid-shuffled image, assuming a constant grid size N gs .
Revised: ( y ins ) , would increase with the number of distinct instances y ins involved in the construction of the grid-shuffled image, assuming a constant grid size N gs .

Original: [Equation]
Revised: [Equation]

Original: Remarks.
Revised: Remarks.

Original: Intuitively, combining crops from more number of distinct instances would increase thecontributions of distinct word-prototypes (Fig. 4B), thereby increasing word-histogram entropy.
Revised: Intuitively, combining crops from more number of distinct instances would increase the contributions of distinct word-prototypes (Fig. 4B), thereby increasing word-histogram entropy.

Original: Thisbehavior would be consistent when distinct instances also come from distinct task categories, as itensures minimal overlap of word-prototypes among instances.
Revised: This behavior would be consistent when distinct instances also come from distinct task categories, as it ensures minimal overlap of word-prototypes among instances.

Original: The assumption of distinct categoriesfor y ins instances is reasonable when goal task has a large number of categories, which usually holdsin common DA benchmarks for object recognition.
Revised: The assumption of distinct categories for y ins instances is reasonable when goal task has a large number of categories, which usually holds in common DA benchmarks for object recognition.

Original: On the other hand, crops from the same imagewould only affect a small set of the same word-prototypes and word-histogram entropy can only bemarginally affected.
Revised: On the other hand, crops from the same image would only affect a small set of the same word-prototypes and word-histogram entropy can only be marginally affected.

Original: While the actual word-histograms may vary with the instances used, the entropy
Revised: While the actual word-histograms may vary with the instances used, the entropy

Original: H e ( y ins ) for a given number of instances y ins would lie in a small range.
Revised: H e ( y ins ) for a given number of instances y ins would lie in a small range.

Original: The pretext task also encourages better prototype-alignment i.e . higher PAS.
Revised: The pretext task also encourages better prototype-alignment i.e . higher PAS.

Original: This is because separability of different pretext classes would improve with sparser word-histograms.
Revised: This is because separability of different pretext classes would improve with sparser word-histograms.

Original: Concretely, cropsfrom multiple instances (in grid-shuffling) would each have a few distinct word-prototypes withsignificant contributions and Insight 3 would be better supported.
Revised: Concretely, crops from multiple instances (in grid-shuffling) would each have a few distinct word-prototypes with significant contributions and Insight 3 would be better supported.

Original: Then, the pretext classes (Fig. 4C)can be easily separated by word-histogram entropy.
Revised: Then, the pretext classes (Fig. 4C) can be easily separated by word-histogram entropy.

Original: Note that we use a learnable classifier since theseintuitions may not hold at the start of training but pretext objectives can still guide the training.
Revised: Note that we use a learnable classifier since these intuitions may not hold at the start of training but pretext objectives can still guide the training.

Revised: The pretext task helps avoid word-level misalignment, i.e . cases where different classes are aligned with the same word-prototype or different samples of a class are aligned to distinct word-prototypes.

Original: Based on Insight 3, we construct an entropy-bin for each number of instances y ins ∈{ 1 , 2 , . . .
Revised: Consider Fig. 4A with the worst-case of misalignment where all classes (class-1, class-2, class-3) are represented by the same word-histogram. Then, in Fig. 4B, the image-level word-histograms would be identical for any no.

Revised: of instances used for patch-shuffling and the pretext task of identifying no. of instances would fail.

Revised: The above example shows, similar to a proof by contradiction, that the pretext task cannot allow word-level misalignment as misalignment would hurt the pretext task performance.

Revised: Based on Insight 3, we construct an entropy-bin for each number of instances y ins ∈{ 1 , 2 , . . .

Original: , N gs }for a novel pretext task of entropy-bin classification.
Revised: , N gs } for a novel pretext task of entropy-bin classification.

Original: Concretely, a pretext classifier f n : R K →C noperates on the output of ψ (Fig. 3) and is trained to predict y ins -class of input grid-shuffled images.
Revised: Concretely, a pretext classifier f n : R K →C n operates on the output of ψ (Fig. 3) and is trained to predict y ins -class of input grid-shuffled images.

Original: Procurement of pretext-task samples.
Revised: Procurement of pretext-task samples.

Original: We illustrate this process in Fig.
Revised: We illustrate this process in Fig.

Original: 4C. The same process isfollowed separately for both source dataset D s and target dataset D t where goal-task category labelsare not required.
Revised: 4C. The same process is followed separately for both source dataset D s and target dataset D t where goal-task category labels are not required.

Original: First, the number of distinct instances y ins is sampled from a uniform distribution
Revised: First, the number of distinct instances y ins is sampled from a uniform distribution

Original: U (1 , N gs ) and serves as the pretext-label.
Revised: U (1 , N gs ) and serves as the pretext-label.

Original: Next, a batch of y ins instances is sampled from (say) thesource dataset D s .
Revised: Next, a batch of y ins instances is sampled from (say) the source dataset D s .

Original: Now, crops are sampled randomly from the y ins instances such that each instancecontributes at least one crop.
Revised: Now, crops are sampled randomly from the y ins instances such that each instance contributes at least one crop.

Original: Finally, these crops are randomly assembled into the grid-shuffledimage denoted by x s,n .
Revised: Finally, these crops are randomly assembled into the grid-shuffled image denoted by x s,n .

Original: The same process can be performed for the target dataset to obtain x t,n .
Revised: The same process can be performed for the target dataset to obtain x t,n .

Original: Insummary, a source-pretext dataset D s,n = { ( x s,n , y ins ) : x s,n ∈ X , y ins ∈ C n } and a target-pretextdataset D t,n = { ( x t,n , y ins ) : x t,n ∈ X , y ins ∈ C n } are extracted. 
Revised: In summary, a source-pretext dataset D s,n = { ( x s,n , y ins ) : x s,n ∈ X , y ins ∈ C n } and a target-pretext dataset D t,n = { ( x t,n , y ins ) : x t,n ∈ X , y ins ∈ C n } are extracted. [Figure]

Original: Training algorithm.
Revised: Training algorithm.

Original: We aim to demonstrate that our proposed approach is complementary toexisting UniDA methods.
Revised: We aim to demonstrate that our proposed approach is complementary to existing UniDA methods.

Original: Thus, we simply add our architecture modification ( i.e . ψ in Fig. 3) and thepretext classifier head f n while keeping the other components from an existing method like [32, 20]. [Table] [Table] [Table]
Revised: Thus, we simply add our architecture modification ( i.e . ψ in Fig. 3) and the pretext classifier head f n while keeping the other components from an existing method like [42, 27].

Original: [Table] [Table] Consider an existing UniDA training algorithm denoted by UniDA-Algo ( D s , D t ) that trains thebackbone h and goal-classifier f g : R K →C s .
Revised: Consider an existing UniDA training algorithm denoted by UniDA-Algo ( D s , D t ) that trains the backbone h and goal-classifier f g : R K → C s .

Original: Additionally, we introduce the pretext-task objectives
Revised: Additionally, we introduce the pretext-task objectives

Original: L s,n = L ce ( f n ◦ h ( x s,n ) , y ins ) and L t,n (defined similarly) for the source and target data respectively(Fig. 5).
Revised: L s,n = L ce ( f n ◦ h ( x s,n ) , y ins ) and L t,n (defined similarly) for the source and target data respectively (Fig.

Original: Here, L ce denotes the standard cross-entropy loss.
Revised: Here, L ce denotes the standard cross-entropy loss.

Original: Formally, the overall objective is,
Revised: Formally, the overall objective is,

Original: [Equation]
Revised: [Equation]

Original: G o a l t a s k i t e r a t

Original: i o n

Original: P r e t e x t t a s k

Original: i t e r a t

Original: i o n G o a l t a s k i t e r a t i o n s h a r e d Here, J ( . ) represents the objective or loss functionand the third term is borrowed from Eq.
Revised: Here, J ( . ) represents the objective or loss function and the third term is borrowed from Eq.

Original: We update only the backbone h and goal-classifier f g using UniDA-Algo while the backbone h along withthe word-prototype layer ψ and pretext-classifier f nare updated via the pretext-task objectives.
Revised: We update only the backbone h and goal-classifier f g using UniDA-Algo while the backbone h along with the word-prototype layer ψ and pretext-classifier f n are updated via the pretext-task objectives.

Original: Intuitively, the word-prototypes are learnt only throughthe pretext-task as they are the implicit constraints,as discussed under Insight 2. 
Revised: Intuitively, the word-prototypes are learnt only through the pretext-task as they are the implicit constraints, as discussed under Insight 2. [Table] [Table] [Table]

Revised: [Table]

Original: Inference.
Revised: Inference.

Original: We discard the pretext-classifier f n anduse only the goal-classifier f g at inference time.
Revised: We discard the pretext-classifier f n and use only the goal-classifier f g at inference time.

Original: Fora fair comparison, we use the same known-unknowndemarcation algorithm asused in UniDA-Algo .
Revised: For a fair comparison, we use the same known-unknown demarcation algorithm as UniDA-Algo .

Original:  Experiments
Revised:  Experiments

Original: Dataset.
Revised: Dataset.

Original: We report results on three different benchmarks.
Revised: We report results on three different benchmarks.

Original: Office-31 [31] contains three domains:
Revised: Office-31 [41] contains three domains:

Original: DSLR (D), Amazon (A), and Webcam (W).
Revised: DSLR (D), Amazon (A), and Webcam (W).

Original: Office-Home [38] is a more difficult benchmark, with 65classes and 4 domains, Artistic (Ar), Clipart (Cl), Product (Pr), and Real-World (Rw).
Revised: Office-Home [49] is a more difficult benchmark, withclasses and 4 domains, Artistic (Ar), Clipart (Cl), Product (Pr), and Real-World (Rw).

Original: DomainNet[29] is the largest DA benchmark and the most challenging due to highly diverse domains and hugeclass-imbalance.
Revised: DomainNet [38] is the largest DA benchmark and the most challenging due to highly diverse domains and huge class-imbalance.

Original: Following [10], we use three subsets, namely Painting (P), Real (R), and Sketch (S).
Revised: Following [12], we use three subsets, namely Painting (P), Real (R), and Sketch (S).

Original: Evaluation.
Revised: Evaluation.

Original: We report the mean results of three runs for each experiment.
Revised: We report the mean results of three runs for each experiment.

Original: Following [32, 20],target-private classes are grouped into a single unknown class for both Open-Set and UniDA.
Revised: Following [42, 27], target-private classes are grouped into a single unknown class for both Open-Set and UniDA.

Original: Wereport H-score metric (HOS), i.e .
Revised: We report H-score metric (HOS), i.e .

Original: the harmonic mean of accuracy of shared and target-private samples.
Revised: the harmonic mean of accuracy of shared and target-private samples.

Original: Implementation Details.
Revised: Implementation Details.

Original: We use two recent prior arts, OVANet [32] and DCC [20], separatelyas UniDA-Algo (Fig. 5).
Revised: We use two recent prior arts, OVANet [42] and DCC [27], separately as UniDA-Algo (Fig. 5).

Original: We initialize ResNet50 with ImageNet-pretrained weights and retain [Table] [Table] [Table]
Revised: We initialize ResNet50 with ImageNet-pretrained weights and retain other hyperparameters from the original baseline ( UniDA-Algo ).

Original: other hyperparameters from the original baseline ( UniDA-Algo ).

Original: Unless otherwise mentioned,
Revised: Unless otherwise mentioned,

Original: UniDA-Algo will be OVANet (for most analysis experiments).
Revised: UniDA-Algo will be OVANet (for most analysis experiments).

Original: We also keep the optimizers, learningrates, and schedulers for both goal task iterations and pretext task iterations same as the baseline.
Revised: We also keep the optimizers, learning rates, and schedulers for both goal task iterations and pretext task iterations same as the baseline.

Original: Wefollow DCC [20] for the dataset and shared-private class splits.
Revised: We follow DCC [27] for the dataset and shared-private class splits.

Original: See Suppl.
Revised: See Suppl.

Original: for complete details.
Revised: for complete details.

Original: 4.1 Comparison with prior arts
Revised: 4.1 Comparison with prior arts

Original: Open-Set DA.
Revised: Open-Set DA.

Original: The benchmark comparisons for OSDA are presented in Table 3 and 4 for OfficeHome and Office-31 benchmarks respectively.
Revised: The benchmark comparisons for OSDA are presented in Table 3 and 4 for OfficeHome and Office-31 benchmarks respectively.

Original: Our method surpasses all current methods, even thosetailored for OSDA [3, 33, 21].
Revised: Our method surpasses all current methods, even those tailored for OSDA [3, 43, 28].

Original: With a 2.2% H-score gain over OVANet [32] and a 4.6% gain over
Revised: With a 2.2% H-score gain over OVANet [42] and a 4.6% gain over

Original: DCC [20], we consistently outperform all OSDA baselines for Office-Home.
Revised: DCC [27], we consistently outperform all OSDA baselines for Office-Home.

Original: Similarly, our SPA ontop of DCC and OVANet improves on Office-31 by 6.4% and 1.3% over the baselines.
Revised: Similarly, our SPA on top of DCC and OVANet improves on Office-31 by 6.4% and 1.3% over the baselines.

Original: Overall, weachieve a superior balance between shared class categorization and private sample identification.
Revised: Overall, we achieve a superior balance between shared class categorization and private sample identification.

Original: UniDA.
Revised: UniDA.

Original: On the Office-31 benchmark (Table 1), the proposed approach outperforms all other methodsin terms of H-score.
Revised: On the Office-31 benchmark (Table 1), the proposed approach outperforms all other methods in terms of H-score.

Original: We improve upon the earlier state-of-the-art methods, DCC [20] and OVANet[32] by 7.0% and 0.4% respectively, again demonstrating a superior balance between shared andprivate sample identification.
Revised: We improve upon the earlier state-of-the-art methods, DCC [27] and OVANet [42] by 7.0% and 0.4% respectively, again demonstrating a superior balance between shared and private sample identification.

Original: Office-Home (Table 2) is a more difficult benchmark, with moreprivate classes than the shared classes (55 vs. 10).
Revised: Office-Home (Table 2) is a more difficult benchmark, with more [Table] [Table] [Table]

Original:  Our approach exhibits a stronger capabilityfor the separation of shared and private classes in this extreme circumstance, benefiting from ourproposed pretext task.
Revised: private classes than the shared classes (55 vs. 10). Our approach exhibits a stronger capability for the separation of shared and private classes in this extreme circumstance, benefiting from our proposed pretext task.

Original: On the Office-Home benchmark, our method combined with DCC yieldsa 4.7% improvement in H-score and an additional 1.4% gain with OVANet.
Revised: On the Office-Home benchmark, our method combined with DCC yields a 4.7% improvement in H-score and an additional 1.4% gain with OVANet.

Original: On the large-scale
Revised: On the large-scale

Original: DomainNet dataset (Table 1), our strategy improves OVANet [32] by 1.5%.
Revised: DomainNet dataset (Table 1), our strategy improves OVANet [42] by 1.5%.

Original: This shows that ourapproach outperforms prior arts in a variety of settings, i.e . varying degrees of openness.
Revised: This shows that our approach outperforms prior arts in a variety of settings, i.e . varying degrees of openness.

Original: 4.2 Discussion
Revised: 4.2 Discussion

Original: a) Ablation study.
Revised: a) Ablation study.

Original: We perform a thorough ablation study on Office-Home for the components of ourapproach (Table 5).
Revised: We perform a thorough ablation study on Office-Home for the components of our approach (Table 5).

Original: First, since our architecture modification presents a small increase in computation,we demonstrate that simply using the arch.
Revised: First, since our architecture modification presents a small increase in computation, we demonstrate that simply using the arch.

Original: mod.
Revised: mod.

Original: only marginally improves the performance (0.2%over OVANet and 0.6% over DCC).
Revised: only marginally improves the performance (0.2% over OVANet and 0.6% over DCC).

Original: Further, simply adding the word-histogram self-entropy loss
Revised: Further, simply adding the word-histogram self-entropy loss

Original: L em (Eq. 6) with the arch.
Revised: L em (Eq. 6) with the arch.

Original: mod.
Revised: mod.

Original: also yields fairly low improvements over the baseline (0.4% over
Revised: also yields fairly low improvements over the baseline (0.4% over

Original: OVANet and 1.2% over DCC).
Revised: OVANet and 1.2% over DCC).

Original: Next, we assess the improvement from the proposed pretext task (butwithout L em ) and observe gains of 1.1% over OVANet and 3.2% over DCC.
Revised: Next, we assess the improvement from the proposed pretext task (but without L em ) and observe gains of 1.1% over OVANet and 3.2% over DCC.

Original: Finally, including L emwith the pretext task gives further gains of 0.5% over OVANet and 1.4% over DCC.
Revised: Finally, including L em with the pretext task gives further gains of 0.5% over OVANet and 1.4% over DCC.

Original: Thus, the arch.mod. and L em independently give marginal gains.
Revised: Thus, the arch. mod. and L em independently give marginal gains.

Original: However, they give significant improvementscombined with our pretext task, underlining the importance of every component.
Revised: However, they give significant improvements combined with our pretext task, underlining the importance of every component.

Original: [Figure]
Revised: [Table]

Original: Effect

Original: [Figure]
Revised: [Table]

Original: varying

Original: [Figure]
Revised: [Table]

Original: Correlation of 
Revised: b) Evaluating clustering of target-private classes. To further support Insight 2, i.e . private class samples are better clustered with our approach, we apply linear evaluation protocol for the target-private classes on the word-prototype features ϕ ( x ) . Here, we use the labels of target-private classes (only for this analysis) to train a linear classifier on the frozen features from ϕ ( x ) and compute the accuracy for target-private classes (Table 6).

Original: [Figure]
Revised: We observe a significant gain (+5.6%) over the baseline

Original: w/ NTR & PAS [Figure]
Revised: OVANet, which indicates that the target-private classes are better clustered with SPA.

Original: b) Correlation between UniDA performance, pretext task performance, PAS and NTR.
Revised: c) Correlation between UniDA performance, pretext task performance, PAS and NTR.

Original: For
Revised: For

Original: UniDA on Office-Home, we study the correlation between goal task performance (HOS), pretexttask performance, negative-transfer-risk (NTR) and prototype-alignment-score (PAS) (averaged over
Revised: UniDA on Office-Home, we study the correlation between goal task performance (HOS), pretext task performance, negative-transfer-risk (NTR) and prototype-alignment-score (PAS) (averaged over

Original: D t ) in Fig.
Revised: D t ) in Fig.

Original: 6C. We observe that goal task performance is positively correlated with both pretext taskperformance and PAS, as in Insight 2.
Revised: 6C. We observe that goal task performance is positively correlated with both pretext task [Figure] B. Effect of varying grid [Figure]

Revised: [Figure]

Original: Further, goal task performance is negatively correlated with
Revised: [Figure] [Figure] C. Correlation of performance w/ NTR & PAS [Figure]

Revised: P r e t e x t

Revised: T a sk

Revised: [Figure]

Revised: N

Revised: [Figure]

Revised: [Figure]

Revised: performance and PAS, as in Insight 2.

Revised: Further, goal task performance is negatively correlated with

Original: NTR which supports Insight 1.
Revised: NTR which supports Insight 1.

Original: Further, compared to the baseline OVANet [32] (dashed curves),our OVANet+SPA (solid curves) achieves better and faster convergence ( ∼ 100 vs. ∼ 300 iterations).
Revised: Further, compared to the baseline OVANet [42] (dashed curves), our OVANet+SPA (solid curves) achieves better and faster convergence ( ∼ 100 vs. ∼ 300 iterations).

Original: Further, we also observe that PAS does not change much when our pretext task is not employed.
Revised: Further, we also observe that PAS does not change much when our pretext task is not employed.

Original: [Table]
Revised: [Table]

Original:  c) Comparison with other pretext tasks.
Revised: + Ours 65. c) Comparison with other pretext tasks.

Original: We compare theeffectiveness of our word-related pretext task, for Open-Set DAand Universal DA on Office-Home, with existing pretext tasksin Table 6.
Revised: We compare the effectiveness of our word-related pretext task, for Open-Set DA and Universal DA on Office-Home, with existing pretext tasks in Table 7. Note that we include our EM loss and architecture modification with each of the pretext tasks for a fair comparison.

Original: First, we compare with dense-output based pretexttasks like colorization [48, 19] and inpainting [28].
Revised: First, we compare with dense-output based pretext tasks like colorization [59, 26] and inpainting [37].

Original: We observemarginal or no improvements with these tasks because theoutput spaces of these tasks are usually domain-dependent ( e.g .inpainting of rainy scenes must consider rain, a domain-specificfactor), which hinders the adaptation.
Revised: We observe marginal or no improvements with these tasks because the output spaces of these tasks are usually domain-dependent ( e.g . inpainting of rainy scenes must consider rain, a domain-specific factor), which hinders the adaptation.

Original: Next, we consider nondense classification tasks like rotation [23, 15], jigsaw [5] andpatch-location [37].
Revised: Next, we consider non-dense classification tasks like rotation [31, 19], jigsaw [7] and patchlocation [48].

Original: While these tasks achieve some marginalgains, our  pretext task outperforms them as it effectively aids UniDA and OSDA by encouraging prototype-alignment (Insight 2).d) Effect of varying degree of openness.
Revised: While these tasks achieve some marginal gains, our proposed pretext task outperforms them as it effectively aids UniDA and Open Set DA by encouraging prototype-alignment (Insight 2). d) Effect of varying degree of openness.

Original: Fig.
Revised: Fig.

Original: 6A studies the effect of altering the degree of opennessfor UniDA in Office-Home.
Revised: 6A studies the effect of altering the degree of openness for UniDA in Office-Home.

Original: Following [32], we report the average results over five scenarios to covera variety of domains while varying the amount of unknown classes.
Revised: Following [42], we report the average results over five scenarios to cover a variety of domains while varying the amount of unknown classes.

Original: Our approach exhibits a lowsensitivity to the degree of openness and consistently outperforms other baselines.e) Effect of grid-size N gs used in grid-shuffling.
Revised: Our approach exhibits a low sensitivity to the degree of openness and consistently outperforms other baselines. e) Effect of grid-size N gs used in grid-shuffling.

Original: For UniDA on Office-Home, we report a sensitivityanalysis for grid-size N gs (Fig. 6B) which controls the number of entropy-bins i.e . number of pretexttask classes.
Revised: For UniDA on Office-Home, we report a sensitivity analysis for grid-size N gs (Fig. 6B) which controls the number of entropy-bins i.e . number of pretext task classes.

Original: We observe a marginal decrease in the performance as the grid-size increases, but itoutperforms the baseline (OVANet) across a wide range of grid-sizes (2x2 to 6x6).
Revised: We observe a marginal decrease in the performance as the grid-size increases, but it outperforms the baseline (OVANet) across a wide range of grid-sizes (2x2 to 6x6).

Original: Mitsuzumi etal.
Revised: Mitsuzumi et al.

Original: [24] show that, as grid-size increases beyond 4x4, shuffling the grid patches significantly altersdomain information.
Revised: [32] show that, as grid-size increases beyond 4x4, shuffling the grid patches significantly alters domain information.

Original: Thus, beyond 4x4, performance drops as more domains are introduced whichmake adaptation difficult.
Revised: Thus, beyond 4x4, performance drops as more domains are introduced which make adaptation difficult.

Original: Another reason is that the pretext task itself becomes more difficult as thegrid-size is increased.
Revised: Another reason is that the pretext task itself becomes more difficult as the grid-size is increased.

Original: Based on this, we choose grid-size 2x2 ( N gs =4) for our experiments.
Revised: Based on this, we choose grid-size 2x2 ( N gs =4) for our experiments.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we address the problem of Universal DA via closed-set Subsidiary Prototype-space
Revised: In this work, we address the problem of Universal DA via closed-set Subsidiary Prototype-space

Original: Alignment (SPA).
Revised: Alignment (SPA).

Original: First, we uncover a tradeoff between negative-transfer-risk and domain-invarianceat different layers of a deep network.
Revised: First, we uncover a tradeoff between negative-transfer-risk and domain-invariance at different layers of a deep network.

Original: While a balance can be struck at a mid-level layer, we drawmotivation from Bag-of-visual-Words (BoW) to introduce explicit word-prototypes followed byword-histogram based classification.
Revised: While a balance can be struck at a mid-level layer, we draw motivation from Bag-of-visual-Words (BoW) to introduce explicit word-prototypes followed by word-histogram based classification.

Original: We realize the closed-set SPA through a novel word-histogrambased pretext task operating in parallel with UniDA objectives.
Revised: We realize the closed-set SPA through a novel word-histogram based pretext task operating in parallel with UniDA objectives.

Original: Building on top of existing UniDAworks, we achieve state-of-the-art results on three UniDA and Open-Set DA benchmarks. 
Revised: Building on top of existing UniDA works, we achieve state-of-the-art results on three UniDA and Open-Set DA benchmarks. Acknowledgements.

Revised: This work was supported by MeitY (Ministry of Electronics and Information

Revised: Technology) project (No. 4(16)2019-ITEA), Govt. of India.

Original: Checklist
Revised: Checklist

Original: For all authors.. (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu- tions and scope?
Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Original: [Yes] See L66-73.

Original:  (b) Did you describe the limitations of your work?[Yes] See Suppl.
Revised: [Yes] (b) Did you describe the limitations of your work?

Original: Sec. (c) Did you discuss any potential negative societal impacts of your work?[Yes] See Suppl. Sec.
Revised: [Yes] See Appendix B (c) Did you discuss any potential negative societal impacts of your work?

Original:  (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Revised: [Yes] See Appendix C (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes] 
Revised: [Yes] 2.

Original: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A] (b) Did you include complete proofs of all theoretical results?
Revised: [N/A] (b) Did you include complete proofs of all theoretical results?

Original: [N/A] 
Revised: [N/A] 3.

Original: If you ran experiments...
Revised: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [No] We promise to release the code
Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] It will be available at https://sites.google.com/view/spa-unida .

Original: and trained weights upon acceptance.

Original: (b) Did you specify all the training details (e g., data splits, hyperparameters, how they were chosen)?
Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Original: [Yes] See L278-290 and Suppl.

Original:  Sec. 4. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?[Yes] See Suppl
Revised: [Yes] See Sec. 4 and Appendix D. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Original: [Table]

Original:  (d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs,[Table] internal cluster, or cloud provider)?
Revised: [Yes] See Appendix E (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Suppl. Sec. 4.[Table]
Revised: [Yes] See Appendix D. 4.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets...[Table] (a) If your work uses existing assets, did you cite the creators?
Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Suppl.

Original: Sec. 4.

Original: [Table] (b) Did you mention the license of the assets?[Yes] See Suppl.
Revised: [Yes] See Appendix D. (b) Did you mention the license of the assets?

Original: Sec. 4.

Original:  (c) Did you include any new assets either in the supplemental material or as a URL?
Revised: [Yes] See Appendix D. (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No] (d) Did you discuss whether and how consent was obtained from people whose data you’re us- ing/curating?
Revised: [No] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Original: [N/A] We used publicly available standard object recognition datasets that do not contain any personally identifiable information (PII).
Revised: [N/A] We used publicly available standard object recognition datasets that do not contain any personally identifiable information (PII).

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content?
Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Original: [N/A] See answer to 4d.
Revised: [N/A] See answer to 4d.

Original: If you used crowdsourcing or conducted research with human subjects... 
Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Original: (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Original: [N/A]
Revised: [N/A]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB)
Revised: In this appendix, we provide more details of our approach, extensive implementation details, additional analyses, limitations and potential negative societal impact. Towards reproducible resesarch, we will publicly release our complete codebase and trained network weights.

Original: approvals, if applicable?
Revised: This supplementary is organized as follows:

Original: [N/A]
Revised: • Section A: Notations (Table 8) • Section B: Limitations • Section C: Potential societal impact • Section D: Implementation details ◦ Baseline details ◦ Compute requirements ◦ Miscellaneous details (Fig. 7) • Section E: Analysis (Table 9, 10)

Original: [Table]
Revised: We summarize the notations used throughout the paper in Table 8. The notations are listed undergroups i.e . models, datasets, samples, spaces and measures. B Limitations

Revised: The proposed approach may be unsuitable for datasets with very less number of classes.

Revised: When number of classes are low, our Insight 3 may not hold, making the pretext task very difficult to learn.

Revised: This may negatively impact the goal task performance as the backbone is shared between the two tasks.

Revised: While this is a limitation of the proposed implementation, a possible solution could be to

Revised: [Table]

Revised: merge some entropy-bins through clustering techniques to form more discriminative pretext classes.

Revised: This limitation can also arise in case of large class-imbalance in the data, as this would also lead to overlapping entropy-bins, and a similar bin-merging solution may be used

Revised: [Table]

Revised: C Potential societal impact

Revised: Our findings may be used to train deep neural networks with minimal supervision by transferring knowledge from supplementary datasets.

Revised: On many datasets with a large amount of annotated data, such as ImageNet, modern deep networks surpass humans [17].

Revised: In many cases where such large-scale related datasets are accessible, our proposed approach can be a proxy to supervision in the target data.

Revised: Our approach has a favorable impact as it can reduce the data collection effort for data-intensive applications.

Revised: This might make technology more accessible to organizations and individuals with limited resources.

Revised: It can also aid applications where data is protected by privacy regulations and hence difficult to collect.

Revised: The negative consequences might include making these systems more available to organizations or individuals who try to utilize them for illegal purposes.

Revised: Our system is also vulnerable to adversarial attacks and lacks interpretability, as do all contemporary deep learning systems.

Revised: While we demonstrate increased performance compared to the state-of-the-art, negative transfer is still possible in extreme cases of domain-shift or category-shift.

Revised: Thus, our technique should not be employed in critical applications or to make significant decisions without human supervision.

Revised: D Implementation details

Revised: Here, we describe the implementation details excluded from the main paper due to the page limit.

Revised: D.1 Baseline details

Revised: OVANet.

Revised: Following prior works [44, 55], we use ResNet50 [18] as our backbone network, which has been pre-trained on ImageNet [40].

Revised: We add a new linear classification layer to replace the previous one.

Revised: We use inverse learning rate decay scheduling to train our models, as described in [44].

Revised: We set the weight for entropy minimization loss, λ =0 .

Revised: 1 for all the settings.

Revised: The value is calculated by

Revised: [Table]

Revised: the outcome of Open-Set DA for Office-31 (Amazon to DSLR) following [44].

Revised: For all experiments, the source and target batch size is 36.

Revised: The starting learning rate for new layers is set to 0.01 and for backbone layers to 0.001.

Revised: Our method is implemented with PyTorch [36].

Revised: DCC.

Revised: We use ResNet50 [18] as the backbone, pretrained on ImageNet [40].

Revised: The classifier is made up of two linear layers, following [55, 12, 43, 5].

Revised: We use Nesterov momentum SGD to optimize the model, which has a momentum of 0.9 and a weight decay of 5e-4.

Revised: The learning rate decreases by a factor of (1 + α iN ) − β , where i and N represent current and global iteration, respectively, and we set α = 10 and β = 0 .

Revised: We use a batch size of 36 and the initial learning rate is set as 1e-4 for

Revised: Office-31, and 1e-3 for Office-Home and DomainNet.

Revised: We use PyTorch for implementation.

Revised: Existing code used.

Revised: • OVANet [42]: https://github.com/VisionLearningGroup/OVANet (MIT license) • DCC [44]: https://github.com/Solacex/Domain-Consensus-Clustering (MIT license)

Revised: • PyTorch [36]: https://pytorch.org/ (BSD-style license)

Revised: Existing datasets used.

Revised: • DomainNet [38]: http://ai.bu.edu/M3SDA (Fair use notice)

Revised: • Office-Home [49]: https://www.hemanthdv.org/officeHomeDataset.html (Fair use notice)

Revised: • Office-31 [41]: https://www.cc.gatech.edu/ ∼ judy/domainadapt (open source)

Revised: D.2 Compute requirements

Revised: For our experiments, we used a local desktop machine with an Intel Core i7-6700K CPU, a single

Revised: Nvidia GTX 1080Ti GPU and 32GB of RAM.

Revised: D.3 Miscellaneous details

Revised: Negative-Transfer-Risk (NTR).

Revised: We introduce a negative-transfer-risk (NTR) γ NTR ( h ) for a given feature extractor h : X → Z , where Z is an intermediate feature-space.

Revised: First, the standard linear evaluation protocol [45] from transfer learning and self-supervised literature is applied on the feature extractor where a linear classifier f : Z → C s is trained on the feature h with the labeled source data.

Revised: Next, following [44], NTR is computed as the known-unknown classification accuracy using a fixed entropy threshold ρ on the linear classifier prediction as: γ NTR ( h ) =

Revised: E ( x,y unk )

Revised: ∼D t 1 ( H t ( f ◦ h ( x ) , ρ )=

Revised: y unk )

Revised: where H t ( f ◦ h ( x ) , ρ )=

Revised: (cid:26) 1; H ( f ◦ h ( x )) >

Revised: ρ 0; otherwise (9) where f = arg min f ′ E ( x s ,y s ) ∈D s CE ( f ′ ◦

Revised: h

Revised: ( x s ) , y s ) is the learned source classifier on features from h .

Revised: Here, H ( . ) computes self-entropy, ρ is a fixed entropy threshold, log( | C s | ) / 2 , where | C s | represents the number of classes, following [44].

Revised: CE represents the standard cross-entropy loss, and y unk represents known-unknown label (0 for known, 1 for unknown).

Revised: We access the known-unknown labels y unk for a subset of target data only for analysis (not for training).

Revised: Pretext dataset procurement.

Revised: We illustrate more examples in Fig.

Revised: 7, based on the procedure given under Insight 3 and in Fig.

Revised: 4C.

Revised: S o u r c e ( l a b e l e d )

Revised: [Figure]

Revised: S o u

Revised: r c e - p r e t e x t

Revised: [Figure]

Revised: T a r g e t ( u n l a b e l e d )

Revised: [Figure]

Revised: [Figure]

Original: c) Did you include the estimated hourly wage paid to participants and the total amount spent on
Revised: T a r g e t - p r e t e x t [Figure] “class [Figure]

Original: participant compensation?
Revised: Variance across different seeds. We highlight the significance of our results by reporting the mean and standard deviation of OS (overall accuracy) and OS* (known classes accuracy) over 3 runs with different random seeds in Table 9 for Open-Set DA on Office-31. We observe low variance with significant performance gains over the baseline.

Original: [N/A]
Revised: [Table]

Revised: Table 10 provides the details of the computational overhead caused by the extra parameters added in the BoW-inspired architecture modification (Sec. 3.4).

Revised: While ∼ 2M additional parameters are required, there is only a marginal increase in the

Revised: MACS (number of multiply-accumulate operations).

Revised: Further, simply adding the architecturemodification only marginally improves UniDA (also shown in Table 5, Sec. 4.2a).