Using Interventions to Improve Out-of-Distribution Generalization of Text-Matching Systems
Abstract
Given a user’s input text, text-matching recommender systems output relevant items by comparing the input text to available items’ description, such as product-toproduct recommendation on e-commerce platforms.
As users’ interests and item inventory are expected to change, it is important for a text-matching system to generalize to data shifts, a task known as out-of-distribution (OOD) generalization.
However, we find that the popular approach of fine-tuning a large, base language model on paired item relevance data (e.g., user clicks) can be counter-productive for OOD generalization.
For a product recommendation task, fine-tuning obtains worse accuracy than the base model when recommending items in a new category or for a future time period.
To explain this generalization failure, we consider an intervention-based importance metric, which shows that a fine-tuned model captures spurious correlations and fails to learn the causal features that determine the relevance between any two text inputs.
Moreover, standard methods for causal regularization do not apply in this setting, because unlike in images, there exist no universally spurious features in a text-matching task (the same token may be spurious or causal depending on the text it is being matched to).
For OOD generalization on text inputs, therefore, we highlight a different goal: avoiding high importance scores for certain features.
We do so using an interventionbased regularizer that constraints the causal effect of any token on the model’s relevance score to be similar to the base model.
Results on Amazon product andquestion recommendation datasets show that our proposed regularizer improves generalization for both in-distribution and OOD evaluation, especially in difficult scenarios when the base model is not accurate.
Introduction
In item-to-item recommendation [1], the goal is to output a list of relevant items given an input item.
Many such recommender systems utilize the text content of an item, such as recommending relevant questions given a question in an online forum [23], suggesting relevant products given a product title [14], or predicting relevant ads given a search engine query [4].
A popular way for training these text-matching recommender systems is to fine-tune text embeddings from a base language model like
BERT using supervised user feedback (such as clicks).
For example, one may use a contrastive loss to ensure that given an input query, embedding of another user-labelled relevant item is closer [6, 28].
However, the generalizability of the fine-tuned embeddings to item distributions beyond the userlabelled data has received little attention.
Distribution shifts are common in recommendation systems, such as when new product categories are added to an e-commerce platform, the list of recommendable items is modified, or the popularity of items changes over time.
As a result, after a recommender model is deployed, it is likely to encounter out-of-distribution data compared to its training set.
For such out-of-distribution data, we find that fine-tuned models using relevance labels can be worse than the pre-trained base model that they start from.
On a product recommendations dataset from
Amazon, while fine-tuning always increases in-distribution accuracy, the accuracy of the fine-tuned model on unseen product categories is lower than that of the base model.
We find a similar result on question-to-question recommendation on online forums ( sentence matching task [17, 22]).
To understand why, we characterize two common shifts in recommender systems: change in input query’s distribution and change in candidate items’ distribution.
Even without any change in the relevance function, these distribution shifts lead to a change in association between relevance score and the input tokens of a model.
As a result, even though standard fine-tuned models obtain high validation accuracy, they overfit to the training distribution by capturing spurious associations (e.g., between brand and a product category).
We characterize model spuriousness through an interventionbased importance score for any tokens subset, which can be interpreted as the causal effect of the token subset on the model’s relevance function.
Using the score, we find that the fine-tuned embedding on Amazon products learns a spuriously high importance for certain tokens while forgetting the rest.
To improve generalizability, we consider a causal graph for the relevance function’s computation and highlight the difficulty of not having universally spurious tokens for the text-matching task.
Instead, we argue that avoiding disproportionately high importance scores for tokens can be a viable way to regularize for OOD generalization.
Specifically, we propose ITVReg , a method that constrains the importance of tokens to be similar to that in the base model, which has been trained on a larger and more diverse set and thus less likely to share the same spurious patterns.
To do so, the method utilizes interventions on the training data (e.g., by masking certain tokens) to create OOD queries.
We evaluate ITVReg on OOD data for product title and question recommendation tasks and find that it improves OOD accuracy of naively finetuned models.
Compared to a baseline of directly regularizing a model’s predictions to the base model, ITVReg performs the best under reasonable
OOD shifts, where exploiting training data is useful.
Interestingly, ITVReg also helps accuracy on an IID test set: it increases accuracy over low-frequency, “tail” items.
Our contributions include, 1)
We characterize how new items break correlation between tokens of the query and candidate items; as a result state-of-the-art models for text-matching can be worse than the base model on which they are fine-tuned.
We propose a regularizer based on an interventional metric of token importance that improves OOD generalization.
Related Work
Our work connects recommender systems, sentence matching, and OOD generalization literature.
Text-matching recommendation systems and Sentence matching.
Among item-to-item recommendation systems, text-matching is a popular technique since many items can be characterized through their text, e.g., product-to-product recommendation on e-commerce websites or questionto-question recommendation on online forums.
Given an input query (such as a product’s title or description), the goal is to find the most relevant items.
State-of-the-art models use dense text retrieval techniques [28, 11], based on a pre-trained base model like BERT [7] for the initial encoding.
Then, either a bi-encoder or cross-encoder architecture [22, 11] is trained for learning the similarity between any two items.
The former is computationally efficient while the latter is more accurate but inefficient [17].
Since recommender systems often deal with a large number of items, we restrict ourselves to bi-encoders.
Contrastive loss with negative mining [31, 6, 10] is a popular way to train a bi-encoder model because recommender systems usually have one-sided feedback on the relevant pairs of items.
Closer to our work, [22] tackled the problem of domain adaptation while our work focuses on the harder domain generalization problem [32].
OOD generalization and fine-tuning.
Domain generalization in the vision literature [32] aims to identify spurious features in image data and remove them from a model’s representation using data augmentation on the spurious feature [29] or through regularization [5].
While recent work has attempted using data augmentations from vision [27], such augmentations are not always useful in text data [30] since it is difficult to obtain universal augmentations.
For instance, in the recommendation scenario, certain tokens (e.g., brand of a product) may be both spurious and causal depending on the user intent (e.g., substitute or accessory for a product).
Given the prevalance of pre-trained base models, another direction is to utilize the base models for OOD generalization since they are trained on larger, diverse data [9].
Recent work proposes fine-tuning-based OOD generalization on image classification [12, 25, 5], but this direction has not been explored for text models or contrastive loss, especially in the recommendations context.
Distribution shifts in recommendation
Let X be the set of input queries and Z be the set of candidate items.
Let the train data be ( X i , Z i , R i ) 1 ...N , where, X i , Z i correspond to the query and item text respectively and R i is the relevance or similarity score between X i , Z i .
We define f θ : X → R N as our encoder, parameterised by θ .
We use θ 0 as parameters for Base model.
We assume that f outputs unit norm embeddings.
For training, state-of-the-art methods [6, 28] embed both queries and items in a common space using a large, pre-trained base encoder like BERT that provides the initial embeddings.
The embeddings are fine-tuned using a supervised loss on the paired relevance labels given below L = max(0 , 1 + f θ ( X ) ⊤ f θ ( Z − ) − f θ ( X ) ⊤ f θ ( Z + )) where X , Z + and Z − denote a query, relevant item, and a non-relevant item respectively.
When ratings or relevance scores are available, one may also use the
MSE loss [17].
For a new input query, the model outputs a vector ˆ R ∈ R M where M is number of items, and r j = f θ ( X )
⊤ f θ ( Z j ) is its relevance score with j th item.
The system is evaluated on its accuracy in predicting the “ground-truth” relevance labels (e.g., metric may be precision at rank k ).
3.1 Types of distribution shift
Method
IID
[Table]
OOD denotes queries from five categories that were not included in the train data.
Fine-tuning on training data results in lower precision than the base model.
We study three shifts: change in queries P ( X ) , items P ( Z ) , and both queries and items P ( X, Z ) .
Change in P(X).
The distribution of queries changes but the set of candidate recommendations may remain the same.
For example, the popularity of certain queries can shoot up due to external events, or the system may be expanded to new queries (e.g., products on another website).
Change in P(Z).
The distribution of candidate items changes while the queries remain the same.
For example, an e-commerce platform may alter the eligibility rules for an item to be recommended, or recommend items from a partner website.
Change in both P(X) and P(Z).
A final scenario is when the distribution of both queries and items changes.
For example, introducing a new item category in an online store leads to new queries that can be accessed by user and also be recommended.
While the criteria for relevance (i.e., user intent) can change over time, for simplicity, we assume that
P ( R | Z, X ) remains constant where R is the true relevance score.
As we show below, even under this favorable assumption, text-matching models can learn patterns that do not generalize.
Example: OOD data on Amazon products.
Let us use the Amazon Titles dataset (see Supp.
D) to illustrate the problem with OOD generalization for fine-tuned models.
Both the input query and candidate items are titles of products on Amazon.com.
We simulate a scenario where P ( X ) is changed by removing queries from five categories from the train data and evaluating the model on those removed queries using precision@1 metric.
Table 1 compares the precision@1 of the base model (MSMarco [20] ) and fine-tuned model that is initialized with the base model.
As expected, the fine-tuned model improves significantly over the base model on IID test data, almost doubling the precision.
However, on the queries from the unseen item categories, the finetuned model performs worse than the base [-model.3.
-]model.
3.2 Explaining the OOD generalization failure: An intervention-based importance score
Explanation through causal graph.
To understand the failure, consider the schematic causal diagram for the data-generating process of the training set.
Figure 1 shows that each query can be broken down into its semantic (causal) component X c and its spurious component X s .
Same for candidate items, denoted by Z c and Z s .
The semantic components X c and Z c together cause the relevance label.
An ideal encoder should only learn the semantic components.
However, since the [Figure] [Figure]
[Figure]
z ##
[Figure]
##p [Figure] e c
[Figure]
ea n e
[Figure]
k
[Figure] [Figure] po [Figure]
Rowenta Z100 Non Toxic Soleplate Cleaner Kit
Base
Finetuned Finetuned model gives dispropotion- high weights to the token " non-semantic components are correlated with the semantic component and can be easier to learn, a fine-tuned encoder may learn the non-semantic components too Specifically, when new queries are introduced, the correlation between X c and X s can be broken, even as P ( R | X, Z ) remains the same. Hence, X s is no longer a good predictor of relevance and any model relying on X s will fail to generalize.
For example, for certain queries in the Amazon data, X s may correspond to the brand of a query product and X c the rest of its title .
In the training data, the brand (e.g., “samsung”) may be correlated with a single product category (e.g., “smartphone”) and thus an encoder may learn a non-zero weight for the brand to determine the relevance score.
However, the correlation may be broken in the evaluation data where the same brand is associated with another product category (e.g., “refrigerator”).
We can reason analogously for a change in P ( Z ) .
Explanation through intervention-based importance score.
OOD generalization failure is more intuitively understood through intervention-based analysis [18], where we change parts of a query and inspect the model’s prediction.
To understand which tokens a model learns as important, we define an importance score s of each token, that corresponds to the causal effect of a token on the model’s relevance score.
To compute s for a token, we mask that token (intervention) and compute the dot product similarity (relevance) of the masked input’s embedding with the original input’s embedding.
[2] s j = 1 − f θ ( X ) ⊺ f θ ( X ′− j ) where X ′− j is X with its j th token masked.
Consider a candidate item in the Amazon dataset, " Rowenta Z100 Non Toxic Soleplate Cleaner Kit ".
Figure 2 shows the importance scores.
The base encoder gives almost equal importance to each token, whereas the finetuned encoder is biased towards the token rowe , possibly because Rowenta products tend to be relevant to other Rowenta products.
This token obtains such a disproportionely high importance score that the item’s representation (average of all tokens’ representation) is defined by it.
Here
rowe ∈
Z s is a spurious token since relevance may depend on the brand rowenta , but not on rowe alone.
When we consider an OOD query having an actual brand called “Rowe”, " Rowe USA
Spoke Wrench - Bagged 09-0001 ", the model matches it to the Rowenta item and other products by
Rowenta (see Suppl.
Table 9), exposing the spurious correlation learnt by the encoder.
The query, “ Soleus Air MS-09 Oscillating Reflective Heater ” is another example.
The importance scores are in
Suppl.
Figure 4.
Again, we see the finetuned encoder is biased towards the token reflective because reflective is a strong matching signal in the train set: products with this token are often marked as relevant to products that have it too.
As a result, the top predicted item by the finetuned model is 3M
Scotchlite Reflective Tape, Silver, 1-Inch by 36-Inch (not a relevant item, see Suppl.
Table 10).
Regularization for OOD generalization
The above analysis indicates that fine-tuning encourages a model to learn high importance for certain tokens while forgetting the rest, which is undesirable if the tokens are spuriously associated with the relevance label.
However, it is non-trivial to identify the spurious versus semantic tokens.
Building a Risk Invariance Predictor.
As we mentioned in Section 3, in text-matching systems, we expect the distribution of queries P ( X ) , distribution of labels P ( Z ) , or both to change from train to test data.
At the same time, we assumed that P ( R | X, Z ) , i.e., the relevance between a query and label remains invariant across distributions.
That is, the relevance function g ( .
| X, Z ) (for bi-encoders it can be written as f ( X )
⊤ f ( Z ) , see Sec 3) remains invariant.
Then the goal is that the relevance [Table] function has similar accuracy on train as well as test data.
We now formally express this intuition of a OOD generalizable relevance function, using the definition of risk invariance from [16].
Let
P be a set of distributions over R, X, Z such that P
( R | X, Z ) remains invariant but the marginal distributions P ( X, Z ) can vary across the distributions.
Then, an optimal predictor is,
Definition 4.1.
Optimal Risk Invariant Predictor Define the risk of predictor g on distribution
P t ∈ P as R P t ( g ) =
E x,z,r ∼ P t l ( g ( x, z ) , r ) where l is the loss function.
Then, the set of risk-invariant predictors obtain the same risk across all distributions P t ∈ P , and set of the optimal risk-invariant predictors is defined as the risk-invariant predictors that obtain minimum risk on all distributions. From the causal graph of Figure
[Table]
we assumed that each query can be broken down into two mutually exclusive subsets of tokens, causal X c and spurious X s where X c (cid:83) X s =
X , such that only X c affects the relevance with a label.
Similarly, for label Z we can get causal Z c and spurious Z s , such that only Z c affects the relevance with a query.
Thus, an intuitive way to achieve risk invariance is to build a predictor g for relevance that only depends on X c and Z c for any query and label,
P ( R | X c , Z c ) .
From OOD generalization literature on images [32, 24], we may use one of these methods to learn X c , Z c from observed data: regularization [15, 13], weighting [19, 29], or data augmentation [32].
But such a solution relies on assumptions that the spurious features (e.g., image background) 1) are universal for all inputs; 2) can change independently of the semantic features.
Problem: Spurious features depends on context.
In text-based recommendations, it is difficult to find (a set of) tokens that are universally spurious, since spuriousness depends on context.
For example, in our example with “samsung” products (“smartphone” and “refrigerator”), the brand was a spurious feature for learning the relevance score.
However, when looking for a smartphone accessory, the brand is no longer spurious (in fact, it is the semantic feature).
Thus, the subset X c for a query may change based on the label, and similarly, the subset Z c for a label may change based on different queries.
The second assumption that tokens can be changed independently is also less plausible.
Possibly spurious tokens like “samsung” can be associated with other parts of the query (e.g, product code or name); it is infeasible to remove them independently.
Thus, while there are distribution shifts in text-based queries and items, it is difficult to find universal spurious features that can be modified to improve accuracy (as also observed empirically in prior work [8]).
Therefore, fully removing the influence of any token can be counter-productive.
Instead, we propose that our goal should be to ensure that no token is under-weighted such that it ceases to be important (ignored) for relevance prediction.
Since finetuning on text-matching models tends to assign very high weights to certain tokens, we reframe our goal: to avoid learning large importance scores for some tokens that may lead to ignoring other tokens, and thus not generalize to new data.
To do so, we regularize the importance scores to be similar to those from the base model, since the base model has been trained on larger, diverse data and is thus unlikely to encode the same spurious patterns.
We present two methods in this direction.
The first is a brute-force attempt, that forces not just the importance scores but the entire prediction of a model to be the same as base model, subject to the strength of a regularization constant (see Supp.
A for details).
The second is described below.
Intervention-based regularization.
Generalizing the importance score from Section 3.2, given a text input, we define an intervention as an independent change to a subset of tokens without affecting the rest of the input.
Further, we consider any intervention that removes information about the token subset, such as masking, deleting, replacing the token subset, etc.
By definition, an intervention deviates from the original distribution P ( X ) that generated the data, thus creating an out-of-distribution sample.
For each input X , we construct an interventional input X ′ using a transformation on a random subset X sub of the tokens.
The key idea behind our regularizer is that for any subset of tokens X sub , its importance score using the finetuned model should be the same as the importance score using the base model.
This is a relaxation of the risk-invariance goal of ensuring that the accuracy of the finetuned model remains the same for X and X ′ .
Using the equation for importance score from Section 3.2, the regularizer can be written as,
[Equation]
We call this the Interventional regularizer ( ITVReg ).
The full training loss (where L ERM can be the contrastive loss from Sec 3) is, L ERM + λ E X
[( f θ ( X ) ⊺ f θ ( X ′ )
− f θ 0 ( X ) ⊺ f θ
( X ′ ))
Evaluation
We study OOD generalization of text-matching models on product-to-product and question-toquestion recommendation.
Initial experiments on ITVReg indicated that masking as the intervention worked the best, so we use masking for all results.
For hyperparameter tuning, see Supp.
G.
We use two product-to-product recommendation datasets, namely AmazonTitles131K collected in 2013 and AmazonTitles1.3M [3] collected in 2014.
We also simulate a categorical shift setting in AmazonTitles131K , with OOD evaluation set termed AmazonCatOOD and the in-distribution evaluation set termed AmazonCatRemoved .
Precision@1 is used for evaluation (details in Supp.
D)
Baselines.
We compare OutReg and ITVReg to the Base model, standard Finetuned model, and two baselines from past work.
1) SimCSE : We adapt SimCSE [8], an augmentation method for pretraining sentence matching models, for our fine-tuning task.
During training, we use dropout (seeds s , s ′ ) to pass an input sentence twice through the model and obtain two embeddings, considered as a “relevant” pair.
The regulariser is ( f ( X, s ) ⊺ f ( X, s ′ ) − 1) 2 . 2) MaskReg : [26] propose deleting a span of words in the input as data augmentation.
To adapt it for comparison to ITVReg , we mask a portion of tokens instead.
The regulariser is ( f ( X ) ⊺ f ( X ′ ) − 1) 2 , where X ′ is masked input.
Results.
Table 2 shows the IID and OOD P@1 metrics for different recommender models.
We first analyze the temporal distribution shift: train on Amazon131K and test on Amazon1.3M .
On
IID evaluation, ITVReg and Finetuned both obtain the highest P@1 while OutReg obtains the lowest P@1.
On OOD evaluation, however, OutReg obtains the highest P@1 followed by ITVReg .
[Figure]
To understand the tradeoff between ITVReg and
OutReg , we study how the accuracy of these models would vary as different amounts of OOD data (from 1.3M dataset) is mixed with IID data (131K validation) for evaluation.
This simulates the realworld scenario where new items are progressively added.
Specifically, we progressively add 40K new items and their relevant queries from 1.3M to create multiple OOD datasets.
As Suppl.
Figure 5 shows,
ITVReg is the only method that performs better than Finetuned (line y = 0 ) on P@1 on all OOD datasets.
As the distribution shift becomes more extreme, (around 24% of 1.3M OOD data)
OutReg surpasses ITVReg indicating that OutReg is more suitable for large distribution shifts.
Next, we consider categorical distribution shift from Table 2.
We obtain similar results as the temporal shift: on the OOD evaluation, OutReg followed by ITVReg are the best performing models.
For P@1 on each category, see Suppl.
Table 8.
While we looked at adding new queries and items, in practice,
OOD shifts are more commonly observed as reweighing of existing items.
Hence, in addition to the average IID accuracy, it is important to analyze accuracy of a model at different levels of item popularity.
We bin the items according to their frequency in five quantiles i.e. 0-20%, ..., 80-100%, where lower quantiles denote low-frequency tail and higher quantiles denote head items.
Comparing the percentage gain in P@1 of methods wrt the Finetuned model (Figure 3), we find that OutReg helps on the tail items but suffers a huge drop in P@1 on head items.
In comparison, ITVReg achieves best of both worlds, with the same gains on tail and significantly lower drop on head items.
Qualitative results for all methods for the two queries from Section 3.2 are in Suppl.
K.
Importance Scores.
We also evaluate importance scores for the two qualitative examples presented in Sec.
3.1 after ITVReg regularisation in Suppl.
Fig 4.
The importance scores for the problematic tokens rowe and reflective are now in a reasonable range of value guided by the Base model.
Question Recommendations We also use the setup of [21] for numbers on question recommendations task.
More details are in Supp.
E
Conclusion
We identified limitations of text-matching recommender systems on OOD generalization, showing that finetuned models often perform worse than the base model on which they were finetuned.
We proposed two regularizers using a causal graph and an interventional measure of token importance.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justification to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[No] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[No] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Output-based regularization
Since both the fine-tuned model and the base model have the same architecture, a straightforward way to avoid very high importance scores is to make the neural network output be the same as the base model.
In a bi-encoder architecture for text-matching, we can implement this by enforcing that the finetuned and base encoders output exactly the same representation for an input.
Given the finetuning f θ ( . ) and base f θ 0 ( . ) encoders and an input query X , the output regularizer ( OutReg ) can be written as, [ f θ ( X ) − f θ 0 ( X )] 2 .
This simple regularizer is expected to do well for extreme distribution shifts where learning from IID data may not be that useful, since it constrains the encoder’s representation to be closer to the base encoder.
However, the same property is the biggest weakness of OutReg .
As a regularization, it is too strong and discourages learning from train data: with a high enough penalty term, the final encoder may be exactly equal to the base encoder.
B Qualitative Analysis
[Figure]
C Temporal Evolution
[Figure]
D Amazon Titles Recommendations
Dataset.
We use two product-to-product recommendation datasets, namely AmazonTitles131K and AmazonTitles1.3M [3].
These datasets contain titles of products that were recommended on a focal product’s page.
Only the product title can be used for recommendation.
There is a time shift though: 131K is collected from the year 2013 and 1.3M is collected from 2014.
As both new queries and labels are added while moving from 131K to 1.3M, this simulates the P ( X, Y ) distribution shift setup.
For each dataset, the same candidate items (131K and 1.3M items respectively) are shared across train and validation datasets.
Each query in the train set has a labelled set of positive items with relevance 1 .
The validation set contains unseen queries and we need to recommend the most relevant items for them.
While going from 131K to 1.3M dataset represents a temporal shift where both queries and items’ distributions change (and possibly other factors), we simulate a more controlled setup using category information.
We exclude queries from the 5 most popular second-level categories from the train set, and use them to construct the OOD evaluation set ( AmazonCatOOD ).
The in-distribution validation set also has the categories excluded ( AmazonCatRemoved ).
Thus, the item distribution P ( Z ) remains the same while the query distribution P ( X ) changes across train and evaluation.
Since X, Z have been treated symmetrically in our setup, this setups results can also be extrapolated to a case where the label distribution P ( Z ) changes while the query distribution P ( X ) remains the same.
Fine-tuning.
We follow the modeling procedure used in a recent state-of-the-art work [6].
We initialize with MSMARCO-DistilBERT-v4 [20] and use contrastive loss with hard negative mining.
Details are in Supp H.1.
All experiments are run for 3 random seeds and the mean and standard deviation are reported.
Evaluation metric.
We use the Precision @1 metric for evaluation.
Given predicted similarities ˆ R ∈ R M and ground truth similarities R (where r i is 1 if its a relevant item, otherwise 0), P @
k is defined as P @ k = 1 k (cid:80) l ∈ rank k ( ˆ R ) R l .
E Question Recommendation Evaluation
Datasets.
We use the setup of [21] for our experiments.
The AskUbuntu and SuperUser data comes from Stack Exchange, a family of technical community support forums.
Both datasets contain around one million pairs of sentences, labeled either zero or one denoting a negative and a positive pair.
Simulating a cold-start scenario with limited items, we present main results on 10% subsample of these datasets.
Results on the complete datasets are in Suppl L.
To study generalization under an extreme distribution shift, we additionally evaluate on a non-technical forum, Quora-Question Pairs dataset [23].
Fine-tuning.
We adopt the same modeling procedure as [22], training with a MSE loss between the ground truth and the predicted relevance score.
We initialize the models using two different Base models: NLI-DistilBERT [20] MSMARCO-DistilBERT-v4 [20] the latter having a substantially higher accuracy on our task.
For more details on training, refer Supp H.2.
Evaluation metric.
We use AUC(0.05) as the evaluation metric as in [21].
AUC(0.05) is the area under the ROC curve when the false positive rate ( fpr ) ranges from 0 to 0.05.
Results.
Table 3 shows the IID and OOD performance of models trained on the two forum datasets.
Let us first consider AUC on evaluation within the technical forums and using the less accurate
NLI-DistilBERT Base model.
On both IID and OOD evaluation, ITVReg obtains the highest
AUC for both datasets.
On OOD evaluation, MaskReg and ITVAug have the second-best AUC for SuperUser and AskUbuntu evaluation sets respectively.
OutReg has comparatively lower AUC because the base model is not accurate: ITVReg obtains > 10 points higher AUC on both OOD datasets.
Using the more accurate MSMARCO-DistilBERT-v4, accuracy of both OutReg and ITVReg increases.
P@1 on OOD data of OutReg is comparable to ITVReg on AskUbuntu evaluation andpoint higher than ITVReg on SuperUser evaluation.
In comparison, MaskReg and SimCSE fail to utilise the better Base model.
When evaluated on an extreme distribution shift ( Quora ), Base model performs the best on OOD evaluation (Table 3) , followed by OutReg .
Finetuning on AskUbuntu or
SuperUser adds no information.
In OOD setup (training on AskUbuntu or Superuser) we can see that base model does the best and fine-tuning brings no gains. [Table]
Summary.
Compared to other methods, ITVReg obtains high IID accuracy and OOD accuracy, thus exploiting the best of the Base model and the training data.
However, under extreme distribution shifts, finetuning training data is not useful and thus OutReg or the Base model is more suitable.
E.1 Ablations
while with our methods ITVReg you get " Channellock 804 4.5-Inch Adjustable Wrench, Chrome " as the top predicted label See Figure ??
for more details and Table 10,9 for qualitative samples
F Dataset Details
Quora, SuperUser,
AskUbuntu
The ratio of positives to negatives in SuperUser, AskUbuntu is 1:100, while in Quora is 4:7.
Amazon131K
In Amazon131K we have categorical information for 99K labels.
The categorical shift experiments are conducted on this filtered data.
G Hyper-parameter Tuning
For SimCSE the only choice of hyperparameter is the λ regularisation coefficient.
We search overvalues of λ i.e. 0.01,0.1,1.0 and select 0.1 as the best value.
For other methods also we fix λ as 0.1.
See Table
[Table]
For MaskReg and ITVReg another hyperparameter choice is the masking fraction of the input.
We search over 2 choices of masking namely masking 50% of input, or 15% of input.
We find that 15% works best for MaskReg while 50% gives best results for ITVReg .
We use these parameters for all experiments.
For MaskReg results are reported in Table 5, while for ITVReg results can be seen in
Table 11,13,12.
We also try running with lower learning rates but higher learning rate gives better numbers and hence we work with 1e-4.
See Table 6 for reference.
[Table]
[Table]
Since we deal with short sentences
[Table]
the max token length is fixed at 32 which allows for bigger batch sizes for Amazon Titles, 250 for Question Recommendation).
For ITVReg and OutReg we report numbers for various values of hyperparameter λ in Table 7.
We can see that for higher values of λ behaves more like the Base model but the numbers for both
ITVReg and OutReg are stable across this wide range of λ .
H Training Details H.1 AmazonTitle Recommendations Training Details
We follow the modeling procedure used in a recent state-of-the-art work [6].
For training, We use contrastive loss with hard negative mining.
we initialise the model as MSMARCO-DistilBERT model [Table] and fine-tune it for 200 epochs with a batch size of 900.
The only difference from [6]’s setup is that we train only till 200 epochs instead of convergence (300-400 epochs) due to computational constraints.
We use Adam optimizer with learning rate of 1e-4.
H.2 Question Recommendations Training Details
We train all the models for 5 epochs with a batch size of 250.
We use learning rate of 1e-4 to fine-tune the model.
We train the model with a MSE loss between the ground truth and the predicted similarities, as done in [22].
For Quora on the 10% subset, we train the model for 20 epochs with lr of 1e-4. [Table] λ .
We can see that both ITVReg and
OutReg behave well in these reasonable range of λ .
OutReg on higher values of hyperparameter imitates the Base model while ITVReg still learns useful information from the data
[Table] I Category-wise P@1 for AmazonCatOOD
We report category wise numbers OOD numbers for AmazonCatOOD (Table 2).
All categories have a similar trend.
Results can be seen in Table 8.
J Evaluation Metrics J.1 Precision@
Given a query, the model outputs a vector ˆ R ∈ R M , where M is the number of items and ˆ r j denotes the similarity between the query and the j th item.
We use the Precision @1 metric for evaluation, interpreted as the fraction of queries where the top-predicted item is in the query’s ground-truth relevant items.
Given predicted similarities ˆ r and ground truth similarities r (where r i is 1 if its a relevant item, otherwise 0), P @
k is defined as P @ k = 1 k (cid:80) [Table] r
[Table]
K Qualitative Samples
We give top 5 predicted labels as qualitative samples for " Soleus Air Oscillating Reflective Heater " query in Table 10 and for query " Rowenta Z100 Non Toxic Soleplate Cleaner " in Table 9.
L Complete Dataset Results
For the complete dataset results, we follow the same setup as described before, with the only difference being number of epochs.
We run Quora for 4 epochs and SuperUser, Askubuntu for 1 epoch.
Results can be seen in Table 11,13,12.
M 10% Dataset Subset Results
Results on 10% subset of the data (most of them are redundant as they overlap with results in main paper) can be seen here: Table 14,16,15.
N Experimental Budget
Computing Infrastructure We use 16GB V100 GPUs for all our experiments
Training Time
Our training time is around 1 hour for each run on Question Recommendation.
Amazon dataset takes 8 hrs for training.
Parameters Of Model We use DistilBERT model for all our experiments. [Table] [Table]
Other models rely on spurious feature i.e. Reflective for predicting items, which leads to wrong results
[Table] [Table] [Table] [Table]
[Table]
MSMARCO-DistilBERT-v
[Table] [Table] [Table]
MSMARCO-DistilBERT-v
[Table]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]