Harmless Transfer Learning for Item Embeddings (2022)

Abstract: Learning embedding layers (for classes, words, items, etc.) is a key component of lots of applications, ranging from natural language processing, recommendation systems to electronic health records, etc. However, the frequency of real-world items follows a long-tail distribution in these applications, causing naive training methods perform poorly on the rare items. A line of previous works address this problem by transferring the knowledge from the frequent items to rare items by introducing an auxiliary transfer loss. However, when defined improperly, the transfer loss may introduce harmful biases and deteriorate the performance. In this work, we propose a harmless transfer learning framework that limits the impact of the potential biases in both the definition and optimization of the transfer loss. On the definition side, we reduce the bias in transfer loss by focusing on the items to which information from high-frequency items can be efficiently transferred. On the optimization side, we leverage a lexicographic optimization framework to efficiently incorporate the information of the transfer loss without hurting the minimization of the main prediction loss function. Our method serves as a plug-in module and significantly boosts the performance on a variety of NLP and recommendation system tasks.

## introduction since the advent of the deep learning era, learnable embedding layers for categorical and discrete features are basic modules for neural networks in many fields, such as natural language processing (nlp) @xcite @xcite , recommendation systems @xcite , electronic health record (e.g. @xcite @xcite . discrete items are first mapped to continuous representations in the feature space through the embedding layers and then processed by other neural modules. however, a key challenge in learning item embeddings is due to the long tail phenomenon @xcite : the frequencies of different items are usually extremely imbalanced and there often exists a large number of rare items that appear only a small number of times. as a result, it is very difficult to properly estimate the embeddings of rare items (e.g. @xcite @xcite @xcite . the rare items often suffer from the under-fitting issue because they appears infrequently in the data @xcite . to solve this long tail problem, transfer learning techniques are proposed (e.g. @xcite @xcite by propagating useful information from highfrequency items to rare items. the idea of these methods is to minimize the linear combination of the main prediction loss ℓ main with an auxiliary transfer loss ℓ transfer which encourages that the embeddings of the high-frequency and rare items follows a similar distribution, and hence allows us to the regularize the training of the rare items using the information from the common items. the basic assumption of the existing transfer learning techniques is that the distribution of the underlying embedding of the high-frequency and rare items are similar, which is necessarily false in practice. as a result, the transfer loss ℓ transfer introduces potentially harmful bias and impacts the overall performance. the goal of this work is to limit the potential harmful information in the transfer learning pipeline. we achieve this following two complementary directions: to improve the definition of ℓ transfer , we leverage the co-appearance information to identify the rare items that are expected to be embedded similarly to the high-frequency items; by applying the transfer loss only between them, we significantly reduce the potential harmful bias. to adaptively balance ℓ main and ℓ transfer , we formulate the transfer learning as a lexicographic (lexico) optimization, in which we minimize ℓ transfer subject to that the minimum of ℓ main is achieved. this allows us to incorporate the information of ℓ transfer , without interfering the minimization of the main loss ℓ main . we provide a simple algorithm for solving the lexico optimization by extending the dynamic barrier algorithm of @xcite to incorporate the item-wise structure in our problem. we verify the performance of our method in multiple applications, from language model, machine translation, named entity recognition to clickthrough rate (ctr) prediction. compared to different baseline transfer learning methods, our method achieves better or comparable results without tuning the hyper-parameters in all the applications. ## problem set: learning with long tail in the following, we first formulate the problem and then introduce the details about our method. we start by introducing the framework of transfer learning for rare item embeddings. assume we have a dataset d regarding a set of discrete items i, such as the words in nlp and the users and products in recommendation systems. a typical deep learning model consists of two following parts: both e and θ are parameters that we learn using the training data. in practice, the items i tend to exhibit a long tail distribution, consisting of both high-frequency, informative items that appear many times in the dataset, and also a large set of rare items that only appear a small number of times. as a result, the embedding vectors of the rare items may not be well estimated and hence deteriorate the performance. to address the long tail problem, a number of works have been proposed to train the parameters {e, θ} by regularizing the main prediction loss with a transfer loss that propagate information from the frequent items to the rare items: min {e,θ} {l(e, θ) := ℓ main (e, θ) + λℓ transfer (e)} , (1) where λ > 0 is a combination coefficient; the ℓ main is the main prediction loss on the training data based on model f θ and the embedding e, which is usually cross-entropy loss, kl divergence, and ℓ transfer is a transfer loss, designed to transfer the knowledge from the frequent items (the source domain) to the rare items (target domain). denote by s and t the items in the source and target domain, respectively. the transfer loss is defined as in practice, the source domain s and target domain t is chosen based on the item frequency with various heuristics. for example, @xcite and @xcite uses the pareto principle @xcite (or 80/20 rule), which places the top 20% high-frequency items in s and the rest in t . further, @xcite define the target domain as the items with frequency lower than a dataset-dependent threshold. ## harmless transferring for rare items the basic assumption that underpins the transfer loss ℓ transfer in (2) is that the distribution of the high-frequency items e s and rare items e i is similar. however, this is not necessarily true in practice. there could be rare items that find no similar highfrequency items and should be embedded differently. when this happens, the transfer loss ℓ transfer may provide biased and harmful information and hence deteriorate the overall performance. in this work, we propose two techniques to limit the potential harmful information in the transfer learning pipeline, by improving both the definition and the optimization of ℓ transfer . 1) in section 3.1, we improve the definition of ℓ transfer in (2) to reduce potential harmful bias. this is made possible by filtering out the irrelevant items from the target domain t based on the coappearance information between the items. 2) in section 3.2, we further limits the potential harmful impact of ℓ transfer by replacing the regularized loss in (1) with a lexicographic (lexico) optimization which minimizes ℓ transfer subject to that the minimum of ℓ main is achieved. this allows us to prioritize the optimization of ℓ main when the gradient direction ℓ main and ℓ transfer are conflicting with each other. we provide a simple algorithm for solving the lexico optimization by extending the dynamic barrier algorithm of @xcite to incorporate the item-wise structure in our problem. ## conclusion this work improves transfer learning for rare items embeddings in multiple tasks. in the future, we plan to 1) extend our method for more general methods and settings in the embedding learning problems, 2) apply our method to datasets with larger-scale vocabulary, and 3) develop an online version algorithm for selecting the target and source domain. in