R eco BERT : A Catalog Language Model for Text-Based Recommendations (2020)

Abstract: Language models that utilize extensive self-supervised pre-training from unlabeled text, have recently shown to significantly advance the state-of-the-art performance in a variety of language understanding tasks. However, it is yet unclear if and how these recent models can be harnessed for conducting text-based recommendations. In this work, we introduce RecoBERT, a BERT-based approach for learning catalog-specialized language models for text-based item recommendations. We suggest novel training and inference procedures for scoring similarities between pairs of items, that don’t require item similarity labels. Both the training and the inference techniques were designed to utilize the unlabeled structure of textual catalogs, and minimize the discrepancy between them. By incorporating four scores during inference, RecoBERT can infer text-based item-to-item similarities more accurately than other techniques. In addition, we introduce a new language understanding task for wine recommendations using similarities based on professional wine reviews. As an additional contribution, we publish annotated recommendations dataset crafted by human wine experts. Finally, we evaluate RecoBERT and compare it to various state-of-the-art NLP models on wine and fashion recommendations tasks.

## introduction recommendation systems are a major component of content discovery in online stores. different recommendation systems are employed across a broad spectrum of domains, such as movies, music, groceries, and more. in each case, the recommendation system is associated with a different catalog of items comprising different descriptors, item properties, and metadata. this work deals with the case of generating item-to-item similarities based on item descriptions. personalized recommender systems make use of either or both collaborative filtering (cf) or content-based (cb) information @xcite . cf approaches build models based on users past behavior @xcite . on the other hand, cb recommenders use item meta-data such as properties, tags, and descriptions in order to build and match user and item profiles @xcite @xcite . a model that utilizes both cf and cb is called a hybrid recommender system. item-to-item recommendations are commonly used in large scale recommender systems such as on netflix (gomez-uribe and @xcite , amazon @xcite , xbox @xcite and many others. commonly found on product details page (pdp), these nonpersonalized recommendation lists are known to drive-up purchases as well as user engagement. similar to personalized recommendation, item similarities can be computed based on user activity, item meta-data or both, using a variety of different models. in a new store, where user data does not exist, item-to-item recommendations are computed using one or more content-based approaches that leverage item meta-data in order to compute item-to-item similarities. the extracted data may include images, videos, textual descriptions, and more. textual content-based recommendation systems leverage textual information about items, such as item descriptions and titles. these models usually rely on natural language processing (nlp) models to compute item-to-item similarities. a naive approach to produce recommendations from textual information is to infer similarities by embedding the textual description (and title) of every item in a latent space @xcite @xcite . item embeddings, that utilize textual descriptions, can be obtained via different types of language models. recently, self-supervised pre-training of language models have revolutionized the field of nlp. these techniques first utilize a self-supervised pretraining of a neural-based model using a large corpus of unlabeled text. then, apply fine-tuning for specific nlp tasks. among the recent selfsupervised pre-trained language models, bert @xcite has emerged as a very powerful method, achieving state-of-the-art results in a variety of nlp tasks such as sentiment analysis @xcite , language inference @xcite , sentence similarities @xcite and more. bert pre-training technique incorporates (1) reconstruction of randomly masked words (known as masked language model), and (2) predicting whether two sentences are consecutive (next sentence prediction). in this work, we build upon bert and introduce a novel technique for self-supervised pre-training of catalog-based language models. in addition, we introduce an inference technique that utilizes the above model for inferring item similarities that can be used for item-to-item recommendations in cold catalogs. hence, we name our technique re-cobert -a bert model adapted for textual based recommendations. recobert pre-training leverages selfsupervision to its fullest by utilizing a combination of a masked language model along with a title-description model. the latter comprises a learning task that reveals relationships between item titles and descriptions. in some cases, these relations can form a summarization task, for which titles are short sentences that summarize the longer descriptions. in other cases, catalogs may comprise items with implicit titles that incorporate a few words that were crafted for each item at hand. for both cases, the title-description task encourages the model to reveal the underlying connections between titles and descriptions, improves language understanding, and therefore yields more accurate embeddings. this results in an improved text-based item similarity performance in cold catalogs. importantly, recobert doesn't require item similarity labels nor usage data. we also introduce a new nlp wine recommendation task, demonstrating recobert's ability to find similar items in very complex domains. the task utilizes a publicly available dataset comprising 120k elaborate wine descriptions written by wine experts. the goal is to produce wine recommendations for each item in the dataset, in the form of other similar wines. we employed a professional wine sommelier to manually craft 1095 recommendations for ∼100 wines that form a "ground-truth" test-set for evaluations. for reproducibility, and as an additional contribution, we made these annotations publicly available foot_0 . importantly, the novel wine recommendations task introduced in this work is different and more complex than most nlp tasks usually considered. the wine reviews incorporate domain-specific semantics, taxonomy, and phrases, as well as picturesque descriptions of tastes, aromas, and colors. arguably, determining similarities between wine reviews is a challenging task, which requires a high level of intelligence and knowledge even to the average human. specifically, compared to the tasks presented in the glue benchmark @xcite , for which the average adult person can easily solve a query in few seconds, determining the similarity of wines based on their reviews may pose a challenge to most people and takes up to a few minutes even to wine enthusiasts and professionals. the main contribution of this paper is threefold: (1) we introduce recobert, a self-supervised training for catalog-based language model. (2) we introduce a novel inference technique that yields item-to-item similarities by leveraging recobert, and compare its performance against relevant baselines. (3) we introduce a novel complex nlp task of wine recommendations and publish a matched labeled test set crafted by a professional sommelier. ## related work recent methods in text-based recommendations suggest a hybrid approach that combines usage data with either traditional or neural-based nlp methods. in @xcite , the authors suggest a hybrid approach for recommendations that utilizes both session data (cf) and textual features from articles extracted by a convolutional neural network (cnn). additionally, in @xcite ; @xcite the authors proposed hierarchical bayesian models for learning a joint representation for textual content and personal ratings, using latent dirichlet allocation (lda), deep au-figure 1 : recobert receives title-description pairs corresponding to positive ("real") and negative ("fake") samples, extracted from a given catalog. (a) during training, the title-description pairs are propagated through the bert backbone and transformed into two feature vectors. these vectors are then fed into the tdm, minimizing a cosine loss between them. (b) in inference, four scores are computed. two scores propagate the seed and candidate items separately ("real" pairs). the other two scores utilize the tdm head and propagate title-description pairs extracted from both seed and candidate items ("fake" pairs). a recently proposed family of transformerbased language models @xcite @xcite uses multiple attention layers and a two-phase training procedure composed of unlabeled pre-training and supervised fine-tuning. these models show great promise in linguistic tasks, and were shown to exceed humanlevel baselines in specific tasks such as machine translation @xcite , question answering @xcite , and other related tasks @xcite . these models utilize sentence embedding techniques @xcite , where a text is encoded into low dimensional vectors that summarize the information in the input text. for example, in universal sentence encoder (use) @xcite , the authors suggest utilizing vectors extracted from a machine translation model for transfer learning to other nlp tasks. lately, @xcite ; @xcite ; aßenmacher and heumann (2020) claimed that human baselines are being surpassed by transformerbased models and others that exploit statistical cues in the well-known glue set @xcite . such models may suffer severe performance degradation when putting to use on real-world problems. hence, some argue that the tasks in the glue dataset no longer suffice for evaluating language understanding models. in this work, we propose a new language task that is much more complicated than the semantic similarity tasks in glue. motivated by extracting item similarities for recommender systems, our task is neither composed of single sentences nor sentence pairs. instead, the goal is to induce semantic similarity between wine items represented by sentence-paragraph pairs. due to the complexity of the wines domain, as well as the professional language and length of the wine reviews, our novel language understanding task requires a high level of intelligence and knowledge that exceeds the average human level. ## methodology let @xmath0. let t be the set of all possible sentences generated by w, including the empty sentence. additionally, let d be the set of all possible paragraphs generated by t . let c := {m i |m @xmath1, where each item is associated with a titledescription pair (titles are sentences, and descriptions are paragraphs). given a catalog c, the task is to infer a similarity function f : c × @xmath2, that scores the similarity between any pair of items s, @xmath3. in particular, f can be used to quantify a similarity score that ranks all the items in the catalog according to their semantic textual similarity with a given seed item @xmath4. ## wine recommendations from reviews we introduce a novel nlp recommendation task of finding wine recommendations from reviews. the task is based on a publicly available dataset from kaggle foot_1 , and a new test set, annotated by a professional wine sommelier. a common obstacle in evaluating similarity models is the lack of a relevant test-set or ground-truth. therefore, as part of this paper's contributions, we made this test publicly available. the kaggle dataset, together with our annotated ground truth, form a new text-based recommendation task that can be further used by others in the future. ## evaluation we evaluate recobert on two datasets: (1) wines catalog and (2) fashion catalog. for each catalog, we train a separate recobert model using 3 https://doi.org/10.5281/zenodo.3653403 the adam optimizer (kingma and ba, 2014) and a batch size of 16. ## discussion and conclusions in this work, we introduce a novel natural language recommendation task along with a novel annotated test set that together contribute to the state-of-theart research of text-based recommenders and language models. we present recobert -a model for text-based item similarity that (a) mitigates the discrepancy between training and inference phases in the classical bert model, by operating on sentence-paragraph pairs, (b) refines the backbone language model to provide more accurate embeddings, improving item similarities under the cosine metric, and (c) utilizes matched cosine scores as part of the inference process. in addition, we show that the unique mechanism behind recobert leads to significant improvements over the other baselines and across all metrics. recobert's preeminence stems from two properties of its tdm loss: first, feeding titledescription pairs allows recobert to apply crossattention between the tokens of both elements entailing an effective dependency between their embeddings. second, by leveraging the tdm task, recobert learns an additional task for revealing the underlying connections between item titles and descriptions, which reinforces the model to better specialize in the domain at hand. compared to other semantic textual similarity tasks, the proposed wine recommendations task, along with our published annotated test set, can shed light on the limitations as well as the key advances of state-of-the-art nlp models for recommendations. in addition, by publishing our annotated wine recommendations dataset, we intend to encourage the community to further explore the boundaries of other nlp models, assessing the ability of machines to understand complex human language.