U ni TR ec: A Unified Text-to-Text Transformer and Joint Contrastive Learning Framework for Text-based Recommendation (2023)

Abstract: Prior study has shown that pretrained language models (PLM) can boost the performance of text-based recommendation. In contrast to previous works that either use PLM to encode user history as a whole input text, or impose an additional aggregation network to fuse multi-turn history representations, we propose a unified local- and global-attention Transformer encoder to better model two-level contexts of user history. Moreover, conditioned on user history encoded by Transformer encoders, our framework leverages Transformer decoders to estimate the language perplexity of candidate text items, which can serve as a straightforward yet significant contrastive signal for user-item text matching. Based on this, our framework, UniTRec, unifies the contrastive objectives of discriminative matching scores and candidate text perplexity to jointly enhance text-based recommendation. Extensive evaluation shows that UniTRec delivers SOTA performance on three text-based recommendation tasks.

## introduction text-based recommendation @xcite @xcite aims to recommend relevant textual content (e.g., news articles, twitter posts) to people based on their behaviors as represented in historical log texts. for instance, engagement recommendation @xcite on social media (e.g., twitter and reddit) helps users discover and engage with interested threads by modeling their browsing history. pretrained language models @xcite have made waves in recent text-based recommendation research @xcite @xcite . the most common practice is using plm encoders (bert family) to learn representations of user history and candidate item texts. recommendation matching scores are computed over the user and item representations and finally optimized by noise contrastive estimation (nce) loss @xcite for ranking multiple candidates. unlike encoding single text, using plm to encode multi-turn texts of user history is nontrivial. existing works @xcite @xcite concatenate multi-turn history texts as a whole input text, then use one plm encoder to learn the holistic user representation. this is a standard plm encoding manner but ignores the relation among history turns, as all word tokens from different history turns are equally attended foot_1 . in contrast, previous studies point out that learning the relation among user history turns is also beneficial @xcite ). another approach is using plm encoders to learn representations from multi-turn history texts, followed by an additional aggregation network to fuse the multi-turn representations @xcite . however, the imposed aggregation networks (with newly initialized parameters) weaken the representation power of plm encoders which are already pretrained on large-scale corpora. this work introduces unitrec, a unified text-totext transformer framework for text-based recommendation. in the encoder component of unitrec, we design local-and global-attention to learn user history representations through tailored attention masking, which aims to jointly model word-level and turn-level relations of user history. unitrec can utilize the full power of plm encoders because it preserves the intact structure of plm encoders without newly imposed parameters. different from most previous works that predict user-candidate matching scores solely based on the representations learned by transformer encoders, we argue that conditioned on user representations learned by transformer encoders, candidate text perplexity (ppl) estimated by pretrained transformer decoders is also a straightforward yet significant signal for text-based recommendation. as shown in figure 1 , we hypothesize that the candidate text perplexity estimated by pretrained lm decoders can directly measure the text matching degree between user history and candidate texts. it is because the perplexity estimates the likelihood of candidate texts based on encoder outputs, which naturally indicates the probabilities of candidate texts given the user history. besides, unitrec can use the last hidden states of transformer decoders to directly predict matching scores. hence, this work unifies the contrastive objectives of discriminative matching scores and candidate text perplexity to jointly enhance text-based recommendation. the contributions of this work are: (1) we propose local-and global-attention to model two-level relation of user history without additional parameters, which enjoys the full power of plm encoders. global attention on turn-level context. over the local self-attention layers, we leverage global self-attention to model the relation among history turns. specifically, tokens from all turns attend to each other in self-attention computation (by setting the mask matrix @xmath0. in this way, transformer encoders can perform global interaction among each token (and turn) to learn turn-level context representations of user history. there are l 2 layers in the global self-attention blocks, which can also be inherited from plm encoders directly. ## experiments we evaluate unitrec on three text-based recommendation tasks: 1) newsrec, to recommend news articles to users based on their browsing history. we use the mind-small dataset @xcite for experiments. 2) quoterec, to recommend quotations to users based on their conversation history. we use the reddit-quotation dataset @xcite for experiments. 3) engagerec, to recommend social media posts for users to engage with based on their comment history. we use the dataset released by @xcite for experiments. detailed dataset statistics is provided in appendix a. implementation details. the unitrec encoder and decoder both consist of 6 transformer layers with 768-dimensional hidden states and 12 attention heads. we set l 1 = 3 and l 2 = 3. we use adamw optimizer @xcite to train unitrec with cosine learning rate decay. baselines. we compare unitrec with competitive baselines: 1) gru4rec @xcite utilizes a gru network to learn multi-turn history. note that we do not consider other methods that use non-text inputs (e.g., user profile, text topic labels). for fair comparison, all baseline models use pretrained 12-layer roberta-base @xcite ) as text encoders to learn embeddings of texts. ## conclusion we present a unified transformer unitrec for textbased recommendation. unitrec learns two-level contexts of multi-turn user history and jointly exploits discriminative matching scores and candidate text perplexity as matching objectives. empirical experiments on three text-based recommendation datasets corroborate the effectiveness of unitrec.