Retrieval-Augmented Controllable Review Generation (2020)

Abstract: In this paper, we study review generation given a set of attribute identifiers which are user ID, product ID and rating. This is a difficult subtask of natural language generation since models are limited to the given identifiers, without any specific descriptive information regarding the inputs, when generating the text. The capacity of these models is thus confined and dependent to how well the models can capture vector representations of attributes. We thus propose to additionally leverage references, which are selected from a large pool of texts labeled with one of the attributes, as textual information that enriches inductive biases of given attributes. With these references, we can now pose the problem as an instance of text-to-text generation, which makes the task easier since texts that are syntactically, semantically similar with the output text are provided as input. Using this framework, we address issues such as selecting references from a large candidate set without textual context and improving the model complexity for generation. Our experiments show that our models improve over previous approaches on both automatic and human evaluation metrics.

## introduction the ultimate goal of opinion mining and sentiment analysis @xcite is to automatically digest opinions of users towards a certain product to accommodate decision making. while some of these opinions are explicitly articulated in product reviews that users write, most of them are unknown since users have not bought most of the products. alternative solutions such as aspect-based sentiment analysis @xcite and recommendation systems @xcite exist, however these only offer superficial outputs that are not as expressive as textual reviews. thus, the task of automatically generating reviews given their attributes such as user and product, or review generation @xcite , is necessary to achieve this goal. most of the previous approaches @xcite have framed review generation as a2t (attribute-to-text problem), where the given input is a non-linguistic data (i.e., attribute identifiers for user, product, and rating) and the output is the review text. in this problem setup, the key challenge is to learn rich representations of the attributes, which are then used to produce the text using either template-based surface realization methods @xcite or neural-based decoders @xcite , as shown in the red box in figure 1 . however, it is difficult to learn these representations merely from the given attribute identifiers since they do not convey any semantics regarding the attributes. our key contribution is at2t (attribute-matched-text-to-text), of augmenting inductive biases of attributes with their matching reference reviews, as illustrated as the blue box in figure 1 . for example, as shown in figure 1 , multiple references together contain inductive biases such as frequently reviewed aspects of the product (e.g., talking about plot and character aspects) or habitual user phrases (e.g., "looking forward to the next book"). these references greatly help text generation since not only do they reinforce the representations learned from the attributes, they also allow the use of techniques used in sequenceto-sequence learning such as attention @xcite and copy @xcite mechanisms. in related problem domains of generating abstractive summaries or dialogue utterances, such bias is introduced by a t2t (text-to-text) approach, of generating an extractive summary first @xcite or retrieving informative prior turns @xcite , then generating final outputs using these as references. central to the framework is reference retrieval, since relevant references provide valuable context, but, in contrast, noisy references rather hinder generation. for reference retrieval in t2t, lexical features, e.g., tf-idf, have been used, assigning relevance based on the degree of word overlap between two texts. in at2t, however, lexical features are not directly applicable or effective. first, unlike t2t where input and output are both texts, our input is a list of identifiers, i.e., (user id, product id, rating). as a result, we cannot expedite the process of finding matching references, as in t2t solutions using lexical features for fast retrieval, e.g., using inverted index. second, lexical similarity cannot fully capture rating, as sentiment lexicons appear in a small portion of text @xcite . for example, flipping a single lexicon (from 'good' to 'bad') from lexically identical sentences can completely invert the rating. one alternative solution, complementing lexical similarity, is to assign additional credits to references labeled with the input rating. however, references -labeled with a different rating, but having useful rating-related context -are forced to be penalized. for example, in figure 1 , no additional credits are assigned to ref#2 labeled with the nearly opposite rating, although it contains useful context for the given rating, e.g., "enjoyed". we later empirically show that neither lexical similarity nor rating accuracy of references guarantee rating semantics of generated reviews. to address these limitations, we propose two approaches: pseudo-supervised and reinforcement learning framework, denoted as sl and rl respectively. first, we expedite matching in sl using identifiers. for efficient retrieval without lexical features, we propose a parametric coarse-filtering approach using attribute identifiers, having constant time complexity for each instance in a candidate pool. second, to generate reviews which are compatible with input rating, we retrieve references which maximize the rating accuracy of generated reviews -rather than references labeled with the input rating. rl enables such retrieval: a retrieval model is trained to maximize rewards including rating accuracy as well as lexical similarity of generated reviews. to validate the effectiveness of at2t, we perform experiments on a dataset consisting of product reviews from amazon books, aligned with their corresponding attributes: user, product and rating @xcite . our experiments using automatic evaluations show that utilizing relevant references hugely helps generation in terms of content similarity, and rating accuracy. moreover, our human evaluations show that our model generates more informative and grammatical texts compared to previous models. augmenting context using references while data-hungry neural models for some task may afford sufficient training resources, some other tasks such as sentence-level classification @xcite and summarization @xcite suffer from limited context, given a single sentence as context. review generation can be viewed as an extreme case of limited context, totally lacking textual context and thus depending solely on a small set of attribute identifiers as input. for text classification tasks, solutions have been to increase the context, by adding inherent and induced metadata such as topics @xcite and translations @xcite . meanwhile references have been used as additional source to augment context in text-to-text generation tasks such as summarization @xcite , machine translation @xcite , or dialogue system @xcite @xcite . references can be seen as a new and effective additional context that introduces inductive biases of attributes that can only be found in texts. however, retrieval task is much harder in our task than in previous tasks, as we have inputs of attribute identifiers having little information for retrieval. this work studies review generation task, where we are given review-specific attributes such as user, product, and rating, i.e., @xmath0, p, r} as input and the corresponding review @xmath1. we reformulate the problem by introducing references. in review domain, references are reviews from the training dataset that are either written by user u or written for product p. that is, references can be reviews from another user but from the same product, i.e., (u , p) where @xmath2, and vice versa foot_0 . this introduces an additional input to our text generation model: a set of n references @xmath3, where ## at2t the new problem setting introduces a major challenge since there can be a large number of references. as most of the references are irrelevant, efficiently and effectively selecting the optimal k references x * = {x * 1 , ..., x * k } ⊆ x is one of the essential sub-tasks to at2t. to this end, in at2t, models consist of (1) reference selection modules (reflect) where we select relevant references x * to given attributes a and (2) a generation module (gen) of utilizing references as inductive biases of the attributes. while, in t2t tasks, lexical features have been used for retrieval, those cannot be directly applied for at2t, having inputs of identifiers without text contents. to overcome such difficulty, we propose two approaches on retrieval stage with different learning schemes which are pseudo-supervised learning (sl) and reinforcement learning (rl). we first introduce a sl method to construct trainable reflect without text contents as inputs. then, we propose to train reflect using rl where we enable the model to effectively preserve rating semantics. we show an overview of our approach in figure 2 . ## sl-reflect given the attributes a and reference candidates x, reflect selects the most relevant k references in t2t tasks having texts as inputs, text-based, non-parametric matching has been used for retrieval, e.g., tf-idf. however, given only identifiers, we cannot use such matching. in this section, we explore a pseudo-supervised learning approach to train parametric retrieval models. relevance pseudo-supervision in contrast to t2t tasks, references should contain relevant contexts to the input rating, to guide gen to generate a rating-consistent review. although references largely comprising of words presented in target review are useful for providing overall contents, lexical similarity does not guarantee semantic relevance especially for rating, e.g., two identical sentences with one word difference such as "good" or "bad" have significant rating differences. to consider rating information, we propose to generate a pseudo-label z x j for each reference candidate x j . we use rating accuracy and lexical similarity which are linearly combined by λ: coarse-grained reflect to maximize computational efficiency, previous approaches in t2t coarsely retrieve the small number of promising references, x + = {x + 1 , ..., x + m } ⊆ x, using efficient matching such as tf-idf or inverted index, then rerank using more effective but expensive matching to select best k references, x * ⊆ x + where @xmath4. instead of text-based matching which is not available in our task, we propose to use attributebased parametric matching, which has o(n ) time complexity and is fully parallelizable. more formally, we match input attributes a with attributes of candidates a @xmath5, ..., a x n }, where a x @xmath6, p x j , r x j }. we encode attribute features using embedding matrices followed by a fully connect layer, and calculate relevance score using inner product between them. we denote concatenation of the attribute vectors for the input identifiers, u/p/r, by @xmath7. for simplicity, we use different superscripts for a to indicate that different embedding matrices are used with the same identifiers, and use different subscripts to indicate different identifiers. fine-grained reflect now that we narrowed down to m n reference candidates, we can afford to use expensive textual input features. fine-grained reflect accepts the references x + as input and outputs a score for each candidate that is used to select the final k references. to get document-level encoding d + j for each candidate x + j , we use a bidirectional lstm @xcite with attention pooling @xcite using a as attention query. we also add a self-attention layer @xcite to further contextualize reference {d + j } m @xmath8. ## rl-reflect sl has the advantage of efficient selection of references by lexical and rating similarity, but this cannot guarantee generated documents would preserve such similarity. in addition, "selection" of reference can be generalized into "composing" multiple documents. we first concretely describe the motivation for introducing rl-reflect using the example presented in figure 1 . pseudo-labels evaluate the lexical and rating similarity of the entire reference document, and encourage to select ref#3 in figure 1 . however, in a hypothetical scenario without ref#3, ref#1 and ref#2 can be "composed", with the former contributing to aspects to be discussed (e.g., "character" and "plot") and the latter to rating (e.g., "enjoyed"), while pseudo-labels may underestimate the importance of both. especially with respect to the goal of attaining rating semantics, pseudo-labels discourage reflect to retrieve ref#2, with nearly opposite rating to the ground truth, though other aspects can guide the generation strongly. instead, to compute the aggregated effect of partial contributions, we employ reinforcement learning (rl): we first sample a set of references, generate a review using the references, and then estimate partial relevance of those by how those are reflected in the generation. as illustrated with the above example, rl is better than sl, especially in scenarios where multiple references collaboratively contribute to the generation, while sl is comparable if some document is dominant in all aspects. specifically, training retrieval models using rl involves (1) sampling a set of references x * from a candidate pool; (2) generating a review y using gen with x * ; (3) calculating a reward for x * based on y and y ; and (4) adapting the sampling towards the direction to maximize the reward. we set reward function by replacing x j with y in the previously defined formula on relevance score (equation 1 ). to obtain rating accuracy of y , we first train a standard sentiment classifier comprising of a bidirectional lstm layer followed by an attention pooling layer, and then predict rating of y using the classifier. for efficient rl training, we use filtered references by coarse-grained reflect, i.e., x + , as candidates. a challenging part is adapting the sampling x * (the 4th step described above) during training, since sampling is a discrete operation which breaks continuity of a function and thus violates differentiability assumption in standard backpropagation algorithm. likelihood-ratio trick, proposed in @xcite , enables the backpropagation of gradients regardless of discrete sampling. our key contribution is to make rl training practical. note, we keep the weights of pretrained gen fixed to keep rl training cost low. first, we use bleu-1 as a proxy of look-ahead exploration. compared to random exploration for all references increasing variance too much, we prioritize exploration to those with high lexical similarity (e.g., bleu-1 no less than 0.2). we empirically found this prioritization reduces the variance without compromising the reference quality much. second, as greedily maximizing for r induces high variance, we maximize r-r instead of r, where r is a sub-optimal reward obtained by a baseline, specifically, p(x + ) @xcite . introducing a baseline performance is known to decrease the variance @xcite , and our experience was consistent. ## gen gen follows an encoder-decoder framework equipped with copying mechanism. first, we encode each reference x * j with l * j words using a bidirectional lstm with attention pooling. for each generation step, decoder uses both reference vectors o ref t and attribute vectors o emb t . as habitual words/phrases of users/products can be reused for generation, we allow gen to copy words from references. training gen we pretrain gen using top-k bleu-1 score references as inputs and cross-entropy on gold reviews as the loss function. when we trained gen using retrieved references by reflect, performance dropped significantly. we suspect this is because even a small amount of irrelevant references misguide training of gen. ## experiments we used the same dataset used in @xcite (amazon book reviews) to evaluate models. each instance in the dataset consists of a review and aligned attributes (user id, product id, and a rating ranging from 1 to 5). statistics of the number of references for each attribute are as follows: (1) minimum number of references is 6, 2, 13k for user, product, rating respectively, (2) maximum number of references is 1265, 351, 405k, and (3) average number of references is 33.34, 8.17, 131k. ## conclusion in this work, we study the problem of review generation guided by reference documents. attributespecific reference reviews provide useful inductive biases of given attributes, and such biases can be explicitly delivered to generated reviews using copying mechanism. inspired by promising results, as future work, we will investigate alternative means of guidance, such as keywords or topic distribution of attributes.