Improving Specificity in Review Response Generation with Data-Driven Data Filtering (2022)

Abstract: Responding to online customer reviews has become an essential part of successfully managing and growing a business both in e-commerce and the hospitality and tourism sectors. Recently, neural text generation methods intended to assist authors in composing responses have been shown to deliver highly fluent and natural looking texts. However, they also tend to learn a strong, undesirable bias towards generating overly generic, one-size-fits-all outputs to a wide range of inputs. While this often results in ‘safe’, high-probability responses, there are many practical settings in which greater specificity is preferable. In this work we examine the task of generating more specific responses for online reviews in the hospitality domain by identifying generic responses in the training data, filtering them and fine-tuning the generation model. We experiment with a range of data-driven filtering methods and show through automatic and human evaluation that, despite a 60% reduction in the amount of training data, filtering helps to derive models that are capable of generating more specific, useful responses.

## introduction sequence-to-sequence (seq2seq) modelling with neural networks has proven to be an extremely popular and effective paradigm for a wide range of conditional text generation tasks @xcite @xcite . more recently, the development of large, pre-trained seq2seq models (e.g. @xcite has lowered the bar on the amount of labelled in-domain data required to train models on a particular task and still achieve highly grammatical and fluent text. however, generative models often tend to produce bland and generic text, which significantly inhibits their potential utility @xcite . this problem is especially prevalent in tasks with valid many-to-one mappings, where generic outputs occur frequently amazing food variety for a coeliac friendly staff and great service. apartment ideal for business trip maybe needs a bit updating for a family stay. will definitely be back for leisure stay. ideally situated. thank you for your glowing review! it is a delight to hear that you enjoyed your visit. we look forward to welcoming you again in the near future. thank you for your great review. our fantastic chefs do their best to cater to all kinds of dietary requirements. often creating off menu dishes when requested. we think they do a brilliant job. we hope to be able to impress you again with our service next time you stay with us. be sure to call us direct for the best rates available. response a: in the training data; in dialogue modelling it has been referred to as the "i don't know" problem @xcite . in this work we consider the task of automatically generating responses to online hospitality reviews. figure 1 provides an example of the task and presents a user-written hotel review along with two potentially valid responses. while response b is highly specific, addressing the opening comment of the review and the positive mention of the service, response a is generic. such a response would be applicable to a broad range of positive reviews, highlighting the many-to-one problem. defining exactly what constitutes a good review response is not straightforward. formal require-ments such as structure, style, intent and grammaticality are all important to consider, however, in this work we focus on content. popular webbased review platforms (e.g. google, tripadvisor, etc.) recommend that responses should address topics raised in the review specifically. such an approach is also supported by the gricean maxims of quantity (be informative) and relation (be relevant) @xcite . thus, we aim to avoid generating generic responses such as response a in figure 1 . yet, given a lack of constraints in response authorship, a significant portion of data that is available from online platforms consists of generic responses which are potentially of little benefit or even detrimental. in order to derive models that are capable of producing more specific, contentful responses, it is essential to mitigate the negative impact of these generic responses in the training data. a simple yet effective method for improving a model's performance toward a specific goal is to increase the amount of training examples that exhibit the associated target quality and decrease the amount that do not. however, depending on the objective, this can be difficult. for example, classifying an arbitrary piece of text for specificity is challenging since there is limited consensus on what exactly constitutes specificity @xcite . nevertheless, we investigate this idea and apply unsupervised scoring techniques to hotel review responses that aim to indicate a text's genericness. given these scores, we infer suitable thresholds and filter out highly generic training data examples. we find that refining the training data and using just 40% of the original training examples allows us to derive models that are capable of producing fewer generic review responses according to both automatic metrics and human evaluation. our code to reproduce the data used and relevant experiments is available on github. foot_0 2 background and related work review response generation thanks to an increasing awareness of the benefits associated with addressing online customer feedback @xcite , there is a growing body of literature on automated review response generation. previous work in this area has considered various domain applications and extended the basic encoder-decoder architecture to incorporate additional contextual information alongside a review text. @xcite generate responses for product reviews on an e-commerce platform using tabular product information as additional context, while @xcite focus on generating responses for smartphone app reviews and incorporate discrete external attribute features, such as the review rating and app category. @xcite later applied the same model to restaurant and hotel reviews in english and german and showed that extensive variability in hospitality responses (compared to app review responses) leads to considerably worse performance according to automatic metrics. combating generic outputs considerable work has been dedicated to mitigating generic outputs in dialogue models. one popular approach is to feed the model additional contextual information in order to encourage more 'contentful' responses. depending on the availability of relevant data, this might include the dialogue history @xcite , free text from an external knowledge source @xcite , or embedded topic signals derived from the input query @xcite . meanwhile, a number of works have focused on improving the model architecture @xcite @xcite or modifying the decoding strategy @xcite . since generic responses occur with high frequency in the dialogue training data they induce a strong, undesirable bias. thus, it also makes sense to tackle this problem at its source. previous work in this direction has aimed to remove uninformative training examples in a conditional framework by performing comparisons between source and target pairs @xcite . in contrast to dialogue data, review-response pairs typically consist of multiple sentences, resembling paragraphs rather than single sentences. this leads to extensive variance in the surface form on both the source and target side, rendering conditional approaches less suitable. for instance, initial investigations revealed that the best-performing approach presented in @xcite identifies only 5% of hospitality review-responses as generic. therefore, in contrast to previous works, we set out to identify generic responses independently of their corresponding source texts to improve our training data. ## methods in order to derive models that are capable of producing fewer generic responses, we consider removing them entirely from the training data. our hypothesis is that generic responses seen often during training encourage the model to learn 'safe' but uninformative responses and are thus detrimental to the model's ability to generate more specific responses. to investigate this, we define three potential methods for scoring a text's genericness within a corpus, operationalising these at the word, sentence and document level. we then derive suitable thresholds for each scoring method and filter training data examples according to their genericness. formally, given a response text in the training corpus @xmath0, we aim to assign a numerical score s, indicating how unique r is in relation to all other responses in the corpus. ## results automatic metrics according to both versions of chrf, all models trained on filtered data sets show considerable gains over the baseline model, trained on the entirety of the data. specifically, chrf computed against the true target shows smaller improvements over the baseline, while chrf computed against the corresponding source shows a relatively large improvement for all experimental models, bringing the degree of overlap in model outputs much more in line with human-written responses. dist-1 shows that intra-textual lexical diversity is consistent against the human-written responses for most models, indicating the lexical repetitions occur within a reasonable range for this relatively restricted domain. thus, there is no indication that our models are getting stuck in repetitive degenerate loops @xcite . self-bleu reveals considerable variation among all models in terms of the diversity between generated responses. according to this metric, the lm ppl filtering method ensures the most diverse response texts, while both the rule-based and bart baseline generate the least diverse texts by far. noticeably, the best scoring models are not quite on par with the diversity of human-written responses. however, this is to be expected given that neural models generally stick to generating higher-frequency words @xcite . this phenomenon is further indicated by a large discrepancy between the counts of unique lexical items observed. in terms of average response length, most models under-generate when compared to the humanwritten ground truth. here, the baseline models fall the shortest, which may be a useful proxy indicating higher genericness. meanwhile, the models trained on the lexical frequency-filtered subset show a tendency to generate longer responses. this may be due to the score being directly related to the word count of a text, despite normalisation. ## conclusion state-of-the-art approaches to conditional text generation involving transfer learning can be adapted to perform a wide range of domain-specific tasks with strong and convincing results. however, the content and quality of a model's outputs largely reflect that of the in-domain data used for fine-tuning. thus, care should be taken when deciding which data to use for training. in this paper we presented three unconditional scoring techniques for identifying and filtering generic responses in a parallel corpus of review-response pairs. results of both automatic and human evaluation revealed that this is an effective approach for helping to reduce the production of generic, one-size-fits-all outputs for review response generation in the hospitality domain, as well as for mobile app reviews. we have also shown that such an approach has potential side effects that must be handled appropriately before being utilised in a real-world scenario.