CHIME : Cross-passage Hierarchical Memory Network for Generative Review Question Answering (2020)

Abstract: We introduce CHIME, a cross-passage hierarchical memory network for question answering (QA) via text generation. It extends XLNet introducing an auxiliary memory module consisting of two components: the context memory collecting cross-passage evidences, and the answer memory working as a buffer continually refining the generated answers. Empirically, we show the efficacy of the proposed architecture in the multi-passage generative QA, outperforming the state-of-the-art baselines with better syntactically well-formed answers and increased precision in addressing the questions of the AmazonQA review dataset. An additional qualitative analysis revealed the interpretability introduced by the memory module.

## introduction with the development of large-scale pre-trained language models (lms) such as bert @xcite , xlnet @xcite , and t5 @xcite , tremendous progress has been made in question answering (qa). fine tuning pre-trained lms on task-specific data has surpassed human performance on qa datasets such as squad @xcite and newsqa @xcite . nevertheless, most existing qa systems largely deal with factoid questions and assume a simplified setup such as multiple-choice questions, retrieving spans of text from given documents, and filling in the blanks. however, in many more realistic situations such as online communities, people tend to ask 'descriptive' questions (e.g., 'how to improve the sound quality of echo dot?'). answering such questions requires the identification, linking, and integration of relevant information scattered over long-form multiple documents for the generation of free-form answers. we are particularly interested in developing a qa system for questions from e-shopping communities using customer reviews. compared to factoid qa systems, building a review qa system faces the following challenges: (1) as opposed to extractive qa where answers can be directly extracted from documents or multiple-choice qa where systems only need to make a selection over a set of pre-defined answers, review qa needs to gather evidence across multiple documents and generate answers in freeform text; (2) while factoid qa mostly centres on 'entities' and only needs to deal with limited types of questions, review qa systems are often presented with a wide variety of 'descriptive' questions; in our work here, we focus on the amazonqa dataset @xcite , which contains a total of 923k questions and most of the questions are associated with 10 reviews and one or more answers. we propose a novel cross-passage hierarchical memory network named chime to address the aforementioned challenges. regular neural qa models search answers by interactively comparing the question and supporting text, which is in line with human cognition in solving factoid questions @xcite . while for opinion questions, the cognition process is deeper: reading larger scale and more complex texts, building cross-text comprehension, continually refine the figure 1 : illustration of the review qa task and the general idea of chime. the example question (the top box) is paired with 10 reviews (left panel) and one or more answers (right upper panel). chime is trained on the (question, review, answer) triplet. during testing, chime is presented with a question and 10 related reviews and generates an answer (right bottom box). both reviews and answers in this example contain contradictory information as highlighted by colors, while the question contains complex sub-questions. chime is able to identify relevant evidence and generate clear answers. in summary, we have made the following contributions: our work is related to the following three lines of research: opinion/review question-answering in opinion or review qa, questions may concern about finding subjective personal experiences or opinions of certain products and services. the amazon qa dataset was first released in (mcauley and yang, 2016) which contains 1.4 million questions (and answers) and 13 million reviews on 191 thousand products collected from amazon product pages. they developed a mixture of expert (moe) model which automatically detects whether a review of a product is relevant to a given query. in their subsequent work, @xcite noticed that users tend to ask for subjective information and answers might also be highly subjective and possibly contradictory. they, therefore, built a new dataset with 800 thousand questions and over 3 million answers from amazon, in which each question is paired with multiple answers, and extended their previous moe model with subjective information such as review rating scores and reviewer's bias incorporated. but they found that subjective information is only effective in predicting 'yes/no' answers to binary questions and does not help in distinguishing 'true' answers from alternatives in open-ended 'descriptive' questions. more recently, @xcite only focused on the yes/no questions in the amazon qa dataset @xcite and trained a binary answer prediction model by leveraging latent aspect-specific representations of both questions and reviews learned by an autoencoder. @xcite focused on factual qa in e-commerce and proposed a product-aware answer generator that combines reviews and product attributes for answer generation, and uses a discriminator to determine whether the generated answer contains question-related facts. @xcite proposed an extractive review-based qa task and manually created just over 2,500 questions and annotated the corresponding answer spans in less than 1,000 reviews relating to laptops and restaurants from the review data of semeval 2016 task 5 foot_0 . they first jointly fine-tuned bert for answer span detection, aspect extraction and aspect sentiment classification on the semeval 2016 task 5 data, and then post-trained bert on over 3 million unlabelled amazon and yelp reviews in order to fuse domain knowledge, and also on squad 1.1 @xcite in order to gain task-relevant but out-of-domain knowledge. @xcite created a subset from the amazon qa product review dataset @xcite , consisting of 923k questions with 3.6m answers and 14m reviews on 156k amazon products. they trained an answerability classifier from 3,297 question-context pairs labeled by mechanical turk and used it to classify answerability for the whole dataset. they then converted the dataset into a span-based format by heuristically creating an answer span from reviews that best answers a question based on users' actual answers, and trained r-net @xcite , which uses a gated self-attention mechanism and pointer networks, to predict answer boundaries. there are few studies using generative models to deal with opinion/review-based qa. multi-passage qa there are mainly two types of methods for multi-passage qa. one is to use retrieval-based methods to first identify text passages that are most likely to contain answer information, and then perform qa on the extracted text passages which are essentially considered as a single passage. the other one is to separately run single-passage qa over each passage, obtaining multiple answer candidates, and then determine the best answer through mutual verification among the answers. examples in the first type of methods include s-net @xcite , multi-passage bert @xcite , and masque @xcite . these models require supporting text passages to be explicitly annotated. s-net @xcite follows an extraction-then-synthesis framework. first, relevant passages are extracted from context using a variant of r-net @xcite , which learns to rank passages and extract the most possible evidence span from the selective passage; then, the evidencenotated selective passage is used for the gru decoder synthesizing answers. in multi-passage bert @xcite , two independent berts were used to perform multi-passage qa. one bert takes the question and a text passage as input and then uses the hidden states of the cls token to train a classifier to determine if the text passage is relevant to the given question. the other bert is used for extracting candidate answers from relevant text passages. the masque model @xcite is a generative reading comprehension approach based on multi-source abstractive summarization. masque uses a joint-learning framework, comprising of a question answerability classifier, a passage ranker, and an answer generator. at each step of answer generation, the decoder chooses a word from the mixture of three distributions derived from a vocabulary, from the question and associated multiple passages. a representative example of the second type of methods is v-net @xcite . the main assumption of v-net is that correct answers often appear in multiple documents with high frequency and similarity, and wrong answers are usually different from each other. therefore, v-net builds a mutual verification mechanism between all answer candidates, which are separately extracted from different passages, to select the best final answer. most existing approaches require explicit annotations of supporting text passages in order to train multi-passage qa models in a supervised way. in our setup here, supporting review passages to a question was unsupervised ranked by bm25, which may introduce noises to qa model training and poses a more significant challenge. memory network memory network has been first proposed to model the relation between a story and a query for qa systems @xcite . apart from its application in qa, memory networks have also achieved great successes in other nlp tasks, such as machine translation @xcite , sentiment analysis @xcite , visual question answering @xcite , social networks @xcite , and summarization @xcite . the main idea of memory networks is to use the attention mechanism to assign different weights to text passages so as to identify the most relevant passages for answer generation @xcite . @xcite proposed a gated memory network to represent facts in different iterations during the learning process to verify the potentially related passages to generate an answer. @xcite used a convolutional architecture to capture attention signals in memory networks. @xcite leveraged the memory network as an information retrieval system to search possible entities in knowledge bases for complex questions. @xcite used the memory network to verify items in knowledge bases as passages and then generate answers. generally speaking, existing memory-network-based qa methods mainly focus on using memory networks to weigh and derive representations of question-aware text passages and knowledge entities for answer generation. we instead explore a novel structure of a hierarchical memory network composing of both context and answer memories for better capturing review context and generating more appropriate answers. ## cross-passage hierarchical memory network (chime) in this section, we first define the review qa task and then present our proposed cross-passage hierarchical memory network (chime). ## experiments in this section, we first introduce the dataset used in our experiments, the baselines for comparison, and the evaluation metrics employed, followed by a discussion over the obtained results and a few examples generated using the different approaches presented. ## conclusions in this paper, we have proposed chime, a cross-passage hierarchical memory network for multi-passage generative review qa. it is built on the xlnet generator @xcite by adding a memory module consisting of a context and a answer memory which guarantees a more accurate refining process for crosspassage evidence collection and answer generation. the sequential process adopted in chime makes it possible to elaborate longer text passages and some straightforward interpretability. we have assessed experimentally a significant quality improvement using different state-of-the-art metrics to measure the lexical and semantic coherence of the generated text. we plan to further extend chime to model with multiple ground truth simultaneously and leverage the available product attributes. appendix a more example outputs generated by various models (a) question: does this come with the straps?/ answer 1: no it doesn't sadly. answer 2: no. you have to buy separate. answer 3: it comes with the white cotton rope ending in knots typical of mayan hammocks but no tree hugger straps or hammock frame. answer 4: no it does not come with the straps or rope. i advise you to purchase parachord with it. xlnet + summary: yes, it comes with the straps and the hammock is included with it! it comes in two pairs of harnesses that are attached to each other, so you can use them on your porch or your bed, or just use them on a bed or bed xlnet + v-net: yes it comes with the straps and the sizing is very good and very easy to do! chime: yes it does. i love this hammock and love the quality of the straps (b) question: has anyone tried baking with this? answer 1: yes you can! designer whey also has recipes that you can make, like cookies, waffles or pancakes and muffins! xlnet + summary: i have never tried it with my own breads but i have had a few good ones and they were great! xlnet + v-net: i haven't tried baking with it, but i've been using this for a couple of weeks now and chime: i haven't tried baking with this, but the flavor is very good and the flavor is very nice and good! (c) question: can you put coals on the lid, like you can with some lodge models? answer 1: yes you can, answer 2: yes. i've done it many times. make sure you get one of those lid-lifters though so you don't pour ash or coals into the pot. answer 3: yes, this dutch oven is specifically designed with a tall lip to cup the coals on top of the lid to allow for baking. this has worked great for me in making cobbler and biscuits. answer 4: never heard of coals on the lid, sorry. it makes a great chili over an open pit bbq. we hang it over a pit in the winter out back and make stews, chili's and pot luck. makes for a fun time especially in a snow storm! answer 5: shouldn't be a problem. lid fits with good seal so no coals or ash should spill inside. xlnet + summary: yes, you can. i don't know if it is possible to put coals on the top of the lid and then place them on top of that xlnet + v-net: yes, you can put coals on the lid, but i don't think you can put coals inside the top, so i don've had any problems with this oven and have not tried any of these models with any problem with mine chime: yes, you can put coals on the lid, but i don't think you can put any coal in this one, as the lid is not designed for that type of use (d) question: in what stores are these sold? answer 1: target and target.com is where i've bought 3 pairs over time. answer 2: different sizes different stores. xlnet + summary: i don't know. i have never used them in my life, but they are very good for the foot, but it is not as good as the ones that are sold at stores xlnet + v-net: i don't know. i've been using them for a couple of years and it works great! they are very good and very good! chime: amazon has them in their store. i have had them for over 2 years and have never had any problems with it working for my feet, especially when table a1 : example outputs to compare the quality of chime answers and xlnets answers.