semi PQA : A Study on Product Question Answering over Semi-structured Data (2022)

Abstract: Product question answering (PQA) aims to automatically address customer questions to improve their online shopping experience. Current research mainly focuses on finding answers from either unstructured text, like product descriptions and user reviews, or structured knowledge bases with pre-defined schemas. Apart from the above two sources, a lot of product information is represented in a semi-structured way, e.g., key-value pairs, lists, tables, json and xml files, etc. These semi-structured data can be a valuable answer source since they are better organized than free text, while being easier to construct than structured knowledge bases. However, little attention has been paid to them. To fill in this blank, here we study how to effectively incorporate semi-structured answer sources for PQA and focus on presenting answers in a natural, fluent sentence. To this end, we present semiPQA: a dataset to benchmark PQA over semi-structured data. It contains 11,243 written questions about json-formatted data covering 320 unique attribute types. Each data point is paired with manually-annotated text that describes its contents, so that we can train a neural answer presenter to present the data in a natural way. We provide baseline results and a deep analysis on the successes and challenges of leveraging semi-structured data for PQA. In general, state-of-the-art neural models can perform remarkably well when dealing with seen attribute types. For unseen attribute types, however, a noticeable drop is observed for both answer presentation and attribute ranking.

## introduction product question answering (pqa) is playing an increasingly important role in e-commerce platforms. it is able to greatly improve the online shopping experience since customers do not need to traverse over the detailed web pages to seek information themselves. traditional approaches built structured knowledge bases for product attributes and mapped customer questions into executable queries @xcite @xcite . in recent years, with the rapid progress of large-scaled pretrained neural models, many research works have achieved promising results by leveraging only unstructured text, like product descriptions, user reviews and community answers @xcite @xcite . lying between these two source types, a lot of product information is often organized in a semistructured form, e.g., key-value pairs, lists and tables from product web pages, json and xml files from internal databases, etc. these semi-structured data can be a valuable answer source since they are better organized and more precise than free text, while being much cheaper to maintain than structured knowledge bases. nonetheless, few research works have ever considered them and there is no public available dataset for its study. this paper aims to fill in this blank and study how to effectively incorporate semi-structured answer sources for pqa and present answers in a natural sentence. to this end, we construct a dataset to benchmark this study. it contains 11,243 product questions about json-formatted semi-structured data foot_0 . the data contains 320 unique attribute types (size, material, color, etc) spanning a diverse set of semi-structured forms like key-value pairs, lists and hierarchies. each data is paired with manually annotated text that describes its contents. for attribute ranking, we build our model upon state-of-the-art pre-trained language models. due to the small size of our training data, we follow the common practice of prefinetuning the attribute ranker on four large-scale qa datasets: natural questions @xcite , amazonqa (mcauley and yang, 2016), newsqa @xcite and squad @xcite . since these are all based on unstructured text, we also experiment with converting semi-structured attributes into text before being passed to the ranker. our results show that text-based qa models are quite robust to semi-structured data representations, and can rank attributes correctly with only keyword matching without the extra order information. for answer presentation, we consider a questionindependent answer presenter, which is less risky than question-dependent presentation while being more flexible than span extraction or multi-choice selection. we evaluate both a template-based system and a neural sequence-to-sequence generation model. each template is one or more sentences with gaps that can be filled with pre-defined rules @xcite . however, semistructured data does not follow any unified schema, so designing rules to cover all possible data forms or unseen attributes is infeasible. our neural generation models are initialized with bart @xcite and t5 @xcite , two representative pretrained models for generative tasks, and fine-tuned on a small set of annotated examples. compared with the template system, we show the neural approach improves not only the fluency, but also the faithfulness of presented answers. finally, we discuss and analyse the challenge of generating factually-correct sentences without hallucinate information, as well as the difficulty of handling unseen attributes in both ranking and answer presentation. ## dataset the data collection contains 3 stages: semistructured attribute collection, text annotation and question sourcing. this section will explain these three stages in order then present the statistics. attribute collection we obtain the semistructured attributes of product information from our internal database. these attributes are aggregated from different providers with varied schema. we select 320 unique attribute types from it, filter out information only for internal use and indicator tags containing no actual information like "lan-guage_tag", "attribute_id" etc. for each of the 320 attribute types, we randomly sample 20 products containing such attribute from 5m products sold in the us market (the 5m products are randomly sampled from different categories), then extract their attribute instances. after removing duplicate ones, we get 3,316 unique attribute instances in the end. we then preprocess them to lower-case all characters, remove emojis and normalize all floats to contain at most 2 decimals. text annotation after obtaining the semistructured attributes, we hire annotators from amazon mechanical turk to write a natural sentence for each attribute instance. we restrict to us-based annotators who @xmath0, out of which more than 97% had been accepted. before the formal annotation, we did a pilot study with 100 samples. without extra information, we find 16% of attributes are not understandable to humans, which indicates proper context is necessary to understand the meanings of attributes. therefore, we also provide the product image and title in the second round of pilot study. by adding the extra information, only 4% of them are not understandable. we then continue with this setting and get all attributes annotated. we also remove all attributes that are not understandable to annotators (usually those that rely on other information to interpret), and end up with 3,191 attribute instances annotated with their description text. question sourcing we collect questions on mechanical turk by present annotators with the image, title and rating of the product plus one of its associated attribute instances. annotators are asked to imagine themselves as potential customers, and their task is to ask four questions about this attribute, which means that the attribute contains the information to answer their question. we explicitly add three criteria that annotators must follow: quesis the body made out of nylon? data (key-value): fabric_type:{ value:"body: nylon/spandex; cup linings:100% polyester;cup pad:100% polyurethane."} text: the body is made of nylon and spandex, the linings in ... question: what kind of devices fit in this? data (list): compatible_devices: {value:"apple ipad mini 4"}; {value:"apple ... text: the product is compatible with apple ipad mini 4, apple ipad air... question: is this metal? data (hierarchy): blade:{material:[{value:"plastic"}],length:[{unit:inches,value:3.0}]} text: the blade on this measures 3 inches and is plastic. ## attribute ranking attribute ranking aims to select the proper attribute that contains the information to answer the userposed question. we start from a tf-idf baseline, which has been shown a strong baseline for sentence matching tasks @xcite . we count the frequency based on the attribute instances on the training set. at test time, we convert question and answer candidate into tf-idf vectors based on the counted frequency, then compute their cosine similarity as the ranking score. following the common practice, we also tried concatenating the question and candidate attribute into one sequence then feeding into the robertabase encoder @xcite , a transformerbased neural model pretrained on billions of text. the final classifier is built on top of the representation of the first [cls] token. the multi selfattention layers of the encoder makes sure each token is able to interact with all other tokens to capture the dependency relations. the model is trained to maximize the likelihood of the positive candidates and minimize that of the negative candidates. as for the input form of the semi-structured attribute, we experimented with 5 forms: (1) nameonly: only use the attribute name as input. (2) value-only: only use the attribute value as input. due to the limited size of our training data, we follow a two-step setting @xcite where the roberta-base model is first fine-tuned on a largescale qa dataset, then fine-tuned on our semipqa training data. this has been shown to improve performance in the low-resource setting @xcite . we consider 4 datasets: (1) nq: the google natural questions dataset. we use its sentence selection version @xcite , where its negative samples are cate-gorized into 4 classes to improve the robustness of the model. it contains 61,186 questions from the google queries for training. (2) amazonqa: qa pairs from the amazon community qa website @xcite . we remove answers containing "i don't know", "i'm not sure" etc, and filter questions more than 32 words and answers more than 64 words. negative candidates are answers about different questions under the same product. it contains 1,065,407 community questions for training. (3) newsqa: qas about news articles @xcite . we convert it into sentence selection and drop the span label. for each question, we sample 5 negative sentences not labeled as correct for training. the training dataset contains 75,473 questions. (4) squad: qas about wikipedia paragraphs @xcite . we treat sentences containing the ground-truth span as positive and other sentences in the same paragraph as negative. the training dataset contains 87,599 questions. notably, all answers in the above 4 datasets are in form of unstructured sentences.. we analyzed the performance under three settings: (1) zeroshot where the model is applied directly to the testsets without using our training data, (2) performance on seen attributes after finetuning on the training data and (2) performance on unseen attributes after finetuning on the training data foot_1 . precision@1 results are shown in figure 1 . we also computed other metrics like map, mrr and hit@5, but they show a similar trend and are omitted for space limit. zeroshot performance the zeroshot results are visualized in the upper part of figure 1 , where we apply the rankers finetuned on different datasets to directly test on our data. as can be seen, when only the attribute name or value is available, the performance is significantly lower than the others, both for neural models and the tf-idf baseline. this suggests we need information from both the attribute name and value to rank attributes properly. neither of them are sufficient by its own. neural models finetuned on unstructured text can generally adapt well to semi-structured data (linerized form), except for the one finetuned on nq which performs poorly compared with others. one reason could be that the negative samples from nq are finer-grained. it must learn to differentiate between sentences containing correct answer spans but talking about irrelevant things, and correct sentences. therefore, it must rely on the sentence structure to infer the meaning and decide whether it is relevant or not @xcite . when directly tested on semi-structured jsons, it cannot easily interpret non-natural sentences. when finetuned on other datasets like newsqa, amazonqa and squad, negative samples are randomly sampled and hardly contain the correct answer span, so the model might only rely on span detection and do not need well-formed sentences. using template-generated text leads to the best zeroshot performance for all models, next come the neuralgenerated text and linearized json which perform slightly worse. among all datasets used for finetuning, amazonqa adapts best for all input formats. this is not surprising considering that it is also about product questions and has the largest data size for finetuning. finetuned performance on seen attributes the finetuned results on seen attributes is visualized in the middle of figure 1 . "roberta" indicates the model is initialized with the roberta-based checkpoint without being finetuned on any other qa datasets in advance. "nq" indicates that the roberta-based model is first finetuned on nq, then finetuned on our training data, same for "ama-zonqa", "newsqa" and "squad". similarly to the zeroshot setting, using only the attribute name or value leads to significantly worse results, although attribute names seem to be more important when finetuning on the training data. using the linearized json format and the template-generated text have the best overall performance, achieving a precision score of over 85%. using more natural and fluent text does not help the ranking performance on seen attributes. although neural generated text are of higher-quality according to human evaluations (to be shown in section 4), this does not make the ranking task easier and leads to performance drop, suggesting that presentation is not a requirement for ranking and can be addressed separately. prefinetuning on large-scale text-based qa datasets also does not help the performance on seen attributes, as the roberta result already achieves similar performance. the model is able to quickly learn the correspondence between questions and seen attributes even with the limited training data. ## answer presentation the first approach we consider for answer presentation is to use handcrafted templates. however, defining a perfect template for each attribute is challenging due to the lack of a standard schema and templates cannot scale to unseen attributes. with this concern, we also experiment with training a neural data-to-text generator trained with annotated text as the target. template system when designing the template system, we aim to capture general rules across different attribute types so that one template can be reusable to other similar attributes. we define each template should contain (1) a precondition specializing when to apply the template, (2) one or several corresponding text with gaps to fill, and (3) a set of rules defining how to fill in the gaps. for example, the following is a template defined from the attribute type are_batteries_required: precondition: applies if the pos tag of the attribute name follows the pattern of be_noun_verbed. rule: (1) if the value is "y" or "yes" or "true": output "it verbs the noun". during template construction, we maintain a template bank starting from empty. as we see more attribute types, we check if any template from the bank can be applied, and if so, whether it generates the correct text or whether we need to manually update the template. otherwise, we create a new template for this attribute type. this process is repeated until we go over all the 320 attribute types three times, to refine, merge and fix the template bank and rules. after these rounds, we end up with a total of 23 distinct templates. attribute value text { value:"gaspowered"} the product is gaspowered. { value:"batteries"} it runs on batteries. { value:"manual" } this doesn't have power. { value:"na" } this doesn't run on any power. neural generator to avoid pre-defined rules and to generalise to unseen attributes, we train a neural generator model initialized either with bart @xcite or t5 @xcite , two stateof-the-art generative models pretrained on large amount of web text with self-supervised objectives. as input, we feed the linearized json-formatted data foot_2 and the output is the annotated text. we further normalize the numbers in both the attribute and text to keep them in a consistent form, to help the model learn their correspondence in the generation task. for example, we turn forms like "1.", "1.0" and "1.00" into 1, and normalize words to numeric values ("one" → "1" etc). to minimize the changes of hallucination in the generation, we also delexicalize words in the an- @xcite @xcite . the tag is the linearized path from the root node (attribute name) to the tag of the value. for example, for the second sentence in ## conclusion in this work, we study how to effectively leverage semi-structured data for product question answering. as there is no public datasets for this problem, we collect a dataset containing manually annotated questions together with description text about semi-structured attributes from our internal database. we present empirical results and findings about two key challenges of this problem: attribute ranking and answer presentation . experiments show that neural models can provide superior text than template systems and perform well for ranking seen attributes, albeit there is still a noticeable drop when it comes to unseen attributes for both ranking