Which is Better for Deep Learning: Python or MATLAB ? Answering Comparative Questions in Natural Language (2021)

Abstract: We present a system for answering comparative questions (Is X better than Y with respect to Z?) in natural language. Answering such questions is important for assisting humans in making informed decisions. The key component of our system is a natural language interface for comparative QA that can be used in personal assistants, chatbots, and similar NLP devices. Comparative QA is a challenging NLP task, since it requires collecting support evidence from many different sources, and direct comparisons of rare objects may be not available even on the entire Web. We take the first step towards a solution for such a task offering a testbed for comparative QA in natural language by probing several methods, making the three best ones available as an online demo.

## introduction comparison of objects of a particular class (e.g., holiday destinations, mobile phones, programming languages) is an essential daily task that many individuals require every day. according to @xcite , comparative questions constitute around 3% of queries submitted to major search engines-a non-negligible amount. answering a comparative question (what is better, x or y?) requires collecting and combining facts and opinions about compared objects from various sources. this challenges general-purpose question answering (qa) systems that rely on finding a direct answer in some existing datasets or extracting from web documents. nowadays, many websites (e.g. diffen, wolphra-malpha, or versus) provide users with a comparison functionality. furthermore, the task of answering comparative questions has recently attracted the attention of the research community @xcite @xcite . most of the current research suggests that an answer to a comparative question not only should indicate the "winner" of comparison but also provide arguments in favor of this decision and arguments that support the alternative choice. therefore, we argue that a comparative qa system should be a combination of an argument mining engine and a dialogue system that mimics a human expert in the field. in this work, we make the first step towards the development of such technology. namely, we develop a comparative question answering system (coqas), an application that consists of a natural language understanding (nlu) module that identifies comparative structures @xcite in free input questions and a natural language generation (nlg) module that constructs an answer. we tested various options for both nlu and nlg parts ranging from a simple template-based generation to transformers-based language models. the main contributions of our work are threefold: (i) we design an evaluation framework for comparative qa, featuring a dataset based on yahoo! answers; (ii) we test several strategies for identification of comparative structures and for answer generation; (iii) we develop an online demo using three answer generation approaches. a demo of the system is available online. foot_0 besides, we release our code and data. ## related work text generation most of the current text natural language generation tasks @xcite are based on se-quence to sequence model architecture @xcite . the existing generation methods are developed by employing attention mechanism @xcite and pointer-generator network @xcite . more recent work on text generation focus on generating natural language using multitask learning from multi-document or multi-passage sources @xcite . however, in our generation task, we have a list of arguments used to build the final answer. this makes our task similar to unsupervised summarization. there exist several approaches for tackling the latter task, e.g. graph-based @xcite and neural models @xcite . a common approach to summarization is based on the textrank graph algorithm @xcite . comparative qa according to @xcite , questions can be divided into 6 coarse and 50 fine-grained categories, such as factoid questions, list questions, or definition questions: we focus on comparative questions. @xcite proposed one of the first works on automatic comparative web search, where each object was submitted as a separate query, then obtained results were compared. opinion mining of comparative sentences is discussed by @xcite and @xcite , yet with no connection to argumentation mining. instead, comparative information needs are partially satisfied by several kinds of industrial systems mentioned above. @xcite proposed comparative argumentative machine (cam) foot_1 , which a comparison system based on extracting and ranking arguments from the web. the authors have conducted a user study on 34 comparison topics, showing that the system is faster and more confident at finding arguments when answering comparative questions in contrast to a keyword-based search. @xcite presented args.me, a search engine for retrieving pro and con arguments given for a given controversial topic. the input to this system is not structured but rather a query in a free textual form. the touché shared task on argument retrieval at clef @xcite featured a related track. the task was to retrieve from a large web corpus documents answering comparative question queries like "what ide is better for java: netbeans or eclipse?". python or matlab for deep learning? objects: -python -matlab aspects: -deep learning sequence tagging: crf / lstm / bert ... natural language understanding cam / args.me / touché ... python pros: -... -... cons: -... matlab pros: -... -... cons: -... we chose python because it was easy to learn and fun to work with. it has a large community of ... comparative argument retrieval templates / ctrl / snippets / ... ## natural language understanding the goal of the nlu module is to identify the objects to compare and comparison structure aspects and predicates if they were specified. we cast this as a sequence labeling task. training dataset to train the nlu, we created comparely, a dataset with comparative sentences manually labeled with objects, aspects, and predicates. first, we extracted comparative sentences for 270 object pairs from the dataset of (not) comparative sentences by @xcite . we extracted them from depcc corpus @xcite using cam. we then performed manual labeling (two annotators) using webanno @xcite . some of the extracted sentences were not comparative, so the annotators were instructed to discard them. the majority of sentences were labeled once, but we also labeled 200 of them multiple times to compute the inter-annotator agreement. the cohen's κ for the aspect labeling is 0.71 (substantial agreement). for predicates and objects, the values are 0.90 and 0.93, respectively-perfect agreement. the dataset consists of 3,004 sentences, each of them has a comparison of two or more distinct objects and at least one aspect or predicate. the average length of sentence is 26.7 words ( method identification of comparative question components (objects, aspects, predicates, or none) is a sequence-labeling task, where the classifier should tag respective tokens in an input question. we test several common baselines starting with simple one-layer bidirectional lstm described by @xcite where the input is encoded with glove @xcite embeddings. for some further improvements, we add conditional random field @xcite we also experiment with transformers @xcite using a pre-trained bert model @xcite and roberta @xcite , which is its modification yielding better performance. for every classifier, during training, we tune hyperparameters by varying a batch size (from 16 to 100) and a learning rate (from 10 -6 to 10 -2 ). to find a proper converge of the training process, we apply two types of learning rate schedulers: linear with warmup and slanted triangular. for the model with the highest achieved f1 (roberta), we employ stochastic weight ensembling @xcite , i.e., we interpolate between the weights obtained by training a certain model with different random seeds. all models were trained on the comparely dataset and tested on its manually relabeled subset of 400 sentences. the overview of the classifiers' effectiveness is shown in ## comparative answer generation based on comparative sentences retrieved by cam, we develop several generation approaches to construct a human-like concise answer: (1) generation with pre-trained transformers-based language models, (2) retrieval of argumentative sentences ranked by cam or textrank, (3) extracting context of sentence retrieved by cam as support for the "winning" object, and (4) entering extracted comparative structures in templates. ## user study to additionally evaluate the proposed answer generation methods, we also collect human assessments in a small user study for the three models with the highest rouge scores (ctrl: question 100, experimental setup for our study, we randomly sampled 30 comparative questions from the yahoo! answers dataset and generated answers using three methods: ctrl: question 100, cam: bullet points, and cam: first snippets. additionally, since we used yahoo!'s "best answers" as ground truth for automatic evaluation, we asked our participants to also assess the quality of the human "best answers". for the user study, we internally recruited five (under-)graduate students. we focused on the two answer evaluation criteria: ## conclusion we present a comparative question answering system targeted at answering comparative questions, such as "is x better than y with respect to z?". our system is based on the comparative argument mining (cam) system-a tool that retrieves from a large corpus textual comparative arguments for two to-be-compared objects. we extend cam with an nlu module that identifies objects and aspects in a user textual query and highlights them in the answer, and a generation module that gives a concise and coherent answer based on the retrieved information. evaluation of generation methods showed that a ctrl-based answer generation has a better performance with respect to rouge-1, and sentence retrieval methods provide superior rouge-2 and rouge-3 scores. we hope that the presented testbed for comparative qa and the set of baseline approaches will pave the way for further research.