z-uo/qasper-squad huggingface dataset:
Paper: CommonGen: A Constrained Text Generation Dataset Towards Generative Commonsense Reasoning
In this section, we formulate our task with mathematical notations and discuss its inherent challenges. The input to the task is a set of $n$ concepts $x=\lbrace c_1,c_2,\dots ,c_n\rbrace \in \mathcal {X}$, where $c_i\in \mathcal {C}$ is a common noun or verb. $\mathcal {X}$ denotes the space of concept-sets and $\mathcal {C}$ stands for the concept vocabulary. The expected output of this task is a simple, grammatical sentence $y\in \mathcal {Y}$, describing a natural scene in our daily-life that covers all given concepts in $x$. Note that other forms of given concepts are also accepted, such as plural forms of nouns and verbs. In addition, we also provide rationales as an optional resource to model the generation process. For each pair of $(x, y)$, a rationale $r$ is a list of sentences that explains the background commonsense knowledge used in the scene recovering process.
Question: What automatic metrics are used for this task?
Paper: CommonGen: A Constrained Text Generation Dataset Towards Generative Commonsense Reasoning
In this section, we present how we build the CommonGen dataset for testing machine commonsense with generative reasoning. The overall data collection process is as follows. 1) We first collect a large amount of high-quality image/video caption sentences from several existing corpora, 2) Then, we compute co-occurrence statistics about concept-sets of different sizes ($3\sim 5$), such that we can find the concept-sets that are more likely to be present in the same scene. 3) Finally, we ask human crowd-workers from AMT to write scenes with rationales for every given concept-set, which serve as our development and test sets. The training set consists of carefully post-processed human-written caption sentences, which have little overlap with dev/test sets. We present the statistics and show its inherent challenges at the end of this section.
Question: Are the models required to also generate rationales?
Paper: CommonGen: A Constrained Text Generation Dataset Towards Generative Commonsense Reasoning
Following the general definition in the largest commonsense knowledge graph, ConceptNet BIBREF11, we understand a concept as a common noun or verb. We aim to test the ability of generating natural scenes with a given set of concepts. The expected concept-sets in our task are supposed to be likely co-occur in natural, daily-life scenes . The concepts in images/videos captions, which usually describe scenes in our daily life, thus possess the desired property. We therefore collect a large amount of caption sentences from a variety of datasets, including VATEX BIBREF4, LSMDC BIBREF12, ActivityNet BIBREF13, and SNLI BIBREF15, forming 1,040,330 sentences in total.
Question: Are the rationales generated after the sentences were written?
Paper: CommonGen: A Constrained Text Generation Dataset Towards Generative Commonsense Reasoning
It is true that the above-mentioned associated caption sentences for each concept-set are human-written and do describe scenes that cover all given concepts. However, they are created under specific contexts (i.e. an image or a video) and thus might be less representative for common sense. To better measure the quality and interpretability of generative reasoners, we need to evaluate them with scenes and rationales created by using concept-sets only as the signals for annotators.
Question: Are the sentences in the dataset written by humans who were shown the concept-sets?
Paper: CommonGen: A Constrained Text Generation Dataset Towards Generative Commonsense Reasoning
We present the statistical information of our final dataset. Firstly, we summarize the basic statistics in Table TABREF9, such as the number of unique concept-sets, scene sentences, and sentence lengths. In total, there are 3,706 unique concepts among all concept-sets, and 3,614/1,018/1,207 in the train/dev/test parts respectively. Note that there are 4% of the dev and 6% of the test concepts never appear in the training data, so we can better understand how well trained models can perform with unseen concepts.
Question: Where do the concept sets come from?
Paper: MMM: Multi-stage Multi-task Learning for Multi-choice Reading Comprehension
Building a system that comprehends text and answers questions is challenging but fascinating, which can be used to test the machine's ability to understand human language BIBREF0, BIBREF1. Many machine reading comprehension (MRC) based question answering (QA) scenarios and datasets have been introduced over the past few years, which differ from each other in various ways, including the source and format of the context documents, whether external knowledge is needed, the format of the answer, to name a few. We can divide these QA tasks into two categories: 1) extractive/abstractive QA such as SQuAD BIBREF2, and HotPotQA BIBREF3. 2) multiple-choice QA (MCQA) tasks such as MultiRC BIBREF4, and MCTest BIBREF5.
Question: How big are improvements of MMM over state of the art?
Paper: MMM: Multi-stage Multi-task Learning for Multi-choice Reading Comprehension
In MCQA, the inputs to the model are a passage, a question, and answer options. The passage, denoted as $P$, consists of a list of sentences. The question and each of the answer options, denoted by $Q$ and $O$, are both single sentences. A MCQA model aims to choose one correct answer from answer options based on $P$ and $Q$.
Question: What out of domain datasets authors used for coarse-tuning stage?
Paper: MMM: Multi-stage Multi-task Learning for Multi-choice Reading Comprehension
Figure FIGREF3 illustrates the model architecture. Specifically, we concatenate the passage, question and one of the answer options into a long sequence. For a question with $n$ answer options, we obtain $n$ token sequences of length $l$. Afterwards, each sequence will be encoded by a sentence encoder to get the representation vector $H \in \mathbb {R}^{d\times l}$, which is then projected into a single value $p=C(H)$ ($p\in \mathbb {R}^{1}$) via a top-level classifier $C$. In this way, we obtain the logit vector $\mathbf {p}=[p_1,p_2,...,p_n]$ for all options of a question, which is then transformed into the probability vector through a softmax layer. We choose the option with highest logit value $p$ as the answer. Cross entropy loss is used as the loss function. We used the pre-trained bidirectional transformer encoder, i.e., BERT and RoBERTa as the sentence encoder. The top-level classifier will be detailed in the next subsection.
Question: What are state of the art methods MMM is compared to?
Answer: FTLM++, BERT-large, XLNet
Paper: MMM: Multi-stage Multi-task Learning for Multi-choice Reading Comprehension
For the top-level classifier upon the sentence encoder, the simplest choice is a two-layer full-connected neural network (FCNN), which consist of one hidden layer with $tanh$ activation and one output layer without activation. This has been widely adopted when BERT is fine-tuned for the down-streaming classification tasks and performs very well BIBREF8. Inspired from the success of the attention network widely used in the span-based QA task BIBREF11, we propose the multi-step attention network (MAN) as our top-level classifier. Similar to the dynamic or multi-hop memory network BIBREF12, BIBREF13, MAN maintains a state and iteratively refines its prediction via the multi-step reasoning.
Question: What four representative datasets are used for bechmark?
Paper: Data Mining in Clinical Trial Text: Transformers for Classification and Question Answering Tasks
Systematic reviews (SR) of randomized controlled trials (RCTs) are regarded as the gold standard for providing information about the effects of interventions to healthcare practitioners, policy makers and members of the public. The quality of these reviews is ensured through a strict methodology that seeks to include all relevant information on the review topic BIBREF0.
Question: What baselines did they consider?
Paper: Data Mining in Clinical Trial Text: Transformers for Classification and Question Answering Tasks
The website systematicreviewtools.com BIBREF6 lists 36 software tools for study selection to date. Some tools are intended for organisational purposes and do not employ PICO classification, such as Covidence BIBREF7. The tool Rayyan uses support vector machines BIBREF8. RobotReviewer uses neural networks, word embeddings and recently also a transformer for named entity recognition (NER) BIBREF9. Question answering systems for PICO data extraction exist based on matching words from knowledge bases, hand-crafted rules and naïve Bayes classification, both on entity and sentence level BIBREF10, BIBREF11, but commonly focus on providing information to practicing clinicians rather than systematic reviewers BIBREF12.
Question: What are the problems related to ambiguity in PICO sentence prediction tasks?
Answer: Some sentences are associated to ambiguous dimensions in the hidden state output
Paper: RelNet: End-to-End Modeling of Entities & Relations
Reasoning about entities and their relations is an important problem for achieving general artificial intelligence. Often such problems are formulated as reasoning over graph-structured representation of knowledge. Knowledge graphs, for example, consist of entities and relations between them BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 . Representation learning BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 and reasoning BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 with such structured representations is an important and active area of research.
Question: How is knowledge retrieved in the memory?
Paper: RelNet: End-to-End Modeling of Entities & Relations
We describe the RelNet model in this section. Figure 1 provides a high-level view of the model. The model is sequential in nature, consisting of the following steps: read text, process it into a dynamic relational memory and then attention conditioned on the question generates the answer. We model the dynamic memory in a fashion similar to Recurrent Entity Networks BIBREF17 and then equip it with an additional relational memory.
Question: How is knowledge stored in the memory?
Answer: entity memory and relational memory.
Paper: RelNet: End-to-End Modeling of Entities & Relations
There is a long line of work in textual question-answering systems BIBREF21 , BIBREF22 . Recent successful approaches use memory based neural networks for question answering, for example BIBREF23 , BIBREF18 , BIBREF24 , BIBREF19 , BIBREF17 . Our model is also a memory network based model and is also related to the neural turing machine BIBREF25 . As described previously, the model is closely related to the Recurrent Entity Networks model BIBREF17 which describes an end-to-end approach to model entities in text but does not directly model relations. Other approaches to question answering use external knowledge, for instance external knowledge bases BIBREF26 , BIBREF11 , BIBREF27 , BIBREF28 , BIBREF9 or external text like Wikipedia BIBREF29 , BIBREF30 .
Question: What are the relative improvements observed over existing methods?
Paper: RelNet: End-to-End Modeling of Entities & Relations
We evaluate the model's performance on the bAbI tasks BIBREF18 , a collection of 20 question answering tasks which have become a benchmark for evaluating memory-augmented neural networks. We compare the performance with the Recurrent Entity Networks model (EntNet) BIBREF17 . Performance is measured in terms of mean percentage error on the tasks.
Question: What is the architecture of the neural network?
Paper: RelNet: End-to-End Modeling of Entities & Relations
We demonstrated an end-to-end trained neural network augmented with a structured memory representation which can reason about entities and relations for question answering. Future work will investigate the performance of these models on more real world datasets, interpreting what the models learn, and scaling these models to answer questions about entities and relations from reading massive text corpora.
Question: What methods is RelNet compared to?
Paper: Modeling Event Background for If-Then Commonsense Reasoning Using Context-aware Variational Autoencoder
Recently, event-centered commonsense knowledge has attracted much attention BIBREF0, BIBREF1, BIBREF2, BIBREF3, because of understanding events is an important component of NLP. Given a daily-life event, human can easily understand it and reason about its causes, effects, and so on. However, it still remains a challenging task for NLP systems. This is partly due to most of them are trained for task-specific datasets or objectives, which results in models that are adapt at finding task-specific underlying correlation patterns but have limited capability in simple and explainable commonsense reasoning BIBREF4.
Question: How do they measure the diversity of inferences?
Answer: by number of distinct n-grams
Paper: Modeling Event Background for If-Then Commonsense Reasoning Using Context-aware Variational Autoencoder
Before specifically describing two dataset —- Event2Mind and Atomic used in this paper as well as the If-Then reasoning task, for clarity, we define the following terminologies:
Question: By how much do they improve the accuracy of inferences over state-of-the-art methods?
Answer: ON Event2Mind, the accuracy of proposed method is improved by  absolute BLUE  2.9,  10.87, 1.79 for xIntent, xReact and oReact respectively.
On Atomic dataset, the accuracy of proposed method is improved by  absolute BLUE 3.95.   4.11, 4.49 for xIntent, xReact and oReact.respectively.
Paper: Modeling Event Background for If-Then Commonsense Reasoning Using Context-aware Variational Autoencoder
Traditional CVAE can model the event-target relation. In other words, given an observed event, CVAE can generate its corresponding targets. While in this paper we model the If-Then reasoning as a [(background), event]-target process. It means that in addition to the observed event, we also want to involve the event background knowledge (which can be learned from event contexts) to generate the reasonable targets.
Question: Which models do they use as baselines on the Atomic dataset?
Paper: Modeling Event Background for If-Then Commonsense Reasoning Using Context-aware Variational Autoencoder
As shown in Figure FIGREF8, CWVAE is mainly composed of four parts: a neural encoder that provides distributed representations of base events/targets, a recognition network for inferring $q_{\phi }(z|x,y)$, $q_{\phi }(z_c|x,c)$ and $q_{\phi }(z|z_{c^{\prime }}, x)$, a prior network for modeling $p_{\theta }(z_{c^{\prime }}|x)$ and $p_{\theta }(z|x, z_{c^{\prime }})$, and a neural decoder that integrates the information from $z$ and $z_{c^{\prime }}$ to generate targets.
Question: How does the context-aware variational autoencoder learn event background information?
Answer:  CWVAE is trained on an auxiliary dataset to learn the event background information by using the context-aware latent variable.  Then, in finetute stage, CWVAE is trained on the task-specific dataset to adapt the event background information to each specific aspect of If-Then inferential target.
Paper: Modeling Event Background for If-Then Commonsense Reasoning Using Context-aware Variational Autoencoder
With the incorporation of $z_{c^{\prime }}$, the original loglikelihood could be decomposed as:
Question: What is the size of the Atomic dataset?
Paper: Comparing Human and Machine Errors in Conversational Speech Transcription
Automatic speech recognition (ASR) systems have seen remarkable advances over the last half-decade from the use of deep, convolutional and recurrent neural network architectures, enabled by a combination of modeling advances, available training data, and increased computational resources. Given these advances, our research group recently embarked on an effort to reach human-level transcription accuracy using state-of-the-art ASR techniques on one of the genres of speech that has historically served as a difficult benchmark task: conversational telephone speech (CTS). About a decade ago, CTS recognition had served as an evaluation task for government-sponsored work in speech recognition, predating the take-over of deep learning approaches and still largely in the GMM-HMM modeling framework BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 . It had proven to be a hard problem, due to the variable nature of conversational pronunciations, speaking styles, and regional accents. Seide at al. BIBREF6 demonstrated that deep networks as acoustic models could achieve significant improvements over GMM-HMM models on CTS data, and more recently researchers at IBM had achieved results on this task that represented a further significant advance BIBREF7 , BIBREF8 over those from a decade ago.
Question: what standard speech transcription pipeline was used?
Paper: An Empirical Comparison of Simple Domain Adaptation Methods for Neural Machine Translation
One of the most attractive features of neural machine translation (NMT) BIBREF0 , BIBREF1 , BIBREF2 is that it is possible to train an end to end system without the need to deal with word alignments, translation rules and complicated decoding algorithms, which are a characteristic of statistical machine translation (SMT) systems. However, it is reported that NMT works better than SMT only when there is an abundance of parallel corpora. In the case of low resource domains, vanilla NMT is either worse than or comparable to SMT BIBREF3 .
Question: How much improvement does their method get over the fine tuning baseline?
Answer: 0.08 points on the 2011 test set, 0.44 points on the 2012 test set, 0.42 points on the 2013 test set for IWSLT-CE.
Paper: An Empirical Comparison of Simple Domain Adaptation Methods for Neural Machine Translation
Besides fine tuning and multi domian NMT using tags, another direction for domain adaptation is using in-domain monolingual data. Either training an in-domain recurrent neural language (RNN) language model for the NMT decoder BIBREF13 or generating synthetic data by back translating target in-domain monolingual data BIBREF5 have been studied.
Question: What kinds of neural networks did they use in this paper?
Paper: An Empirical Comparison of Simple Domain Adaptation Methods for Neural Machine Translation
All the methods that we compare are simple and do not need any modifications to the NMT system.
Question: How did they use the domain tags?
Paper: Combining Search with Structured Data to Create a More Engaging User Experience in Open Domain Dialogue
The Alexa Prize funded 12 international teams to compete to create a conversational agent that can discuss any topic for at least 20 minutes. UCSC's Slugbot was one of these funded teams. The greatest challenges with the competition arise directly from the potential for ongoing mixed-initiative multi-turn dialogues, which do not follow a particular plan or pursue a particular fixed information need. This paper describes some of the lessons we learned building SlugBot for the 2017 Alexa Prize, particularly focusing on the challenges of integrating content found via search with content from structured data in order to carry on an ongoing, coherent, open-domain, mixed-initiative conversation. SlugBot's conversations over the semi-finals user evaluation averaged 8:17 minutes.
Question: Why mixed initiative multi-turn dialogs are the greatest challenge in building open-domain conversational agents?
Paper: Identifying and Understanding User Reactions to Deceptive and Trusted Social News Sources
As the reliance on social media as a source of news increases and the reliability of sources is increasingly debated, it is important to understand how users react to various sources of news. Most studies that investigate misinformation spread in social media focus on individual events and the role of the network structure in the spread BIBREF0 , BIBREF1 , BIBREF2 or detection of false information BIBREF3 . These studies have found that the size and shape of misinformation cascades within a social network depends heavily on the initial reactions of the users. Other work has focused on the language of misinformation in social media BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 to detect types of deceptive news.
Question: How is speed measured?
Paper: Identifying and Understanding User Reactions to Deceptive and Trusted Social News Sources
In this section, we describe our approach to classify user reactions into one of eight types of discourse: agreement, answer, appreciation, disagreement, elaboration, humor, negative reaction, or question, or as none of the given labels, which we call “other”, using linguistically-infused neural network models.
Question: What is the architecture of their model?
Paper: Identifying and Understanding User Reactions to Deceptive and Trusted Social News Sources
We use a manually annotated Reddit dataset from Zhang et al. zhang2017characterizing to train our reaction classification model. Annotations from 25 crowd-workers labelled the primary discourse act for 101,525 comments within 9,131 comment threads on Reddit. The Reddit IDs, but not the text content of the comments themselves, were released with the annotations. So we collected the content of Reddit posts and comments from a public archive of Reddit posts and comments. Some content was deleted prior to archival, so the dataset shown in Table TABREF3 is a subset of the original content. Despite the inability to capture all of the original dataset, Table TABREF3 shows a similar distribution between our dataset and the original.
Question: What are the nine types?
Paper: Discriminative Acoustic Word Embeddings: Recurrent Neural Network-Based Approaches
Many speech processing tasks – such as automatic speech recognition or spoken term detection – hinge on associating segments of speech signals with word labels. In most systems developed for such tasks, words are broken down into sub-word units such as phones, and models are built for the individual units. An alternative, which has been considered by some researchers, is to consider each entire word segment as a single unit, without assigning parts of it to sub-word units. One motivation for the use of whole-word approaches is that they avoid the need for sub-word models. This is helpful since, despite decades of work on sub-word modeling BIBREF0 , BIBREF1 , it still poses significant challenges. For example, speech processing systems are still hampered by differences in conversational pronunciations BIBREF2 . A second motivation is that considering whole words at once allows us to consider a more flexible set of features and reason over longer time spans.
Question: How do they represent input features of their model to train embeddings?
Paper: Discriminative Acoustic Word Embeddings: Recurrent Neural Network-Based Approaches
We next briefly describe the most closely related prior work.
Question: Which dimensionality do they use for their embeddings?
Paper: Discriminative Acoustic Word Embeddings: Recurrent Neural Network-Based Approaches
Question: Which dataset do they use?