z-uo/qasper-squad huggingface dataset:
Paper: Neural Semantic Parsing in Low-Resource Settings with Back-Translation and Meta-Learning
Following the back-translation paradigm BIBREF3 , BIBREF4 , we have a semantic parser, which maps a natural language question INLINEFORM0 to a logical form INLINEFORM1 , and a question generator, which maps INLINEFORM2 to INLINEFORM3 . The semantic parser works for the primary task, and the question generator mainly works for generating pseudo datapoints. We start the training process by applying the rule INLINEFORM4 to a set of natural language questions INLINEFORM5 . The resulting dataset is considered as the training data to initialize both the semantic parser and the question generator. Afterwards, both models are improved following the back-translation protocol that target sequences should follow the real data distribution, yet source sequences can be generated with noises. This is based on the consideration that in an encoder-decoder model, the decoder is more sensitive to the data distribution than the encoder. We use datapoints from both models to train the semantic parser because a logical form is structural which follows a grammar, whose distribution is similar to the ground truth.
Question: What datasets are used in this paper?
Paper: Distant Supervision and Noisy Label Learning for Low Resource Named Entity Recognition: A Study on Hausa and Yor\`ub\'a
Named Entity Recognition (NER) is a classification task that identifies words in a text that refer to entities (such as dates, person, organization and location names). It is a core task of natural language processing and a component for many downstream applications like search engines, knowledge graphs and personal assistants. For high-resource languages like English, this is a well-studied problem with complex state-of-the-art systems reaching close to or above 90% F1-score on the standard datasets CoNLL03 BIBREF0 and Ontonotes BIBREF1. In recent years, research has been extended to a larger pool of languages including those of developing countries BIBREF2, BIBREF3, BIBREF4, BIBREF5. Often, for these languages (like Hausa and Yorùbá studied here), there exists a large population with access to digital devices and internet (and therefore digital text), but natural language processing (NLP) tools do not support them.
Question: How much labeled data is available for these two languages?
Paper: Distant Supervision and Noisy Label Learning for Low Resource Named Entity Recognition: A Study on Hausa and Yor\`ub\'a
Hausa language is the second most spoken indigenous language in Africa with over 40 million native speakers BIBREF20, and one of the three major languages in Nigeria, along with Igbo and Yorùbá. The language is native to the Northern part of Nigeria and the southern part of Niger, and it is widely spoken in West and Central Africa as a trade language in eight other countries: Benin, Ghana, Cameroon, Togo, Côte d'Ivoire, Chad, Burkina Faso, and Sudan. Hausa has several dialects but the one regarded as standard Hausa is the Kananci spoken in the ancient city of Kano in Nigeria. Kananci is the dialect popularly used in many local (e.g VON news) and international news media such as BBC, VOA, DW and Radio France Internationale. Hausa is a tone language but the tones are often ignored in writings, the language is written in a modified Latin alphabet. Despite the popularity of Hausa as an important regional language in Africa and it's popularity in news media, it has very little or no labelled data for common NLP tasks such as text classification, named entity recognition and question answering.
Question: What was performance of classifiers before/after using distant supervision?
Answer: Bi-LSTM: For low resource <17k clean data: Using distant supervision resulted in huge boost of F1 score (1k eg. ~9 to ~36 wit distant supervision)
BERT: <5k clean data boost of F1 (1k eg. ~32 to ~47 with distant supervision)
Paper: Distant Supervision and Noisy Label Learning for Low Resource Named Entity Recognition: A Study on Hausa and Yor\`ub\'a
The Hausa data used in this paper is part of the LORELEI language pack. It consists of Broad Operational Language Translation (BOLT) data gathered from news sites, forums, weblogs, Wikipedia articles and twitter messages. We use a split of 10k training and 1k test instances. Due to the Hausa data not being publicly available at the time of writing, we could only perform a limited set of experiments on it.
Question: What classifiers were used in experiments?
Paper: Distant Supervision and Noisy Label Learning for Low Resource Named Entity Recognition: A Study on Hausa and Yor\`ub\'a
In this work, we rely on two sources of distant supervision chosen for its ease of application:
Question: In which countries are Hausa and Yor\`ub\'a spoken?
Paper: Monitoring stance towards vaccination in twitter messages
In the light of increased vaccine hesitance in various countries, consistent monitoring of public beliefs and opinions about the national immunization program is important. Besides performing qualitative research and surveys, real-time monitoring of social media data about vaccination is a valuable tool to this end. The advantage is that one is able to detect and respond to possible vaccine concerns in a timely manner, that it generates continuous data and that it consists of unsolicited, voluntary user-generated content.
Question: What is the agreement score of their annotated dataset?
Paper: Monitoring stance towards vaccination in twitter messages
We set out to curate a corpus of tweets annotated for their stance towards vaccination, and to employ this corpus to train a machine learning classifier to distinguish tweets with a negative stance towards vaccination from other tweets. In the following, we will describe the stages of data acquisition, from collection to labeling.
Question: What is the size of the labelled dataset?
Paper: Monitoring stance towards vaccination in twitter messages
We queried Twitter messages that refer to a vaccination-related key term from TwiNL , a database with IDs of Dutch Twitter messages from January 2012 onwards BIBREF22. In contrast to the open Twitter Search API, which only allows one to query tweets posted within the last seven days, TwiNL makes it possible to collect a much larger sample of Twitter posts, ranging several years.
Question: Which features do they use to model Twitter messages?
Paper: Monitoring stance towards vaccination in twitter messages
The stance towards vaccination was categorized into `Negative’, `Neutral’, `Positive’ and `Not clear’. The latter category was essential, as some posts do not convey enough information about the stance of the writer. In addition to the four-valued stance classes we included separate classes grouped under relevance, subject and sentiment as annotation categories. With these additional categorizations we aimed to obtain a precise grasp of all possibly relevant tweet characteristics in relation to vaccination, which could help in a machine learning setting.
Question: Do they allow for messages with vaccination-related key terms to be of neutral stance?
Paper: Cross-Lingual Machine Reading Comprehension
Machine Reading Comprehension (MRC) has been a popular task to test the reading ability of the machine, which requires to read text material and answer the questions based on it. Starting from cloze-style reading comprehension, various neural network approaches have been proposed and massive progresses have been made in creating large-scale datasets and neural models BIBREF0, BIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5. Though various types of contributions had been made, most works are dealing with English reading comprehension. Reading comprehension in other than English has not been well-addressed mainly due to the lack of large-scale training data.
Question: How big are the datasets used?
Answer: Evaluation datasets used:
CMRC 2018 - 18939 questions, 10 answers
DRCD - 33953 questions, 5 answers
NIST MT02/03/04/05/06/08 Chinese-English - Not specified

Source language train data:
SQuAD - Not specified
Paper: Cross-Lingual Machine Reading Comprehension
Machine Reading Comprehension (MRC) has been a trending research topic in recent years. Among various types of MRC tasks, span-extraction reading comprehension has been enormously popular (such as SQuAD BIBREF4), and we have seen a great progress on related neural network approaches BIBREF11, BIBREF12, BIBREF13, BIBREF3, BIBREF14, especially those were built on pre-trained language models, such as BERT BIBREF7. While massive achievements have been made by the community, reading comprehension in other than English has not been well-studied mainly due to the lack of large-scale training data.
Question: Is this a span-based (extractive) QA task?
Paper: Cross-Lingual Machine Reading Comprehension
In this section, we illustrate back-translation approaches for cross-lingual machine reading comprehension, which is natural and easy to implement.
Question: Are the contexts in a language different from the questions?
Paper: Modeling Multi-Action Policy for Task-Oriented Dialogues
In a task-oriented dialogue system, the dialogue manager policy module predicts actions usually in terms of dialogue acts and domain specific slots. It is a crucial component that influences the efficiency (e.g., the conciseness and smoothness) of the communication between the user and the agent. Both supervised learning (SL) BIBREF0, BIBREF1, BIBREF2, BIBREF3, BIBREF4 and reinforcement learning (RL) approaches BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9 have been adopted to learn policies. SL learns a policy to predict acts given the dialogue state. Recent work BIBREF10, BIBREF11 also used SL as pre-training for RL to mitigate the sample inefficiency of RL approaches and to reduce the number of interactions. Sequence2Sequence (Seq2Seq) BIBREF12 approaches have also been adopted in user simulators to produce user acts BIBREF13. These approaches typically assume that the agent can only produce one act per turn through classification. Generating only one act per turn significantly limits what an agent can do in a turn and leads to lengthy dialogues, making tracking of state and context throughout the dialogue harder. An example in Table TABREF3 shows how the agent can produce both an inform and a multiple_choice act, reducing the need for additional turns. The use of multiple actions has previously been used in interaction managers that keep track of the floor (who is speaking right now) BIBREF14, BIBREF15, BIBREF16, but the option of generating multiple acts simultaneously at each turn for dialogue policy has been largely ignored, and only explored in simulated scenarios without real data BIBREF17.
Question: What datasets are used for training/testing models? 
Answer: Microsoft Research dataset containing movie, taxi and restaurant domains.
Paper: Modeling Multi-Action Policy for Task-Oriented Dialogues
The proposed policy network adopts an encoder-decoder architecture (Figure FIGREF5). The input to the encoder is the current-turn dialogue state, which follows BIBREF19's definition. It contains policy actions from the previous turn, user dialogue acts from the current turn, user requested slots, the user informed slots, the agent requested slots and agent proposed slots. We treat the dialogue state as a sequence and adopt a GRU BIBREF20 to encode it. The encoded dialogue state is a sequence of vectors $\mathbf {E} = (e_0, \ldots , e_l)$ and the last hidden state is $h^{E}$. The CAS decoder recurrently generates tuples at each step. It takes $h^{E}$ as initial hidden state $h_0$. At each decoding step, the input contains the previous (continue, act, slots) tuple $(c_{t-1},a_{t-1},s_{t-1})$. An additional vector $k$ containing the number of results from the knowledge base (KB) query and the current turn number is given as input. The output of the decoder at each step is a tuple $(c, a, s)$, where $c \in \lbrace \langle \text{continue} \rangle , \langle \text{stop} \rangle , \langle \text{pad} \rangle \rbrace $, $a \in A$ (one act from the act set), and $s \subset S$ (a subset from the slot set).
Question: How better is gCAS approach compared to other approaches?
Answer: For entity  F1 in the movie, taxi and restaurant domain it results in scores of 50.86, 64, and 60.35. For success, it results it outperforms in the movie and restaurant domain with scores of 77.95 and 71.52
Paper: Modeling Multi-Action Policy for Task-Oriented Dialogues
As shown in Figure FIGREF7, the gated CAS cell contains three sequentially connected units for outputting continue, act, and slots respectively.
Question: What is specific to gCAS cell?
Answer: It has three sequentially connected units to output continue, act and slots generating multi-acts in a doble recurrent manner.
Paper: Incorporating Context and External Knowledge for Pronoun Coreference Resolution
The question of how human beings resolve pronouns has long been of interest to both linguistics and natural language processing (NLP) communities, for the reason that pronoun itself has weak semantic meaning BIBREF0 and brings challenges in natural language understanding. To explore solutions for that question, pronoun coreference resolution BIBREF1 was proposed. As an important yet vital sub-task of the general coreference resolution task, pronoun coreference resolution is to find the correct reference for a given pronominal anaphor in the context and has been shown to be crucial for a series of downstream tasks BIBREF2 , including machine translation BIBREF3 , summarization BIBREF4 , information extraction BIBREF5 , and dialog systems BIBREF6 .
Question: What dataset do they evaluate their model on?
Paper: Incorporating Context and External Knowledge for Pronoun Coreference Resolution
Following the conventional setting BIBREF1 , the task of pronoun coreference resolution is defined as: for a pronoun $p$ and a candidate noun phrase set ${\mathcal {N}}$ , the goal is to identify the correct non-pronominal references set ${\mathcal {C}}$ . the objective is to maximize the following objective function: 
Question: What is the source of external knowledge?
Answer: counts of predicate-argument tuples from English Wikipedia
Paper: Improving Textual Network Embedding with Global Attention via Optimal Transport
 When performing network embedding, one maps network nodes into vector representations that reside in a low-dimensional latent space. Such techniques seek to encode topological information of the network into the embedding, such as affinity BIBREF0 , local interactions (e.g, local neighborhoods) BIBREF1 , and high-level properties such as community structure BIBREF2 . Relative to classical network-representation learning schemes BIBREF3 , network embeddings provide a more fine-grained representation that can be easily repurposed for other downstream applications (e.g., node classification, link prediction, content recommendation and anomaly detection).
Question: Which of their proposed attention methods works better overall?
Paper: Improving Textual Network Embedding with Global Attention via Optimal Transport
 We introduce basic notation and definitions used in this work.
Question: Which dataset of texts do they use?
Paper: Improving Textual Network Embedding with Global Attention via Optimal Transport
Question: Do they measure how well they perform on longer sequences specifically?
Paper: Improving Textual Network Embedding with Global Attention via Optimal Transport
 To capture both the topological information (network structure INLINEFORM0 ) and the semantic information (text content INLINEFORM1 ) in the textual network embedding, we explicitly model two types of embeddings for each node INLINEFORM2 : ( INLINEFORM3 ) the topological embedding INLINEFORM4 , and ( INLINEFORM5 ) the semantic embedding INLINEFORM6 . The final embedding is constructed by concatenating the topological and semantic embeddings, i.e., INLINEFORM7 . We consider the topological embedding INLINEFORM8 as a static property of the node, fixed regardless of the context. On the other hand, the semantic embedding INLINEFORM9 dynamically depends on the context, which is the focus of this study.
Question: Which other embeddings do they compare against?
Paper: A Multi-Task Learning Framework for Extracting Drugs and Their Interactions from Drug Labels
Preventable adverse drug reactions (ADRs) introduce a growing concern in the modern healthcare system as they represent a large fraction of hospital admissions and play a significant role in increased health care costs BIBREF0 . Based on a study examining hospital admission data, it is estimated that approximately three to four percent of hospital admissions are caused by adverse events BIBREF1 ; moreover, it is estimated that between 53% and 58% of these events were due to medical errors BIBREF2 (and are therefore considered preventable). Such preventable adverse events have been cited as the eighth leading cause of death in the U.S., with an estimated fatality rate of between 44,000 and 98,000 each year BIBREF3 . As drug-drug interactions (DDIs) may lead to preventable ADRs, being able to extract DDIs from structured product labeling (SPL) documents for prescription drugs is an important effort toward effective dissemination of drug safety information. The Text Analysis Conference (TAC) is a series of workshops aimed at encouraging research in natural language processing (NLP) and related applications by providing large test collections along with a standard evaluation procedure. The Drug-Drug Interaction Extraction from Drug Labels track of TAC 2018 BIBREF4 , organized by the U.S. Food and Drug Administration (FDA) and U.S. National Library of Medicine (NLM), is established with the goal of transforming the contents of SPLs into a machine-readable format with linkage to standard terminologies.
Question: What were the sizes of the test sets?
Answer: Test set 1 contained 57 drug labels and 8208 sentences and test set 2 contained 66 drug labels and 4224 sentences
Paper: A Multi-Task Learning Framework for Extracting Drugs and Their Interactions from Drug Labels
Herein, we describe the training and testing data involved in this task and the metrics used for evaluation. In Section SECREF5 , we describe our modeling approach, our deep learning architecture, and our training procedure.
Question: What training data did they use?
Paper: Domain Agnostic Real-Valued Specificity Prediction
The specificity of a sentence measures its “quality of belonging or relating uniquely to a particular subject” BIBREF0 . It is often pragmatically defined as the level of detail in the sentence BIBREF1 , BIBREF2 . When communicating, specificity is adjusted to serve the intentions of the writer or speaker BIBREF3 . In the examples below, the second sentence is clearly more specific than the first one:
Question: What domains do they experiment with?
Paper: Transfer in Deep Reinforcement Learning using Knowledge Graphs
Text adventure games, in which players must make sense of the world through text descriptions and declare actions through natural language, can provide a stepping stone toward more real-world environments where agents must communicate to understand the state of the world and affect change in the world. Despite the steadily increasing body of research on text-adventure games BIBREF0, BIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6, BIBREF7, and in addition to the ubiquity of deep reinforcement learning applications BIBREF8, BIBREF9, teaching an agent to play text-adventure games remains a challenging task. Learning a control policy for a text-adventure game requires a significant amount of exploration, resulting in training runs that take hundreds of thousands of simulations BIBREF2, BIBREF7.
Question: What games are used to test author's methods?
Paper: Transfer in Deep Reinforcement Learning using Knowledge Graphs
Text-adventure games, in which an agent must interact with the world entirely through natural language, provide us with two challenges that have proven difficult for deep reinforcement learning to solve BIBREF2, BIBREF4, BIBREF7: (1) The agent must act based only on potentially incomplete textual descriptions of the world around it. The world is thus partially observable, as the agent does not have access to the state of the world at any stage. (2) the action space is combinatorially large—a consequence of the agent having to declare commands in natural language. These two problems together have kept commercial text adventure games out of the reach of existing deep reinforcement learning methods, especially given the fact that most of these methods attempt to train on a particular game from scratch.
Question: How is the domain knowledge transfer represented as knowledge graph?
Paper: Language Independent Sequence Labelling for Opinion Target Extraction
Opinion Mining and Sentiment Analysis (OMSA) are crucial for determining opinion trends and attitudes about commercial products, companies reputation management, brand monitoring, or to track attitudes by mining social media, etc. Furthermore, given the explosion of information produced and shared via the Internet, especially in social media, it is simply not possible to keep up with the constant flow of new information by manual methods.
Question: What was the baseline?
Paper: Language Independent Sequence Labelling for Opinion Target Extraction
Early approaches to Opinion Target Extraction (OTE) were unsupervised, although later on the vast majority of works have been based on supervised and deep learning models. To the best of our knowledge, the first work on OTE was published by BIBREF8 . They created a new task which consisted of generating overviews of the main product features from a collection of customer reviews on consumer electronics. They addressed such task using an unsupervised algorithm based on association mining. Other early unsupervised approaches include BIBREF9 which used a dependency parser to obtain more opinion targets, and BIBREF10 which aimed at extracting opinion targets in newswire via Semantic Role Labelling. From a supervised perspective, BIBREF11 presented an approach which learned the opinion target candidates and a combination of dependency and part-of-speech (POS) paths connecting such pairs. Their results improved the baseline provided by BIBREF8 . Another influential work was BIBREF12 , an unsupervised algorithm called Double Propagation which roughly consists of incrementally augmenting a set of seeds via dependency parsing.
Question: Which datasets are used?
Answer: ABSA SemEval 2014-2016 datasets
Yelp Academic Dataset
Wikipedia dumps
Paper: Language Independent Sequence Labelling for Opinion Target Extraction
Three ABSA editions were held within the SemEval Evaluation Exercises between 2014 and 2016. The ABSA 2014 and 2015 tasks consisted of English reviews only, whereas in the 2016 task 7 more languages were added. Additionally, reviews from four domains were collected for the various sub-tasks across the three editions, namely, Consumer Electronics, Telecommunications, Museums and Restaurant reviews. In any case, the only constant in each of the ABSA editions was the inclusion, for the Opinion Target Extraction (OTE) sub-task, of restaurant reviews for every language. Thus, for the experiments presented in this paper we decided to focus on the restaurant domain across 6 languages and the three different ABSA editions. Similarly, this section will be focused on reviewing the OTE results for the restaurant domain.
Question: Which six languages are experimented with?
Paper: Language Independent Sequence Labelling for Opinion Target Extraction
The work presented in this research note requires the following resources: (i) Aspect Based Sentiment Analysis (ABSA) data for training and testing; (ii) large unlabelled corpora to obtain semantic distributional features from clustering lexicons; and (iii) a sequence labelling system. In this section we will describe each of the resources used.
Question: What shallow local features are extracted?