z-uo/qasper-squad huggingface dataset:
Paper: Embracing data abundance: BookTest Dataset for Reading Comprehension
Since humans amass more and more generally available data in the form of unstructured text it would be very useful to teach machines to read and comprehend such data and then use this understanding to answer our questions. A significant amount of research has recently focused on answering one particular kind of questions the answer to which depends on understanding a context document. These are cloze-style questions BIBREF0 which require the reader to fill in a missing word in a sentence. An important advantage of such questions is that they can be generated automatically from a suitable text corpus which allows us to produce a practically unlimited amount of them. That opens the task to notoriously data-hungry deep-learning techniques which now seem to outperform all alternative approaches.
Question: How does their ensemble method work?
Paper: Embracing data abundance: BookTest Dataset for Reading Comprehension
A natural way of testing a reader's comprehension of a text is to ask her a question the answer to which can be deduced from the text. Hence the task we are trying to solve consists of answering a cloze-style question, the answer to which depends on the understanding of a context document provided with the question. The model is also provided with a set of possible answers from which the correct one is to be selected. This can be formalized as follows:
Question: How large are the improvements of the Attention-Sum Reader model when using the BookTest dataset?
Answer: Answer with content missing: (Table 2) Accuracy of best AS reader results including ensembles are 78.4 and 83.7 when trained on BookTest compared to 71.0 and 68.9 when trained on CBT for Named endity and Common noun respectively.
Paper: Embracing data abundance: BookTest Dataset for Reading Comprehension
We will now briefly review what datasets for text comprehension have been published up to date and look at models which have been recently applied to solving the task we have just described.
Question: How do they show there is space for further improvement?
Paper: Argumentation Mining in User-Generated Web Discourse
The art of argumentation has been studied since the early work of Aristotle, dating back to the 4th century BC BIBREF0 . It has been exhaustively examined from different perspectives, such as philosophy, psychology, communication studies, cognitive science, formal and informal logic, linguistics, computer science, educational research, and many others. In a recent and critically well-acclaimed study, Mercier.Sperber.2011 even claim that argumentation is what drives humans to perform reasoning. From the pragmatic perspective, argumentation can be seen as a verbal activity oriented towards the realization of a goal BIBREF1 or more in detail as a verbal, social, and rational activity aimed at convincing a reasonable critic of the acceptability of a standpoint by putting forward a constellation of one or more propositions to justify this standpoint BIBREF2 .
Question: Do they report results only on English data?
Paper: Argumentation Mining in User-Generated Web Discourse
We create a new corpus which is, to the best of our knowledge, the largest corpus that has been annotated within the argumentation mining field to date. We choose several target domains from educational controversies, such as homeschooling, single-sex education, or mainstreaming. A novel aspect of the corpus is its coverage of different registers of user-generated Web content, such as comments to articles, discussion forum posts, blog posts, as well as professional newswire articles.
Question: What argument components do the ML methods aim to identify?
Paper: Argumentation Mining in User-Generated Web Discourse
Let us first present some definitions of the term argumentation itself. [p. 3]Ketcham.1917 defines argumentation as “the art of persuading others to think or act in a definite way. It includes all writing and speaking which is persuasive in form.” According to MacEwan.1898, “argumentation is the process of proving or disproving a proposition. Its purpose is to induce a new belief, to establish truth or combat error in the mind of another.” [p. 2]Freeley.Steinberg.2008 narrow the scope of argumentation to “reason giving in communicative situations by people whose purpose is the justification of acts, beliefs, attitudes, and values.” Although these definitions vary, the purpose of argumentation remains the same – to persuade others.
Question: Which machine learning methods are used in experiments?
Answer: Structural Support Vector Machine
Paper: Argumentation Mining in User-Generated Web Discourse
Despite the missing consensus on the ultimate argumentation theory, various argumentation models have been proposed that capture argumentation on different levels. Argumentation models abstract from the language level to a concept level that stresses the links between the different components of an argument or how arguments relate to each other BIBREF26 . Bentahar.et.al.2010 propose a taxonomy of argumentation models, that is horizontally divided into three categories – micro-level models, macro-level models, and rhetorical models.
Question: How is the data in the new corpus come sourced?
Paper: Argumentation Mining in User-Generated Web Discourse
The above-mentioned models focus basically only on one dimension of the argument, namely the logos dimension. According to the classical Aristotle's theory BIBREF0 , argument can exist in three dimensions, which are logos, pathos, and ethos. Logos dimension represents a proof by reason, an attempt to persuade by establishing a logical argument. For example, syllogism belongs to this argumentation dimension BIBREF34 , BIBREF25 . Pathos dimension makes use of appealing to emotions of the receiver and impacts its cognition BIBREF35 . Ethos dimension of argument relies on the credibility of the arguer. This distinction will have practical impact later in section SECREF51 which deals with argumentation on the Web.
Question: What argumentation phenomena encounter in actual data are now accounted for by this work?
Paper: Argumentation Mining in User-Generated Web Discourse
We conclude the theoretical section by presenting one (micro-level) argumentation model in detail – a widely used conceptual model of argumentation introduced by Toulmin.1958, which we will henceforth denote as the Toulmin's original model. This model will play an important role later in the annotation studies (section SECREF51 ) and experimental work (section SECREF108 ). The model consists of six parts, referred as argument components, where each component plays a distinct role.
Question: What challenges do different registers and domains pose to this task?
Paper: A Multi Purpose and Large Scale Speech Corpus in Persian and English for Speaker and Speech Recognition: The Deepmine Database
Nowadays deep learning techniques outperform the other conventional methods in most of the speech-related tasks. Training robust deep neural networks for each task depends on the availability of powerful processing GPUs, as well as standard and large scale datasets. In text-independent speaker verification, large-scale datasets are available, thanks to the NIST SRE evaluations and other data collection projects such as VoxCeleb BIBREF0.
Question: who transcribed the corpus?
Paper: A Multi Purpose and Large Scale Speech Corpus in Persian and English for Speaker and Speech Recognition: The Deepmine Database
DeepMine is publicly available for everybody with a variety of licenses for different users. It was collected using crowdsourcing BIBREF4. The data collection was done using an Android application. Each respondent installed the application on his/her personal device and recorded several phrases in different sessions. The Android application did various checks on each utterance and if it passed all of them, the respondent was directed to the next phrase. For more information about data collection scenario, please refer to BIBREF4.
Question: how was the speech collected?
Answer: The speech was collected from respondents using an android application.
Paper: A Multi Purpose and Large Scale Speech Corpus in Persian and English for Speaker and Speech Recognition: The Deepmine Database
In order to clean-up the database, the main post-processing step was to filter out problematic utterances. Possible problems include speaker word insertions (e.g. repeating some part of a phrase), deletions, substitutions, and involuntary disfluencies. To detect these, we implemented an alignment stage, similar to the second alignment stage in the LibriSpeech project BIBREF5. In this method, a custom decoding graph was generated for each phrase. The decoding graph allows for word skipping and word insertion in the phrase.
Question: what accents are present in the corpus?
Paper: A Multi Purpose and Large Scale Speech Corpus in Persian and English for Speaker and Speech Recognition: The Deepmine Database
After processing the database and removing problematic respondents and utterances, 1969 respondents remained in the database, with 1149 of them being male and 820 female. 297 of the respondents could not read English and have therefore read only the Persian prompts. About 13200 sessions were recorded by females and similarly, about 9500 sessions by males, i.e. women are over-represented in terms of sessions, even though their number is 17% smaller than that of males. Other useful statistics related to the database are shown in Table TABREF4.
Question: what evaluation protocols are provided?
Paper: A Multi Purpose and Large Scale Speech Corpus in Persian and English for Speaker and Speech Recognition: The Deepmine Database
The DeepMine database consists of three parts. The first one contains fixed common phrases to perform text-dependent speaker verification. The second part consists of random sequences of words useful for text-prompted speaker verification, and the last part includes phrases with word- and phoneme-level transcription, useful for text-independent speaker verification using a random phrase (similar to Part4 of RedDots). This part can also serve for Persian ASR training. Each part is described in more details below. Table TABREF11 shows the number of unique phrases in each part of the database. For the English text-dependent part, the following phrases were selected from part1 of the RedDots database, hence the RedDots can be used as an additional training set for this part:
Question: what age range is in the data?
Paper: A Multi Purpose and Large Scale Speech Corpus in Persian and English for Speaker and Speech Recognition: The Deepmine Database
This part contains a set of fixed phrases which are used to verify speakers in text-dependent mode. Each speaker utters 5 Persian phrases, and if the speaker can read English, 5 phrases selected from Part1 of the RedDots database are also recorded.
Question: what is the source of the data?
Paper: Extractive Summarization of EHR Discharge Notes
Summarization of patient information is essential to the practice of medicine. Clinicians must synthesize information from diverse data sources to communicate with colleagues and provide coordinated care. Examples of clinical summarization are abundant in practice; patient handoff summaries facilitate provider shift change, progress notes provide a daily status update for a patient, oral case presentations enable transfer of information from overnight admission to the care team and attending, and discharge summaries provide information about a patient's hospital visit to their primary care physician and other outpatient providers BIBREF0 .
Question: what topics did they label?
Answer: Demographics Age, DiagnosisHistory, MedicationHistory, ProcedureHistory, Symptoms/Signs, Vitals/Labs, Procedures/Results, Meds/Treatments, Movement, Other.
Demographics, Diagnosis History, Medication History, Procedure History, Symptoms, Labs, Procedures, Treatments, Hospital movements, and others
Paper: Extractive Summarization of EHR Discharge Notes
In the broader field of summarization, automization was meant to standardize output while also saving time and effort. Pioneering strategies in summarization started by extracting "significant" sentences in the whole corpus to build an abstract where "significant" sentences were defined by the number of frequently occurring words BIBREF6 . These initial methods did not consider word meaning or syntax at either the sentence or paragraph level, which made them crude at best. More advanced extractive heuristics like topic modeling BIBREF7 , cue word dictionary approaches BIBREF8 , and title methods BIBREF9 for scoring content in a sentence followed soon after. For example, topic modeling extends initial frequency methods by assigning topics scores by frequency of topic signatures, clustering sentences with similar topics, and finally extracting the centroid sentence, which is considered the most representative sentence BIBREF10 . Recently, abstractive summarization approaches using sequence-to-sequence methods have been developed to generate new text that synthesizes original text BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 ; however, the field of abstractive summarization is quite young.
Question: did they compare with other extractive summarization methods?
Paper: Extractive Summarization of EHR Discharge Notes
MIMIC-III is a freely available, deidentified database containing electronic health records of patients admitted to an Intensive Care Unit (ICU) at Beth Israel Deaconess Medical Center between 2001 and 2012. The database contains all of the notes associated with each patient's time spent in the ICU as well as 55,177 discharge reports and 4,475 discharge addendums for 41,127 distinct patients. Only the original discharge reports were included in our analyses. Each discharge summary was divided into sections (Date of Birth, Sex, Chief Complaint, Major Surgical or Invasive Procedure, History of Present Illness, etc.) using a regular expression.
Question: what datasets were used?
Paper: How Document Pre-processing affects Keyphrase Extraction Performance
 This work is licensed under a Creative Commons Attribution 4.0 International Licence. Licence details: http://creativecommons.org/licenses/by/4.0/
Question: what levels of document preprocessing are looked at?
Answer: Level 1, Level 2 and Level 3.
Paper: How Document Pre-processing affects Keyphrase Extraction Performance
The SemEval-2010 benchmark dataset BIBREF0 is composed of 244 scientific articles collected from the ACM Digital Library (conference and workshop papers). The input papers ranged from 6 to 8 pages and were converted from PDF format to plain text using an off-the-shelf tool. The only preprocessing applied is a systematic dehyphenation at line breaks and removal of author-assigned keyphrases. Scientific articles were selected from four different research areas as defined in the ACM classification, and were equally distributed into training (144 articles) and test (100 articles) sets. Gold standard keyphrases are composed of both author-assigned keyphrases collected from the original PDF files and reader-assigned keyphrases provided by student annotators.
Question: what keyphrase extraction models were reassessed?
Answer: Answer with content missing: (LVL1, LVL2, LVL3) 
- Stanford CoreNLP
- Optical Character Recognition (OCR) system, ParsCIT 
- further abridge the input text from level 2 preprocessed documents to the following: title, headers, abstract, introduction, related work, background and conclusion.
Paper: How Document Pre-processing affects Keyphrase Extraction Performance
We re-implemented five keyphrase extraction models : the first two are commonly used as baselines, the third is a resource-lean unsupervised graph-based ranking approach, and the last two were among the top performing systems in the SemEval-2010 keyphrase extraction task BIBREF0 . We note that two of the systems are supervised and rely on the training set to build their classification models. Document frequency counts are also computed on the training set. Stemming is applied to allow more robust matching. The different keyphrase extraction models are briefly described below:
Question: how many articles are in the dataset?
Paper: A Corpus for Detecting High-Context Medical Conditions in Intensive Care Patient Notes Focusing on Frequently Readmitted Patients
With the widespread adoption of electronic health records (EHRs), medical data are being generated and stored digitally in vast quantities BIBREF0. While much EHR data are structured and amenable to analysis, there appears to be limited homogeneity in data completeness and quality BIBREF1, and it is estimated that the majority of healthcare data are being generated in unstructured, text-based format BIBREF2. The generation and storage of these unstructured data come concurrently with policy initiatives that seek to utilize preventative measures to reduce hospital admission and readmission BIBREF3.
Question: Is this dataset publicly available for commercial use?
Paper: A Corpus for Detecting High-Context Medical Conditions in Intensive Care Patient Notes Focusing on Frequently Readmitted Patients
We have created a dataset of discharge summaries and nursing notes, all in the English language, with a focus on frequently readmitted patients, labeled with 15 clinical patient phenotypes believed to be associated with risk of recurrent Intensive Care Unit (ICU) readmission per our domain experts (co-authors LAC, PAT, DAG) as well as the literature. BIBREF10 BIBREF11 BIBREF12
Question: How many different phenotypes are present in the dataset?
Answer: Thirteen different phenotypes are present in the dataset.
Paper: A Corpus for Detecting High-Context Medical Conditions in Intensive Care Patient Notes Focusing on Frequently Readmitted Patients
Clinical researchers teamed with junior medical residents in collaboration with more senior intensive care physicians to carry out text annotation over the period of one year BIBREF13. Operators were grouped to facilitate the annotation of notes in duplicate, allowing for cases of disagreement between operators. The operators within each team were instructed to work independently on note annotation. Clinical texts were annotated in batches which were time-stamped on their day of creation, when both operators in a team completed annotation of a batch, a new batch was created and transferred to them.
Question: What are 10 other phenotypes that are annotated?
Answer: Adv. Heart Disease, Adv. Lung Disease, Alcohol Abuse, Chronic Neurologic Dystrophies, Dementia, Depression, Developmental Delay, Obesity, Psychiatric disorders and Substance Abuse
Paper: A Deeper Look into Sarcastic Tweets Using Deep Convolutional Neural Networks
Sarcasm is defined as “a sharp, bitter, or cutting expression or remark; a bitter gibe or taunt”. As the fields of affective computing and sentiment analysis have gained increasing popularity BIBREF0 , it is a major concern to detect sarcastic, ironic, and metaphoric expressions. Sarcasm, especially, is key for sentiment analysis as it can completely flip the polarity of opinions. Understanding the ground truth, or the facts about a given event, allows for the detection of contradiction between the objective polarity of the event (usually negative) and its sarcastic characteristic by the author (usually positive), as in “I love the pain of breakup”. Obtaining such knowledge is, however, very difficult.
Question: What are the state of the art models?
Paper: A Deeper Look into Sarcastic Tweets Using Deep Convolutional Neural Networks
NLP research is gradually evolving from lexical to compositional semantics BIBREF10 through the adoption of novel meaning-preserving and context-aware paradigms such as convolutional networks BIBREF11 , recurrent belief networks BIBREF12 , statistical learning theory BIBREF13 , convolutional multiple kernel learning BIBREF14 , and commonsense reasoning BIBREF15 . But while other NLP tasks have been extensively investigated, sarcasm detection is a relatively new research topic which has gained increasing interest only recently, partly thanks to the rise of social media analytics and sentiment analysis. Sentiment analysis BIBREF16 and using multimodal information as a new trend BIBREF17 , BIBREF18 , BIBREF19 , BIBREF20 , BIBREF14 is a popular branch of NLP research that aims to understand sentiment of documents automatically using combination of various machine learning approaches BIBREF21 , BIBREF22 , BIBREF20 , BIBREF23 .
Question: Which benchmark datasets are used?
Paper: A Deeper Look into Sarcastic Tweets Using Deep Convolutional Neural Networks
Sarcasm detection is an important subtask of sentiment analysis BIBREF27 . Since sarcastic sentences are subjective, they carry sentiment and emotion-bearing information. Most of the studies in the literature BIBREF28 , BIBREF29 , BIBREF9 , BIBREF30 include sentiment features in sarcasm detection with the use of a state-of-the-art sentiment lexicon. Below, we explain how sentiment information is key to express sarcastic opinions and the approach we undertake to exploit such information for sarcasm detection.
Question: What are the network's baseline features?
Answer:  The features extracted from CNN.
Paper: Adaptively Sparse Transformers
The Transformer architecture BIBREF0 for deep neural networks has quickly risen to prominence in NLP through its efficiency and performance, leading to improvements in the state of the art of Neural Machine Translation BIBREF1, BIBREF2, as well as inspiring other powerful general-purpose models like BERT BIBREF3 and GPT-2 BIBREF4. At the heart of the Transformer lie multi-head attention mechanisms: each word is represented by multiple different weighted averages of its relevant context. As suggested by recent works on interpreting attention head roles, separate attention heads may learn to look for various relationships between tokens BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9.
Question: What tasks are used for evaluation?
Answer: four machine translation tasks: German -> English, Japanese -> English, Romanian -> English, English -> German
Paper: Adaptively Sparse Transformers
In NMT, the Transformer BIBREF0 is a sequence-to-sequence (seq2seq) model which maps an input sequence to an output sequence through hierarchical multi-head attention mechanisms, yielding a dynamic, context-dependent strategy for propagating information within and across sentences. It contrasts with previous seq2seq models, which usually rely either on costly gated recurrent operations BIBREF15, BIBREF16 or static convolutions BIBREF17.
Question: HOw does the method perform compared with baselines?
Answer: On the datasets DE-EN, JA-EN, RO-EN, and EN-DE, the baseline achieves 29.79, 21.57, 32.70, and 26.02  BLEU score, respectively. The 1.5-entmax achieves  29.83, 22.13, 33.10, and 25.89 BLEU score, which is a difference of +0.04, +0.56, +0.40, and -0.13 BLEU score versus the baseline. The α-entmax achieves 29.90, 21.74, 32.89, and 26.93 BLEU score, which is a difference of +0.11, +0.17, +0.19, +0.91 BLEU score versus the baseline.
Paper: Adaptively Sparse Transformers
The softmax mapping (Equation DISPLAY_FORM8) is elementwise proportional to $\exp $, therefore it can never assign a weight of exactly zero. Thus, unnecessary items are still taken into consideration to some extent. Since its output sums to one, this invariably means less weight is assigned to the relevant items, potentially harming performance and interpretability BIBREF18. This has motivated a line of research on learning networks with sparse mappings BIBREF19, BIBREF20, BIBREF21, BIBREF22. We focus on a recently-introduced flexible family of transformations, $\alpha $-entmax BIBREF23, BIBREF14, defined as:
Question: How does their model improve interpretability compared to softmax transformers?