z-uo/qasper-squad huggingface dataset:
Paper: Multi-class Hierarchical Question Classification for Multiple Choice Science Exams
Question classification typically makes use of a combination of syntactic, semantic, surface, and embedding methods. Syntactic patterns BIBREF18 , BIBREF19 , BIBREF20 , BIBREF21 and syntactic dependencies BIBREF3 have been shown to improve performance, while syntactically or semantically important words are often expanding using Wordnet hypernyms or Unified Medical Language System categories (for the medical domain) to help mitigate sparsity BIBREF22 , BIBREF23 , BIBREF24 . Keyword identification helps identify specific terms useful for classification BIBREF25 , BIBREF3 , BIBREF26 . Similarly, named entity recognizers BIBREF6 , BIBREF27 or lists of semantically related words BIBREF6 , BIBREF24 can also be used to establish broad topics or entity categories and mitigate sparsity, as can word embeddings BIBREF28 , BIBREF29 . Here, we empirically demonstrate many of these existing methods do not transfer to the science domain.
Question: What previous methods is their model compared to?
Paper: Multi-class Hierarchical Question Classification for Multiple Choice Science Exams
Questions: We make use of the 7,787 science exam questions of the Aristo Reasoning Challenge (ARC) corpus BIBREF31 , which contains standardized 3rd to 9th grade science questions from 12 US states from the past decade. Each question is a 4-choice multiple choice question. Summary statistics comparing the complexity of ARC and TREC questions are shown in Table TABREF5 .
Question: Did they use a crowdsourcing platform?
Paper: Multi-class Hierarchical Question Classification for Multiple Choice Science Exams
We identified 5 common models in previous work primarily intended for learned classifiers rather than hand-crafted rules. We adapt these models to a multi-label hierarchical classification task by training a series of one-vs-all binary classifiers BIBREF34 , one for each label in the taxonomy. With the exception of the CNN and BERT models, following previous work BIBREF19 , BIBREF3 , BIBREF8 we make use of an SVM classifier using the LIBSvM framework BIBREF35 with a linear kernel. Models are trained and evaluated from coarse to fine levels of taxonomic specificity. At each level of taxonomic evaluation, a set of non-overlapping confidence scores for each binary classifier are generated and sorted to produce a list of ranked label predictions. We evaluate these ranks using Mean Average Precision BIBREF36 . ARC questions are evaluated using the standard 3,370 questions for training, 869 for development, and 3,548 for testing.
Question: How was the dataset collected?
Answer: from 3rd to 9th grade science questions collected from 12 US states
Used from  science exam questions of the Aristo Reasoning Challenge (ARC) corpus.
Paper: Neural Collective Entity Linking
 This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http://creativecommons.org/licenses/by/4.0/
Question: Which datasets do they use?
Paper: Neural Collective Entity Linking
We denote INLINEFORM0 as a set of entity mentions in a document INLINEFORM1 , where INLINEFORM2 is either a word INLINEFORM3 or a mention INLINEFORM4 . INLINEFORM5 is the entity graph for document INLINEFORM6 derived from the given knowledge base, where INLINEFORM7 is a set of entities, INLINEFORM8 denotes the relatedness between INLINEFORM9 and higher values indicate stronger relations. Based on INLINEFORM10 , we extract a subgraph INLINEFORM11 for INLINEFORM12 , where INLINEFORM13 denotes the set of candidate entities for INLINEFORM14 . Note that we don't include the relations among candidates of the same mention in INLINEFORM15 because these candidates are mutually exclusive in disambiguation.
Question: How effective is their NCEL approach overall?
Paper: Neural Collective Entity Linking
Similar to previous work BIBREF24 , we use the prior probability INLINEFORM0 of entity INLINEFORM1 conditioned on mention INLINEFORM2 both as a local feature and to generate candidate entities: INLINEFORM3 . We compute INLINEFORM4 based on statistics of mention-entity pairs from: (i) Wikipedia page titles, redirect titles and hyperlinks, (ii) the dictionary derived from a large Web Corpus BIBREF27 , and (iii) the YAGO dictionary with a uniform distribution BIBREF22 . We pick up the maximal prior if a mention-entity pair occurs in different resources. In experiments, to optimize for memory and run time, we keep only top INLINEFORM5 entities based on INLINEFORM6 . In the following two sections, we will present the key components of NECL, namely feature extraction and neural network for collective entity linking.
Question: How do they verify generalization ability?
Answer: By calculating Macro F1 metric at the document level.
by evaluating their model on five different benchmarks
Paper: Neural Collective Entity Linking
The main goal of NCEL is to find a solution for collective entity linking using an end-to-end neural model, rather than to improve the measurements of local textual similarity or global mention/entity relatedness. Therefore, we use joint embeddings of words and entities at sense level BIBREF28 to represent mentions and its contexts for feature extraction. In this section, we give a brief description of our embeddings followed by our features used in the neural model.
Question: Do they only use adjacent entity mentions or use more than that in some cases (next to adjacent)?
Answer: NCEL considers only adjacent mentions.
More than that in some cases (next to adjacent)
Paper: To lemmatize or not to lemmatize: how word normalisation affects ELMo performance in word sense disambiguation
Deep contextualised representations of linguistic entities (words and/or sentences) are used in many current state-of-the-art NLP systems. The most well-known examples of such models are arguably ELMo BIBREF0 and BERT BIBREF1.
Question: Do the authors mention any downside of lemmatizing input before training ELMo?
Paper: To lemmatize or not to lemmatize: how word normalisation affects ELMo performance in word sense disambiguation
ELMo contextual word representations are learned in an unsupervised way through language modelling BIBREF0. The general architecture consists of a two-layer BiLSTM on top of a convolutional layer which takes character sequences as its input. Since the model uses fully character-based token representations, it avoids the problem of out-of-vocabulary words. Because of this, the authors explicitly recommend not to use any normalisation except tokenization for the input text. However, as we show below, while this is true for English, for other languages feeding ELMo with lemmas instead of raw tokens can improve WSD performance.
Question: What other examples of morphologically-rich languages do the authors give?
Paper: To lemmatize or not to lemmatize: how word normalisation affects ELMo performance in word sense disambiguation
For the experiments described below, we trained our own ELMo models from scratch. For English, the training corpus consisted of the English Wikipedia dump from February 2017. For Russian, it was a concatenation of the Russian Wikipedia dump from December 2018 and the full Russian National Corpus (RNC). The RNC texts were added to the Russian Wikipedia dump so as to make the Russian training corpus more comparable in size to the English one (Wikipedia texts would comprise only half of the size). As Table TABREF3 shows, the English Wikipedia is still two times larger, but at least the order is the same.
Question: Why is lemmatization not necessary in English?
Answer: Advanced neural architectures and contextualized embedding models learn how to handle spelling and morphology variations.
Paper: To lemmatize or not to lemmatize: how word normalisation affects ELMo performance in word sense disambiguation
We used two WSD datasets for evaluation:
Question: How big was the corpora they trained ELMo on?
Answer: 2174000000, 989000000
2174 million tokens for English and 989 million tokens for Russian
Paper: Generating Descriptions from Structured Data Using a Bifocal Attention Mechanism and Gated Orthogonalization
Rendering natural language descriptions from structured data is required in a wide variety of commercial applications such as generating descriptions of products, hotels, furniture, etc., from a corresponding table of facts about the entity. Such a table typically contains {field, value} pairs where the field is a property of the entity (e.g., color) and the value is a set of possible assignments to this property (e.g., color = red). Another example of this is the recently introduced task of generating one line biography descriptions from a given Wikipedia infobox BIBREF0 . The Wikipedia infobox serves as a table of facts about a person and the first sentence from the corresponding article serves as a one line description of the person. Figure FIGREF2 illustrates an example input infobox which contains fields such as Born, Residence, Nationality, Fields, Institutions and Alma Mater. Each field further contains some words (e.g., particle physics, many-body theory, etc.). The corresponding description is coherent with the information contained in the infobox.
Question: What metrics are used for evaluation?
Paper: Generating Descriptions from Structured Data Using a Bifocal Attention Mechanism and Gated Orthogonalization
Natural Language Generation has always been of interest to the research community and has received a lot of attention in the past. The approaches for NLG range from (i) rule based approaches (e.g., BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 ) (ii) modular statistical approaches which divide the process into three phases (planning, selection and surface realization) and use data driven approaches for one or more of these phases BIBREF15 , BIBREF16 , BIBREF17 , BIBREF18 , BIBREF19 (iii) hybrid approaches which rely on a combination of handcrafted rules and corpus statistics BIBREF20 , BIBREF21 , BIBREF22 and (iv) the more recent neural network based models BIBREF1 .
Question: Do they use pretrained embeddings?
Paper: Generating Descriptions from Structured Data Using a Bifocal Attention Mechanism and Gated Orthogonalization
As input we are given an infobox INLINEFORM0 , which is a set of pairs INLINEFORM1 where INLINEFORM2 corresponds to field names and INLINEFORM3 is the sequence of corresponding values and INLINEFORM4 is the total number of fields in INLINEFORM5 . For example, INLINEFORM6 could be one such pair in this set. Given such an input, the task is to generate a description INLINEFORM7 containing INLINEFORM8 words. A simple solution is to treat the infobox as a sequence of fields followed by the values corresponding to the field in the order of their appearance in the infobox. For example, the infobox could be flattened to produce the following input sequence (the words in bold are field names which act as delimiters)
Question: What dataset is used?
Answer: English WIKIBIO, French WIKIBIO , German WIKIBIO
Paper: Generating Descriptions from Structured Data Using a Bifocal Attention Mechanism and Gated Orthogonalization
Intuitively, when a human writes a description from a table she keeps track of information at two levels. At the macro level, it is important to decide which is the appropriate field to attend to next and at a micro level (i.e., within a field) it is important to know which values to attend to next. To capture this behavior, we use a bifocal attention mechanism as described below.
Question: What is a bifocal attention mechanism?
Paper: Combating Adversarial Misspellings with Robust Word Recognition
Despite the rapid progress of deep learning techniques on diverse supervised learning tasks, these models remain brittle to subtle shifts in the data distribution. Even when the permissible changes are confined to barely-perceptible perturbations, training robust models remains an open challenge. Following the discovery that imperceptible attacks could cause image recognition models to misclassify examples BIBREF0 , a veritable sub-field has emerged in which authors iteratively propose attacks and countermeasures.
Question: What does the "sensitivity" quantity denote?
Answer: The expected number of unique outputs a word recognition system assigns to a set of adversarial perturbations
Paper: Combating Adversarial Misspellings with Robust Word Recognition
Several papers address adversarial attacks on NLP systems. Changes to text, whether word- or character-level, are all perceptible, raising some questions about what should rightly be considered an adversarial example BIBREF8 , BIBREF9 . BIBREF10 address the reading comprehension task, showing that by appending distractor sentences to the end of stories from the SQuAD dataset BIBREF11 , they could cause models to output incorrect answers. Inspired by this work, BIBREF12 demonstrate an attack that breaks entailment systems by replacing a single word with either a synonym or its hypernym. Recently, BIBREF13 investigated the problem of producing natural-seeming adversarial examples, noting that adversarial examples in NLP are often ungrammatical BIBREF14 .
Question: What end tasks do they evaluate on?
Answer: Sentiment analysis and paraphrase detection under adversarial attacks
Paper: Combating Adversarial Misspellings with Robust Word Recognition
To tackle character-level adversarial attacks, we introduce a simple two-stage solution, placing a word recognition model ( $W$ ) before the downstream classifier ( $C$ ). Under this scheme, all inputs are classified by the composed model $C \circ W$ . This modular approach, with $W$ and $C$ trained separately, offers several benefits: (i) we can deploy the same word recognition model for multiple downstream classification tasks/models; and (ii) we can train the word recognition model with larger unlabeled corpora.
Question: What is a semicharacter architecture?
Answer: A semi-character based RNN (ScRNN) treats the first and last characters individually, and is agnostic to the ordering of the internal characters
Paper: Combating Adversarial Misspellings with Robust Word Recognition
We now describe semi-character RNNs for word recognition, explain their limitations, and suggest techniques to improve them.
Question: Do they experiment with offering multiple candidate corrections and voting on the model output, since this seems highly likely to outperform a one-best correction?
Paper: Combating Adversarial Misspellings with Robust Word Recognition
In computer vision, an important factor determining the success of an adversary is the norm constraint on the perturbations allowed to an image ( $|| \bf x - \bf x^{\prime }||_{\infty } < \epsilon $ ). Higher values of $\epsilon $ lead to a higher chance of mis-classification for at least one $\bf x^{\prime }$ . Defense methods such as quantization BIBREF22 and thermometer encoding BIBREF23 try to reduce the space of perturbations available to the adversary by making the model invariant to small changes in the input.
Question: Why is the adversarial setting appropriate for misspelling recognition?
Answer: Adversarial misspellings are a real-world problem
Paper: Combating Adversarial Misspellings with Robust Word Recognition
Suppose we are given a classifier $C: \mathcal {S} \rightarrow \mathcal {Y}$ which maps natural language sentences $s \in \mathcal {S}$ to a label from a predefined set $y \in \mathcal {Y}$ . An adversary for this classifier is a function $A$ which maps a sentence $s$ to its perturbed versions $\lbrace s^{\prime }_1, s^{\prime }_2, \ldots , s^{\prime }_{n}\rbrace $ such that each $s^{\prime }_i$ is close to $s$ under some notion of distance between sentences. We define the robustness of classifier $C$ to the adversary $A$ as: 
Question: Why do they experiment with RNNs instead of transformers for this task?
Paper: Combating Adversarial Misspellings with Robust Word Recognition
In this section, we first discuss our experiments on the word recognition systems.
Question: How do the backoff strategies work?
Answer: In pass-through, the recognizer passes on the possibly misspelled word, backoff to neutral word backs off to a word with similar distribution across classes and backoff to background model backs off to a more generic word recognition model trained with larger and less specialized corpus.
Pass-through passes the possibly misspelled word as is, backoff to neutral word backs off to a word with similar distribution across classes and backoff to background model backs off to a more generic word recognition model trained with larger and less specialized corpus.
Backoff to "a" when an UNK-predicted word is encountered, backoff to a more generic word recognition model when the model predicts UNK
Paper: A Bayesian Model of Multilingual Unsupervised Semantic Role Induction
Semantic Role Labeling (SRL) has emerged as an important task in Natural Language Processing (NLP) due to its applicability in information extraction, question answering, and other NLP tasks. SRL is the problem of finding predicate-argument structure in a sentence, as illustrated below:
Question: What baseline model is used?
Paper: A Bayesian Model of Multilingual Unsupervised Semantic Role Induction
As established in previous work BIBREF7 , BIBREF8 , we use a standard unsupervised SRL setup, consisting of the following steps:
Question: Which additional latent variables are used in the model?
Paper: A Bayesian Model of Multilingual Unsupervised Semantic Role Induction
We use the Bayesian model of garg2012unsupervised as our base monolingual model. The semantic roles are predicate-specific. To model the role ordering and repetition preferences, the role inventory for each predicate is divided into Primary and Secondary roles as follows:
Question: Which parallel corpora are used?
Paper: A Bayesian Model of Multilingual Unsupervised Semantic Role Induction
The multilingual model uses word alignments between sentences in a parallel corpus to exploit role correspondences across languages. We make copies of the monolingual model for each language and add additional crosslingual latent variables (CLVs) to couple the monolingual models, capturing crosslingual semantic role patterns. Concretely, when training on parallel sentences, whenever the head words of the arguments are aligned, we add a CLV as a parent of the two corresponding role variables. Figure FIGREF16 illustrates this model. The generative process, as explained below, remains the same as the monolingual model for the most part, with the exception of aligned roles which are now generated by both the monolingual process as well as the CLV.
Question: Overall, does having parallel data improve semantic role induction across multiple languages?
Paper: A Bayesian Model of Multilingual Unsupervised Semantic Role Induction
The inference problem consists of predicting the role labels and CLVs (the hidden variables) given the predicate, its voice, and syntactic features of all the identified arguments (the visible variables). We use a collapsed Gibbs-sampling based approach to generate samples for the hidden variables (model parameters are integrated out). The sample counts and the priors are then used to calculate the MAP estimate of the model parameters.
Question: Do they add one latent variable for each language pair in their Bayesian model?
Paper: A Bayesian Model of Multilingual Unsupervised Semantic Role Induction
Following the setting of titovcrosslingual, we evaluate only on the arguments that were correctly identified, as the incorrectly identified arguments do not have any gold semantic labels. Evaluation is done using the metric proposed by lang2011unsupervised, which has 3 components: (i) Purity (PU) measures how well an induced cluster corresponds to a single gold role, (ii) Collocation (CO) measures how well a gold role corresponds to a single induced cluster, and (iii) F1 is the harmonic mean of PU and CO. For each predicate, let INLINEFORM0 denote the total number of argument instances, INLINEFORM1 the instances in the induced cluster INLINEFORM2 , and INLINEFORM3 the instances having label INLINEFORM4 in gold annotations. INLINEFORM5 , INLINEFORM6 , and INLINEFORM7 . The score for each predicate is weighted by the number of its argument instances, and a weighted average is computed over all the predicates.
Question: What does an individual model consist of?
Paper: A Bayesian Model of Multilingual Unsupervised Semantic Role Induction
We use the same baseline as used by lang2011unsupervised which has been shown to be difficult to outperform. This baseline assigns a semantic role to a constituent based on its syntactic function, i.e. the dependency relation to its head. If there is a total of INLINEFORM0 clusters, INLINEFORM1 most frequent syntactic functions get a cluster each, and the rest are assigned to the INLINEFORM2 th cluster.
Question: Do they improve on state-of-the-art semantic role induction?
Paper: Evaluating Tag Recommendations for E-Book Annotation Using a Semantic Similarity Metric
When people shop for books online in e-book stores such as, e.g., the Amazon Kindle store, they enter search terms with the goal to find e-books that meet their preferences. Such e-books have a variety of metadata such as, e.g., title, author or keywords, which can be used to retrieve e-books that are relevant to the query. As a consequence, from the perspective of e-book publishers and editors, annotating e-books with tags that best describe the content and which meet the vocabulary of users (e.g., when searching and reviewing e-books) is an essential task BIBREF0 .
Question: how many tags do they look at?
Paper: Evaluating Tag Recommendations for E-Book Annotation Using a Semantic Similarity Metric
In this section, we describe our dataset as well as our tag recommendation approaches we propose to annotate e-books.
Question: which algorithm was the highest performer?
Answer: A hybrid model consisting of best performing popularity-based approach with the best similarity-based approach
Paper: Evaluating Tag Recommendations for E-Book Annotation Using a Semantic Similarity Metric
Our dataset contains two sources of data, one to generate tag recommendations and another one to evaluate tag recommendations. HGV GmbH has collected all data sources and we provide the dataset statistics in Table TABREF3 .
Question: how is diversity measured?
Paper: Evaluating Tag Recommendations for E-Book Annotation Using a Semantic Similarity Metric
We implement three types of tag recommendation approaches, i.e., (i) popularity-based, (ii) similarity-based (i.e., using content information), and (iii) hybrid approaches. Due to the lack of personalized tags (i.e., we do not know which user has assigned a tag), we do not implement other types of algorithms such as collaborative filtering BIBREF8 . In total, we evaluate 19 different algorithms to recommend tags for annotating e-books.
Question: how large is the vocabulary?
Paper: Evaluating Tag Recommendations for E-Book Annotation Using a Semantic Similarity Metric
In this section, we describe our evaluation protocol as well as the measures we use to evaluate and compare our tag recommendation approaches.
Question: what dataset was used?
Answer:  E-book annotation data: editor tags, Amazon search terms, and  Amazon review keywords.
Paper: Evaluating Tag Recommendations for E-Book Annotation Using a Semantic Similarity Metric
For evaluation, we use the third set of e-book annotations, namely Amazon review keywords. As described in Section SECREF1 , these review keywords are extracted from the Amazon review texts and thus, reflect the users' vocabulary. We evaluate our approaches for the 2,896 e-books, for whom we got review keywords. To follow common practice for tag recommendation evaluation BIBREF14 , we predict the assigned review keywords (= our test set) for respective e-books.
Question: what algorithms did they use?