z-uo/qasper-squad huggingface dataset:
Paper: Annotating and normalizing biomedical NEs with limited knowledge
Named Entity Recognition (ner) is considered a necessary first step in the linguistic processing of any new domain, as it facilitates the development of applications showing co-occurrences of domain entities, cause-effect relations among them, and, eventually, it opens the (still to be reached) possibility of understanding full text content. On the other hand, Biomedical literature and, more specifically, clinical texts, show a number of features as regards ner that pose a challenge to NLP researchers BIBREF0: (1) the clinical discourse is characterized by being conceptually very dense; (2) the number of different classes for nes is greater than traditional classes used with, for instance, newswire text; (3) they show a high formal variability for nes (actually, it is rare to find entities in their “canonical form”); and, (4) this text type contains a great number of ortho-typographic errors, due mainly to time constraints when drafted.
Question: What does their system consist of?
Paper: Annotating and normalizing biomedical NEs with limited knowledge
As it is common in resource-based system development, special effort has been devoted to the creation of the set of resources used by the system. These are mainly two —a flat subset of the snomed ct medical ontology, and the library and a part of the contextual regexp grammars developed by BIBREF6 FSL:2018 for a previous competition on abbreviation resolution in clinical texts written in Spanish. The process of creation and/or adaptation of these resources is described in this section.
Question: What are the two PharmaCoNER subtasks?
Answer: Entity identification with offset mapping and concept indexing
Paper: An analysis of the utility of explicit negative examples to improve the syntactic abilities of neural language models
intro
Question: What neural language models are explored?
Paper: An analysis of the utility of explicit negative examples to improve the syntactic abilities of neural language models
The most common evaluation metric of an LM is perplexity. Although neural LMs achieve impressive perplexity BIBREF9, it is an average score across all tokens and does not inform the models' behaviors on linguistically challenging structures, which are rare in the corpus. This is the main motivation to separately evaluate the models' syntactic robustness by a different task.
Question: How do they perform data augmentation?
Answer: They randomly sample sentences from Wikipedia that contains an object RC and add them to training data
Paper: An analysis of the utility of explicit negative examples to improve the syntactic abilities of neural language models
task As introduced in Section intro, the task for a model is to assign a higher probability to the grammatical sentence over the ungrammatical one, given a pair of minimally different sentences at a critical position affecting the grammaticality. For example, (UNKREF3) and (UNKREF5) only differ at a final verb form, and to assign a higher probability to (UNKREF3), models need to be aware of the agreement dependency between author and laughs over an RC.
Question: What proportion of negative-examples do they use?
Paper: Dialectometric analysis of language variation in Twitter
Dialects are language varieties defined across space. These varieties can differ in distinct linguistic levels (phonetic, morphosyntactic, lexical), which determine a particular regional speech BIBREF0 . The extension and boundaries (always diffuse) of a dialect area are obtained from the variation of one or many features such as, e.g., the different word alternations for a given concept. Typically, the dialect forms plotted on a map appear as a geographical continuum that gradually connects places with slightly different diatopic characteristics. A dialectometric analysis aims at a computational approach to dialect distribution, providing quantitative linguistic distances between locations BIBREF1 , BIBREF2 , BIBREF3 .
Question: Do the authors mention any possible confounds in their study?
Paper: Dialectometric analysis of language variation in Twitter
Our corpus consists of approximately 11 million geotagged tweets produced in Europe in Spanish language between October 2014 and June 2016. (Although we will focus on Spain, we will not consider in this work the speech of the Canary Islands due to difficulties with the data extraction). The classification of tweets is accomplished by applying the Compact Language Detector (CLD) BIBREF16 to our dataset. CLD exhibits accurate benchmarks and is thus good for our purposes, although a different detector might be used BIBREF17 . We have empirically checked that when CLD determines the language with a probability of at least 60% the results are extremely reliable. Therefore, we only take into account those tweets for which the probability of being written in Spanish is greater than INLINEFORM0 . Further, we remove unwanted characters, such as hashtags or at-mentions, using Twokenize BIBREF18 , a tokenizer designed for Twitter text in English, adapted to our goals.
Question: What are the characteristics of the city dialect?
Answer: Lexicon of the cities tend to use most forms of a particular concept
Paper: Dialectometric analysis of language variation in Twitter
The dialectometric differences are quantified between regions defined with the aid of our cells. For this purpose we take into account two metrics, which we now briefly discuss.
Question: What are the characteristics of the rural dialect?
Answer: It uses particular forms of a concept rather than all of them uniformly
Paper: Bag of Tricks for Efficient Text Classification
Text classification is an important task in Natural Language Processing with many applications, such as web search, information retrieval, ranking and document classification BIBREF0 , BIBREF1 . Recently, models based on neural networks have become increasingly popular BIBREF2 , BIBREF3 , BIBREF4 . While these models achieve very good performance in practice, they tend to be relatively slow both at train and test time, limiting their use on very large datasets.
Question: What are their baseline methods?
Paper: Bag of Tricks for Efficient Text Classification
A simple and efficient baseline for sentence classification is to represent sentences as bag of words (BoW) and train a linear classifier, e.g., a logistic regression or an SVM BIBREF5 , BIBREF7 . However, linear classifiers do not share parameters among features and classes. This possibly limits their generalization in the context of large output space where some classes have very few examples. Common solutions to this problem are to factorize the linear classifier into low rank matrices BIBREF12 , BIBREF10 or to use multilayer neural networks BIBREF13 , BIBREF14 .
Question: Which datasets are used for evaluation?
Paper: BLiMP: A Benchmark of Linguistic Minimal Pairs for English
Current neural networks for language understanding rely heavily on unsupervised pretraining tasks like language modeling. However, it is still an open question what degree of knowledge state-of-the-art language models (LMs) acquire about different linguistic phenomena. Many recent studies BIBREF0, BIBREF1, BIBREF2 have advanced our understanding in this area by evaluating LMs' preferences between minimal pairs of sentences, as in Example SECREF1. However, these studies have used different analysis metrics and focused on a small set of linguistic paradigms, making a big-picture comparison between these studies limited.
Question: Which of the model yields the best performance?
Paper: BLiMP: A Benchmark of Linguistic Minimal Pairs for English
The objective of a language model is to give a probability distribution over the possible strings of a language. Language models can be built on neural network models or non-neural network models. Due to their unsupervised nature, they can be trained without external annotations. More recently, neural network based language modeling has been shown to be a strong pretraining task for natural language understanding tasks BIBREF6, BIBREF7, BIBREF8, BIBREF9. Some recent models, such as BERT BIBREF9 use closely related tasks such as masked language modeling.
Question: What is the performance of the models on the tasks?
Answer: Overall accuracy per model is: 5-gram (60.5), LSTM (68.9), TXL (68.7), GPT-2 (80.1)
Paper: BLiMP: A Benchmark of Linguistic Minimal Pairs for English
A large number of recent studies has used acceptability judgments to reveal what neural networks know about grammar. One branch of this literature has focused on using minimal pairs to infer whether LMs learn about specific linguistic phenomena. Table TABREF4 gives a summary of work that has studied linguistic phenomena in this way. For instance, linzen2016assessing look closely at minimal pairs contrasting subject-verb agreement. marvin2018targeted look at a larger set of phenomena, including negative polarity item licensing and reflexive licensing. However, a relatively small set of phenomena is covered by these studies, to the exclusion of well-studied phenomena in linguistics such as control and raising, ellipsis, distributional restrictions on quantifiers, and countless others. This is likely due to the labor-intensive nature of collecting examples that exhibit informative grammatical phenomena and their acceptability judgments.
Question: How is the data automatically generated?
Paper: Clustering of Medical Free-Text Records Based on Word Embeddings
Processing of free-text clinical records play an important role in computer-supported medicine BIBREF0 , BIBREF1 . A detailed description of symptoms, examination and an interview is often stored in an unstructured way as free-text, hard to process but rich in important information. Some attempts of processing medical notes exist for English, while for other languages the problem is still challenging BIBREF2 .
Question: Do they fine-tune the used word embeddings on their medical texts?
Paper: Clustering of Medical Free-Text Records Based on Word Embeddings
The clustering method is developed and validated on a new dataset of free-text clinical records of about 100,000 visits.
Question: Which word embeddings do they use to represent medical visits?
Paper: Clustering of Medical Free-Text Records Based on Word Embeddings
In this section we describe our algorithm for visits clustering. The clustering is derived in four steps.
Question: Do they explore similarity of texts across different doctors?
Paper: Clustering of Medical Free-Text Records Based on Word Embeddings
As there are no generally available terminological resources for Polish medical texts, the first step of data processing was aimed at automatic identification of the most frequently used words and phrases. The doctors' notes are usually rather short and concise, so we assumed that all frequently appearing phrases are domain related and important for text understanding. The notes are built mostly from noun phrases so it was decided to extract simple noun phrases which consist of a noun optionally modified by a sequence of adjectives (in Polish they can occur both before and after a noun) or by another noun in the genitive. We only extracted sequences that can be interpreted as phrases in Polish, i.e. nouns and adjectives have to agree in case, number and gender.
Question: Which clustering technique do they use on partients' visits texts?
Paper: A Constructive Prediction of the Generalization Error Across Scales
Question: What is proof that proposed functional form approximates well generalization error in practice?
Paper: A Constructive Prediction of the Generalization Error Across Scales
Question: How is proposed functional form constructed for some model?
Paper: Character-Based Text Classification using Top Down Semantic Model for Sentence Representation
Recently, deep learning has been particularly successful in speech and image as an automatic feature extractor BIBREF1 , BIBREF2 , BIBREF3 , however deep learning's application to text as an automatic feature extractor has not been always successful BIBREF0 even compared to simple linear models with BoW or TF-IDF feature representation. In many experiments when the text is polished like news articles or when the dataset is small, BoW or TF-IDF is still the state-of-art representation compared to sent2vec or paragraph2vec BIBREF4 representation using deep learning models like RNN (Recurrent Neural Network) or CNN (Convolution Neural Network) BIBREF0 . It is only when the dataset becomes large or when the words are noisy and non-standardized with misspellings, text emoticons and short-forms that deep learning models which learns the sentence-level semantics start to outperform BoW representation, because under such circumstances, BoW representation can become extremely sparse and the vocabulary size can become huge. It becomes clear that for large, complex data, a large deep learning model with a large capacity can extract a better sentence-level representation than BoW sentence representation. However, for small and standardized news-like dataset, a direct word counting TF-IDF sentence representation is superior. Then the question is can we design a deep learning model that performs well for both simple and complex, small and large datasets? And when the dataset is small and standardized, the deep learning model should perform comparatively well as BoW? With that problem in mind, we designed TDSM (Top-Down-Semantic-Model) which learns a sentence representation that carries the information of both the BoW-like representation and RNN style of sentence-level semantic which performs well for both simple and complex, small and large datasets.
Question: What other non-neural baselines do the authors compare to? 
Answer: bag of words, tf-idf, bag-of-means
Paper: A Better Variant of Self-Critical Sequence Training
Self-Critical Sequence Training(SCST), upon its release, has been a popular way to train sequence generation models. While originally proposed for image captioning task, SCST not only has become the new standard for training captioning models BIBREF0, BIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9, but also has been applied to many other tasks, like video captioningBIBREF10, BIBREF11, BIBREF12, reading comprehensionBIBREF13, summarizationBIBREF14, BIBREF15, BIBREF16, BIBREF17, image paragraph generationBIBREF18, speech recognitionBIBREF19.
Question: How much better is performance of proposed approach compared to greedy decoding baseline?
Paper: A Better Variant of Self-Critical Sequence Training
MIXER BIBREF22 is the first to use REINFORCE algorithm for sequence generation training. They use a learned function approximator to get the baseline.
Question: What environment is used for self-critical sequence training?
Paper: A Better Variant of Self-Critical Sequence Training
The goal of SCST, for example in captioning, is to maximize the expected CIDEr score of generated captions.
Question: What baseline function is used in REINFORCE algorithm?
Paper: A Better Variant of Self-Critical Sequence Training
The success of SCST comes from better gradient variance reduction introduced by the greedy decoding baseline. In our variant, we use the baseline proposed in BIBREF21 to achieve even better variance reduction.
Question: What baseline model is used for comparison?
Paper: From 'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo Project
This paper reports on the history, progress, and lessons from the Aristo project, a six-year quest to answer grade-school and high-school science exams. Aristo has recently surpassed 90% on multiple choice questions from the 8th Grade New York Regents Science Exam (see Figure FIGREF6). We begin by offering several perspectives on why this achievement is significant for NLP and for AI more broadly.
Question: Is Aristo just some modern NLP model (ex. BERT) finetuned od data specific for this task?
Paper: From 'F' to 'A' on the N.Y. Regents Science Exams: An Overview of the Aristo Project
In 1950, Alan Turing proposed the now well-known Turing Test as a possible test of machine intelligence: If a system can exhibit conversational behavior that is indistinguishable from that of a human during a conversation, that system could be considered intelligent (BID1). As the field of AI has grown, the test has become less meaningful as a challenge task for several reasons. First, its setup is not well defined (e.g., who is the person giving the test?). A computer scientist would likely know good distinguishing questions to ask, while a random member of the general public may not. What constraints are there on the interaction? What guidelines are provided to the judges? Second, recent Turing Test competitions have shown that, in certain formulations, the test itself is gameable; that is, people can be fooled by systems that simply retrieve sentences and make no claim of being intelligent (BID2;BID3). John Markoff of The New York Times wrote that the Turing Test is more a test of human gullibility than machine intelligence. Finally, the test, as originally conceived, is pass/fail rather than scored, thus providing no measure of progress toward a goal, something essential for any challenge problem.
Question: On what dataset is Aristo system trained?
Answer: Aristo Corpus
Regents 4th
Regents 8th
Regents `12th
ARC-Easy
ARC-challenge
Paper: Opinion Recommendation using Neural Memory Model
Offering a channel for customers to share opinions and give scores to products and services, review websites have become a highly influential information source that customers refer to for making purchase decisions. Popular examples include IMDB on the movie domain, Epinions on the product domain, and Yelp on the service domain. Figure FIGREF4 shows a screenshot of a restaurant review page on Yelp.com, which offers two main types of information. First, an overall rating score is given under the restaurant name; second, detailed user reviews are listed below the rating.
Question: Does they focus on any specific product/service domain?
Paper: Opinion Recommendation using Neural Memory Model
Sentiment Analysis. Our task is related to document-level sentiment classification BIBREF1 , which is to infer the sentiment polarity of a given document. Recently, various neural network models are used to capture the sentimental information automatically, including convolutional neural networks BIBREF9 , recursive neural network BIBREF10 and recurrent neural network BIBREF11 , BIBREF12 , which have been shown to achieve competitive results across different benchmarks. Different from binary classification, review rating prediction aims to predict the numeric rating of a given review. PangL05 pioneered this task by regarding it as a classification/regression problem. Most subsequent work focuses on designing effective textural features of reviews BIBREF13 , BIBREF14 , BIBREF15 . Recently, TangQLY15 proposed a neural network model to predict the rating score by using both lexical semantic and user model.
Question: What are the baselines?
Paper: Improving Quality and Efficiency in Plan-based Neural Data-to-Text Generation
In the data-to-text generation task (D2T), the input is data encoding facts (e.g., a table, a set of tuples, or a small knowledge graph), and the output is a natural language text representing those facts. In neural D2T, the common approaches train a neural end-to-end encoder-decoder system that encodes the input data and decodes an output text. In recent work BIBREF0 we proposed to adopt ideas from “traditional” language generation approaches (i.e. BIBREF1, BIBREF2, BIBREF3) that separate the generation into a planning stage that determines the order and structure of the expressed facts, and a realization stage that maps the plan to natural language text. We show that by breaking the task this way, one can achieve the same fluency of neural generation systems while being able to better control the form of the generated text and to improve its correctness by reducing missing facts and “hallucinations”, common in neural systems.
Question: How is fluency of generated text evaluated?
Paper: Improving Quality and Efficiency in Plan-based Neural Data-to-Text Generation
We provide a brief overview of the step-by-step system. See BIBREF0 for further details. The system works in two stages. The first stage (planning) maps the input facts (encoded as a directed, labeled graph, where nodes represent entities and edges represent relations) to text plans, while the second stage (realization) maps the text plans to natural language text.
Question: How is faithfulness of the resulting text evaluated?
Paper: Improving Quality and Efficiency in Plan-based Neural Data-to-Text Generation
The data-to-plan component in BIBREF0 exhaustively generates all possible plans, scores them using a heuristic, and chooses the highest scoring one for realization. While this is feasible with the small input graphs in the WebNLG challenge BIBREF4, it is also very computationally intensive, growing exponentially with the input size. We propose an alternative planner which works in linear time in the size of the graph and remains verifiable: generated plans are guaranteed to represent the input faithfully.
Question: How are typing hints suggested?
Paper: Improving Quality and Efficiency in Plan-based Neural Data-to-Text Generation
In BIBREF0, the sentence plan trees were linearized into strings that were then fed to a neural machine translation decoder (OpenNMT) BIBREF6 with a copy mechanism. This linearization process is lossy, in the sense that the linearized strings do not explicitly distinguish between symbols that represent entities (e.g., BARACK_OBAMA) and symbols that represent relations (e.g., works-for). While this information can be deduced from the position of the symbol within the structure, there is a benefit in making it more explicit. In particular, the decoder needs to act differently when decoding relations and entities: entities are copied, while relations need to be verbalized. By making the typing information explicit to the decoder, we make it easier for it to generalize this behavior distinction and apply it also for unseen entities and relations. We thus expect the typing information to be especially useful for the unseen part of the evaluation set.
Question: What is the effectiveness plan generation?
Paper: Improving Quality and Efficiency in Plan-based Neural Data-to-Text Generation
While the plan generation stage is guaranteed to be faithful to the input, the translation process from plans to text is based on a neural seq2seq model and may suffer from known issues with such models: hallucinating facts that do not exist in the input, repeating facts, or dropping facts. While the clear mapping between plans and text helps to reduce these issues greatly, the system in BIBREF0 still has 2% errors of these kinds.
Question: How is neural planning component trained?
Paper: Rotations and Interpretability of Word Embeddings: the Case of the Russian Language
Word embeddings are frequently used in NLP tasks. In vector space models every word from the source corpus is represented by a dense vector in $\mathbb {R}^d$ , where the typical dimension $d$ varies from tens to hundreds. Such embedding maps similar (in some sense) words to close vectors. These models are based on the so called distributional hypothesis: similar words tend to occur in similar contexts BIBREF0 . Some models also use letter trigrams or additional word properties such as morphological tags.
Question: How do they evaluate interpretability in this paper?
Paper: Improving Visually Grounded Sentence Representations with Self-Attention
Recent NLP studies have thrived on distributional hypothesis. More recently, there have been efforts in applying the intuition to larger semantic units, such as sentences, or documents. However, approaches based on distributional semantics are limited by the grounding problem BIBREF0 , which calls for techniques to ground certain conceptual knowledge in perceptual information.
Question: How much better performing is the proposed method over the baselines?