z-uo/qasper-squad huggingface dataset:
Paper: Visual Question: Predicting If a Crowd Will Agree on the Answer
The remainder of the paper is organized into four sections. We first describe a study where we investigate: 1) How much answer diversity arises for visual questions? and 2) Why do people disagree (Section SECREF4 )? Next, we explore the following two questions: 1) Given a novel visual question, can a machine correctly predict whether multiple independent members of a crowd would supply the same answer? and 2) If so, what insights does our machine-learned system reveal regarding what humans are most likely to agree about (Section SECREF5 )? In the following section, we propose a novel resource allocation system for efficiently capturing the diversity of all answers for a set of visual questions (Section SECREF6 ). Finally, we end with concluding remarks (Section SECREF7 ).
Question: How is the data used for training annotated?
Answer: The number of redundant answers to collect from the crowd is predicted to efficiently capture the diversity of all answers from all visual questions.
Paper: A Multi-Task Architecture on Relevance-based Neural Query Translation
CLIR systems retrieve documents written in a language that is different from search query language BIBREF0 . The primary objective of CLIR is to translate or project a query into the language of the document repository BIBREF1 , which we refer to as Retrieval Corpus (RC). To this end, common CLIR approaches translate search queries using a Machine Translation (MT) model and then use a monolingual IR system to retrieve from RC. In this process, a translation model is treated as a black box BIBREF2 , and it is usually trained on a sentence level parallel corpus, which we refer to as Translation Corpus (TC).
Question: what quantitative analysis is done?
Answer: Answer with content missing: (Evaluation section) Given that in CLIR the primary goal is to get a better ranked list of documents against a translated query, we only report Mean Average Precision (MAP).
Paper: A Multi-Task Architecture on Relevance-based Neural Query Translation
We train NMT with RAT to achieve better query translations. We improve a recently proposed NMT baseline, Transformer, that achieves state-of-the-art results for sentence pairs in some languages BIBREF8 . We discuss Transformer, RAT, and our multi-task learning architecture that achieves balanced translation.
Question: what are the baselines?
Paper: A Question-Entailment Approach to Question Answering
With the availability of rich data on users' locations, profiles and search history, personalization has become the leading trend in large-scale information retrieval. However, efficiency through personalization is not yet the most suitable model when tackling domain-specific searches. This is due to several factors, such as the lexical and semantic challenges of domain-specific data that often include advanced argumentation and complex contextual information, the higher sparseness of relevant information sources, and the more pronounced lack of similarities between users' searches.
Question: Do they report results only on English data?
Paper: A Question-Entailment Approach to Question Answering
In this section we define the RQE task and describe related work at the intersection of question answering, question similarity and textual inference.
Question: What machine learning and deep learning methods are used for RQE?
Paper: Neural User Simulation for Corpus-based Policy Optimisation for Spoken Dialogue Systems
Spoken Dialogue Systems (SDS) allow human-computer interaction using natural speech. Task-oriented dialogue systems, the focus of this work, help users achieve goals such as finding restaurants or booking flights BIBREF0 .
Question: by how much did nus outperform abus?
Answer: Average success rate is higher by 2.6 percent points.
Paper: Neural User Simulation for Corpus-based Policy Optimisation for Spoken Dialogue Systems
A Task-Oriented SDS is typically designed according to a structured ontology, which defines what the system can talk about. In a system recommending restaurants the ontology defines those attributes of a restaurant that the user can choose, called informable slots (e.g. different food types, areas and price ranges), the attributes that the user can request, called requestable slots (e.g. phone number or address) and the restaurants that it has data about. An attribute is referred to as a slot and has a corresponding value. Together these are referred to as a slot-value pair (e.g. area=north).
Question: what corpus is used to learn behavior?
Paper: Text Classification based on Word Subspace with Term-Frequency
Text classification has become an indispensable task due to the rapid growth in the number of texts in digital form available online. It aims to classify different texts, also called documents, into a fixed number of predefined categories, helping to organize data, and making easier for users to find the desired information. Over the past three decades, many methods based on machine learning and statistical models have been applied to perform this task, such as latent semantic analysis (LSA), support vector machines (SVM), and multinomial naive Bayes (MNB).
Question: Which dataset has been used in this work?
Answer: The Reuters-8 dataset (with stop words removed)
Paper: Text Classification based on Word Subspace with Term-Frequency
In this section, we outline relevant work towards text classification. We start by describing how text data is conventionally represented using the bag-of-words model and then follow to describe the conventional methods utilized in text classification.
Question: What can word subspace represent?
Answer: Word vectors, usually in the context of others within the same class
Paper: A Neural Topic-Attention Model for Medical Term Abbreviation Disambiguation
Medical text mining is an exciting area and is becoming attractive to natural language processing (NLP) researchers. Clinical notes are an example of text in the medical area that recent work has focused on BIBREF0, BIBREF1, BIBREF2. This work studies abbreviation disambiguation on clinical notes BIBREF3, BIBREF4, specifically those used commonly by physicians and nurses. Such clinical abbreviations can have a large number of meanings, depending on the specialty BIBREF5, BIBREF6. For example, the term MR can mean magnetic resonance, mitral regurgitation, mental retardation, medical record and the general English Mister (Mr.). Table TABREF1 illustrates such an example. Abbreviation disambiguation is an important task in medical text understanding task BIBREF7. Successful recognition of the abbreviations in the notes can contribute to downstream tasks such as classification, named entity recognition, and relation extraction BIBREF7.
Question: How big are improvements of small-scale unbalanced datasets when sentence representation is enhanced with topic information?
Answer: it has 0.024 improvement in accuracy comparing to ELMO Only and  0.006 improvement in F1 score  comparing to ELMO Only too
Paper: A Neural Topic-Attention Model for Medical Term Abbreviation Disambiguation
Question: To what baseline models is proposed model compared?
Paper: A Neural Topic-Attention Model for Medical Term Abbreviation Disambiguation
We conducted a comprehensive comparison with the baseline models, and some of them were never investigated for the abbreviation disambiguation task. We applied traditional features by simply taking the TF-IDF features as the inputs into the classic classifiers. Deep features are also considered: a Doc2vec model BIBREF19 was pre-trained using Gensim and these word embeddings were applied to initialize deep models and fine-tuned.
Question: How big is dataset for testing?
Answer: 30 terms, each term-sanse pair has around 15 samples for testing
Paper: A Neural Topic-Attention Model for Medical Term Abbreviation Disambiguation
Question: What existing dataset is re-examined and corrected for training?
Paper: Learning Multi-Sense Word Distributions using Approximate Kullback-Leibler Divergence
Language modelling in its inception had one-hot vector encoding of words. However, it captures only alphabetic ordering but not the word semantic similarity. Vector space models helps to learn word representations in a lower dimensional space and also captures semantic similarity. Learning word embedding aids in natural language processing tasks such as question answering and reasoning BIBREF0, stance detection BIBREF1, claim verification BIBREF2.
Question: What are the qualitative experiments performed on benchmark datasets?
Answer: Spearman correlation values of GM_KL model evaluated on the benchmark word similarity datasets.
Evaluation results of GM_KL model on the entailment datasets such as entailment pairs dataset created from WordNet, crowdsourced dataset of 79 semantic relations labelled as entailed or not and annotated distributionally similar nouns dataset.
Paper: Learning Multi-Sense Word Distributions using Approximate Kullback-Leibler Divergence
Probabilistic representation of words helps one model uncertainty in word representation, and polysemy. Given a corpus $V$, containing a list of words each represented as $w$, the probability density for a word $w$ can be represented as a mixture of Gaussians with $C$ components BIBREF10.
Question: How does this approach compare to other WSD approaches employing word embeddings?
Paper: Gender Representation in French Broadcast Corpora and Its Impact on ASR Performance
In recent years, gender has become a hot topic within the political, societal and research spheres. Numerous studies have been conducted in order to evaluate the presence of women in media, often revealing their under-representation, such as the Global Media Monitoring Project BIBREF0. In the French context, the CSA BIBREF1 produces a report on gender representation in media on a yearly basis. The 2017 report shows that women represent 40% of French media speakers, with a significant drop during high-audience hours (6:00-8:00pm) reaching a value of only 29%. Another large scale study confirmed this trend with an automatic analysis of gender in French audiovisuals streams, highlighting a huge variation across type of shows BIBREF2.
Question: What tasks did they use to evaluate performance for male and female speakers?
Paper: Gender Representation in French Broadcast Corpora and Its Impact on ASR Performance
The ever growing use of machine learning in science has been enabled by several progresses among which the exponential growth of data available. The quality of a system now depends mostly on the quality and quantity of the data it has been trained on. If it does not discard the importance of an appropriate architecture, it reaffirms the fact that rich and large corpora are a valuable resource. Corpora are research contributions which do not only allow to save and observe certain phenomena or validate a hypothesis or model, but are also a mandatory part of the technology development. This trend is notably observable within the NLP field, where industrial technologies, such as Apple, Amazon or Google vocal assistants now reach high performance level partly due to the amount of data possessed by these companies BIBREF9.
Question: What is the goal of investigating NLP gender bias specifically in the news broadcast domain and Anchor role?
Paper: Gender Representation in French Broadcast Corpora and Its Impact on ASR Performance
The gender issue has returned to the forefront of the media scene in recent years and with the emergence of AI technologies in our daily lives, gender bias has become a scientific topic that researchers are just beginning to address. Several studies revealed the existence of gender bias in AI technologies such as face recognition (GenderShades BIBREF17), NLP (word embeddings BIBREF5 and semantics BIBREF6) and machine translation (BIBREF18, BIBREF7). The impact of the training data used within these deep-learning algorithms is therefore questioned.
Question: Which corpora does this paper analyse?
Paper: Gender Representation in French Broadcast Corpora and Its Impact on ASR Performance
This section is organized as follows: we first present the data we are working on. In a second time we explain how we proceed to describe the gender representation in our corpus and introduce the notion of speaker's role. The third subsection introduces the ASR system and metrics used to evaluate gender bias in performance.
Question: How many categories do authors define for speaker role?
Paper: Gender Representation in French Broadcast Corpora and Its Impact on ASR Performance
Our data consists of two sets used to train and evaluate our automatic speech recognition system. Four major evaluation campaigns have enabled the creation of wide corpora of French broadcast speech: ESTER1 BIBREF13, ESTER2 BIBREF14, ETAPE BIBREF15 and REPERE BIBREF16. These four collections contain radio and/or TV broadcasts aired between 1998 and 2013 which are used by most academic researchers in ASR. Show duration varies between 10min and an hour. As years went by and speech processing research was progressing, the difficulty of the tasks augmented and the content of these evaluation corpora changed. ESTER1 and ESTER2 mainly contain prepared speech such as broadcast news, whereas ETAPE and REPERE consists also of debates and entertainment shows, spontaneous speech introducing more difficulty in its recognition.
Question: How big is imbalance in analyzed corpora?
Paper: Gender Representation in French Broadcast Corpora and Its Impact on ASR Performance
We first describe the gender representation in training data. Gender representation is measured in terms of number of speakers, number of utterances (or speech turns), and turn lengths (descriptive statistics are given in Section SECREF16). Each speech turn was mapped to its speaker in order to associate it with a gender.
Question: What are four major corpora of French broadcast?
Paper: Applying a Pre-trained Language Model to Spanish Twitter Humor Prediction
- !`Socorro, me ha picado una víbora!
Question: What did the best systems use for their model?
Paper: Applying a Pre-trained Language Model to Spanish Twitter Humor Prediction
The Humor Analysis based on Human Annotation (HAHA) 2019 BIBREF1 competition asked for analysis of two tasks in the Spanish language based on a corpus of publicly collected data described in Castro et al. BIBREF2 :
Question: What were their results on the classification and regression tasks
Answer: F1 score result of 0.8099
Paper: Winograd Schemas and Machine Translation
A Winograd schema (Levesque, Davis, and Morgenstern 2012) is a pair of sentences, or of short texts, called the elements of the schema, that satisfy the following constraints:
Question: Do the authors conduct experiments on the tasks mentioned?
Paper: Winograd Schemas and Machine Translation
In many cases, the identification of the referent of the prounoun in a Winograd schema is critical for finding the correct translation of that pronoun in a different language. Therefore, Winograd schemas can be used as a very difficult challenge for the depth of understanding achieved by a machine translation program.
Question: Did they collect their own datasets?
Paper: Winograd Schemas and Machine Translation
No one familiar with the state of the art in machine translation technology or the state of the art of artificial intelligence generally will be surprised to learn that currently machine translation program are unable to solve these Winograd schema challenge problems.
Question: What data do they look at?
Paper: Winograd Schemas and Machine Translation
The masculine and feminine plural pronouns are distinguished in the Romance languages (French, Spanish, Italian, Portuguese etc.) and in Semitic languages (Arabic, Hebrew, etc.) I have consulted with native speakers and experts in these languages about the degree to which the gender distinction is observed in practice. The experts say that in French, Spanish, Italian, and Portuguese, the distinction is very strictly observed; the use of a masculine pronoun for a feminine antecedent is jarringly wrong to a native or fluent speaker. “Les filles ont chanté une chanson et ils ont dansé” sounds as wrong to a French speaker as “The girl sang a song and he danced” sounds to an English speaker; in both cases, the hearer will interpret the pronoun as referrinig to some other persons or person, who is male. In Hebrew and Arabic, this is much less true; in speech, and even, increasingly, in writing, the masculine pronoun is often used for a feminine antecedent.
Question: What language do they explore?
Paper: On the effectiveness of feature set augmentation using clusters of word embeddings
Many research attempts have proposed novel features that improve the performance of learning algorithms in particular tasks. Such features are often motivated by domain knowledge or manual labor. Although useful and often state-of-the-art, adapting such solutions on NLP systems across tasks can be tricky and time-consuming BIBREF0 . Therefore, simple yet general and powerful methods that perform well across several datasets are valuable BIBREF1 .
Question: Do they report results only on English datasets?
Paper: On the effectiveness of feature set augmentation using clusters of word embeddings
Word embeddings associate words with dense, low-dimensional vectors. Recently, several models have been proposed in order to obtain these embeddings. Among others, the skipgram (skipgram) model with negative sampling BIBREF7 , the continuous bag-of-words (cbow) model BIBREF7 and Glove (glove) BIBREF8 have been shown to be effective. Training those models requires no annotated data and can be done using big amounts of text. Such a model can be seen as a function INLINEFORM0 that projects a word INLINEFORM1 in a INLINEFORM2 -dimensional space: INLINEFORM3 , where INLINEFORM4 is predefined. Here, we focus on applications using data from Twitter, which pose several difficulties due to being particularly short, using creative vocabulary, abbreviations and slang.
Question: Which hyperparameters were varied in the experiments on the four tasks?
Answer: number of clusters, seed value in clustering, selection of word vectors, window size and dimension of embedding
Paper: On the effectiveness of feature set augmentation using clusters of word embeddings
We evaluate the proposed approach for augmenting the feature space in four tasks: (i) NER segmentation, (ii) NER classification, (iii) fine-grained sentiment classification and (iv) fine-grained sentiment quantification. The next sections present the evaluation settings we used. For each of the tasks, we use the designated training sets to train the learning algorithms, and we report the scores of the evaluation measures used in the respective test parts.
Question: Which other hyperparameters, other than number of clusters are typically evaluated in this type of research?
Paper: On the effectiveness of feature set augmentation using clusters of word embeddings
NER concerns the classification of textual segments in a predefined set of categories, like persons, organization and locations. We use the data of the last competition in NER for Twitter which released as a part of the 2nd Workshop on Noisy User-generated Text BIBREF10 . More specifically, the organizers provided annotated tweets with 10 named-entity types (person, movie, sportsteam, product etc.) and the task comprised two sub-tasks: 1) the detection of entity bounds and 2) the classification of an entity into one of the 10 types. The evaluation measure for both sub-tasks is the F INLINEFORM0 measure.
Question: How were the cluster extracted? 
Answer: Word clusters are extracted using k-means on word embeddings
Paper: Compound Probabilistic Context-Free Grammars for Grammar Induction
 Grammar induction is the task of inducing hierarchical syntactic structure from data. Statistical approaches to grammar induction require specifying a probabilistic grammar (e.g. formalism, number and shape of rules), and fitting its parameters through optimization. Early work found that it was difficult to induce probabilistic context-free grammars (PCFG) from natural language data through direct methods, such as optimizing the log likelihood with the EM algorithm BIBREF0 , BIBREF1 . While the reasons for the failure are manifold and not completely understood, two major potential causes are the ill-behaved optimization landscape and the overly strict independence assumptions of PCFGs. More successful approaches to grammar induction have thus resorted to carefully-crafted auxiliary objectives BIBREF2 , priors or non-parametric models BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , and manually-engineered features BIBREF7 , BIBREF8 to encourage the desired structures to emerge.
Question: what were the evaluation metrics?
Answer: Unlabeled sentence-level F1, perplexity, grammatically judgment performance
Paper: Compound Probabilistic Context-Free Grammars for Grammar Induction
 We consider context-free grammars (CFG) consisting of a 5-tuple INLINEFORM0 where INLINEFORM1 is the distinguished start symbol, INLINEFORM2 is a finite set of nonterminals, INLINEFORM3 is a finite set of preterminals, INLINEFORM6 is a finite set of terminal symbols, and INLINEFORM7 is a finite set of rules of the form,
Question: what are the state of the art methods?
Paper: Compound Probabilistic Context-Free Grammars for Grammar Induction
 A compound probability distribution BIBREF19 is a distribution whose parameters are themselves random variables. These distributions generalize mixture models to the continuous case, for example in factor analysis which assumes the following generative process,
Question: what english datasets were used?
Answer: Answer with content missing: (Data section) Penn Treebank (PTB)
Paper: Compound Probabilistic Context-Free Grammars for Grammar Induction
Question: which chinese datasets were used?
Answer: Answer with content missing: (Data section) Chinese with version 5.1 of the Chinese Penn Treebank (CTB)
Paper: Characterizing Political Fake News in Twitter by its Meta-Data
10pt
Question: What were their distribution results?
Answer: Distributions of Followers, Friends and URLs are significantly different between the set of tweets containing fake news and those non containing them, but for Favourites, Mentions, Media, Retweets and Hashtags they are not significantly different
Paper: Characterizing Political Fake News in Twitter by its Meta-Data
While fake news, understood as deliberately misleading pieces of information, have existed since long ago (e.g. it is not unusual to receive news falsely claiming the death of a celebrity), the term reached the mainstream, particularly so in politics, during the 2016 presidential election in the United States BIBREF0 . Since then, governments and corporations alike (e.g. Google BIBREF1 and Facebook BIBREF2 ) have begun efforts to tackle fake news as they can affect political decisions BIBREF3 . Yet, the ability to define, identify and stop fake news from spreading is limited.
Question: How did they determine fake news tweets?
Answer: an expert annotator determined if the tweet fell under a specific category
Paper: Characterizing Political Fake News in Twitter by its Meta-Data
Our research is connected to different strands of academic knowledge related to the phenomenon of fake news. In relation to Computer Science, a recent survey by Conroy and colleagues BIBREF10 identifies two popular approaches to single-out fake news. On the one hand, the authors pointed to linguistic approaches consisting in using text, its linguistic characteristics and machine learning techniques to automatically flag fake news. On the other, these researchers underscored the use of network approaches, which make use of network characteristics and meta-data, to identify fake news.
Question: What is their definition of tweets going viral?
Answer: Viral tweets are the ones that are retweeted more than 1000 times
those that contain a high number of retweets