z-uo/qasper-squad huggingface dataset:
Paper: Feature-Dependent Confusion Matrices for Low-Resource NER Labeling with Noisy Labels
We assume a low-resource setting with a small set of gold standard annotated data $C$ consisting of instances with features $x$ and corresponding, clean labels $y$. Additionally, a large set of noisy instances $(x,\hat{y}) \in N$ is available. This can be obtained e.g. from weak or distant supervision. In a multi-class classification setting, we can learn the probability of a label $y$ having a specific class given the feature $x$ as
Question: How they evaluate their approach?
Answer: They  evaluate newly proposed models in several low-resource settings across different languages with real, distantly supervised data with non-synthetic noise
Paper: Multilingual Twitter Corpus and Baselines for Evaluating Demographic Bias in Hate Speech Recognition
While document classification models should be objective and independent from human biases in documents, research have shown that the models can learn human biases and therefore be discriminatory towards particular demographic groups BIBREF0, BIBREF1, BIBREF2. The goal of fairness-aware document classifiers is to train and build non-discriminatory models towards people no matter what their demographic attributes are, such as gender and ethnicity. Existing research BIBREF0, BIBREF3, BIBREF4, BIBREF5, BIBREF1 in evaluating fairness of document classifiers focus on the group fairness BIBREF6, which refers to every demographic group has equal probability of being assigned to the positive predicted document category.
Question: How large is the corpus?
Answer: It contains 106,350 documents
Paper: Multilingual Twitter Corpus and Baselines for Evaluating Demographic Bias in Hate Speech Recognition
We assemble the annotated datasets for hate speech classification. To narrow down the data sources, we limit our dataset sources to the unique online social media site, Twitter. We have requested 16 published Twitter hate speech datasets, and finally obtained 7 of them in five languages. By using the Twitter streaming API, we collected the tweets annotated by hate speech labels and their corresponding user profiles in English BIBREF14, BIBREF19, BIBREF20, Italian BIBREF15, Polish BIBREF16, Portuguese BIBREF18, and Spanish BIBREF17. We binarize all tweets' labels (indicating whether a tweet has indications of hate speech), allowing to merge the different label sets and reduce the data sparsity.
Question: Which document classifiers do they experiment with?
Paper: Multilingual Twitter Corpus and Baselines for Evaluating Demographic Bias in Hate Speech Recognition
We consider four user factors of age, race, gender and geographic location. For location, we inference two granularities, country and US region, but only experiment with the country attribute. While the demographic attributes can be inferred through tweets BIBREF22, BIBREF8, we intentionally exclude the contents from the tweets if they infer these user attributes, in order to make the evaluation of fairness more reliable and independent. If users were grouped based on attributes inferred from their text, then any differences in text classification across those groups could be related to the same text. Instead, we infer attributes from public user profile information (i.e., description, name and photo).
Question: How large is the dataset?
Answer: over 104k documents
Paper: Variational Cross-domain Natural Language Generation for Spoken Dialogue Systems
Conventional spoken dialogue systems (SDS) require a substantial amount of hand-crafted rules to achieve good interaction with users. The large amount of required engineering limits the scalability of these systems to settings with new or multiple domains. Recently, statistical approaches have been studied that allow natural, efficient and more diverse interaction with users without depending on pre-defined rules BIBREF0 , BIBREF1 , BIBREF2 .
Question: What evaluation metrics are used to measure diversity?
Paper: Variational Cross-domain Natural Language Generation for Spoken Dialogue Systems
The VAE is a generative latent variable model. It uses a neural network (NN) to generate $\hat{x}$ from a latent variable $z$ , which is sampled from the prior $p_{\theta }(z)$ . The VAE is trained such that $\hat{x}$ is a sample of the distribution $p_{D}(x)$ from which the training data was collected. Generative latent variable models have the form $p_{\theta }(x)=\int _{z}p_{\theta }(x|z)p_{\theta }(z) dz$ . In a VAE an NN, called the decoder, models $p_{\theta }(x|z)$ and would ideally be trained to maximize the expectation of the above integral $E\left[p_{\theta }(x)\right]$ . Since this is intractable, the VAE uses another NN, called the encoder, to model $q_{\phi }(z|x)$ which should approximate the posterior $p_{\theta }(z|x)$ . The NNs in the VAE are trained to maximise the variational lower bound (VLB) to $z$0 , which is given by: 
Question: How is some information lost in the RNN-based generation models?
Paper: Attributed Multi-Relational Attention Network for Fact-checking URL Recommendation
While social media sites provide users with the revolutionized communication medium by bringing the communication efficiency to a new level, they can be easily misused for widely spreading misinformation and fake news. Fake news and misinformation have been a long-standing issue for various purposes such as political propaganda BIBREF0 and financial propaganda BIBREF1. To fight against fake news, traditional publishers employed human editors to manually and carefully check the content of news articles to maintain their reputation. However, social media provided a new way to spread news, which lead to broader information sources and expanded audience (i.e., anyone can be a media and create news). In particular, users share news articles with their own opinion or read articles shared by their friends from whatever the source of news is with mostly blind trust BIBREF2 or with their own ideologies BIBREF3, BIBREF4. Although social media posts usually have a very short life cycle, the unprecedented amount of fake news may lead to a catastrophic impact on both individuals and society. Besides from misleading users with false information BIBREF4, widely propagated fake news could even cause trust crisis of entire news ecosystem BIBREF5, even further affecting both the cyberspace and physical space.
Question: What is the model accuracy?
Paper: Attributed Multi-Relational Attention Network for Fact-checking URL Recommendation
In this section, we briefly review related works and position our work within the following areas: (1) fake news and misinformation; (2) advancements in recommender systems; and (3) graph convolutional networks.
Question: How do the authors define fake news?
Paper: Attributed Multi-Relational Attention Network for Fact-checking URL Recommendation
Fake news has attracted considerable attention since it is related to our daily life and has become a serious problem related to multiple areas such as politics BIBREF0 and finance BIBREF1. Social media sites have become one of popular mediums to propagate fake news and misinformation. The dominant line of work in this topic is fake news detection BIBREF15 which was mostly formulated as a binary classification problem. Researchers began to incorporate social context and other features for identifying fake news at an early stage and preventing it from diffusion on the social network BIBREF5, BIBREF7. Some other researchers focus on investigating the propagation patterns of fake news in social network BIBREF16, BIBREF17. BIBREF18 also studied fake news intervention. Unlike most previous works, we follow the direction of BIBREF12 and propose to build a personalized recommender system for promoting the fact-checking article circulation to debunk fake news.
Question: What dataset is used?
Paper: Learn to Code-Switch: Data Augmentation using Copy Mechanism on Language Modeling
Language mixing has been a common phenomenon in multilingual communities. It is motivated in response to social factors as a way of communication in a multicultural society. From a sociolinguistic perspective, individuals do code-switching in order to construct an optimal interaction by accomplishing the conceptual, relational-interpersonal, and discourse-presentational meaning of conversation BIBREF0 . In its practice, the variation of code-switching will vary due to the traditions, beliefs, and normative values in the respective communities. A number of studies BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 found that code-switching is not produced indiscriminately, but follows syntactic constraints. Many linguists formulated various constraints to define a general rule for code-switching BIBREF1 , BIBREF3 , BIBREF4 . However, the constraints are not enough to make a good generalization of real code-switching constraints, and they have not been tested in large-scale corpora for many language pairs.
Question: Did they use other evaluation metrics?
Paper: Learn to Code-Switch: Data Augmentation using Copy Mechanism on Language Modeling
The synthetic code-switching generation approach was introduced by adapting equivalence constraint on monolingual sentence pairs during the decoding step on an automatic speech recognition (ASR) model BIBREF5 . BIBREF10 explored Functional Head Constraint, which was found to be more restrictive than the Equivalence Constraint, but complex to be implemented, by using a lattice parser with a weighted finite-state transducer. BIBREF11 extended the RNN by adding POS information to the input layer and factorized output layer with a language identifier. Then, Factorized RNN networks were combined with an n-gram backoff model using linear interpolation BIBREF12 . BIBREF13 added syntactic and semantic features to the Factorized RNN networks. BIBREF14 adapted an effective curriculum learning by training a network with monolingual corpora of both languages, and subsequently train on code-switched data. A further investigation of Equivalence Constraint and Curriculum Learning showed an improvement in language modeling BIBREF6 . A multi-task learning approach was introduced to train the syntax representation of languages by constraining the language generator BIBREF9 .
Question: What was their perplexity score?
Answer: Perplexity score 142.84 on dev and 138.91 on test
Paper: Learn to Code-Switch: Data Augmentation using Copy Mechanism on Language Modeling
We use a sequence to sequence (Seq2Seq) model in combination with pointer and copy networks BIBREF7 to align and choose words from the monolingual sentences and generate a code-switching sentence. The models' input is the concatenation of the two monolingual sentences, denoted as INLINEFORM0 , and the output is a code-switched sentence, denoted as INLINEFORM1 . The main assumption is that almost all, the token present in the code-switching sentence are also present in the source monolingual sentences. Our model leverages this property by copying input tokens, instead of generating vocabulary words. This approach has two major advantages: (1) the learning complexity decreases since it relies on copying instead of generating; (2) improvement in generalization, the copy mechanism could produce words from the input that are not present in the vocabulary.
Question: What languages are explored in this paper?
Paper: Learn to Code-Switch: Data Augmentation using Copy Mechanism on Language Modeling
Instead of generating words from a large vocabulary space using a Seq2Seq model with attention BIBREF17 , pointer-generator network BIBREF7 is proposed to copy words from the input to the output using an attention mechanism and generate the output sequence using decoders. The network is depicted in Figure FIGREF1 . For each decoder step, a generation probability INLINEFORM0 INLINEFORM1 [0,1] is calculated, which weights the probability of generating words from the vocabulary, and copying words from the source text. INLINEFORM2 is a soft gating probability to decide whether generating the next token from the decoder or copying the word from the input instead. The attention distribution INLINEFORM3 is a standard attention with general scoring BIBREF17 . It considers all encoder hidden states to derive the context vector. The vocabulary distribution INLINEFORM4 is calculated by concatenating the decoder state INLINEFORM5 and the context vector INLINEFORM6 . DISPLAYFORM0 
Question: What parallel corpus did they use?
Answer: Parallel monolingual corpus in English and Mandarin
Paper: Normalized and Geometry-Aware Self-Attention Network for Image Captioning
Automatically generating captions for images, namely image captioning BIBREF0, BIBREF1, has emerged as a prominent research problem at the intersection of computer vision (CV) and natural language processing (NLP). This task is challenging as it requires to first recognize the objects in the image, the relationships between them, and finally properly organize and describe them in natural language.
Question: What datasets are used for experiments on three other tasks?
Paper: VQABQ: Visual Question Answering by Basic Questions
Visual Question Answering (VQA) is a challenging and young research field, which can help machines achieve one of the ultimate goals in computer vision, holistic scene understanding BIBREF1 . VQA is a computer vision task: a system is given an arbitrary text-based question about an image, and then it should output the text-based answer of the given question about the image. The given question may contain many sub-problems in computer vision, e.g.,
Question: In which setting they achieve the state of the art?
Answer: in open-ended task esp. for counting-type questions
Paper: VQABQ: Visual Question Answering by Basic Questions
The following two important reasons motivate us to do Visual Question Answering by Basic Questions (VQABQ). First, recently most of VQA works only emphasize more on the image part, the visual features, but put less effort on the question part, the text features. However, image and question features both are important for VQA. If we only focus on one of them, we probably cannot get the good performance of VQA in the near future. Therefore, we should put our effort more on both of them at the same time. In BIBREF7 , they proposed a novel co-attention mechanism that jointly performs image-guided question attention and question-guided image attention for VQA. BIBREF7 also proposed a hierarchical architecture to represent the question, and construct image-question co-attention maps at the word level, phrase level and question level. Then, these co-attended features are combined with word level, phrase level and question level recursively for predicting the final answer of the query question based on the input image. BIBREF8 is also a recent work focusing on the text-based question part, text feature. In BIBREF8 , they presented a reasoning network to update the question representation iteratively after the question interacts with image content each time. Both of BIBREF7 , BIBREF8 yield better performance than previous works by doing more effort on the question part.
Question: What accuracy do they approach with their proposed method?
Paper: VQABQ: Visual Question Answering by Basic Questions
Recently, there are many papers BIBREF0 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 have proposed methods to solve the VQA issue. Our method involves in different areas in machine learning, natural language processing (NLP) and computer vision. The following, we discuss recent works related to our approach for solving VQA problem.
Question: What they formulate the question generation as?
Answer: LASSO optimization problem
Paper: VQABQ: Visual Question Answering by Basic Questions
We propose a new dataset, called Basic Question Dataset (BQD), generated by our basic question generation algorithm. BQD is the first basic question dataset. Regarding the BQD, the dataset format is $\lbrace Image,~MQ,~3~(BQ + corresponding~similarity~score)\rbrace $ . All of our images are from the testing images of MS COCO dataset BIBREF30 , the MQ, main questions, are from the testing questions of VQA, open-ended, dataset BIBREF0 , the BQ, basic questions, are from the training and validation questions of VQA, open-ended, dataset BIBREF0 , and the corresponding similarity score of BQ is generated by our basic question generation method, referring to Section 5. Moreover, we also take the multiple-choice questions in VQA dataset BIBREF0 to do the same thing as above. Note that we remove the repeated questions in the VQA dataset, so the total number of questions is slightly less than VQA dataset BIBREF0 . In BQD, we have 81434 images, 244302 MQ and 732906 (BQ + corresponding similarity score). At the same time, we also exploit BQD to do VQA and achieve the competitive accuracy compared to state-of-the-art.
Question: What two main modules their approach consists of?
Paper: A Full End-to-End Semantic Role Labeler, Syntax-agnostic Over Syntax-aware?
This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http://creativecommons.org/licenses/by/4.0/ Semantic role labeling (SRL) is a shallow semantic parsing, which is dedicated to identifying the semantic arguments of a predicate and labeling them with their semantic roles. SRL is considered as one of the core tasks in the natural language processing (NLP), which has been successfully applied to various downstream tasks, such as information extraction BIBREF0 , question answering BIBREF1 , BIBREF2 , machine translation BIBREF3 , BIBREF4 .
Question: Are there syntax-agnostic SRL models before?
Paper: A Full End-to-End Semantic Role Labeler, Syntax-agnostic Over Syntax-aware?
SRL includes two subtasks: predicate identification/disambiguation and argument identification/labeling. Since the CoNLL-2009 dataset provides the gold predicates, most previous neural SRL systems use a default model to perform predicate disambiguation and focus on argument identification/labeling. Despite nearly all SRL work adopted the pipeline model with two or more components, Zhao2008Parsing and zhao-jair-2013 presented an end-to-end solution for the entire SRL task with a word pair classifier. Following the same formulization, we propose the first neural SRL system that uniformly handles the tasks of predicate disambiguation and argument identification/labeling.
Question: What is the biaffine scorer?
Paper: Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisis
Social media are sometimes used to disseminate hateful messages. In Europe, the current surge in hate speech has been linked to the ongoing refugee crisis. Lawmakers and social media sites are increasingly aware of the problem and are developing approaches to deal with it, for example promising to remove illegal messages within 24 hours after they are reported BIBREF0 .
Question: What languages are were included in the dataset of hateful content?
Paper: Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisis
For the purpose of building a classifier, warner2012 define hate speech as “abusive speech targeting specific group characteristics, such as ethnic origin, religion, gender, or sexual orientation”. More recent approaches rely on lists of guidelines such as a tweet being hate speech if it “uses a sexist or racial slur” BIBREF2 . These approaches are similar in that they leave plenty of room for personal interpretation, since there may be differences in what is considered offensive. For instance, while the utterance “the refugees will live off our money” is clearly generalising and maybe unfair, it is unclear if this is already hate speech. More precise definitions from law are specific to certain jurisdictions and therefore do not capture all forms of offensive, hateful speech, see e.g. matsuda1993. In practice, social media services are using their own definitions which have been subject to adjustments over the years BIBREF3 . As of June 2016, Twitter bans hateful conduct.
Question: How was reliability measured?
Paper: Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisis
As previously mentioned, there is no German hate speech corpus available for our needs, especially not for the very recent topic of the refugee crisis in Europe. We therefore had to compile our own corpus. We used Twitter as a source as it offers recent comments on current events. In our study we only considered the textual content of tweets that contain certain keywords, ignoring those that contain pictures or links. This section provides a detailed description of the approach we used to select the tweets and subsequently annotate them.
Question: How did the authors demonstrate that showing a hate speech definition caused annotators to partially align their own opinion with the definition?
Paper: Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisis
In order to assess the reliability of the hate speech definitions on social media more comprehensively, we developed two online surveys in a between-subjects design. They were completed by 56 participants in total (see Table TABREF7 ). The main goal was to examine the extent to which non-experts agree upon their understanding of hate speech given a diversity of social media content. We used the Twitter definition of hateful conduct in the first survey. This definition was presented at the beginning, and again above every tweet. The second survey did not contain any definition. Participants were randomly assigned one of the two surveys.
Question: What definition was one of the groups was shown?
Paper: Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisis
Since the surveys were completed by 56 participants, they resulted in 1120 annotations. Table TABREF7 shows some summary statistics.
Question: Was the degree of offensiveness taken as how generally offensive the text was, or how personally offensive it was to the annotator?
Answer: Personal thought of the annotator.
Paper: Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisis
This paper describes the creation of our hate speech corpus and offers first insights into the low agreement among users when it comes to identifying hateful messages. Our results imply that hate speech is a vague concept that requires significantly better definitions and guidelines in order to be annotated reliably. Based on the present findings, we are planning to develop a new coding scheme which includes clear-cut criteria that let people distinguish hate speech from other content.
Question: How were potentially hateful messages identified?
Paper: Fair is Better than Sensational:Man is to Doctor as Woman is to Doctor
Word embeddings are distributed representations of texts which capture similarities between words. Beside improving a wide variety of NLP tasks, the power of word embeddings is often also tested intrinsically. Together with the idea of training word embeddings, BIBREF0 introduced the idea of testing the soundness of embedding spaces via the analogy task. Proportional analogies are equations of the form INLINEFORM0 , or simply A is to B as C is to D. Given the terms INLINEFORM1 , the model must return the word that correctly stands for INLINEFORM2 in the given analogy. A most classic example is man is to king as woman is to X, where the model is expected to return queen, by subtracting “manness" from the concept of king to obtain some general royalty, and then re-adding some “womanness" to obtain the concept of queen ( INLINEFORM3 ).
Question: Which embeddings do they detect biases in?
Answer: Word embeddings trained on GoogleNews and Word embeddings trained on Reddit dataset