qinchuanhui/UDA-QA huggingface dataset:
Question: What accuracy do they approach with their proposed method?

Evidence:
raw_evidence: Accordingly to the Table 2 , 43% of testing questions from VQA dataset cannot find the proper basic questions from VQA training and validation datasets, and there are some failed examples about this case in Table 6 . We also discover that a lot of questions in VQA training and validation datasets are almost the same. This issue reduces the diversity of basic question dataset. Although we only have 57% of testing questions can benefit from the basic questions, our method still can improve the state-of-the-art accuracy BIBREF7 from 60.32% to 60.34%, referring to Table 4 and 5 . Then, we have 142093 testing questions, so that means the number of correctly answering questions of our method is more than state-of-the-art method 28 questions. In other words, if we have well enough basic question dataset, we can increase accuracy more, especially in the counting-type question, referring to Table 4 and 5 . Because the Co-Attention Mechanism is good at localizing, the counting-type question is improved more than others. So, based on our experiment, we can conclude that basic question can help accuracy obviously.

Answers:
our method still can improve the state-of-the-art accuracy BIBREF7 from 60.32% to 60.34%
Question: What they formulate the question generation as?

Evidence:
raw_evidence: Our idea is the BQ generation for MQ and, at the same time, we only want the minimum number of BQ to represent the MQ, so modeling our problem as $LASSO$ optimization problem is an appropriate way:

Answers:
LASSO optimization problem
Question: What two main modules their approach consists of?

Evidence:
raw_evidence: The Co-Attention mechanism inspires us to build part of our VQABQ model, illustrated by Figure 2 . In the VQABQ model, there are two main modules, the basic question generation module (Module 1) and co-attention visual question answering module (Module 2). We take the query question, called the main question (MQ), encoded by Skip-Thought Vectors BIBREF9 , as the input of Module 1. In the Module 1, we encode all of the questions, also by Skip-Thought Vectors, from the training and validation sets of VQA BIBREF0 dataset as a 4800 by 215623 dimension basic question (BQ) matrix, and then solve the LASSO optimization problem, with MQ, to find the 3 BQ of MQ. These BQ are the output of Module 1. Moreover, we take the MQ, BQ and the given image as the input of Module 2, the VQA module with co-attention mechanism, and then it can output the final answer of MQ. We claim that the BQ can help Module 2 get the correct answer to increase the VQA accuracy. In this work, our main contributions are summarized below:

Answers:
the basic question generation module (Module 1) and co-attention visual question answering module (Module 2)
Question: Are there syntax-agnostic SRL models before?

Evidence:
raw_evidence: Typically, SRL task can be put into two categories: constituent-based (i.e., phrase or span) SRL and dependency-based SRL. This paper will focus on the latter one popularized by CoNLL-2008 and 2009 shared tasks BIBREF5 , BIBREF6 . Most conventional SRL systems relied on sophisticated handcraft features or some declarative constraints BIBREF7 , BIBREF8 , which suffers from poor efficiency and generalization ability. A recently tendency for SRL is adopting neural networks methods attributed to their significant success in a wide range of applications BIBREF9 , BIBREF10 . However, most of those works still heavily resort to syntactic features. Since the syntactic parsing task is equally hard as SRL and comes with its own errors, it is better to get rid of such prerequisite as in other NLP tasks. Accordingly, marcheggiani2017 presented a neural model putting syntax aside for dependency-based SRL and obtain favorable results, which overturns the inherent belief that syntax is indispensable in SRL task BIBREF11 .

Answers:
Yes
Question: What is the biaffine scorer?

Evidence:
raw_evidence: Typically, to predict and label arguments for a given predicate, a role classifier is employed on top of the BiLSTM encoder. Some work like BIBREF17 shows that incorporating the predicate's hidden state in their role classifier enhances the model performance, while we argue that a more natural way to incorporate the syntactic information carried by the predicate is to employ the attentional mechanism. Our model adopts the recently introduced biaffine attention BIBREF14 to enhance our role scorer. Biaffine attention is a natural extension of bilinear attention BIBREF18 which is widely used in neural machine translation (NMT).

Answers:
biaffine attention BIBREF14
Question: What languages are were included in the dataset of hateful content?

Evidence:
raw_evidence: As previously mentioned, there is no German hate speech corpus available for our needs, especially not for the very recent topic of the refugee crisis in Europe. We therefore had to compile our own corpus. We used Twitter as a source as it offers recent comments on current events. In our study we only considered the textual content of tweets that contain certain keywords, ignoring those that contain pictures or links. This section provides a detailed description of the approach we used to select the tweets and subsequently annotate them.

Answers:
German
Question: How was reliability measured?

Evidence:
raw_evidence: Even among researchers familiar with the definitions outlined above, there was still a low level of agreement (Krippendorff's INLINEFORM0 ). This supports our claim that a clearer definition is necessary in order to be able to train a reliable classifier. The low reliability could of course be explained by varying personal attitudes or backgrounds, but clearly needs more consideration.

Answers:
level of agreement (Krippendorff's INLINEFORM0 )
Question: How did the authors demonstrate that showing a hate speech definition caused annotators to partially align their own opinion with the definition?

Evidence:
raw_evidence: Participants who were shown the definition were more likely to suggest to ban the tweet. In fact, participants in group one very rarely gave different answers to questions one and two (18 of 500 instances or 3.6%). This suggests that participants in that group aligned their own opinion with the definition.

Answers:
participants in group one very rarely gave different answers to questions one and two (18 of 500 instances or 3.6%)
Question: What definition was one of the groups was shown?

Evidence:
raw_evidence: In order to assess the reliability of the hate speech definitions on social media more comprehensively, we developed two online surveys in a between-subjects design. They were completed by 56 participants in total (see Table TABREF7 ). The main goal was to examine the extent to which non-experts agree upon their understanding of hate speech given a diversity of social media content. We used the Twitter definition of hateful conduct in the first survey. This definition was presented at the beginning, and again above every tweet. The second survey did not contain any definition. Participants were randomly assigned one of the two surveys.

Answers:
Twitter definition of hateful conduct
Question: Was the degree of offensiveness taken as how generally offensive the text was, or how personally offensive it was to the annotator?

Evidence:
raw_evidence: The surveys consisted of 20 tweets presented in a random order. For each tweet, each participant was asked three questions. Depending on the survey, participants were asked (1) to answer (yes/no) if they considered the tweet hate speech, either based on the definition or based on their personal opinion. Afterwards they were asked (2) to answer (yes/no) if the tweet should be banned from Twitter. Participants were finally asked (3) to answer how offensive they thought the tweet was on a 6-point Likert scale from 1 (Not offensive at all) to 6 (Very offensive). If they answered 4 or higher, the participants had the option to state which particular words they found offensive.

Answers:
Personal thought of the annotator.
Question: How were potentially hateful messages identified?

Evidence:
raw_evidence: To find a large amount of hate speech on the refugee crisis, we used 10 hashtags that can be used in an insulting or offensive way. Using these hashtags we gathered 13 766 tweets in total, roughly dating from February to March 2016. However, these tweets contained a lot of non-textual content which we filtered out automatically by removing tweets consisting solely of links or images. We also only considered original tweets, as retweets or replies to other tweets might only be clearly understandable when reading both tweets together. In addition, we removed duplicates and near-duplicates by discarding tweets that had a normalised Levenshtein edit distance smaller than .85 to an aforementioned tweet. A first inspection of the remaining tweets indicated that not all search terms were equally suited for our needs. The search term #Pack (vermin or lowlife) found a potentially large amount of hate speech not directly linked to the refugee crisis. It was therefore discarded. As a last step, the remaining tweets were manually read to eliminate those which were difficult to understand or incomprehensible. After these filtering steps, our corpus consists of 541 tweets, none of which are duplicates, contain links or pictures, or are retweets or replies.

Answers:
10 hashtags that can be used in an insulting or offensive way
Question: Which embeddings do they detect biases in?

Evidence:
raw_evidence: For both word2vec BIBREF0 and gensim BIBREF7 we adapted the code so that the input terms of the analogy query are allowed to be returned. Throughout this article, we use two different embedding spaces. The first is the widely used representation built on GoogleNews BIBREF8 . The second is taken from BIBREF2 , and was trained on a Reddit dataset BIBREF9 .

Answers:
Word embeddings trained on GoogleNews and Word embeddings trained on Reddit dataset
Question: What does their system consist of?

Evidence:
raw_evidence: In this paper, in spite of previous statements, we present a system that uses rule-based and dictionary-based methods combined (in a way we prefer to call resource-based). Our final goals in the paper are two-fold: on the one hand, to describe our system, developed for the PharmaCoNER shared task, dealing with the annotation of some of the nes in health records (namely, pharmacological, chemical and biomedical entities) using a revisited version of rule- and dictionary-based approaches; and, on the other hand, to give pause for thought about the quality of datasets (and, thus, the fairness) with which systems of this type are evaluated, and to highlight the key role of resource-based systems in the validation and consolidation of both the annotation guidelines and the human annotation practices.

Answers:
rule-based and dictionary-based methods
Question: What are the two PharmaCoNER subtasks?

Evidence:
raw_evidence: Although the competition proposes two different scenarios, in fact, both are guided by the snomed ct ontology —for subtask 1, entities must be identified with offsets and mapped to a predefined set of four classes (PROTEINAS, NORMALIZABLES, NO_NORMALIZABLES and UNCLEAR); for subtask 2, a list of all snomed ct ids (sctid) for entities occurring in the text must be given, which has been called concept indexing by the shared task organizers. Moreover, PharmaCoNER organizers decided to promote snomed ct substance ids over product, procedure or other possible interpretations also available in this medical ontology for a given entity. This selection must be done even if the context clearly refers to a different concept, according to the annotation guidelines (henceforth, AnnotGuide) and the praxis. Finally, PROTEINAS is ranked as the first choice for substances in this category.

Answers:
Entity identification with offset mapping and concept indexing
Question: What neural language models are explored?

Evidence:
raw_evidence: Since our focus in this paper is an additional loss exploiting negative examples (Section method), we fix the baseline LM throughout the experiments. Our baseline is a three-layer LSTM-LM with 1,150 hidden units at internal layers trained with the standard cross-entropy loss. Word embeddings are 400-dimensional, and input and output embeddings are tied BIBREF11. Deviating from some prior work BIBREF0, BIBREF1, we train LMs at sentence level as in sequence-to-sequence models BIBREF12. This setting has been employed in some previous work BIBREF3, BIBREF6.

Answers:
LSTM-LM
Question: How do they perform data augmentation?

Evidence:
raw_evidence: We first inspect the frequencies of object and subject RCs in the training data, by parsing them with the state-of-the-art Berkeley neural parser BIBREF19. In total, while subject RCs occur 373,186 times, object RCs only occur 106,558 times. We create three additional training datasets by adding sentences involving object RCs to the original Wikipedia corpus (Section lm). To this end, we randomly pick up 30 million sentences from Wikipedia (not overlapped to any sentences in the original corpus), parse by the same parser, and filter sentences containing an object RC, amounting to 680,000 sentences. Among the test cases about object RCs, we compare accuracies on subject-verb agreement, to make a comparison with subject RCs. We also evaluate on “animate only” subset, which has a correspondence to the test cases for subject RC with only differences in word order and inflection (like (UNKREF45) and (UNKREF46); see footnote FOOTREF47). Of particular interest to us is the accuracy on these animate cases. Since the vocabularies are exactly the same, we hypothesize that the accuracy will reach the same level as that on subject RCs with our augmentation.

Answers:
They randomly sample sentences from Wikipedia that contains an object RC and add them to training data
Question: Do the authors mention any possible confounds in their study?

Evidence:
raw_evidence: After averaging over all concepts, we lose information on the lexical variation that each concept presents but on the other hand one can now investigate which regions show similar geolectal variation, yielding well defined linguistic varieties. Those cells that have similar colors in either figure FIGREF16 or figure FIGREF17 are expected to be ascribed to the same dialect zone. Thus, we can distinguish two main regions or clusters in the maps. The purple background covers most of the map and represents rural regions with small, scattered population. Our analysis shows that this group of cells possesses more specific words in their lexicon. In contrast, the green and yellow cells form a second cluster that is largely concentrated on the center and along the coastline, which correspond to big cities and industrialized areas. In these cells, the use of standard Spanish language is widespread due probably to school education, media, travelers, etc. The character of its vocabulary is more uniform as compared with the purple group. While the purple cluster prefer particular utterances, the lexicon of the urban group includes most of the keywords. Importantly, we emphasize that both distance measures (cosine similarity and Jensen-Shanon) give rise to the same result, with little discrepancies on the numerical values that are not significant. The presence of two Twitter superdialects (urban and rural) has been recently suggested BIBREF10 based on a machine learning approach. Here, we arrive at the same conclusion but with a totally distinct model and corpus. The advantage of our proposal is that it may serve as a useful tool for dialectometric purposes.

Answers:
Yes
Question: What are the characteristics of the city dialect?

Evidence:
raw_evidence: After averaging over all concepts, we lose information on the lexical variation that each concept presents but on the other hand one can now investigate which regions show similar geolectal variation, yielding well defined linguistic varieties. Those cells that have similar colors in either figure FIGREF16 or figure FIGREF17 are expected to be ascribed to the same dialect zone. Thus, we can distinguish two main regions or clusters in the maps. The purple background covers most of the map and represents rural regions with small, scattered population. Our analysis shows that this group of cells possesses more specific words in their lexicon. In contrast, the green and yellow cells form a second cluster that is largely concentrated on the center and along the coastline, which correspond to big cities and industrialized areas. In these cells, the use of standard Spanish language is widespread due probably to school education, media, travelers, etc. The character of its vocabulary is more uniform as compared with the purple group. While the purple cluster prefer particular utterances, the lexicon of the urban group includes most of the keywords. Importantly, we emphasize that both distance measures (cosine similarity and Jensen-Shanon) give rise to the same result, with little discrepancies on the numerical values that are not significant. The presence of two Twitter superdialects (urban and rural) has been recently suggested BIBREF10 based on a machine learning approach. Here, we arrive at the same conclusion but with a totally distinct model and corpus. The advantage of our proposal is that it may serve as a useful tool for dialectometric purposes.

Answers:
Lexicon of the cities tend to use most forms of a particular concept
Question: What are the characteristics of the rural dialect?

Evidence:
raw_evidence: After averaging over all concepts, we lose information on the lexical variation that each concept presents but on the other hand one can now investigate which regions show similar geolectal variation, yielding well defined linguistic varieties. Those cells that have similar colors in either figure FIGREF16 or figure FIGREF17 are expected to be ascribed to the same dialect zone. Thus, we can distinguish two main regions or clusters in the maps. The purple background covers most of the map and represents rural regions with small, scattered population. Our analysis shows that this group of cells possesses more specific words in their lexicon. In contrast, the green and yellow cells form a second cluster that is largely concentrated on the center and along the coastline, which correspond to big cities and industrialized areas. In these cells, the use of standard Spanish language is widespread due probably to school education, media, travelers, etc. The character of its vocabulary is more uniform as compared with the purple group. While the purple cluster prefer particular utterances, the lexicon of the urban group includes most of the keywords. Importantly, we emphasize that both distance measures (cosine similarity and Jensen-Shanon) give rise to the same result, with little discrepancies on the numerical values that are not significant. The presence of two Twitter superdialects (urban and rural) has been recently suggested BIBREF10 based on a machine learning approach. Here, we arrive at the same conclusion but with a totally distinct model and corpus. The advantage of our proposal is that it may serve as a useful tool for dialectometric purposes.

Answers:
It uses particular forms of a concept rather than all of them uniformly
Question: What are their baseline methods?

Evidence:
raw_evidence: Figure 1 shows a simple linear model with rank constraint. The first weight matrix $A$ is a look-up table over the words. The word representations are then averaged into a text representation, which is in turn fed to a linear classifier. The text representation is an hidden variable which can be potentially be reused. This architecture is similar to the cbow model of mikolov2013efficient, where the middle word is replaced by a label. We use the softmax function $f$ to compute the probability distribution over the predefined classes. For a set of $N$ documents, this leads to minimizing the negative log-likelihood over the classes: $ -\frac{1}{N} \sum _{n=1}^N y_n \log ( f (BAx_n)), $

Answers:
simple linear model with rank constraint, Hierarchical softmax, N-gram features
Question: Which of the model yields the best performance?

Evidence:
raw_evidence: An LM's overall performance on *X can be measured simply by taking the proportion of correct predictions across the 67,000 minimal pairs from all paradigms. GPT-2 achieves the highest score and the $n$-gram the lowest. Transformer-XL and the LSTM LM perform in the middle, and at roughly the same level as each other. All models perform well below estimated human agreement (as described in Section SECREF11). The $n$-gram model's poor overall performance confirms *X is not solvable from co-occurrence information alone. Rather, success at *X is driven by the more abstract features learned by neural networks. There are no categories in which the $n$-gram approaches human performance.

Answers:
GPT-2
Question: What is the performance of the models on the tasks?

Evidence:
raw_evidence: An LM's overall performance on *X can be measured simply by taking the proportion of correct predictions across the 67,000 minimal pairs from all paradigms. GPT-2 achieves the highest score and the $n$-gram the lowest. Transformer-XL and the LSTM LM perform in the middle, and at roughly the same level as each other. All models perform well below estimated human agreement (as described in Section SECREF11). The $n$-gram model's poor overall performance confirms *X is not solvable from co-occurrence information alone. Rather, success at *X is driven by the more abstract features learned by neural networks. There are no categories in which the $n$-gram approaches human performance.
We report the 12-category accuracy results for all models and human evaluation in Table TABREF14.

Answers:
Overall accuracy per model is: 5-gram (60.5), LSTM (68.9), TXL (68.7), GPT-2 (80.1)
Question: How is the data automatically generated?

Evidence:
raw_evidence: To create minimal pairs exemplifying a wide array of linguistic contrasts, it is necessary to artificially generate all datasets. This ensures both that we have sufficient unacceptable examples, and that the data is fully controlled, allowing for repeated isolation of a single linguistic phenomenon in each paradigm BIBREF30. The data generation scripts use a basic template to create each paradigm, pulling from a vocabulary of over 3000 words annotated for morphological, syntactic, and semantic features needed to create grammatical and semantically felicitous sentences. Examples SECREF6 and SECREF6 show one such template for the `acceptable' and `unacceptable' sentences within a pair: the sole difference between them is the underlined word, which differs only in whether the anaphor agrees in number with its antecedent. Our generation codebase and scripts are freely available.
This generation procedure is not without limitations, and despite the very detailed vocabulary we use, implausible sentences are occasionally generated (e.g., `Sam ran around some glaciers'). In these cases, though, both the acceptable and unacceptable sentences will be equally implausible given world knowledge, so any difference in the probability assigned to them is still due to the intended grammatical contrast.

Answers:
 The data generation scripts use a basic template to create each paradigm, pulling from a vocabulary of over 3000 words annotated for morphological, syntactic, and semantic features needed to create grammatical and semantically felicitous sentences.