qinchuanhui/UDA-QA huggingface dataset:
Question: How do they incorporate human advice?

Evidence:
raw_evidence: While most relational learning methods restrict the human to merely annotating the data, we go beyond and request the human for advice. The intuition is that we as humans read certain patterns and use them to deduce the nature of the relation between two entities present in the text. The goal of our work is to capture such mental patterns of the humans as advice to the learning algorithm. We modified the work of Odom et al. odomAIME15,odomAAAI15 to learn RDNs in the presence of advice. The key idea is to explicitly represent advice in calculating gradients. This allows the system to trade-off between data and advice throughout the learning phase, rather than only consider advice in initial iterations. Advice, in particular, become influential in the presence of noisy or less amout of data. A few sample advice rules in English (these are converted to first-order logic format and given as input to our algorithm) are presented in Table TABREF11 . Note that some of the rules are “soft" rules in that they are not true in many situations. Odom et al. odomAAAI15 weigh the effect of the rules against the data and hence allow for partially correct rules.

Answers:
by converting human advice to first-order logic format and use as an input to calculate gradient
Question: What do they learn jointly?

Evidence:
raw_evidence: To address our next question, we assessed our pipeline when learning relations independently (i.e., individually) versus learning relations jointly within the RDN, displayed in Table TABREF22 . Recall and F1 are omitted for conciseness – the conclusions are the same across all metrics. Joint learning appears to help in about half of the relations (8/14). Particularly, in person category, joint learning with gold standard outperforms their individual learning counterparts. This is due to the fact that some relations such as parents, spouse, siblings etc. are inter-related and learning them jointly indeed improves performance. Hence Q2 can be answered affirmatively for half the relations.

Answers:
relations
Question: Is this an English-language dataset?

Evidence:
raw_evidence: Within this scenario, it is crucial to analyse people attitudes towards rumours in social media and to resolve their veracity as soon as possible. Several approaches have been proposed to check the rumour veracity in social media BIBREF1 . This paper focus on a stance-based analysis of event-related rumours, following the approach proposed at SemEval-2017 in the new RumourEval shared task (Task 8, sub-task A) BIBREF2 . In this task English tweets from conversation threads, each associated to a newsworthy event and the rumours around it, are provided as data. The goal is to determine whether a tweet in the thread is supporting, denying, querying, or commenting the original rumour which started the conversation. It can be considered a stance classification task, where we have to predict the user's stance towards the rumour from a tweet, in the context of a given thread. This task has been defined as open stance classification task and is conceived as a key step in rumour resolution, by providing an analysis of people reactions towards an emerging rumour BIBREF0 , BIBREF3 . The task is also different from detecting stance towards a specific target entity BIBREF4 .

Answers:
Yes
Question: What affective-based features are used?

Evidence:
raw_evidence: We used the following affective resources relying on different emotion models.
Emolex: it contains 14,182 words associated with eight primary emotion based on the Plutchik model BIBREF10 , BIBREF11 .
EmoSenticNet(EmoSN): it is an enriched version of SenticNet BIBREF12 including 13,189 words labeled by six Ekman's basic emotion BIBREF13 , BIBREF14 .
Dictionary of Affect in Language (DAL): includes 8,742 English words labeled by three scores representing three dimensions: Pleasantness, Activation and Imagery BIBREF15 .
Affective Norms for English Words (ANEW): consists of 1,034 English words BIBREF16 rated with ratings based on the Valence-Arousal-Dominance (VAD) model BIBREF17 .
Linguistic Inquiry and Word Count (LIWC): this psycholinguistic resource BIBREF18 includes 4,500 words distributed into 64 emotional categories including positive (PosEMO) and negative (NegEMO).

Answers:
affective features provided by different emotion models such as Emolex, EmoSenticNet, Dictionary of Affect in Language, Affective Norms for English Words and Linguistics Inquiry and Word Count
Question: What conversation-based features are used?

Evidence:
raw_evidence: Conversation Based Features
These features are devoted to exploit the peculiar characteristics of the dataset, which have a tree structure reflecting the conversation thread.
Text Similarity to Source Tweet: Jaccard Similarity of each tweet with its source tweet.
Text Similarity to Replied Tweet: the degree of similarity between the tweet with the previous tweet in the thread (the tweet is a reply to that tweet).
Tweet Depth: the depth value is obtained by counting the node from sources (roots) to each tweet in their hierarchy.

Answers:
Text Similarity to Source Tweet, Text Similarity to Replied Tweet, Tweet Depth
Question: What are the evaluation metrics used?

Evidence:
raw_evidence: When testing across languages, we report accuracy for two setups: average accuracy over each single-language model (Avg), and accuracy obtained when training on the concatenation of all languages but the target one (All). The latter setting is also used for the embeddings model. We report accuracy for all experiments.

Answers:
average accuracy over each single-language model (Avg), and accuracy obtained when training on the concatenation of all languages but the target one (All)
Question: Do they report results only on English datasets?

Evidence:
raw_evidence: Using the Twitter Application Program Interface (API) and the IUNI OSoMeBIBREF46 (a service which provides searchable access to the Twitter “Gardenhose”, a 10% sample of all daily tweets), we search for tweets that matched both “diagnos*” and “depress*.” The resulting set of tweets are then filtered for matching the expressions “i”, “diagnos*”, “depres*” in that order in a case-insensitive manner allowing insertions to match the greatest variety of diagnosis statements, e.g. a tweet that states “I was in fact just diagnosed with clinical depression” would match. Finally, to ensure we are only including true self-referential statements of a depression diagnosis, a team of 3 experts manually removed quotes, jokes, and external references. For each qualifying diagnosis tweet we retrieve the timeline of the corresponding Twitter user using the Twitter user_timeline API endpoint . Subsequently, we remove all non-English tweets (Twitter API machine-detected“lang” field), all retweets, and tweets that contain “diagnos*” or “depress*”, but not a valid diagnosis statement. The resulting Depressed cohort contains 1,207 individuals and 1,759,644 tweets ranging from from May 2008 to September 2018.

Answers:
Yes
Question: What datasets or tasks do they conduct experiments on?

Evidence:
raw_evidence: Transformer is a versatile and powerful modeling tool and widely is used in various natural language process tasks. In order to verify the effectiveness of our method (i.e., Multi-linear attention) replacing multi-head attention in Transformer, we carry out two NLP tasks named language modeling (LM) and neural machine translation (NMT). Complete code for running experiments will be released after the paper is accepted, while the key code which is about our method can be found in Supplementary Materials F.
Specially, we take Transformer, the open source state-of-the art language modeling architecture, and replace the standard multi-head attention layers with our Multi-linear attention. Then, we test different model configurations on the PTB BIBREF25 , WikiText-103 BIBREF26 and One-Billion Word benchmark BIBREF27 datasets and report the results in Table 1 and Table 2 .
The goal is to map an input sequence $s=(x_1,x_2,\ldots ,x_n)$ representing a phrase in one language, to an output sequence $y=(y_1,y_2,\ldots , y_m)$ representing the same phrase in a different language. In this task, we have trained the Transformer model BIBREF2 on WMT 2016 English-German dataset BIBREF36 . Sentences were tokenized using the SentencePiece . For our experiments, we have replaced each of the attention layers with Multi-linear attention. For evaluation we used beam search with a beam size of 5 and length penalty $\alpha $ = $0.6$ . In this section, we only compared the results with Transformer BIBREF2 . Our results are summarized in Table 3 . $*$ indicates that the result is our own implementation.

Answers:
Language Modeling (LM), PTB BIBREF25 , WikiText-103 BIBREF26 and One-Billion Word benchmark BIBREF27 datasets, neural machine translation (NMT), WMT 2016 English-German dataset
Question: How authors create adversarial test set to measure model robustness?

Evidence:
raw_evidence: To evaluate model robustness, we devise a test set consisting of ‘adversarial’ examples, i.e, perturbed examples that can potentially change the base model's prediction. These could stem from paraphrasing a sentence, e.g., lexical and syntactical changes. We use two approaches described in literature: back-translation and noisy sequence autoencoder. Note that these examples resemble black-box attacks but are not intentionally designed to fool the system and hence, we use the term 'adversarial' broadly. We use these techniques to produce many paraphrases and find a subset of utterances that though very similar to the original test set, result in wrong predictions. We will measure the model robustness against such changes.

Answers:
we devise a test set consisting of ‘adversarial’ examples, i.e, perturbed examples that can potentially change the base model's prediction. , We use two approaches described in literature: back-translation and noisy sequence autoencoder.
Question: Do they compare with the MAML algorithm?

Evidence:
raw_evidence: We compare our method to the following baselines: (1) Single-task CNN: training a CNN model for each task individually; (2) Single-task FastText: training one FastText model BIBREF23 with fixed embeddings for each individual task; (3) Fine-tuned the holistic MTL-CNN: a standard transfer-learning approach, which trains one MTL-CNN model on all the training tasks offline, then fine-tunes the classifier layer (i.e. $\mathrm {M}^{(cls)}$ Figure 1 (a)) on each target task; (4) Matching Network: a metric-learning based few-shot learning model trained on all training tasks; (5) Prototypical Network: a variation of matching network with different prediction function as Eq. 9 ; (6) Convex combining all single-task models: training one CNN classifier on each meta-training task individually and taking the encoder, then for each target task training a linear combination of all the above single-task encoders with Eq. ( 24 ). This baseline can be viewed as a variation of our method without task clustering. We initialize all models with pre-trained 100-dim Glove embeddings (trained on 6B corpus) BIBREF24 .

Answers:
No
Question: What baseline is used?

Evidence:
raw_evidence: The baseline classifier uses a linear Support Vector Machine BIBREF7 , which is suited for a high number of features. We use a text classification framework for German BIBREF8 that has been used successfully for sentiment analysis before.

Answers:
SVM
Question: What topic clusters are identified by LDA?

Evidence:
raw_evidence: For offensive language detection in Twitter, users addressed in tweets might be an additional relevant signal. We assume it is more likely that politicians or news agencies are addressees of offensive language than, for instance, musicians or athletes. To make use of such information, we obtain a clustering of user ids from our Twitter background corpus. From all tweets in our stream from 2016 or 2017, we extract those tweets that have at least two @-mentions and all of the @-mentions have been seen at least five times in the background corpus. Based on the resulting 1.8 million lists of about 169,000 distinct user ids, we compute a topic model with INLINEFORM0 topics using Latent Dirichlet Allocation BIBREF3 . For each of the user ids, we extract the most probable topic from the inferred user id-topic distribution as cluster id. This results in a thematic cluster id for most of the user ids in our background corpus grouping together accounts such as American or German political actors, musicians, media websites or sports clubs (see Table TABREF17 ). For our final classification approach, cluster ids for users mentioned in tweets are fed as a second input in addition to (sub-)word embeddings to the penultimate dense layer of the neural network model.

Answers:
Clusters of Twitter user ids from accounts of American or German political actors, musicians, media websites or sports club
Question: What are the near-offensive language categories?

Evidence:
raw_evidence: As introduced above, the `One Million Post' corpus provides annotation labels for more than 11,000 user comments. Although there is no directly comparable category capturing `offensive language' as defined in the shared task, there are two closely related categories. From the resource, we extract all those comments in which a majority of the annotators agree that they contain either `inappropriate' or `discriminating' content, or none of the aforementioned. We treat the first two cases as examples of `offense' and the latter case as examples of `other'. This results in 3,599 training examples (519 offense, 3080 other) from on the `One Million Post' corpus. We conduct pre-training of the neural model as a binary classification task (similar to the Task 1 of GermEval 2018)

Answers:
inappropriate, discriminating
Question: How do they generate the auxiliary sentence?

Evidence:
raw_evidence: Construction of the auxiliary sentence
For simplicity, we mainly describe our method with TABSA as an example.
We consider the following four methods to convert the TABSA task into a sentence pair classification task:
The sentence we want to generate from the target-aspect pair is a question, and the format needs to be the same. For example, for the set of a target-aspect pair (LOCATION1, safety), the sentence we generate is “what do you think of the safety of location - 1 ?"
For the NLI task, the conditions we set when generating sentences are less strict, and the form is much simpler. The sentence created at this time is not a standard sentence, but a simple pseudo-sentence, with (LOCATION1, safety) pair as an example: the auxiliary sentence is: “location - 1 - safety".
For QA-B, we add the label information and temporarily convert TABSA into a binary classification problem ( INLINEFORM0 ) to obtain the probability distribution. At this time, each target-aspect pair will generate three sequences such as “the polarity of the aspect safety of location - 1 is positive", “the polarity of the aspect safety of location - 1 is negative", “the polarity of the aspect safety of location - 1 is none". We use the probability value of INLINEFORM1 as the matching score. For a target-aspect pair which generates three sequences ( INLINEFORM2 ), we take the class of the sequence with the highest matching score for the predicted category.
The difference between NLI-B and QA-B is that the auxiliary sentence changes from a question to a pseudo-sentence. The auxiliary sentences are: “location - 1 - safety - positive", “location - 1 - safety - negative", and “location - 1 - safety - none".
After we construct the auxiliary sentence, we can transform the TABSA task from a single sentence classification task to a sentence pair classification task. As shown in Table TABREF19 , this is a necessary operation that can significantly improve the experimental results of the TABSA task.

Answers:
The sentence we want to generate from the target-aspect pair is a question, and the format needs to be the same., For the NLI task, the conditions we set when generating sentences are less strict, and the form is much simpler., For QA-B, we add the label information and temporarily convert TABSA into a binary classification problem ( INLINEFORM0 ) to obtain the probability distribution, auxiliary sentence changes from a question to a pseudo-sentence
Question: Have any baseline model been trained on this abusive language dataset?

Evidence:

Answers:
No
Question: How big are this dataset and catalogue?

Evidence:
raw_evidence: The size of the training datasets varies considerably from 469 posts to 17 million; a difference of four orders of magnitude. Differences in size partly reflect different annotation approaches. The largest datasets are from proprietary data sharing agreements with platforms. Smaller datasets tend to be carefully collected and then manually annotated. There are no established guidelines for how large an abusive language training dataset needs to be. However, smaller datasets are problematic because they contain too little linguistic variation and increase the likelihood of overfitting. Rizoiu et al.BIBREF61 train detection models on only a proportion of the Davidson et al. and Waseem training datasets and show that this leads to worse performance, with a lower F1-Score, particularly for `data hungry' deep learning approaches BIBREF61. At the same time, `big' datasets alone are not a panacea for the challenges of abusive content classification. Large training datasets which have been poorly sampled, annotated with theoretically problematic categories or inexpertly and unthoughtfully annotated, could still lead to the development of poor classification systems.

Answers:
 from 469 posts to 17 million
Question: What is open website for cataloguing abusive language data?

Evidence:
raw_evidence: Most detection systems rely on having the right training dataset, reflecting one of the most widely accepted mantras in computer science: Garbage In, Garbage Out. Put simply: to have systems which can detect and classify abusive online content effectively, one needs appropriate datasets with which to train them. However, creating training datasets is often a laborious and non-trivial task – and creating datasets which are non-biased, large and theoretically-informed is even more difficult (BIBREF0 p. 189). We address this issue by examining and reviewing publicly available datasets for abusive content detection, which we provide access to on a new dedicated website, hatespeechdata.com.

Answers:
hatespeechdata.com
Question: how many speeches are in the dataset?

Evidence:
raw_evidence: The UNGA speeches dataset, compiled by Baturo et al. UNGAspeeches, contains the text from 7,507 speeches given between 1970-2015 inclusive. Over the course of this period a variety of topics are discussed, with many debated throughout (such as nuclear disarmament). Although the linguistic style has changed over this period, the context of these speeches constrains the variability to the formal domain. Before training the model, the dataset is split into 283,593 paragraphs, cleaned by removing paragraph deliminators and other excess noise, and tokenized using the spaCy tokenizer BIBREF4 .

Answers:
7,507
Question: How big is the provided treebank?

Evidence:
raw_evidence: Recently bhat-EtAl:2017:EACLshort provided a CS dataset for the evaluation of their parsing models which they trained on the Hindi and English Universal Dependency (UD) treebanks. We extend this dataset by annotating 1,448 more sentences. Following bhat-EtAl:2017:EACLshort we first sampled CS data from a large set of tweets of Indian language users that we crawled from Twitter using Tweepy–a Twitter API wrapper. We then used a language identification system trained on ICON dataset (see Section "Preliminary Tasks" ) to filter Hindi-English CS tweets from the crawled Twitter data. Only those tweets were selected that satisfied a minimum ratio of 30:70(%) code-switching. From this dataset, we manually selected 1,448 tweets for annotation. The selected tweets are thoroughly checked for code-switching ratio. For POS tagging and dependency annotation, we used Version 2 of Universal dependency guidelines BIBREF21 , while language tags are assigned based on the tag set defined in BIBREF22 , BIBREF23 . The dataset was annotated by two expert annotators who have been associated with annotation projects involving syntactic annotations for around 10 years. Nonetheless, we also ensured the quality of the manual annotations by carrying an inter-annotator agreement analysis. We randomly selected a dataset of 150 tweets which were annotated by both annotators for both POS tagging and dependency structures. The inter-annotator agreement has a 96.20% accuracy for POS tagging and a 95.94% UAS and a 92.65% LAS for dependency parsing.

Answers:
1448 sentences more than the dataset from Bhat et al., 2017
Question: is the dataset balanced across the four languages?

Evidence:
raw_evidence: The data stems from a joint ADAPT-Microsoft project. An overview of the provided dataset is given in Table TABREF16 . Notice that the available amount of data differs per language.

Answers:
No
Question: what evaluation metrics were used?

Evidence:
raw_evidence: We decided to evaluate our model using weighted F1-score, i.e., the per-class F1 score is calculated and averaged by weighting each label by its support. Notice, since our setup deviates from the shared task setup (single-label versus multi-label classification), the final evaluation metric is different. We will report on weighted F1-score for the development and test set (with simple macro averaging), but use Exact-Accuracy and Micro F1 over all labels when presenting official results on the test sets. The latter two metrics were part of the official evaluation metrics. For details we refer the reader to the shared task overview paper BIBREF5 .

Answers:
weighted F1-score
Question: what dataset was used?

Evidence:
raw_evidence: The data stems from a joint ADAPT-Microsoft project. An overview of the provided dataset is given in Table TABREF16 . Notice that the available amount of data differs per language.

Answers:
The dataset from a joint ADAPT-Microsoft project
Question: What is the size of ACL-ARC datasets?

Evidence:
raw_evidence: ACL-ARC is a dataset of citation intents released by BIBREF7 . The dataset is based on a sample of papers from the ACL Anthology Reference Corpus BIBREF15 and includes 1,941 citation instances from 186 papers and is annotated by domain experts in the NLP field. The data was split into three standard stratified sets of train, validation, and test with 85% of data used for training and remaining 15% divided equally for validation and test. Each citation unit includes information about the immediate citation context, surrounding context, as well as information about the citing and cited paper. The data includes six intent categories outlined in Table 2 .

Answers:
includes 1,941 citation instances from 186 papers
Question: How is quality of annotation measured?

Evidence:
raw_evidence: To control the quality, we ensured that a single annotator annotates maximum 120 headlines (this protects the annotators from reading too many news headlines and from dominating the annotations). Secondly, we let only annotators who geographically reside in the U.S. contribute to the task.
We test the annotators on a set of $1,100$ test questions for the first phase (about 10% of the data) and 500 for the second phase. Annotators were required to pass 95%. The questions were generated based on hand-picked non-ambiguous real headlines through swapping out relevant words from the headline in order to obtain a different annotation, for instance, for “Djokovic happy to carry on cruising”, we would swap “Djokovic” with a different entity, the cue “happy” to a different emotion expression.

Answers:
Annotators went through various phases to make sure their annotations did not deviate from the mean.