z-uo/qasper-squad huggingface dataset:
Paper: Depressed individuals express more distorted thinking on social media
Depression is a leading contributor to the burden of disability worldwideBIBREF0, BIBREF1, with some evidence that disability attributed to depression is rising, particularly among youthBIBREF2, BIBREF3. A key challenge in reducing the prevalence of depression has been that it is often under-recognizedBIBREF4 as well as under-treatedBIBREF5. Cognitive-behavioral therapy (CBT), is the most widely researched psychotherapy for depression. It is equivalent to antidepressant medications in its short-term efficacy and evidences superior outcomes in the long-termBIBREF6, BIBREF7. The cognitive theory underlying CBT argues that the ways in which individuals process and interpret information about themselves and their world is directly related to the onset, maintenance, and recurrence of their depressionBIBREF8, BIBREF9. This model is consistent with information processing accounts of mood regulationBIBREF10 and its dynamicsBIBREF11, as well as basic research that supports the role of cognitive reappraisal and language in emotion regulationBIBREF12, BIBREF13, BIBREF14, BIBREF15.
Question: Do they report results only on English datasets?
Paper: A Tensorized Transformer for Language Modeling
In NLP, Neural language model pre-training has shown to be effective for improving many tasks BIBREF0 , BIBREF1 . Transformer BIBREF2 is based solely on the attention mechanism, and dispensing with recurrent and convolutions entirely. At present, this model has received extensive attentions and plays an key role in many neural language models, such as BERT BIBREF0 , GPT BIBREF3 and Universal Transformer BIBREF4 . However, in Transformer based model, a lot of model parameters may cause problems in training and deploying these parameters in a limited resource setting. Thus, the compression of large neural pre-training language model has been an essential problem in NLP research.
Question: What datasets or tasks do they conduct experiments on?
Paper: Improving Robustness of Task Oriented Dialog Systems
In computer vision, it is well known that otherwise competitive models can be "fooled" by adding intentional noise to the input images BIBREF0, BIBREF1. Such changes, imperceptible to the human eye, can cause the model to reverse its initial correct decision on the original input. This has also been studied for Automatic Speech Recognition (ASR) by including hidden commands BIBREF2 in the voice input. Devising such adversarial examples for machine learning algorithms, in particular for neural networks, along with defense mechanisms against them, has been of recent interest BIBREF3. The lack of smoothness of the decision boundaries BIBREF4 and reliance on weakly correlated features that do not generalize BIBREF5 seem to be the main reasons for confident but incorrect predictions for instances that are far from the training data manifold. Among the most successful techniques to increase resistance to such attacks is perturbing the training data and enforcing the output to remain the same BIBREF4, BIBREF6. This is expected to improve the smoothing of the decision boundaries close to the training data but may not help with points that are far from them.
Question: How big is performance improvement proposed methods are used?
Answer: Data augmentation (es)  improved Adv es by 20% comparing to baseline 
Data augmentation (cs) improved Adv cs by 16.5% comparing to baseline
Data augmentation (cs+es) improved both Adv cs and Adv es by at least 10% comparing to baseline 
All models show improvements over adversarial sets
Paper: Improving Robustness of Task Oriented Dialog Systems
block = [text width=15em, text centered]
Question: How authors create adversarial test set to measure model robustness?
Paper: Diverse Few-Shot Text Classification with Multiple Metrics
Few-shot learning (FSL) BIBREF0 , BIBREF1 , BIBREF2 aims to learn classifiers from few examples per class. Recently, deep learning has been successfully exploited for FSL via learning meta-models from a large number of meta-training tasks. These meta-models can be then used for rapid-adaptation for the target/meta-testing tasks that only have few training examples. Examples of such meta-models include: (1) metric-/similarity-based models, which learn contextual, and task-specific similarity measures BIBREF3 , BIBREF4 , BIBREF5 ; and (2) optimization-based models, which receive the input of gradients from a FSL task and predict either model parameters or parameter updates BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 .
Question: Do they compare with the MAML algorithm?
Paper: Transfer Learning from LDA to BiLSTM-CNN for Offensive Language Detection in Twitter
User-generated content in forums, blogs, and social media not only contributes to a deliberative exchange of opinions and ideas but is also contaminated with offensive language such as threats and discrimination against people, swear words or blunt insults. The automatic detection of such content can be a useful support for moderators of public platforms as well as for users who could receive warnings or would be enabled to filter unwanted content.
Question: By how much does transfer learning improve performance on this task?
Answer: In task 1 best transfer learning strategy improves F1 score by 4.4% and accuracy score by 3.3%, in task 2 best transfer learning strategy improves F1 score by 2.9% and accuracy score by 1.7%
Paper: Transfer Learning from LDA to BiLSTM-CNN for Offensive Language Detection in Twitter
Automatic detection of offensive language is a well-studied phenomenon for the English language. Initial works on the detection of `hostile messages' have been published already during the 1990s BIBREF4 . An overview of recent approaches comparing the different task definitions, feature sets and classification methods is given by Schmidt.2017. A major step forward to support the task was the publication of a large publicly available, manually annotated dataset by Yahoo research BIBREF5 . They provide a classification approach for detection of abusive language in Yahoo user comments using a variety of linguistic features in a linear classification model. One major result of their work was that learning text features from comments which are temporally close to the to-be-predicted data is more important than learning features from as much data as possible. This is especially important for real-life scenarios of classifying streams of comment data. In addition to token-based features, Xiang.2012 successfully employed topical features to detect offensive tweets. We will build upon this idea by employing topical data in our transfer learning setup. Transfer learning recently has gained a lot of attention since it can be easily applied to neural network learning architectures. For instance, Howard.2018 propose a generic transfer learning setup for text classification based on language modeling for pre-training neural models with large background corpora. To improve offensive language detection for English social media texts, a transfer learning approach was recently introduced by Felbo.2017. Their `deepmoji' approach relies on the idea to pre-train a neural network model for an actual offensive language classification task by using emojis as weakly supervised training labels. On a large collection of millions of randomly collected English tweets containing emojis, they try to predict the specific emojis from features obtained from the remaining tweet text. We will follow this idea of transfer learning to evaluate it for offensive language detection in German Twitter data together with other transfer learning strategies.
Question: What baseline is used?
Answer: SVM
Paper: Transfer Learning from LDA to BiLSTM-CNN for Offensive Language Detection in Twitter
Organizers of GermEval 2018 provide training and test datasets for two tasks. Task 1 is a binary classification for deciding whether or not a German tweet contains offensive language (the respective category labels are `offense' and `other'). Task 2 is a multi-class classification with more fine-grained labels sub-categorizing the same tweets into either `insult', `profanity', `abuse', or `other'.
Question: What topic clusters are identified by LDA?
Answer: Clusters of Twitter user ids from accounts of American or German political actors, musicians, media websites or sports club
Paper: Transfer Learning from LDA to BiLSTM-CNN for Offensive Language Detection in Twitter
Since the provided dataset for offensive language detection is rather small, we investigate the potential of transfer learning to increase classification performance. For this, we use the following labeled as well as unlabeled datasets.
Question: What are the near-offensive language categories?
Paper: Utilizing BERT for Aspect-Based Sentiment Analysis via Constructing Auxiliary Sentence
Sentiment analysis (SA) is an important task in natural language processing. It solves the computational processing of opinions, emotions, and subjectivity - sentiment is collected, analyzed and summarized. It has received much attention not only in academia but also in industry, providing real-time feedback through online reviews on websites such as Amazon, which can take advantage of customers' opinions on specific products or services. The underlying assumption of this task is that the entire text has an overall polarity.
Question: How much do they outperform previous state-of-the-art?
Answer: On subtask 3 best proposed model has F1 score of 92.18 compared to best previous F1 score of 88.58.
On subtask 4 best proposed model has 85.9, 89.9 and 95.6 compared to best previous results of 82.9, 84.0 and 89.9 on 4-way, 3-way and binary aspect polarity.
Paper: Utilizing BERT for Aspect-Based Sentiment Analysis via Constructing Auxiliary Sentence
In this section, we describe our method in detail.
Question: How do they generate the auxiliary sentence?
Paper: Directions in Abusive Language Training Data: Garbage In, Garbage Out
Abusive online content, such as hate speech and harassment, has received substantial attention over the past few years for its malign social effects. Left unchallenged, abusive content risks harminng those who are targeted, toxifying public discourse, exacerbating social tensions and could lead to the exclusion of some groups from public spaces. As such, systems which can accurately detect and classify online abuse at scale, in real-time and without bias are of central interest to tech companies, policymakers and academics.
Question: Have any baseline model been trained on this abusive language dataset?
Paper: Directions in Abusive Language Training Data: Garbage In, Garbage Out
The volume of research examining the social and computational aspects of abusive content detection has expanded prodigiously in the past five years. This has been driven by growing awareness of the importance of the Internet more broadly BIBREF1, greater recognition of the harms caused by online abuse BIBREF2, and policy and regulatory developments, such as the EU's Code of Conduct on Hate, the UK Government's `Online Harms' white paper BIBREF3, Germany's NetzDG laws, the Public Pledge on Self-Discipline for the Chinese Internet Industry, and France's anti-hate regulation BIBREF2. In 2020 alone, three computer science venues will host workshops on online hate (TRAC and STOC at LREC, and WOAH at EMNLP), and a shared task at 2019's SemEval on online abuse detection reports that 800 teams downloaded the training data and 115 submitted detection systems BIBREF4. At the same time, social scientific interventions have also appeared, deepening our understanding of how online abuse spreads BIBREF5 and how its harmful impact can be mitigated and challenged BIBREF6.
Question: How big are this dataset and catalogue?
Paper: Directions in Abusive Language Training Data: Garbage In, Garbage Out
Relevant publications have been identified from four sources to identify training datasets for abusive content detection:
Question: What is open website for cataloguing abusive language data?
Paper: Automated Speech Generation from UN General Assembly Statements: Mapping Risks in AI Generated Texts
The rise of Artificial Intelligence (AI) brings many potential benefits to society, as well as significant risks. These risks take a variety of forms, from autonomous weapons and sophisticated cyber-attacks, to the more subtle techniques of societal manipulation. In particular, the threat this technology poses to maintaining peace and political stability is especially relevant to the United Nations (UN) and other international organisations. In terms of applications of AI, several major risks to peace and political stability have been identified, including: the use of automated surveillance platforms to suppress dissent; fake news reports with realistic fabricated video and audio; and the manipulation of information availability BIBREF0 .
Question: how many speeches are in the dataset?
Paper: Universal Dependency Parsing for Hindi-English Code-switching
Code-switching (henceforth CS) is the juxtaposition, within the same speech utterance, of grammatical units such as words, phrases, and clauses belonging to two or more different languages BIBREF0 . The phenomenon is prevalent in multilingual societies where speakers share more than one language and is often prompted by multiple social factors BIBREF1 . Moreover, code-switching is mostly prominent in colloquial language use in daily conversations, both online and offline.
Question: How big is the provided treebank?
Answer: 1448 sentences more than the dataset from Bhat et al., 2017
Paper: Universal Dependency Parsing for Hindi-English Code-switching
As preliminary steps before parsing of CS data, we need to identify the language of tokens and normalize and/or back-transliterate them to enhance the parsing performance. These steps are indispensable for processing CS data and without them the performance drops drastically as we will see in Results Section. We need normalization of non-standard word forms and back-transliteration of Romanized Hindi words for addressing out-of-vocabulary problem, and lexical and syntactic ambiguity introduced due to contracted word forms. As we will train separate normalization and back-transliteration models for Hindi and English, we need language identification for selecting which model to use for inference for each word form separately. Moreover, we also need language information for decoding best word sequences.
Question: What is LAS metric?
Paper: ALL-IN-1: Short Text Classification with One Model for All Languages
Customer feedback analysis is the task of classifying short text messages into a set of predefined labels (e.g., bug, request). It is an important step towards effective customer support.
Question: is the dataset balanced across the four languages?
Paper: ALL-IN-1: Short Text Classification with One Model for All Languages
Motivated by the goal to evaluate how good a single model for multiple languages fares, we decided to build a very simple model that can handle any of the four languages. We aimed at an approach that does not require any language-specific processing (beyond tokenization) nor requires any parallel data. We set out to build a simple baseline, which turned out to be surprisingly effective. Our model is depicted in Figure FIGREF7 .
Question: what evaluation metrics were used?
Paper: ALL-IN-1: Short Text Classification with One Model for All Languages
In this section we first describe the IJCNLP 2017 shared task 4 including the data, the features, model and evaluation metrics.
Question: what dataset was used?
Answer: The dataset from a joint ADAPT-Microsoft project
Paper: Structural Scaffolds for Citation Intent Classification in Scientific Publications
Citations play a unique role in scientific discourse and are crucial for understanding and analyzing scientific work BIBREF0 , BIBREF1 . They are also typically used as the main measure for assessing impact of scientific publications, venues, and researchers BIBREF2 . The nature of citations can be different. Some citations indicate direct use of a method while some others merely serve as acknowledging a prior work. Therefore, identifying the intent of citations (Figure 1 ) is critical in improving automated analysis of academic literature and scientific impact measurement BIBREF1 , BIBREF3 . Other applications of citation intent classification are enhanced research experience BIBREF4 , information retrieval BIBREF5 , summarization BIBREF6 , and studying evolution of scientific fields BIBREF7 .
Question: What are the citation intent labels in the datasets?
Answer: Background, extends, uses, motivation, compare/contrast, and future work for the ACL-ARC dataset. Background, method, result comparison for the SciCite dataset.
Paper: Structural Scaffolds for Citation Intent Classification in Scientific Publications
We propose a neural multitask learning framework for classification of citation intents. In particular, we introduce and use two structural scaffolds, auxiliary tasks related to the structure of scientific papers. The auxiliary tasks may not be of interest by themselves but are used to inform the main task. Our model uses a large auxiliary dataset to incorporate this structural information available in scientific documents into the citation intents. The overview of our model is illustrated in Figure 2 .
Question: What is the size of ACL-ARC datasets?
Paper: Aff2Vec: Affect--Enriched Distributional Word Representations
 This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http://creativecommons.org/licenses/by/4.0/ Affect refers to the experience of a feeling or emotion BIBREF0 , BIBREF1 . This definition includes emotions, sentiments, personality, and moods. The importance of affect analysis in human communication and interactions has been discussed by Picard Picard1997. Historically, affective computing has focused on studying human communication and reactions through multi-modal data gathered through various sensors. The study of human affect from text and other published content is an important topic in language understanding. Word correlation with social and psychological processes is discussed by Pennebaker Pennebaker2011. Preotiuc-Pietro et al. perspara17nlpcss studied personality and psycho-demographic preferences through Facebook and Twitter content. Sentiment analysis in Twitter with a detailed discussion on human affect BIBREF2 and affect analysis in poetry BIBREF3 have also been explored. Human communication not only contains semantic and syntactic information but also reflects the psychological and emotional states. Examples include the use of opinion and emotion words BIBREF4 . The analysis of affect in interpersonal communication such as emails, chats, and longer written articles is necessary for various applications including the study of consumer behavior and psychology, understanding audiences and opinions in computational social science, and more recently for dialogue systems and conversational agents. This is a open research space today.
Question: Is the affect of a word affected by context?
Paper: Unsupervised Cross-lingual Representation Learning at Scale
The goal of this paper is to improve cross-lingual language understanding (XLU), by carefully studying the effects of training unsupervised cross-lingual representations at a very large scale. We present XLM-R, a transformer-based multilingual masked language model pre-trained on text in 100 languages, which obtains state-of-the-art performance on cross-lingual classification, sequence labeling and question answering.
Question: asdfasdaf
Paper: Unsupervised Cross-lingual Representation Learning at Scale
From pretrained word embeddings BIBREF11, BIBREF12 to pretrained contextualized representations BIBREF13, BIBREF14 and transformer based language models BIBREF15, BIBREF0, unsupervised representation learning has significantly improved the state of the art in natural language understanding. Parallel work on cross-lingual understanding BIBREF16, BIBREF14, BIBREF1 extends these systems to more languages and to the cross-lingual setting in which a model is learned in one language and applied in other languages.
Question: asdfasdf
Paper: Unsupervised Cross-lingual Representation Learning at Scale
In this section, we present the training objective, languages, and data we use. We follow the XLM approach BIBREF1 as closely as possible, only introducing changes that improve performance at scale.
Question: asdfasd
Paper: Unsupervised Cross-lingual Representation Learning at Scale
We use a Transformer model BIBREF2 trained with the multilingual MLM objective BIBREF0, BIBREF1 using only monolingual data. We sample streams of text from each language and train the model to predict the masked tokens in the input. We apply subword tokenization directly on raw text data using Sentence Piece BIBREF26 with a unigram language model BIBREF27. We sample batches from different languages using the same sampling distribution as BIBREF1, but with $\alpha =0.3$. Unlike BIBREF1, we do not use language embeddings, which allows our model to better deal with code-switching. We use a large vocabulary size of 250K with a full softmax and train two different models: XLM-R Base (L = 12, H = 768, A = 12, 270M params) and XLM-R (L = 24, H = 1024, A = 16, 550M params). For all of our ablation studies, we use a BERTBase architecture with a vocabulary of 150K tokens. Appendix SECREF8 goes into more details about the architecture of the different models referenced in this paper.
Question: asdf
Paper: GoodNewsEveryone: A Corpus of News Headlines Annotated with Emotions, Semantic Roles, and Reader Perception
Research in emotion analysis from text focuses on mapping words, sentences, or documents to emotion categories based on the models of Ekman1992 or Plutchik2001, which propose the emotion classes of joy, sadness, anger, fear, trust, disgust, anticipation and surprise. Emotion analysis has been applied to a variety of tasks including large scale social media mining BIBREF0, literature analysis BIBREF1, BIBREF2, lyrics and music analysis BIBREF3, BIBREF4, and the analysis of the development of emotions over time BIBREF5.
Question: How is quality of annotation measured?
Answer: Annotators went through various phases to make sure their annotations did not deviate from the mean.
Paper: LinkNBed: Multi-Graph Representation Learning with Entity Linkage
Reasoning over multi-relational data is a key concept in Artificial Intelligence and knowledge graphs have appeared at the forefront as an effective tool to model such multi-relational data. Knowledge graphs have found increasing importance due to its wider range of important applications such as information retrieval BIBREF0 , natural language processing BIBREF1 , recommender systems BIBREF2 , question-answering BIBREF3 and many more. This has led to the increased efforts in constructing numerous large-scale Knowledge Bases (e.g. Freebase BIBREF4 , DBpedia BIBREF5 , Google's Knowledge graph BIBREF6 , Yago BIBREF7 and NELL BIBREF8 ), that can cater to these applications, by representing information available on the web in relational format.
Question: On what data is the model evaluated?