z-uo/qasper-squad huggingface dataset:
Paper: Adversarial Examples with Difficult Common Words for Paraphrase Identification
Paraphrase identification is to determine whether a pair of sentences are paraphrases of each other BIBREF0. It is important for applications such as duplicate post matching on social media BIBREF1, plagiarism detection BIBREF2, and automatic evaluation for machine translation BIBREF3 or text summarization BIBREF4.
Question: How much in experiments is performance improved for models trained with generated adversarial examples?
Answer: Answer with content missing: (Table 1) The performance of all the target models raises significantly, while that on the original
examples remain comparable (e.g. the overall accuracy of BERT on modified examples raises from 24.1% to 66.0% on Quora)
Paper: Adversarial Examples with Difficult Common Words for Paraphrase Identification
Paraphrase identification can be viewed as a problem of sentence matching. Recently, many deep models for sentence matching have been proposed and achieved great advancements on benchmark datasets. Among those, some approaches encode each sentence independently and apply a classifier on the embeddings of two sentences BIBREF10, BIBREF11, BIBREF12. In addition, some models make strong interactions between two sentences by jointly encoding and matching sentences BIBREF5, BIBREF13, BIBREF14 or hierarchically extracting matching features from the interaction space of the sentence pair BIBREF15, BIBREF16, BIBREF6. Notably, BERT pre-trained on large-scale corpora achieved even better results BIBREF7. In this paper, we study the robustness of recent typical deep models for paraphrase identification and generate new adversarial examples for revealing their robustness issues and improving their robustness.
Question: How much dramatically results drop for models on generated adversarial examples?
Paper: Adversarial Examples with Difficult Common Words for Paraphrase Identification
Many methods have been proposed to find different types of adversarial examples for NLP tasks. We focus on those that can be applied to paraphrase identification. Some of them generate adversarial examples by adding semantic-preserving perturbations to the input sentences. BIBREF17 added perturbations to word embeddings. BIBREF18, BIBREF19, BIBREF20, BIBREF21, BIBREF22 employed several character-level or word-level manipulations. BIBREF23 used syntactically controlled paraphrasing, and BIBREF24 paraphrased sentences with extracted rules. However, for some tasks including paraphrase identification, adversarial examples can be semantically different from original sentences, to study other robustness issues tailored to the corresponding tasks.
Question: What is discriminator in this generative adversarial setup?
Paper: Adversarial Examples with Difficult Common Words for Paraphrase Identification
For a certain type of adversarial examples, adversarial attacks or adversarial example generation aim to find examples that are within the defined type and make existing models fail. Some work has no access to the target model until an adversarial dataset is generated BIBREF28, BIBREF26, BIBREF23, BIBREF24, BIBREF29, BIBREF27. However, in many cases including ours, finding successful adversarial examples, i.e. examples on which the target model fails, is challenging, and employing an attack algorithm with access to the target model during generation is often necessary to ensure a high success rate.
Question: What are benhmark datasets for paraphrase identification?
Paper: Gender Representation in Open Source Speech Resources
1.1em
Question: What representations are presented by this paper?
Paper: Gender Representation in Open Source Speech Resources
1.1.1em
Question: What corpus characteristics correlate with more equitable gender balance?
Paper: Gender Representation in Open Source Speech Resources
1.1.1.1em
Question: What natural languages are represented in the speech resources studied?
Paper: A Neural Approach to Discourse Relation Signal Detection
The development of formal frameworks for the analysis of discourse relations has long gone hand in hand with work on signaling devices. The analysis of discourse relations is also closely tied to what a discourse structure should look like and what discourse goals should be fulfilled in relation to the interpretation of discourse relations BIBREF0. Earlier work on the establishment of inventories of discourse relations and their formalization (BIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6) relied on the existence of `discourse markers' (DMs) or `connectives', including conjunctions such as because or if, adverbials such as however or as a result, and coordinations such as but, to identify and distinguish relations such as condition in SECREF1, concession in SECREF1, cause in SECREF1, or contrast, result etc., depending on the postulated inventory of relations (signals for these relations as identified by human analysts are given in bold; examples come from the GUM corpus BIBREF7, presented in Section SECREF3).
Question: How is the delta-softmax calculated?
Answer: Answer with content missing: (Formula) Formula is the answer.
Paper: A Neural Approach to Discourse Relation Signal Detection
A straightforward approach to identifying discourse relation signals in corpora with discourse parses is to extract frequency counts for all lexical types or lemmas and cross-tabulate them with discourse relations, such as sentences annotated as cause, elaboration, etc. (e.g. BIBREF21, BIBREF22, BIBREF17). Table TABREF4, reproduced from BIBREF17, illustrates this approach for the Russian RST Treebank.
Question: Are some models evaluated using this metric, what are the findings?
Paper: A Neural Approach to Discourse Relation Signal Detection
Discourse relation signals are broadly classified into two categorizes: anchored signals and unanchored signals. By `anchoring' we refer to associating signals with concrete token spans in texts. Intuitively, most of the signals are anchorable since they correspond to certain token spans. However, it is also possible for a discourse relation to be signaled but remain unanchored. Results from BIBREF27 indicated that there are several signaled but unanchored relations such as preparation and background since they are high-level discourse relations that capture and correspond to genre features such as interview layout in interviews where the conversation is constructed as a question-answer scheme, and are thus rarely anchored to tokens.
Question: Where does proposed metric differ from juman judgement?
Paper: A Neural Approach to Discourse Relation Signal Detection
In order to study open-ended signals anchored to concrete tokens, we use the signal-annotated subset of the freely available Georgetown University Multilayer (GUM) Corpus BIBREF7 from BIBREF27. Our choice to use a multi-genre RST-annotated corpus rather than using PDTB, which also contains discourse relation signal annotation to a large extent is motivated by three reasons: The first reason is that we wish to explore the full range of potential signals, as laid out in the work on the Signalling Corpus BIBREF12, BIBREF34, whereas PDTB annotates only a subset of the possible cues identified by human annotators. Secondly, the use of RST as a framework allows us to examine discourse relations at all hierarchical levels, including long distance, high-level relations between structures as large as paragraphs or sections, which often have different types of signals allowing their identification. Finally, although the entire GUM corpus is only about half the size of RST-DT (109K tokens), using GUM offers the advantage of a more varied range of genres than PDTB and RST-SC, both of which annotate Wall Street Journal data.
Question: Where does proposed metric overlap with juman judgement?
Paper: Citation Text Generation
The output of the world's scientists doubles roughly every nine years BIBREF0, and their pace is quickening. As a result, scientists and other experts must devote significant time to the difficult task of literature review, or coming to understand the context in which they work. Might artificial intelligence help to reduce that time?
Question: Which baseline performs best?
Paper: Citation Text Generation
Given the important research challenges posed by the citation text generation task, along with the potential social benefits of its solutions, let us continue with a formalization of the problem. Citation text generation is the task of generating a natural language citing sentence which explains the relationship between two documents. Examples of such citing sentences can be found in scientific documents as in-text citations to a previous work. Thus, we will formally distinguish one document as the source document, from which we will draw citing sentences which reference the cited document.
Question: Which baselines are explored?
Paper: Citation Text Generation
We explore two basic styles of model for citation text generation. Following current work in neural text generation, we fine-tune the predictions of a large pre-trained language model to the citation text generation task. Additionally, we investigate approximate nearest neighbor methods to retrieve plausible human-authored citation sentences from the training data.
Question: What is the size of the corpus?
Paper: Citation Text Generation
Recent work has shown that adapting large pre-trained language models to text generation tasks yields strong results BIBREF8. Due to its widespread use in text generation, we investigate the GPT model of BIBREF6 for the citation text generation task. GPT2 is a transformer model trained on 40 gigabytes of internet text with a language modeling objective BIBREF9. The adaptation process, called fine-tuning, involves continued training of the model on the target objective, in our case citation text generation.
Question: How was the evaluation corpus collected?
Paper: Machine Translation with Cross-lingual Word Embeddings
Methods for machine translations have been studied for years, and at the same time algorithms to generate word embeddings are becoming more and more accurate. Still, there is a lot of research aiming at unifying word embeddings across multiple languages. In this experience we try a technique for machine translation that relates word embeddings between two different languages. Based on the literature we found that it is possible to infer missing dictionary entries using distributed representations of words and phrases. One way of doing it is to create a linear mapping between the two vector spaces of two different languages. In order to achieve this, we first built two dictionaries of the two different languages. Next, we learned a function that projects the first vector space to the second one. In this way, we are able to translate every word belonging to the first language into the second one. Once we obtain the translated word embedding, we output the most similar word vector as the translation. The word embeddings were learnt using the Skip Gram method proposed by (Mikolov et al., 2013a). An example of how the method would work is reported in figure 1 and figure 2. After creating the word embeddings from the two dictionaries, we plotted the numbers in the two graphs using PCA. Figure 3 reports the results after creating a linear mapping between the embeddings from the two languages. You can see how similar words are closer together.
Question: Are any machine translation sysems tried with these embeddings, what is the performance?
Paper: Machine Translation with Cross-lingual Word Embeddings
In recent years, various models for learning cross-lingual representations have been proposed. Two main broad categories with some related papers are identified here:
Question: Are any experiments performed to try this approach to word embeddings?
Paper: Meteorologists and Students: A resource for language grounding of geographical descriptors
Language grounding, i.e., understanding how words and expressions are anchored in data, is one of the initial tasks that are essential for the conception of a data-to-text (D2T) system BIBREF0 , BIBREF1 . This can be achieved through different means, such as using heuristics or machine learning algorithms on an available parallel corpora of text and data BIBREF2 to obtain a mapping between the expressions of interest and the underlying data BIBREF3 , getting experts to provide these mappings, or running surveys on writers or readers that provide enough data for the application of mapping algorithms BIBREF4 .
Question: Which two datasets does the resource come from?
Answer: two surveys by two groups - school students and meteorologists to draw on a map a polygon representing a given geographical descriptor
Paper: SocialNLP EmotionX 2019 Challenge Overview: Predicting Emotions in Spoken Dialogues and Chats
Emotions are a central component of our existence as human beings, and are manifested by physiological and psychological changes that often affect behavior and action. Emotions involve a complicated interplay of mind, body, language, and culture BIBREF0.
Question: What model was used by the top team?
Paper: SocialNLP EmotionX 2019 Challenge Overview: Predicting Emotions in Spoken Dialogues and Chats
The two datasets used for the challenge are Friends and EmotionPush, part of the EmotionLines corpus BIBREF4. The datasets contain English-language dialogues of varying lengths. For the competition, we provided 1,000 labeled dialogues from each dataset for training, and 240 unlabeled dialogues from each dataset for evaluation. The Friends dialogues are scripts taken from the American TV sitcom (1994-2004). The EmotionPush dialogues are from Facebook Messenger chats by real users which have been anonymized to ensure user privacy. For both datasets, dialogue lengths range from 5 to 24 lines each. A breakdown of the lengths of the dialogues is shown in Table .
Question: What was the baseline?
Paper: SocialNLP EmotionX 2019 Challenge Overview: Predicting Emotions in Spoken Dialogues and Chats
NLP tasks require plenty of data. Due to the relatively small number of samples in our datasets, we added more labeled data using a technique developed in BIBREF7 that was used by the winning team in Kaggle's Toxic Comment Classification Challenge BIBREF8. The augmented datasets are similar to the original data files, but include additional machine-computed utterances for each original utterance. We created the additional utterances using the Google Translate API. Each original utterance was first translated from English into three target languages (German, French, and Italian), and then translated back into English. The resulting utterances were included together in the same object with the original utterance. These “duplex translations” can sometimes result in the original sentence, but many times variations are generated that convey the same emotions. Table shows an example utterance (labeled with “Joy”) after augmentation.
Question: What is the size of the second dataset?
Answer: 1 000 labeled dialogues for training and 240 unlabeled dialogues for evaluation
Paper: SocialNLP EmotionX 2019 Challenge Overview: Predicting Emotions in Spoken Dialogues and Chats
A dedicated website for the competition was set up. The website included instructions, the registration form, schedule, and other relevant details. Following registration, participants were able to download the training datasets.
Question: How large is the first dataset?
Answer: 1 000 labeled dialogues for training and 240 unlabeled dialogues for evaluation
Paper: SocialNLP EmotionX 2019 Challenge Overview: Predicting Emotions in Spoken Dialogues and Chats
A total of eleven teams submitted their evaluations, and are presented in the online leaderboard. Seven of the teams also submitted technical reports, the highlights of which are summarized below. More details are available in the relevant reports.
Question: Who was the top-scoring team?
Answer: IDEA
Paper: Mixed Membership Word Embeddings for Computational Social Science
Word embedding models, which learn to encode dictionary words with vector space representations, have been shown to be valuable for a variety of natural language processing (NLP) tasks such as statistical machine translation BIBREF2 , part-of-speech tagging, chunking, and named entity recogition BIBREF3 , as they provide a more nuanced representation of words than a simple indicator vector into a dictionary. These models follow a long line of research in data-driven semantic representations of text, including latent semantic analysis BIBREF4 and its probabilistic extensions BIBREF5 , BIBREF6 . In particular, topic models BIBREF7 have found broad applications in computational social science BIBREF8 , BIBREF9 and the digital humanities BIBREF10 , where interpretable representations reveal meaningful insights. Despite widespread success at NLP tasks, word embeddings have not yet supplanted topic models as the method of choice in computational social science applications. I speculate that this is due to two primary factors: 1) a perceived reliance on big data, and 2) a lack of interpretability. In this work, I develop new models to address both of these limitations.
Question: What supervised learning tasks are attempted with these representations?
Paper: Mixed Membership Word Embeddings for Computational Social Science
In this section, I provide the necessary background on word embeddings, as well as on topic models and mixed membership models. Traditional language models aim to predict words given the contexts that they are found in, thereby forming a joint probabilistic model for sequences of words in a language. BIBREF19 developed improved language models by using distributed representations BIBREF20 , in which words are represented by neural network synapse weights, or equivalently, vector space embeddings.
Question: What is MRR?
Paper: Mixed Membership Word Embeddings for Computational Social Science
To design an interpretable word embedding model for small corpora, we identify novel connections between word embeddings and topic models, and adapt advances from topic modeling. Following the distributional hypothesis BIBREF23 , the skip-gram's word embeddings parameterize discrete probability distributions over words INLINEFORM0 which tend to co-occur, and tend to be semantically coherent – a property leveraged by the Gaussian LDA model of BIBREF21 . This suggests that these discrete distributions can be reinterpreted as topics INLINEFORM1 . We thus reinterpret the skip-gram as a parameterization of a certain supervised naive Bayes topic model (Table TABREF2 , top-right). In this topic model, input words INLINEFORM2 are fully observed “cluster assignments,” and the words in INLINEFORM3 's contexts are a “document.” The skip-gram differs from this supervised topic model only in the parameterization of the “topics” via word vectors which encode the distributions with a log-bilinear model. Note that although the skip-gram is discriminative, in the sense that it does not jointly model the input words INLINEFORM4 , we are here equivalently interpreting it as encoding a “conditionally generative” process for the context given the words, in order to develop probabilistic models that extend the skip-gram.
Question: Which techniques for word embeddings and topic models are used?
Paper: Mixed Membership Word Embeddings for Computational Social Science
The goals of our experiments were to study the relative merits of big data and domain-specific small data, to validate the proposed methods, and to study their applicability for computational social science research.
Question: Why is big data not appropriate for this task?
Answer: Training embeddings from small-corpora can increase the performance of some tasks
Paper: Mixed Membership Word Embeddings for Computational Social Science
I first measured the effectiveness of the embeddings at the skip-gram's training task, predicting context words INLINEFORM0 given input words INLINEFORM1 . This task measures the methods' performance for predictive language modeling. I used four datasets of sociopolitical, scientific, and literary interest: the corpus of NIPS articles from 1987 – 1999 ( INLINEFORM2 million), the U.S. presidential state of the Union addresses from 1790 – 2015 ( INLINEFORM3 ), the complete works of Shakespeare ( INLINEFORM4 ; this version did not contain the Sonnets), and the writings of black scholar and activist W.E.B. Du Bois, as digitized by Project Gutenberg ( INLINEFORM5 ). For each dataset, I held out 10,000 INLINEFORM6 pairs uniformly at random, where INLINEFORM7 , and aimed to predict INLINEFORM8 given INLINEFORM9 (and optionally, INLINEFORM10 ). Since there are a large number of classes, I treat this as a ranking problem, and report the mean reciprocal rank. The experiments were repeated and averaged over 5 train/test splits.
Question: What is an example of a computational social science NLP task?
Answer: Visualization of State of the union addresses
Paper: Multimodal Differential Network for Visual Question Generation
To understand the progress towards multimedia vision and language understanding, a visual Turing test was proposed by BIBREF0 that was aimed at visual question answering BIBREF1 . Visual Dialog BIBREF2 is a natural extension for VQA. Current dialog systems as evaluated in BIBREF3 show that when trained between bots, AI-AI dialog systems show improvement, but that does not translate to actual improvement for Human-AI dialog. This is because, the questions generated by bots are not natural (human-like) and therefore does not translate to improved human dialog. Therefore it is imperative that improvement in the quality of questions will enable dialog agents to perform well in human interactions. Further, BIBREF4 show that unanswered questions can be used for improving VQA, Image captioning and Object Classification.
Question: Do they report results only on English datasets?
Paper: Multimodal Differential Network for Visual Question Generation
Generating a natural and engaging question is an interesting and challenging task for a smart robot (like chat-bot). It is a step towards having a natural visual dialog instead of the widely prevalent visual question answering bots. Further, having the ability to ask natural questions based on different contexts is also useful for artificial agents that can interact with visually impaired people. While the task of generating question automatically is well studied in NLP community, it has been relatively less studied for image-related natural questions. This is still a difficult task BIBREF5 that has gained recent interest in the community.
Question: What were the previous state of the art benchmarks?
Paper: Multimodal Differential Network for Visual Question Generation
In this section, we clarify the basis for our approach of using exemplars for question generation. To use exemplars for our method, we need to ensure that our exemplars can provide context and that our method generates valid exemplars.
Question: How/where are the natural question generated?
Paper: Multimodal Differential Network for Visual Question Generation
The task in visual question generation (VQG) is to generate a natural language question INLINEFORM0 , for an image INLINEFORM1 . We consider a set of pre-generated context INLINEFORM2 from image INLINEFORM3 . We maximize the conditional probability of generated question given image and context as follows: DISPLAYFORM0 
Question: What is the input to the differential network?
Paper: Multimodal Differential Network for Visual Question Generation
The proposed Multimodal Differential Network (MDN) consists of a representation module and a joint mixture module.
Question: How do the authors define a differential network?