z-uo/qasper-squad huggingface dataset:
Paper: Common Voice: A Massively-Multilingual Speech Corpus
The Common Voice project is a response to the current state of affairs in speech technology, in which training data is either prohibitively expensive or unavailable for most languages BIBREF0. We believe that speech technology (like all technology) should be decentralized and open, and the Common Voice project achieves this goal via a mix of community building, open source tooling, and a permissive licensing scheme. The corpus is designed to organically scale to new languages as community members use the provided tools to translate the interface, submit text sentences, and finally record and validate voices in their new language . The project was started with an initial focus on English in July 2017 and then in June 2018 was made available for any language.
Question: What crowdsourcing platform is used for data collection and data validation?
Paper: Common Voice: A Massively-Multilingual Speech Corpus
Some notable multilingual speech corpora include VoxForge BIBREF1, Babel BIBREF2, and M-AILABS BIBREF3. Even though the Babel corpus contains high-quality data from 22 minority languages, it is not released under an open license. VoxForge is most similar to Common Voice in that it is community-driven, multilingual (17 languages), and released under an open license (GNU General Public License). However, the VoxForge does not have a sustainable data collection pipeline compared to Common Voice, and there is no data validation step in place. M-AILABS data contains 9 language varieties with a modified BSD 3-Clause License, however there is no community-driven aspect. Common Voice is a sustainable, open alternative to these projects which allows for collection of minority and majority languages alike.
Question: How is validation of the data performed?
Paper: Common Voice: A Massively-Multilingual Speech Corpus
The data presented in this paper was collected and validated via Mozilla's Common Voice initiative. Using either the Common Voice website or iPhone app, contributors record their voice by reading sentences displayed on the screen (see Figure (FIGREF5)). The recordings are later verified by other contributors using a simple voting system. Shown in Figure (FIGREF6), this validation interface has contributors mark $<$audio,transcript$>$ pairs as being either correct (up-vote) or incorrect (down-vote).
Question: Is audio data per language balanced in dataset?
Paper: Weakly Supervised Domain Detection
Text classification is a fundamental task in Natural Language processing which has been found useful in a wide spectrum of applications ranging from search engines enabling users to identify content on websites, sentiment and social media analysis, customer relationship management systems, and spam detection. Over the past several years, text classification has been predominantly modeled as a supervised learning problem (e.g., BIBREF0 , BIBREF1 , BIBREF2 ) for which appropriately labeled data must be collected. Such data is often domain-dependent (i.e., covering specific topics such as those relating to “Business” or “Medicine”) and a classifier trained using data from one domain is likely to perform poorly on another. For example, the phrase “the mouse died quickly” may indicate negative sentiment in a customer review describing the hand-held pointing device or positive sentiment when describing a laboratory experiment performed on a rodent. The ability to handle a wide variety of domains has become more pertinent with the rise of data-hungry machine learning techniques like neural networks and their application to a plethora of textual media ranging from news articles to twitter, blog posts, medical journals, Reddit comments, and parliamentary debates BIBREF0 , BIBREF3 , BIBREF4 , BIBREF5 .
Question: What is the performance of their model?
Paper: Weakly Supervised Domain Detection
Our work lies at the intersection of multiple research areas, including domain adaptation, representation learning, multiple instance learning, and topic modeling. We review related work below.
Question: Which text genres did they experiment with?
Paper: Weakly Supervised Domain Detection
We formulate domain detection as a multilabel learning problem. Our model is trained on samples of document-label pairs. Each document consists of INLINEFORM0 sentences INLINEFORM1 and is associated with discrete labels INLINEFORM2 . In this work, domain labels are not annotated manually but extrapolated from Wikipedia (see Section SECREF6 for details). In a non-MIL framework, a model typically learns to predict document labels by directly conditioning on its sentence representations INLINEFORM3 or their aggregate. In contrast, INLINEFORM4 under MIL is a learned function INLINEFORM5 of latent instance-level labels, i.e., INLINEFORM6 . A MIL classifier will therefore first produce domain scores for all instances (aka sentences), and then learn to integrate instance scores into a bag (aka document) prediction.
Question: What domains are detected in this paper?
Answer: Answer with content missing: (Experimental setup not properly rendered) In our experiments we used seven target domains: “Business and Commerce” (BUS), “Government and Politics” (GOV), “Physical and Mental Health” (HEA), “Law and Order” (LAW),
“Lifestyle” (LIF), “Military” (MIL), and “General Purpose” (GEN). Exceptionally, GEN does
not have a natural root category.
Paper: Harry Potter and the Action Prediction Challenge from Natural Language
Natural language processing (nlp) has achieved significant advances in reading comprehension tasks BIBREF0 , BIBREF1 . These are partially due to embedding methods BIBREF2 , BIBREF3 and neural networks BIBREF4 , BIBREF5 , BIBREF6 , but also to the availability of new resources and challenges. For instance, in cloze-form tasks BIBREF7 , BIBREF8 , the goal is to predict the missing word given a short context. weston2015towards presented baBI, a set of proxy tasks for reading comprenhension. In the SQuAD corpus BIBREF9 , the aim is to answer questions given a Wikipedia passage. 2017arXiv171207040K introduce NarrativeQA, where answering the questions requires to process entire stories. In a related line, 2017arXiv171011601F use fictional crime scene investigation data, from the CSI series, to define a task where the models try to answer the question: ‘who committed the crime?’.
Question: Why do they think this task is hard?  What is the baseline performance?
Answer: 1. there may be situations where more than one action is reasonable, and also because writers tell a story playing with elements such as surprise or uncertainty.
2. Macro F1 = 14.6 (MLR, length 96 snippet)
Weighted F1 = 31.1 (LSTM, length 128 snippet)
Paper: Harry Potter and the Action Prediction Challenge from Natural Language
To build an action prediction corpus, we need to: (1) consider the set of actions, and (2) collect data where these occur. Data should come from different users, to approximate a real natural language task. Also, it needs to be annotated, determining that a piece of text ends up triggering an action. These tasks are however time consuming, as they require annotators to read vast amounts of large texts. In this context, machine comprehension resources usually establish a compromise between their complexity and the costs of building them BIBREF7 , BIBREF13 .
Question: Isn't simple word association enough to predict the next spell?
Paper: Harry Potter and the Action Prediction Challenge from Natural Language
We rely on an intuitive idea that uses transcripts from the Harry Potter world to build up a corpus for textual action prediction. The domain has a set of desirable properties to evaluate reading comprehension systems, which we now review.
Question: Do they literally just treat this as "predict the next spell that appears in the text"?
Paper: Harry Potter and the Action Prediction Challenge from Natural Language
The number of occurrences of spells in the original Harry Potter books is small (432 occurrences), which makes it difficult to train and test a machine learning model. However, the amount of available fan fiction for this saga allows to create a large corpus. For hpac, we used fan fiction (and only fan fiction texts) from https://www.fanfiction.net/book/Harry-Potter/ and a version of the crawler by milli2016beyond. We collected Harry Potter stories written in English and marked with the status ‘completed’. From these we extracted a total of 82 836 spell occurrences, that we used to obtain the scene descriptions. Table 2 details the statistics of the corpus (see also Appendix "Corpus distribution" ). Note that similar to Twitter corpora, fan fiction stories can be deleted over time by users or admins, causing losses in the dataset.
Question: How well does a simple bag-of-words baseline do?
Paper: Finding Dominant User Utterances And System Responses in Conversations
There are several existing works that focus on modelling conversation using prior human to human conversational data BIBREF0 , BIBREF1 , BIBREF2 . BIBREF3 models the conversation from pairs of consecutive tweets. Deep learning based approaches have also been used to model the dialog in an end to end manner BIBREF4 , BIBREF5 . Memory networks have been used by Bordes et al Bor16 to model goal based dialog conversations. More recently, deep reinforcement learning models have been used for generating interactive and coherent dialogs BIBREF6 and negotiation dialogs BIBREF7 .
Question: Do they study frequent user responses to help automate modelling of those?
Paper: Finding Dominant User Utterances And System Responses in Conversations
The notion of adjacency pairs was introduced by Sacks et al SSE74 to formalize the structure of a dialog. Adjacency pairs have been used to analyze the semantics of the dialog in computational linguistics community BIBREF9 . Clustering has been used for different tasks related to conversation. BIBREF10 considers the task of discovering dialog acts by clustering the raw utterances. We aim to obtain the frequent adjacency pairs through clustering.
Question: How do they divide text into utterances?
Paper: Finding Dominant User Utterances And System Responses in Conversations
In this section we describe our approach SimCluster that performs clustering in the two domains simultaneously and ensures that the generated clusters can be aligned with each other. We will describe the model in section SECREF9 and the algorithm in Section SECREF11 .
Question: Do they use the same distance metric for both the SimCluster and K-means algorithm?
Paper: Finding Dominant User Utterances And System Responses in Conversations
We consider a problem setting where we are given a collection of pairs of consecutive utterances, with vector representations INLINEFORM0 where INLINEFORM1 s are in speaker 1's domain and INLINEFORM2 s are in speaker 2's domain. We need to simultaneously cluster the utterances in their respective domains to minimize the variations within each domain and also ensure that the clusters for both domains are close together.
Question: How do they generate the synthetic dataset?
Answer: using generative process
Paper: Ask the Right Questions: Active Question Reformulation with Reinforcement Learning
Web and social media have become primary sources of information. Users' expectations and information seeking activities co-evolve with the increasing sophistication of these resources. Beyond navigation, document retrieval, and simple factual question answering, users seek direct answers to complex and compositional questions. Such search sessions may require multiple iterations, critical assessment, and synthesis BIBREF0 .
Question: how are multiple answers from multiple reformulated questions aggregated?
Paper: Seeing Things from a Different Angle: Discovering Diverse Perspectives about Claims
Understanding most nontrivial claims requires insights from various perspectives. Today, we make use of search engines or recommendation systems to retrieve information relevant to a claim, but this process carries multiple forms of bias. In particular, they are optimized relative to the claim (query) presented, and the popularity of the relevant documents returned, rather than with respect to the diversity of the perspectives presented in them or whether they are supported by evidence.
Question: What is the average length of the claims?
Answer: Average claim length is 8.9 tokens.
Paper: Seeing Things from a Different Angle: Discovering Diverse Perspectives about Claims
In this section we provide a closer look into the challenge and propose a collection of tasks that move us closer to substantiated perspective discovery. To clarify our description we use to following notation. Let INLINEFORM0 indicate a target claim of interest (for example, the claims INLINEFORM1 and INLINEFORM2 in Figure FIGREF6 ). Each claim INLINEFORM3 is addressed by a collection of perspectives INLINEFORM4 that are grouped into clusters of equivalent perspectives. Additionally, each perspective INLINEFORM5 is supported, relative to INLINEFORM6 , by at least one evidence paragraph INLINEFORM7 , denoted INLINEFORM8 .
Question: What debate websites did they look at?
Paper: Seeing Things from a Different Angle: Discovering Diverse Perspectives about Claims
In this section we describe a multi-step process, constructed with detailed analysis, substantial refinements and multiple pilots studies.
Question: What crowdsourcing platform did they use?
Paper: Seeing Things from a Different Angle: Discovering Diverse Perspectives about Claims
We now provide a brief summary of [wave]390P[wave]415e[wave]440r[wave]465s[wave]485p[wave]525e[wave]535c[wave]595t[wave]610r[wave]635u[wave]660m. The dataset contains about INLINEFORM0 claims with a significant length diversity (Table TABREF19 ). Additionally, the dataset comes with INLINEFORM1 perspectives, most of which were generated through paraphrasing (step 2b). The perspectives which convey the same point with respect to a claim are grouped into clusters. On average, each cluster has a size of INLINEFORM2 which shows that, on average, many perspectives have equivalents. More granular details are available in Table TABREF19 .
Question: Which machine baselines are used?
Paper: Seeing Things from a Different Angle: Discovering Diverse Perspectives about Claims
We perform a closer investigation of the abilities required to solve the stance classification task. One of the authors went through a random subset of claim-perspectives pairs and annotated each with the abilities required in determining their stances labels. We follow the common definitions used in prior work BIBREF37 , BIBREF38 . The result of this annotation is depicted in Figure FIGREF24 . As can be seen, the problem requires understanding of common-sense, i.e., an understanding that is commonly shared among humans and rarely gets explicitly mentioned in the text. Additionally, the task requires various types of coreference understanding, such as event coreference and entity coreference.
Question: What challenges are highlighted?
Paper: Seeing Things from a Different Angle: Discovering Diverse Perspectives about Claims
In this section we provide empirical analysis to address the tasks. We create a split of 60%/15%/25% of the data train/dev/test. In order to make sure our baselines are not overfitting to the keywords of each topic (the “topic” annotation from Section SECREF20 ), we make sure to have claims with the same topic fall into the same split.
Question: What debate topics are included in the dataset?
Answer: Ethics, Gender, Human rights, Sports, Freedom of Speech, Society, Religion, Philosophy, Health, Culture, World, Politics, Environment, Education, Digital Freedom, Economy, Science and Law
Paper: Pay More Attention - Neural Architectures for Question-Answering
Enabling machines to understand natural language is one of the key challenges to achieve artificially intelligent systems. Asking machines questions and getting a meaningful answer adds value to us since it automatizes knowledge acquisition efforts drastically. Apple's Siri and Amazon's Echo are two such examples of mass market products capable of machine comprehension that has led to a paradigm shift on how consumers' interact with machines.
Question: By how much, the proposed method improves BiDAF and DCN on SQuAD dataset?
Answer: In terms of F1 score, the Hybrid approach improved by 23.47% and 1.39% on BiDAF and DCN respectively. The DCA approach improved by 23.2% and 1.12% on BiDAF and DCN respectively.
Paper: Creating and Characterizing a Diverse Corpus of Sarcasm in Dialogue
Irony and sarcasm in dialogue constitute a highly creative use of language signaled by a large range of situational, semantic, pragmatic and lexical cues. Previous work draws attention to the use of both hyperbole and rhetorical questions in conversation as distinct types of lexico-syntactic cues defining diverse classes of sarcasm BIBREF0 .
Question: Do they report results only on English datasets?
Paper: Creating and Characterizing a Diverse Corpus of Sarcasm in Dialogue
There has been relatively little theoretical work on sarcasm in dialogue that has had access to a large corpus of naturally occurring examples. Gibbs00 analyzes a corpus of 62 conversations between friends and argues that a robust theory of verbal irony must account for the large diversity in form. He defines several subtypes, including rhetorical questions and hyperbole:
Question: What are the linguistic differences between each class?
Answer: Each class has different patterns in adjectives, adverbs and verbs for sarcastic and non-sarcastic classes
Paper: Creating and Characterizing a Diverse Corpus of Sarcasm in Dialogue
We first replicated the pattern-extraction experiments of LukinWalker13 on their dataset using AutoSlog-TS BIBREF13 , a weakly-supervised pattern learner that extracts lexico-syntactic patterns associated with the input data. We set up the learner to extract patterns for both sarcastic and not-sarcastic utterances. Our first discovery is that we can classify not-sarcastic posts with very high precision, ranging between 80-90%.
Question: What simple features are used?
Paper: Creating and Characterizing a Diverse Corpus of Sarcasm in Dialogue
The goal of collecting additional corpora for rhetorical questions and hyperbole is to increase the diversity of the corpus, and to allow us to explore the semantic differences between sarcastic and not-sarcastic utterances when particular lexico-syntactic cues are held constant. We hypothesize that identifying surface-level cues that are instantiated in both sarcastic and not sarcastic posts will force learning models to find deeper semantic cues to distinguish between the classes.
Question: What lexico-syntactic cues are used to retrieve sarcastic utterances?
Paper: Brazilian Lyrics-Based Music Genre Classification Using a BLSTM Network
Music is part of the day-to-day life of a huge number of people, and many works try to understand the best way to classify, recommend, and identify similarities between songs. Among the tasks that involve music classification, genre classification has been studied widely in recent years BIBREF0 since musical genres are the main top-level descriptors used by music dealers and librarians to organize their music collections BIBREF1.
Question: what is the source of the song lyrics?
Paper: Brazilian Lyrics-Based Music Genre Classification Using a BLSTM Network
Several works have been carried out to add textual information to genre and mood classification. Fell and Sporleder BIBREF6 used several handcraft features, such as vocabulary, style, semantics, orientation towards the world, and song structure to obtain performance gains on three different classification tasks: detecting genre, distinguishing the best and the worst songs, and determining the approximate publication time of a song. The experiments in genre classification focused on eight genres: Blues, Rap, Metal, Folk, R&B, Reggae, Country, and Religious. Only lyrics in English were included and they used an SVM with the default settings for the classification.
Question: what genre was the most difficult to classify?
Paper: Brazilian Lyrics-Based Music Genre Classification Using a BLSTM Network
In this chapter we present all the major steps we have taken, from obtaining the dataset to the proposed approach to address the automatic music genre classification problem.
Question: what word embedding techniques did they experiment with?
Paper: Brazilian Lyrics-Based Music Genre Classification Using a BLSTM Network
In order to obtain a large number of Brazilian music lyrics, we created a crawler to navigate into the Vagalume website, extracting, for each musical genre, all the songs by all the listed authors. The implementation of a crawler was necessary because, although the Vagalume site provides an API, it is only for consultation and does not allow obtaining large amounts of data. The crawler was implemented using Scrapy, an open-source and collaborative Python library to extract data from websites.
Question: what genres do they songs fall under?
Answer: Gospel, Sertanejo, MPB, Forró, Pagode, Rock, Samba, Pop, Axé, Funk-carioca, Infantil, Velha-guarda, Bossa-nova and Jovem-guarda
Paper: A Robust Hybrid Approach for Textual Document Classification
Text document classification is an important task for diverse natural language processing based applications. Traditional machine learning approaches mainly focused on reducing dimensionality of textual data to perform classification. This although improved the overall classification accuracy, the classifiers still faced sparsity problem due to lack of better data representation techniques. Deep learning based text document classification, on the other hand, benefitted greatly from the invention of word embeddings that have solved the sparsity problem and researchers’ focus mainly remained on the development of deep architectures. Deeper architectures, however, learn some redundant features that limit the performance of deep learning based solutions. In this paper, we propose a two stage text document classification methodology which combines traditional feature engineering with automatic feature engineering (using deep learning). The proposed methodology comprises a filter based feature selection (FSE) algorithm followed by a deep convolutional neural network. This methodology is evaluated on the two most commonly used public datasets, i.e., 20 Newsgroups data and BBC news data. Evaluation results reveal that the proposed methodology outperforms the state-of-the-art of both the (traditional) machine learning and deep learning based text document classification methodologies with a significant margin of 7.7% on 20 Newsgroups and 6.6% on BBC news datasets.
Question: Is the filter based feature selection (FSE) a form of regularization?
Paper: AvgOut: A Simple Output-Probability Measure to Eliminate Dull Responses
Many modern dialogue generation models use a sequence-to-sequence architecture as their backbone BIBREF0, following its success when applied to Machine Translation (MT) BIBREF1. However, dialogue tasks also have a requirement different from that of MT: the response not only has to be "correct" (coherent and relevant), but also needs to be diverse and informative. However, seq2seq has been reported by many previous works to have low corpus-level diversity BIBREF2, BIBREF3, BIBREF0, BIBREF4, as it tends to generate safe, terse, and uninformative responses, such as "I don't know.". These responses unnecessarily make a dialogue system much less interactive than it should be.
Question: To what other competitive baselines is this approach compared?
Answer: LSTMs with and without attention, HRED, VHRED with and without attention, MMI and Reranking-RL