z-uo/qasper-squad huggingface dataset:
Paper: VAIS Hate Speech Detection System: A Deep Learning based Approach for System Combination
Content in the dataset that provided in this HSD task is very diverse. Words having the same meaning were written in various types (teen code, non tone, emojis,..) depending on the style of users. Dataset was crawled from various sources with multiple text encodes. In order to make it easy for training, all types of encoding need to be unified. This cleaning module will be used in two processes: cleaning data before training and cleaning input in inferring phase. Following is the data processing steps that we use:
Question: What is private dashboard?
Answer: Private dashboard is leaderboard where competitors can see results after competition is finished - on hidden part of test set (private test set).
Paper: VAIS Hate Speech Detection System: A Deep Learning based Approach for System Combination
Social comment dataset has high variety, the core idea is using multiple model architectures to handle data in many viewpoints. In our system, we use five different model architectures combining many types of CNN, and RNN. Each model will use some types of word embedding or handle directly sentence embedding to achieve the best general result. Source code of five models is extended from the GitHub repository
Question: What is public dashboard?
Answer: Public dashboard where competitors can see their results during competition, on part of the test set (public test set).
Paper: VAIS Hate Speech Detection System: A Deep Learning based Approach for System Combination
Ensemble methods is a machine learning technique that combines several base models in order to produce one optimal predictive model. Have the main three types of ensemble methods including Bagging, Boosting and Stacking. In this system, we use the Stacking method. In this method, the output of each model is not only class id but also the probability of each class in the set of three classes. This probability will become a feature for the ensemble model. The stacking ensemble model here is a simple full-connection model with input is all of probability that output from sub-model. The output is the probability of each class.
Question: What dataset do they use?
Answer: They used Wiki Vietnamese language and Vietnamese newspapers to pretrain embeddings and dataset provided in HSD task to train model (details not mentioned in paper).
Paper: Yoga-Veganism: Correlation Mining of Twitter Health Data
The main motivation of this work has been started with a question "What do people do to maintain their health?"– some people do balanced diet, some do exercise. Among diet plans some people maintain vegetarian diet/vegan diet, among exercises some people do swimming, cycling or yoga. There are people who do both. If we want to know the answers of the following questions– "How many people follow diet?", "How many people do yoga?", "Does yogi follow vegetarian/vegan diet?", may be we could ask our acquainted person but this will provide very few intuition about the data. Nowadays people usually share their interests, thoughts via discussions, tweets, status in social media (i.e. Facebook, Twitter, Instagram etc.). It's huge amount of data and it's not possible to go through all the data manually. We need to mine the data to get overall statistics and then we will also be able to find some interesting correlation of data.
Question: Do the authors report results only on English data?
Paper: Yoga-Veganism: Correlation Mining of Twitter Health Data
Tweet messages are retrieved from the Twitter source by utilizing the Twitter API and stored in Kafka topics. The Producer API is used to connect the source (i.e. Twitter) to any Kafka topic as a stream of records for a specific category. We fetch data from a source (Twitter), push it to a message queue, and consume it for further analysis. Fig. FIGREF2 shows the overview of Twitter data collection using Kafka.
Question: What other interesting correlations are observed?
Paper: Joint Learning of Sentence Embeddings for Relevance and Entailment
Let us consider the goal of building machine reasoning systems based on knowledge from fulltext data like encyclopedic articles, scientific papers or news articles. Such machine reasoning systems, like humans researching a problem, must be able to recover evidence from large amounts of retrieved but mostly irrelevant information and judge the evidence to decide the answer to the question at hand.
Question: what were the baselines?
Paper: Joint Learning of Sentence Embeddings for Relevance and Entailment
Formally, the Hypothesis Evaluation task is to build a function INLINEFORM0 , where INLINEFORM1 is a binary label (no towards yes) and INLINEFORM2 is a hypothesis instance in the form of question text INLINEFORM3 and a set of INLINEFORM4 evidence texts INLINEFORM5 as extracted from an evidence-carrying corpus.
Question: what is the state of the art for ranking mc test answers?
Paper: Joint Learning of Sentence Embeddings for Relevance and Entailment
Our main aim is to propose a solution to the Argus Task, where the Argus system BIBREF7 BIBREF5 is to automatically analyze and answer questions in the context of the Augur prediction market platform. In a prediction market, users pose questions about future events whereas others bet on the yes or no answer, with the assumption that the bet price reflects the real probability of the event. At a specified moment (e.g. after the date of a to-be-predicted sports match), the correct answer is retroactively determined and the bets are paid off. At a larger volume of questions, determining the bet results may present a significant overhead for running of the market. This motivates the Argus system, which should partially automate this determination — deciding questions related to recent events based on open news sources.
Question: what is the size of the introduced dataset?
Answer: 2427
Paper: Joint Learning of Sentence Embeddings for Relevance and Entailment
The AI2 Elementary School Science Questions (no-diagrams variant) released by the Allen Institute cover 855 basic four-choice questions regarding high school science and follows up to the Allen AI Science Kaggle challenge. The vocabulary includes scientific jargon and named entities, and many questions are not factoid, requiring real-world reasoning or thought experiments.
Question: what datasets did they use?
Paper: MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learning
In recent years, Transformer has been remarkably adept at sequence learning tasks like machine translation BIBREF0, BIBREF1, text classification BIBREF2, BIBREF3, language modeling BIBREF4, BIBREF5, etc. It is solely based on an attention mechanism that captures global dependencies between input tokens, dispensing with recurrence and convolutions entirely. The key idea of the self-attention mechanism is updating token representations based on a weighted sum of all input representations.
Question: What evaluation metric is used?
Paper: MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learning
Like other sequence-to-sequence models, MUSE also adopts an encoder-decoder framework. The encoder takes a sequence of word embeddings $(x_1, \cdots , x_n)$ as input where $n$ is the length of input. It transfers word embeddings to a sequence of hidden representation ${z} = (z_1, \cdots , z_n)$. Given ${z}$, the decoder is responsible for generating a sequence of text $(y_1, \cdots , y_m)$ token by token.
Question: What datasets are used?
Paper: MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learning
Self-attention is responsible for learning representations of global context. For a given input sequence $X$, it first projects $X$ into three representations, key $K$, query $Q$, and value $V$. Then, it uses a self-attention mechanism to get the output representation:
Question: What are three main machine translation tasks?
Paper: MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learning
We introduce convolution operations into MUSE to capture local context. To learn contextual sequence representations in the same hidden space, we choose depth-wise convolution BIBREF9 (we denote it as DepthConv in the experiments) as the convolution operation because it includes two separate transformations, namely, point-wise projecting transformation and contextual transformation. It is because that original convolution operator is not separable, but DepthConv can share the same point-wise projecting transformation with self-attention mechanism. We choose dynamic convolution BIBREF10, the best variant of DepthConv, as our implementation.
Question: How big is improvement in performance over Transformers?
Paper: Aspect Term Extraction with History Attention and Selective Transformation
Aspect-Based Sentiment Analysis (ABSA) involves detecting opinion targets and locating opinion indicators in sentences in product review texts BIBREF0 . The first sub-task, called Aspect Term Extraction (ATE), is to identify the phrases targeted by opinion indicators in review sentences. For example, in the sentence “I love the operating system and preloaded software”, the words “operating system” and “preloaded software” should be extracted as aspect terms, and the sentiment on them is conveyed by the opinion word “love”. According to the task definition, for a term/phrase being regarded as an aspect, it should co-occur with some “opinion words” that indicate a sentiment polarity on it BIBREF1 .
Question: How do they determine the opinion summary?
Paper: Aspect Term Extraction with History Attention and Selective Transformation
Given a sequence INLINEFORM0 of INLINEFORM1 words, the ATE task can be formulated as a token/word level sequence labeling problem to predict an aspect label sequence INLINEFORM2 , where each INLINEFORM3 comes from a finite label set INLINEFORM4 which describes the possible aspect labels. As shown in the example below:
Question: Do they explore how useful is the detection history and opinion summary?
Paper: Aspect Term Extraction with History Attention and Selective Transformation
As shown in Figure FIGREF3 , our model contains two key components, namely Truncated History-Attention (THA) and Selective Transformation Network (STN), for capturing aspect detection history and opinion summary respectively. THA and STN are built on two LSTMs that generate the initial word representations for the primary ATE task and the auxiliary opinion detection task respectively. THA is designed to integrate the information of aspect detection history into the current aspect feature to generate a new history-aware aspect representation. STN first calculates a new opinion representation conditioned on the current aspect candidate. Then, we employ a bi-linear attention network to calculate the opinion summary as the weighted sum of the new opinion representations, according to their associations with the current aspect representation. Finally, the history-aware aspect representation and the opinion summary are concatenated as features for aspect prediction of the current time step.
Question: Which dataset(s) do they use to train the model?
Paper: Aspect Term Extraction with History Attention and Selective Transformation
All the components in the proposed framework are differentiable. Thus, our framework can be efficiently trained with gradient methods. We use the token-level cross-entropy error between the predicted distribution INLINEFORM0 ( INLINEFORM1 ) and the gold distribution INLINEFORM2 as the loss function: DISPLAYFORM0 
Question: By how much do they outperform state-of-the-art methods?
Paper: Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset
Voice-based “personal assistants" such as Apple's SIRI, Microsoft's Cortana, Amazon Alexa, and the Google Assistant have finally entered the mainstream. This development is generally attributed to major breakthroughs in speech recognition and text-to-speech (TTS) technologies aided by recent progress in deep learning BIBREF0, exponential gains in compute power BIBREF1, BIBREF2, and the ubiquity of powerful mobile devices. The accuracy of machine learned speech recognizers BIBREF3 and speech synthesizers BIBREF4 are good enough to be deployed in real-world products and this progress has been driven by publicly available labeled datasets. However, conspicuously absent from this list is equal progress in machine learned conversational natural language understanding (NLU) and generation (NLG). The NLU and NLG components of dialog systems starting from the early research work BIBREF5 to the present commercially available personal assistants largely rely on rule-based systems. The NLU and NLG systems are often carefully programmed for very narrow and specific cases BIBREF6, BIBREF7. General understanding of natural spoken behaviors across multiple dialog turns, even in single task-oriented situations, is by most accounts still a long way off. In this way, most of these products are very much hand crafted, with inherent constraints on what users can say, how the system responds and the order in which the various subtasks can be completed. They are high precision but relatively low coverage. Not only are such systems unscalable, but they lack the flexibility to engage in truly natural conversation.
Question: What is the average number of turns per dialog?
Paper: Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset
BIBREF14 discuss the major features and differences among the existing offerings in an exhaustive and detailed survey of available corpora for data driven learning of dialog systems. One important distinction covered is that of human-human vs. human-machine dialog data, each having its advantages and disadvantages. Many of the existing task-based datasets have been generated from deployed dialog systems such as the Let’s Go Bus Information System BIBREF15 and the various Dialog State Tracking Challenges (DSTCs) BIBREF16. However, it is doubtful that new data-driven systems built with this type of corpus would show much improvement since they would be biased by the existing system and likely mimic its limitations BIBREF17. Since the ultimate goal is to be able to handle complex human language behaviors, it would seem that human-human conversational data is the better choice for spoken dialog system development BIBREF13. However, learning from purely human-human based corpora presents challenges of its own. In particular, human conversation has a different distribution of understanding errors and exhibits turn-taking idiosyncrasies which may not be well suited for interaction with a dialog system BIBREF17, BIBREF14.
Question: What baseline models are offered?
Paper: Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset
The WOz framework, first introduced by BIBREF12 as a methodology for iterative design of natural language interfaces, presents a more effective approach to human-human dialog collection. In this setup, users are led to believe they are interacting with an automated assistant but in fact it is a human behind the scenes that controls the system responses. Given the human-level natural language understanding, users quickly realize they can comfortably and naturally express their intent rather than having to modify behaviors as is normally the case with a fully automated assistant. At the same time, the machine-oriented context of the interaction, i.e. the use of TTS and slower turn taking cadence, prevents the conversation from becoming fully fledged, overly complex human discourse. This creates an idealized spoken environment, revealing how users would openly and candidly express themselves with an automated assistant that provided superior natural language understanding.
Question: Which six domains are covered in the dataset?
Paper: Using word embeddings to improve the discriminability of co-occurrence text networks
The ability to construct complex and diverse linguistic structures is one of the main features that set us apart from all other species. Despite its ubiquity, some language aspects remain unknown. Topics such as language origin and evolution have been studied by researchers from diverse disciplines, including Linguistic, Computer Science, Physics and Mathematics BIBREF0, BIBREF1, BIBREF2. In order to better understand the underlying language mechanisms and universal linguistic properties, several models have been developed BIBREF3, BIBREF4. A particular language representation regards texts as complex systems BIBREF5. Written texts can be considered as complex networks (or graphs), where nodes could represent syllables, words, sentences, paragraphs or even larger chunks BIBREF5. In such models, network edges represent the proximity between nodes, e.g. the frequency of the co-occurrence of words. Several interesting results have been obtained from networked models, such as the explanation of Zipf's Law as a consequence of the least effort principle and theories on the nature of syntactical relationships BIBREF6, BIBREF7.
Question: What other natural processing tasks authors think could be studied by using word embeddings?
Paper: Using word embeddings to improve the discriminability of co-occurrence text networks
Complex networks have been used in a wide range of fields, including in Social Sciences BIBREF13, Neuroscience BIBREF14, Biology BIBREF15, Scientometry BIBREF16 and Pattern Recognition BIBREF17, BIBREF18, BIBREF19, BIBREF20. In text analysis, networks are used to uncover language patterns, including the origins of the ever present Zipf's Law BIBREF21 and the analysis of linguistic properties of natural and unknown texts BIBREF22, BIBREF23. Applications of network science in text mining and text classification encompasses applications in semantic analysis BIBREF24, BIBREF25, BIBREF26, BIBREF27, authorship attribution BIBREF28, BIBREF29 and stylometry BIBREF28, BIBREF30, BIBREF31. Here we focus in the stylometric analysis of texts using complex networks.
Question: What is the reason that traditional co-occurrence networks fail in establishing links between similar words whenever they appear distant in the text?
Paper: Using word embeddings to improve the discriminability of co-occurrence text networks
To represent texts as networks, we used the so-called word adjacency network representation BIBREF35, BIBREF28, BIBREF32. Typically, before creating the networks, the text is pre-processed. An optional pre-processing step is the removal of stopwords. This step is optional because such words include mostly article and prepositions, which may be artlessly represented by network edges. However, in some applications – including the authorship attribution task – stopwords (or function words) play an important role in the stylistic characterization of texts BIBREF32. A list of stopwords considered in this study is available in the Supplementary Information.
Question: Do the use word embeddings alone or they replace some previous features of the model with word embeddings?
Answer: They use it as addition to previous model - they add new edge between words if word embeddings are similar.
Paper: Using word embeddings to improve the discriminability of co-occurrence text networks
In Section SECREF13, we probe whether the inclusion of virtual edges is able to improve the performance of the traditional co-occurrence network-based classification in a usual stylometry task. While the focus of this paper is not to perform a systematic analysis of different methods comprising the adopted network, we consider two variations in the adopted methodology. In Section SECREF19, we consider the use of stopwords and the adoption of a local thresholding process to establish different criteria to create new virtual edges.
Question: On what model architectures are previous co-occurence networks based?
Paper: e-SNLI-VE-2.0: Corrected Visual-Textual Entailment with Natural Language Explanations
Inspired by textual entailment BIBREF0, Xie BIBREF1 introduced the visual-textual entailment (VTE) task, which considers semantic entailment between a premise image and a textual hypothesis. Semantic entailment consists in determining if the hypothesis can be concluded from the premise, and assigning to each pair of (premise image, textual hypothesis) a label among entailment, neutral, and contradiction. In Figure FIGREF3, the label for the first image-sentence pair is entailment, because the hypothesis states that “a bunch of people display different flags”, which can be clearly derived from the image. On the contrary, the second image-sentence pair is labelled as contradiction, because the hypothesis stating that “people [are] running a marathon” contradicts the image with static people.
Question: Is model explanation output evaluated, what metric was used?
Paper: e-SNLI-VE-2.0: Corrected Visual-Textual Entailment with Natural Language Explanations
The goal of VTE is to determine if a textual hypothesis $H_{text}$ can be concluded, given the information in a premise image $P_{image}$ BIBREF1. There are three possible labels:
Question: How many annotators are used to write natural language explanations to SNLI-VE-2.0?
Paper: e-SNLI-VE-2.0: Corrected Visual-Textual Entailment with Natural Language Explanations
In this work, we only collect new labels for the neutral pairs in the validation and test sets of SNLI-VE. While the procedure of re-annotation is generic, we limit our re-annotation to these splits as a first step to verify the difference in performance that current models have when evaluated on the corrected test set as well as the effect of model selection on the corrected validation set. We leave for future work re-annotation of the training set, which would likely lead to training better VTE models. We also chose not to re-annotate entailment and contradiction classes, as their error rates are much lower ($<$1% as reported by Vu BIBREF3).
Question: How many natural language explanations are human-written?
Answer: Totally 6980 validation and test image-sentence pairs have been corrected.
Paper: e-SNLI-VE-2.0: Corrected Visual-Textual Entailment with Natural Language Explanations
Since we decreased the error rate of labels in the validation and test set, we are interested in the performance of a VTE model when using the corrected sets.
Question: How much is performance difference of existing model between original and corrected corpus?
Paper: e-SNLI-VE-2.0: Corrected Visual-Textual Entailment with Natural Language Explanations
To tackle SNLI-VE, Xie BIBREF1 used EVE (for “Explainable Visual Entailment”), a modified version of the BUTD architecture, the winner of the Visual Question Answering (VQA) challenge in 2017 BIBREF5. Since the EVE implementation is not available at the time of this work, we used the original BUTD architecture, with the same hyperparameters as reported in BIBREF1.
Question: What is the class with highest error rate in SNLI-VE?
Paper: An Analysis of Word2Vec for the Italian Language
In order to make human language comprehensible to a computer, it is obviously essential to provide some word encoding. The simplest approach is the one-hot encoding, where each word is represented by a sparse vector with dimension equal to the vocabulary size. In addition to the storage need, the main problem of this representation is that any concept of word similarity is completely ignored (each vector is orthogonal and equidistant from each other). On the contrary, the understanding of natural language cannot be separated from the semantic knowledge of words, which conditions a different closeness between them. Indeed, the semantic representation of words is the basic problem of Natural Language Processing (NLP). Therefore, there is a necessary need to code words in a space that is linked to their meaning, in order to facilitate a machine in potential task of “understanding" it. In particular, starting from the seminal work BIBREF0, words are usually represented as dense distributed vectors that preserve their uniqueness but, at the same time, are able to encode the similarities.
Question: What is the dataset used as input to the Word2Vec algorithm?
Answer: Italian Wikipedia and Google News extraction producing final vocabulary of 618224 words