z-uo/qasper-squad huggingface dataset:
Paper: Multimodal Differential Network for Visual Question Generation
The role of decoder is to predict the probability for a question, given INLINEFORM0 . RNN provides a nice way to perform conditioning on previous state value using a fixed length hidden vector. The conditional probability of a question token at particular time step INLINEFORM1 is modeled using an LSTM as used in machine translation BIBREF38 . At time step INLINEFORM2 , the conditional probability is denoted by INLINEFORM3 , where INLINEFORM4 is the hidden state of the LSTM cell at time step INLINEFORM5 , which is conditioned on all the previously generated words INLINEFORM6 . The word with maximum probability in the probability distribution of the LSTM cell at step INLINEFORM7 is fed as an input to the LSTM cell at step INLINEFORM8 as shown in part 3 of Figure FIGREF4 . At INLINEFORM9 , we are feeding the output of the mixture module to LSTM. INLINEFORM10 are the predicted question tokens for the input image INLINEFORM11 . Here, we are using INLINEFORM12 and INLINEFORM13 as the special token START and STOP respectively. The softmax probability for the predicted question token at different time steps is given by the following equations where LSTM refers to the standard LSTM cell equations: INLINEFORM14 
Question: How do the authors define exemplars?
Paper: CoupleNet: Paying Attention to Couples with Coupled Attention for Relationship Recommendation
The social web has become a common means for seeking romantic companionship, made evident by the wide assortment of online dating sites that are available on the Internet. As such, the notion of relationship recommendation systems is not only interesting but also highly applicable. This paper investigates the possibility and effectiveness of a deep learning based relationship recommendation system. An overarching research question is whether modern artificial intelligence (AI) techniques, given social profiles, can successfully approximate successful relationships and measure the relationship compatibility of two users.
Question: Is this a task other people have worked on?
Paper: CoupleNet: Paying Attention to Couples with Coupled Attention for Relationship Recommendation
This section provides an overview of the main contributions of this work.
Question: Where did they get the data for this project?
Paper: #MeToo on Campus: Studying College Sexual Assault at Scale Using Data Reported on Social Media
Sexual harassment is defined as "bullying or coercion of a sexual nature, or the unwelcome or inappropriate promise of rewards in exchange for sexual favors." In fact, it is an ongoing problem in the U.S., especially within the higher education community. According to the National Sexual Violence Resource Center (NSRVC), one in five women and one in sixteen men are sexually assaulted while they are attending college. In addition to the prevalence of campus sexual harassment, it has been shown to have detrimental effects on student's well-being, including health-related disorders and psychological distress BIBREF0, BIBREF1. However, these studies on college sexual misconduct usually collect data based on questionnaires from a small sample of the college population, which might not be sufficiently substantial to capture the big picture of sexual harassment risk of the entire student body.
Question: Which major geographical regions are studied?
Answer: Northeast U.S, South U.S., West U.S. and Midwest U.S.
Paper: #MeToo on Campus: Studying College Sexual Assault at Scale Using Data Reported on Social Media
Previous works for sexual misconduct in academia and workplace dated back to last few decades, when researchers studied the existence, as well as psychometric and demographic insights regarding this social issue, based on survey and official data BIBREF2, BIBREF3, BIBREF4. However, these methods of gathering data are limited in scale and might be influenced by the psychological and cognitive tendencies of respondents not to provide faithful answers BIBREF5.
Question: How strong is the correlation between the prevalence of the #MeToo movement and official reports [of sexual harassment]?
Answer: 0.9098 correlation
Paper: #MeToo on Campus: Studying College Sexual Assault at Scale Using Data Reported on Social Media
In this study, we limit the sample size to the followers identified as English speakers in the U.S. News Top 200 National Universities. We utilize the Jefferson-Henrique script, a web scraper designed for Twitter to retrieve a total of over 300,000 #MeToo tweets from October 15th, when Alyssa Milano posted the inceptive #MeToo tweet, to November 15th of 2017 to cover a period of a month when the trend was on the rise and attracting mass concerns. Since the lists of the followers of the studied colleges might overlap and many Twitter users tend to reiterate other's tweets, simply putting all the data collected together could create a major redundancy problem. We extract unique users and tweets from the combined result set to generate a dataset of about 60,000 unique tweets, pertaining to 51,104 unique users.
Question: How are the topics embedded in the #MeToo tweets extracted?
Answer: Using Latent Dirichlet Allocation on TF-IDF transformed from the corpus
Paper: #MeToo on Campus: Studying College Sexual Assault at Scale Using Data Reported on Social Media
We pre-process the Twitter textual data to ensure that its lexical items are to a high degree lexically comparable to those of natural language. This is done by performing sentiment-aware tokenization, spell correction, word normalization, segmentation (for splitting hashtags) and annotation. The implemented tokenizer with SentiWordnet corpus BIBREF13 is able to avoid splitting expressions or words that should be kept intact (as one token), and identify most emoticons, emojis, expressions such as dates, currencies, acronyms, censored words (e.g. s**t), etc. In addition, we perform modifications on the extracted tokens. For spelling correction, we compose a dictionary for the most commonly seen abbreviations, censored words and elongated words (for emphasis, e.g. "reallyyy"). The Viterbi algorithm is used for word segmentation, with word statistics (unigrams and bigrams) computed from the NLTK English Corpus to obtain the most probable segmentation posteriors from the unigrams and bigrams probabilities. Moreover, all texts are lower-cased, and URLs, emails and mentioned usernames are replaced with common designated tags so that they would not need to be annotated by the semantic parser.
Question: How many tweets are explored in this paper?
Paper: #MeToo on Campus: Studying College Sexual Assault at Scale Using Data Reported on Social Media
The meta-statistics on the college demographics regarding enrollment, geographical location, private/public categorization and male-to-female ratio are obtained. Furthermore, we acquire the Campus Safety and Security Survey dataset from the official U.S. Department of Education website and use rape-related cases statistic as an attribute to complete the data for our linear regression model. The number of such reported cases by these 200 colleges in 2015 amounts to 2,939.
Question: Which geographical regions correlate to the trend?
Answer: Northeast U.S., West U.S. and South U.S.
Paper: #MeToo on Campus: Studying College Sexual Assault at Scale Using Data Reported on Social Media
We examine other features regarding the characteristics of the studied colleges, which might be significant factors of sexual harassment. Four factual attributes pertaining to the 200 colleges are extracted from the U.S. News Statistics, which consists of Undergraduate Enrollment, Male/Female Ratio, Private/Public, and Region (Northeast, South, West, and Midwest). We also use the normalized rape-related cases count (number of cases reported per student enrolled) from the stated government resource as another attribute to examine the proximity of our dataset to the official one. This feature vector is then fitted in a linear regression to predict the normalized #metoo users count (number of unique users who posted #MeToo tweets per student enrolled) for each individual college.
Question: How many followers did they analyze?
Paper: S-Net: From Answer Extraction to Answer Generation for Machine Reading Comprehension
Machine reading comprehension BIBREF0 , BIBREF1 , which attempts to enable machines to answer questions after reading a passage or a set of passages, attracts great attentions from both research and industry communities in recent years. The release of the Stanford Question Answering Dataset (SQuAD) BIBREF0 and the Microsoft MAchine Reading COmprehension Dataset (MS-MARCO) BIBREF1 provides the large-scale manually created datasets for model training and testing of machine learning (especially deep learning) algorithms for this task. There are two main differences in existing machine reading comprehension datasets. First, the SQuAD dataset constrains the answer to be an exact sub-span in the passage, while words in the answer are not necessary in the passages in the MS-MARCO dataset. Second, the SQuAD dataset only has one passage for a question, while the MS-MARCO dataset contains multiple passages.
Question: What two components are included in their proposed framework?
Answer: evidence extraction and answer synthesis
Paper: S-Net: From Answer Extraction to Answer Generation for Machine Reading Comprehension
Benchmark datasets play an important role in recent progress in reading comprehension and question answering research. BIBREF4 release MCTest whose goal is to select the best answer from four options given the question and the passage. CNN/Daily-Mail BIBREF5 and CBT BIBREF6 are the cloze-style datasets in which the goal is to predict the missing word (often a named entity) in a passage. Different from above datasets, the SQuAD dataset BIBREF0 whose answer can be much longer phrase is more challenging. The answer in SQuAD is a segment of text, or span, from the corresponding reading passage. Similar to the SQuAD, MS-MARCO BIBREF1 is the reading comprehension dataset which aims to answer the question given a set of passages. The answer in MS-MARCO is generated by human after reading all related passages and not necessarily sub-spans of the passages.
Question: Which framework they propose in this paper?
Paper: S-Net: From Answer Extraction to Answer Generation for Machine Reading Comprehension
Following the overview in Figure 1 , our approach consists of two parts as evidence extraction and answer synthesis. The two parts are trained in two stages. The evidence extraction part aims to extract evidence snippets related to the question and passage. The answer synthesis part aims to generate the answer based on the extracted evidence snippets. We propose a multi-task learning framework for the evidence extraction shown in Figure 15 , and use the sequence-to-sequence model with additional features of the start and end positions of the evidence snippet for the answer synthesis shown in Figure 3 .
Question: Why MS-MARCO is different from SQuAD?
Paper: IIIDYT at SemEval-2018 Task 3: Irony detection in English tweets
Sentiment analysis and emotion recognition, as two closely related subfields of affective computing, play a key role in the advancement of artificial intelligence BIBREF0 . However, the complexity and ambiguity of natural language constitutes a wide range of challenges for computational systems.
Question: Did they experiment with pre-training schemes?
Paper: IIIDYT at SemEval-2018 Task 3: Irony detection in English tweets
For the shared task, a balanced dataset of 2,396 ironic and 2,396 non-ironic tweets is provided. The ironic corpus was constructed by collecting self-annotated tweets with the hashtags #irony, #sarcasm and #not. The tweets were then cleaned and manually checked and labeled, using a fine-grained annotation scheme BIBREF3 . The corpus comprises different types of irony:
Question: What were their results on the test set?
Paper: IIIDYT at SemEval-2018 Task 3: Irony detection in English tweets
The goal of the subtask A was to build a binary classification system that predicts if a tweet is ironic or non-ironic. In the following sections, we first describe the dataset provided for the task and our pre-processing pipeline. Later, we lay out the proposed model architecture, our experiments and results.
Question: What is the size of the dataset?
Paper: IIIDYT at SemEval-2018 Task 3: Irony detection in English tweets
Representation learning approaches usually require extensive amounts of data to derive proper results. Moreover, previous studies have shown that initializing representations using random values generally causes the performance to drop. For these reasons, we rely on pre-trained word embeddings as a means of providing the model the adequate setting. We experiment with GloVe BIBREF14 for small sizes, namely 25, 50 and 100. This is based on previous work showing that representation learning models based on convolutional neural networks perform well compared to traditional machine learning methods with a significantly smaller feature vector size, while at the same time preventing over-fitting and accelerates computation (e.g BIBREF2 .
Question: What was the baseline model?
Paper: Asking and Answering Questions to Evaluate the Factual Consistency of Summaries
Automatic summarization aims to produce summaries that are succinct, coherent, relevant, and — crucially — factually correct. Recent progress in conditional text generation has led to models that can generate fluent, topical summaries BIBREF2. However, model-generated summaries frequently contain factual inconsistencies, limiting their applicability BIBREF3.
Question: What models are evaluated with QAGS?
Paper: Asking and Answering Questions to Evaluate the Factual Consistency of Summaries
Standard approaches to evaluating generated text are primarily based on counting $n$-gram overlap. These methods assume access to one or more reference texts, and score a generated summary based on the precision and recall of all reference $n$-grams in the generated summary. We briefly describe the most common metrics in this family, and refer readers to BIBREF9 for further discussion.
Question: Do they use crowdsourcing to collect human judgements?
Paper: Deep Text-to-Speech System with Seq2Seq Model
Traditional text-to-speech (TTS) systems are composed of complex pipelines BIBREF0 , these often include accoustic frontends, duration model, acoustic prediction model and vocoder models. The complexity of the TTS problem coupled with the requirement for deep domain expertise means these systems are often brittle in design and results in un-natural synthesized speech.
Question: Which dataset(s) do they evaluate on?
Paper: Deep Text-to-Speech System with Seq2Seq Model
Neural text-to-speech systems have garnered large research interest in the past 2 years. The first to fully explore this avenue of research was Google's tacotron BIBREF1 system. Their architecture based off the original Seq2Seq framework. In addition to encoder/decoder RNNs from the original Seq2Seq , they also included a bottleneck prenet module termed CBHG, which is composed of sets of 1-D convolution networks followed by highway residual layers. The attention mechanism follows the original Seq2Seq BIBREF7 mechanism (often termed Bahdanau attention). This is the first work to propose training a Seq2Seq model to convert text to mel spectrogram, which can then be converted to audio wav via iterative algorithms such as Griffin Lim BIBREF8 .
Question: Which modifications do they make to well-established Seq2seq architectures?
Answer: Replacing attention mechanism to query-key attention, and adding a loss to make the attention mask as diagonal as possible
Paper: Deep Text-to-Speech System with Seq2Seq Model
The architecture of our model utilizes RNN-based Seq2Seq model for generating mel spectrogram from text. The architecture is similar to that of Tacotron 2 BIBREF4 . The generated mel spetrogram can either be inverted via iterative algorithms such as Griffin Lim, or through more complicated neural vocoder networks such as a mel spectrogram conditioned Wavenet BIBREF11 .
Question: How do they measure the size of models?
Paper: Deep Text-to-Speech System with Seq2Seq Model
The encoder acts to encoder the input text sequence into a compact hidden representation, which is consumed by the decoder at every decoding step. The encoder is composed of a INLINEFORM0 -dim embedding layer that maps the input sequence into a dense vector. This is followed by a 1-layer bidirectional LSTM/GRU with INLINEFORM1 hidden dim ( INLINEFORM2 hidden dim total for both directions). two linear projections layers project the LSTM/GRU hidden output into two vectors INLINEFORM3 and INLINEFORM4 of the same INLINEFORM5 -dimension, these are the key and value vectors. DISPLAYFORM0 
Question: Do they reduce the number of parameters in their architecture compared to other direct text-to-speech models?
Paper: Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach
Supervised text classification has achieved great success in the past decades due to the availability of rich training data and deep learning techniques. However, zero-shot text classification ($\textsc {0shot-tc}$) has attracted little attention despite its great potential in real world applications, e.g., the intent recognition of bank consumers. $\textsc {0shot-tc}$ is challenging because we often have to deal with classes that are compound, ultra-fine-grained, changing over time, and from different aspects such as topic, emotion, etc.
Question: Do they use pretrained models?
Paper: Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach
The $\textsc {0shot-tc}$ problem was modeled in a too restrictive vision. Firstly, most work only explored a single task, which was mainly topic categorization, e.g., BIBREF1, BIBREF2, BIBREF3. We argue that this is only the tiny tip of the iceberg for $\textsc {0shot-tc}$. Secondly, there is often a precondition that a part of classes are seen and their labeled instances are available to train a model, as we define here as Definition-Restrictive:
Question: What are their baseline models?
Paper: Build Fast and Accurate Lemmatization for Arabic
Lemmatization is the process of finding the base form (or lemma) of a word by considering its inflected forms. Lemma is also called dictionary form, or citation form, and it refers to all words having the same meaning.
Question: How was speed measured?
Answer: how long it takes the system to lemmatize a set number of words
Paper: Build Fast and Accurate Lemmatization for Arabic
Arabic is the largest Semitic language spoken by more than 400 million people. It's one of the six official languages in the United Nations, and the fifth most widely spoken language after Chinese, Spanish, English, and Hindi. Arabic has a very rich morphology, both derivational and inflectional. Generally, Arabic words are derived from a root that uses three or more consonants to define a broad meaning or concept, and they follow some templatic morphological patterns. By adding vowels, prefixes and suffixes to the root, word inflections are generated. For instance, the word ÙØ³ÙÙØªØ­ÙÙ> (wsyftHwn) “and they will open” has the triliteral root ÙØªØ­> (ftH), which has the basic meaning of opening, has prefixes ÙØ³> (ws) “and will”, suffixes ÙÙ> (wn) “they”, stem ÙÙØªØ­> (yftH) “open”, and lemma ÙØªØ­> (ftH) “the concept of opening”.
Question: What were their accuracy results on the task?
Answer: 97.32%
Paper: Build Fast and Accurate Lemmatization for Arabic
To make the annotated data publicly available, we selected 70 news articles from Arabic WikiNews site https://ar.wikinews.org/wiki. These articles cover recent news from year 2013 to year 2015 in multiple genres (politics, economics, health, science and technology, sports, arts, and culture.) Articles contain 18,300 words, and they are evenly distributed among these 7 genres with 10 articles per each.
Question: What is the state of the art?
Paper: Build Fast and Accurate Lemmatization for Arabic
We were inspired by the work done by BIBREF8 for segmenting Arabic words out of context. They achieved an accuracy of almost 99%; slightly better than state-of-the-art system for segmentation (MADAMIRA) which considers surrounding context and many linguistic features. This system shows enhancements in both Machine Translation, and Information Retrieval tasks BIBREF9 . This work can be considered as an extension to word segmentation.
Question: How was the dataset annotated?
Paper: Build Fast and Accurate Lemmatization for Arabic
Data was formatted in a plain text format where sentences are written in separate lines and words are separated by spaces, and the outputs of MADAMIRA and our system are compared against the undiacritized lemma for each word. For accurate results, all differences were revised manually to accept cases that should not be counted as errors (different writings of foreign names entities for example as in ÙÙÙØº ÙÙÙØºØ ÙÙÙØ¬ ÙÙÙØ¬> (hwng kwng, hwnj kwnj) “Hong Kong”, or more than one accepted lemma for some function words, e.g the lemmas ÙÙØ ÙÙÙ Ø§> (fy, fymA) are both valid for the function word ÙÙÙ Ø§> (fymA) “while”).
Question: What is the size of the dataset?
Paper: Build Fast and Accurate Lemmatization for Arabic
Most of the lemmatization errors in our system are due to fact that we use the most common diacritization of words without considering their contexts which cannot solve the ambiguity in cases like nouns and adjectives that share the same diacritization forms, for example the word Ø£ÙØ§Ø¯ÙÙ ÙØ©> (AkAdymyp) can be either a noun and its lemma is Ø£ÙØ§Ø¯ÙÙ ÙØ©> (AkAdymyp) “academy”, or an adjective and its lemma is Ø£ÙØ§Ø¯ÙÙ Ù> (AkAdymy) “academic”. Also for MADAMIRA, errors in selecting the correct Part-of-Speech (POS) for ambiguous words, and foreign named entities.
Question: Where did they collect their dataset from?
Paper: Joint Contextual Modeling for ASR Correction and Language Understanding
Goal-oriented dialogue systems aim to automatically identify the intent of the user as expressed in natural language, extract associated arguments or slots, and take actions accordingly to satisfy the user’s requests BIBREF0. In such systems, the speakers' utterances are typically recognized using an ASR system. Then the intent of the speaker and related slots are identified from the recognized word sequence using an LU component. Finally, a dialogue manager (DM) interacts with the user (not necessarily in natural language) and helps the user achieve the task that the system is designed to support. As a result, the quality of ASR systems has a direct impact on downstream tasks such as LU and DM. This becomes more evident in cases where a generic ASR is used, instead of a domain-specific one BIBREF1.
Question: How much in-domain data is enough for joint models to outperform baselines?
Paper: Joint Contextual Modeling for ASR Correction and Language Understanding
Word Confusion Networks: A compact and normalized class of word lattices, called word confusion networks (WCNs) were initially proposed for improving ASR performance BIBREF5. WCNs are much smaller than ASR lattices but have better or comparable word and oracle accuracy, and because of this they have been used for many tasks, including SLU BIBREF6. However, to the best of our knowledge they have not been used with Neural Semantic Parsers implemented by Recurrent Neural Networks (RNNs) or similar architectures. The closest work would be BIBREF7, who propose to traverse an input lattice in topological order and use the RNN hidden state of the lattice final state as the dense vector representing the entire lattice. However, word confusion networks provide a much better and more efficient solution thanks to token alignments. We use this idea to first infer WCNs from ASR n-best and then directly use them for ASR correction and LU in joint fashion.
Question: How many parameters does their proposed joint model have?