z-uo/qasper-squad huggingface dataset:
Paper: Classifying movie genres by analyzing text reviews
By only reading a single text review of a movie it can be difficult to say what the genre of that movie is, but by using text mining techniques on thousands of movie reviews is it possible to predict the genre?
Question: what was the baseline?
Answer: There is no baseline.
Paper: Classifying movie genres by analyzing text reviews
In this section all relevant theory and methodology is described. Table TABREF1 lists basic terminology and a short description of their meaning.
Question: how many movie genres do they explore?
Paper: Classifying movie genres by analyzing text reviews
Data preprocessing is important when working with text data because it can reduce the number of features and it formats the data into the desired form BIBREF2 .
Question: what evaluation metrics are discussed?
Paper: News-Driven Stock Prediction With Attention-Based Noisy Recurrent State Transition
Stock movement prediction is a central task in computational and quantitative finance. With recent advances in deep learning and natural language processing technology, event-driven stock prediction has received increasing research attention BIBREF0, BIBREF1. The goal is to predict the movement of stock prices according to financial news. Existing work has investigated news representation using bag-of-words BIBREF2, named entities BIBREF3, event structures BIBREF4 or deep learning BIBREF1, BIBREF5.
Question: How big is dataset used?
Answer: 553,451 documents
Paper: News-Driven Stock Prediction With Attention-Based Noisy Recurrent State Transition
There has been a line of work predicting stock markets using text information from daily news. We compare this paper with previous work from the following two perspectives.
Question: What is dataset used for news-driven stock movement prediction?
Paper: Learning Open Information Extraction of Implicit Relations from Reading Comprehension Datasets
Open Information Extraction (OpenIE) is the NLP task of generating (subject, relation, object) tuples from unstructured text e.g. “Fed chair Powell indicates rate hike” outputs (Powell, indicates, rate hike). The modifier open is used to contrast IE research in which the relation belongs to a fixed set. OpenIE has been shown to be useful for several downstream applications such as knowledge base construction BIBREF0 , textual entailment BIBREF1 , and other natural language understanding tasks BIBREF2 . In our previous example an extraction was missing: (Powell, works for, Fed). Implicit extractions are our term for this type of tuple where the relation (“works for” in this example) is not contained in the input sentence. In both colloquial and formal language, many relations are evident without being explicitly stated. However, despite their pervasiveness, there has not been prior work targeted at implicit predicates in the general case. Implicit information extractors for some specific implicit relations such as noun-mediated relations, numerical relations, and others BIBREF3 , BIBREF4 , BIBREF5 have been researched. While specific extractors are important, there are a multiplicity of implicit relation types and it would be intractable to categorize and design extractors for each one.
Question: How much better does this baseline neural model do?
Answer: The model outperforms at every point in the
implicit-tuples PR curve reaching almost 0.8 in recall
Paper: AMR-to-text Generation with Synchronous Node Replacement Grammar
Abstract Meaning Representation (AMR) BIBREF0 is a semantic formalism encoding the meaning of a sentence as a rooted, directed graph. AMR uses a graph to represent meaning, where nodes (such as “boy”, “want-01”) represent concepts, and edges (such as “ARG0”, “ARG1”) represent relations between concepts. Encoding many semantic phenomena into a graph structure, AMR is useful for NLP tasks such as machine translation BIBREF1 , BIBREF2 , question answering BIBREF3 , summarization BIBREF4 and event detection BIBREF5 .
Question: What is the SemEval-2016 task 8?
Paper: MGNC-CNN: A Simple Approach to Exploiting Multiple Word Embeddings for Sentence Classification
Neural models have recently gained popularity for Natural Language Processing (NLP) tasks BIBREF0 , BIBREF1 , BIBREF2 . For sentence classification, in particular, Convolution Neural Networks (CNN) have realized impressive performance BIBREF3 , BIBREF4 . These models operate over word embeddings, i.e., dense, low dimensional vector representations of words that aim to capture salient semantic and syntactic properties BIBREF1 .
Question: How much faster is training time for MGNC-CNN over the baselines?
Paper: MGNC-CNN: A Simple Approach to Exploiting Multiple Word Embeddings for Sentence Classification
Question: What are the baseline models?
Answer: MC-CNN
MVCNN
CNN
Paper: MGNC-CNN: A Simple Approach to Exploiting Multiple Word Embeddings for Sentence Classification
We first review standard one-layer CNN (which exploits a single set of embeddings) for sentence classification BIBREF3 , and then propose our augmentations, which exploit multiple embedding sets.
Question: By how much of MGNC-CNN out perform the baselines?
Answer: In terms of Subj the Average MGNC-CNN is better than the average score of baselines by 0.5.  Similarly, Scores of SST-1, SST-2, and TREC where MGNC-CNN has similar improvements. 
In case of Irony the difference is about 2.0.
Paper: MGNC-CNN: A Simple Approach to Exploiting Multiple Word Embeddings for Sentence Classification
Stanford Sentiment Treebank Stanford Sentiment Treebank (SST) BIBREF14 . This concerns predicting movie review sentiment. Two datasets are derived from this corpus: (1) SST-1, containing five classes: very negative, negative, neutral, positive, and very positive. (2) SST-2, which has only two classes: negative and positive. For both, we remove phrases of length less than 4 from the training set.
Question: What dataset/corpus is this evaluated over?
Paper: MGNC-CNN: A Simple Approach to Exploiting Multiple Word Embeddings for Sentence Classification
We consider three sets of word embeddings for our experiments: (i) word2vec is trained on 100 billion tokens of Google News dataset; (ii) GloVe BIBREF18 is trained on aggregated global word-word co-occurrence statistics from Common Crawl (840B tokens); and (iii) syntactic word embedding trained on dependency-parsed corpora. These three embedding sets happen to all be 300-dimensional, but our model could accommodate arbitrary and variable sizes. We pre-trained our own syntactic embeddings following BIBREF8 . We parsed the ukWaC corpus BIBREF19 using the Stanford Dependency Parser v3.5.2 with Stanford Dependencies BIBREF20 and extracted (word, relation+context) pairs from parse trees. We “collapsed" nodes with prepositions and notated inverse relations separately, e.g., “dog barks" emits two tuples: (barks, nsubj_dog) and (dog, nsubj INLINEFORM0 _barks). We filter words and contexts that appear fewer than 100 times, resulting in INLINEFORM1 173k words and 1M contexts. We trained 300d vectors using word2vecf with default parameters.
Question: What are the comparable alternative architectures?
Paper: Hooks in the Headline: Learning to Generate Headlines with Controlled Styles
Every good article needs a good title, which should not only be able to condense the core meaning of the text, but also sound appealing to the readers for more exposure and memorableness. However, currently even the best Headline Generation (HG) system can only fulfill the above requirement yet performs poorly on the latter. For example, in Figure FIGREF2, the plain headline by an HG model “Summ: Leopard Frog Found in New York City” is less eye-catching than the style-carrying ones such as “What's That Chuckle You Hear? It May Be the New Frog From NYC.”
Question: Which state-of-the-art model is surpassed by 9.68% attraction score?
Paper: Hooks in the Headline: Learning to Generate Headlines with Controlled Styles
Our work is related to summarization and text style transfer.
Question: What is increase in percentage of humor contained in headlines generated with TitleStylist method (w.r.t. baselines)?
Answer: Humor in headlines (TitleStylist vs Multitask baseline):
Relevance: +6.53% (5.87 vs 5.51)
Attraction: +3.72% (8.93 vs 8.61)
Fluency: 1,98% (9.29 vs 9.11)
Paper: Hooks in the Headline: Learning to Generate Headlines with Controlled Styles
Headline generation is a very popular area of research. Traditional headline generation methods mostly focus on the extractive strategies using linguistic features and handcrafted rules BIBREF7, BIBREF8, BIBREF9, BIBREF10, BIBREF11, BIBREF12, BIBREF13. To enrich the diversity of the extractive summarization, abstractive models were then proposed. With the help of neural networks, BIBREF14 proposed attention-based summarization (ABS) to make BIBREF15's framework of summarization more powerful. Many recent works extended ABS by utilizing additional features BIBREF16, BIBREF17, BIBREF18, BIBREF19, BIBREF20, BIBREF21, BIBREF22. Other variants of the standard headline generation setting include headlines for community question answering BIBREF23, multiple headline generation BIBREF24, user-specific generation using user embeddings in recommendation systems BIBREF25, bilingual headline generation BIBREF26 and question-style headline generation BIBREF27.
Question: How is attraction score measured?
Paper: Hooks in the Headline: Learning to Generate Headlines with Controlled Styles
Our work is also related to text style transfer, which aims to change the style attribute of the text while preserving its content. First proposed by BIBREF31, it has achieved great progress in recent years BIBREF32, BIBREF33, BIBREF34, BIBREF35, BIBREF36, BIBREF37, BIBREF38. However, all these methods demand a text corpus for the target style; however, in our case, it is expensive and technically challenging to collect news headlines with humor and romance styles, which makes this category of methods not applicable to our problem.
Question: How is presence of three target styles detected?
Paper: Hooks in the Headline: Learning to Generate Headlines with Controlled Styles
The model is trained on a source dataset $S$ and target dataset $T$. The source dataset $S=\lbrace (\mathbf {a^{(i)}},\mathbf {h^{(i)}})\rbrace _{i=1}^N$ consists of pairs of a news article $\mathbf {a}$ and its plain headline $\mathbf {h}$. We assume that the source corpus has a distribution $P(A, H)$, where $A=\lbrace \mathbf {a^{(i)}}\rbrace _{i=1}^N$, and $H=\lbrace \mathbf {h^{(i)}}\rbrace _{i=1}^N$. The target corpus $T=\lbrace \mathbf {t^{(i)}}\rbrace _{i=1}^{M}$ comprises of sentences $\mathbf {t}$ written in a specific style (e.g., humor). We assume that it conforms to the distribution $P(T)$.
Question: How is fluency automatically evaluated?
Paper: Twitter-Network Topic Model: A Full Bayesian Treatment for Social Network and Text Modeling
Emergence of web services such as blog, microblog and social networking websites allows people to contribute information publicly. This user-generated information is generally more personal, informal and often contains personal opinions. In aggregate, it can be useful for reputation analysis of entities and products, natural disasters detection, obtaining first-hand news, or even demographic analysis. Twitter, an easily accessible source of information, allows users to voice their opinions and thoughts in short text known as tweets.
Question: What are the measures of "performance" used in this paper?
Paper: Towards Language Agnostic Universal Representations
Anecdotally speaking, fluent bilingual speakers rarely face trouble translating a task learned in one language to another. For example, a bilingual speaker who is taught a math problem in English will trivially generalize to other known languages. Furthermore there is a large collection of evidence in linguistics arguing that although separate lexicons exist in multilingual speakers the core representations of concepts and theories are shared in memory BIBREF2 , BIBREF3 , BIBREF4 . The fundamental question we're interested in answering is on the learnability of these shared representations within a statistical framework.
Question: What are the languages they consider in this paper?
Answer: The languages considered were English, Chinese, German, Russian, Arabic, Spanish, French
Paper: Towards Language Agnostic Universal Representations
Our work attempts to unite universal (task agnostic) representations with multilingual (language agnostic) representations BIBREF9 , BIBREF10 . The recent trend in universal representations has been moving away from context-less unsupervised word embeddings to context-rich representations. Deep contextualized word representations (ELMo) trains an unsupervised language model on a large corpus of data and applies it to a large set of auxiliary tasks BIBREF9 . These unsupervised representations boosted the performance of models on a wide array of tasks. Along the same lines BIBREF10 showed the power of using latent representations of translation models as features across other non-translation tasks. In general, initializing models with pre-trained language models shows promise against the standard initialization with word embeddings. Even further, BIBREF11 show that an unsupervised language model trained on a large corpus will contain a neuron that strongly correlates with sentiment without ever training on a sentiment task implying that unsupervised language models maybe picking up informative and structured signals.
Question: Did they experiment with tasks other than word problems in math?
Answer: They experimented with sentiment analysis and natural language inference task
Paper: Is preprocessing of text really worth your time for online comment classification?
Lately, there has been enormous increase in User Generated Contents (UGC) on the online platforms such as newsgroups, blogs, online forums and social networking websites. According to the January 2018 report, the number of active users in Facebook, YouTube, WhatsApp, Facebook Messenger and WeChat was more than 2.1, 1.5, 1.3, 1.3 and 0.98 billions respectively BIBREF1 . The UGCs, most of the times, are helpful but sometimes, they are in bad taste usually posted by trolls, spammers and bullies. According to a study by McAfee, 87% of the teens have observed cyberbullying online BIBREF2 . The Futures Company found that 54% of the teens witnessed cyber bullying on social media platforms BIBREF3 . Another study found 27% of all American internet users self-censor their online postings out of fear of online harassment BIBREF4 . Filtering toxic comments is a challenge for the content providers as their appearances result in the loss of subscriptions. In this paper, we will be using toxic and abusive terms interchangeably to represent comments which are inappropriate, disrespectful, threat or discriminative.
Question: Do they report results only on English data?
Paper: Is preprocessing of text really worth your time for online comment classification?
A large number of studies have been done on comment classification in the news, finance and similar other domains. One such study to classify comments from news domain was done with the help of mixture of features such as the length of comments, uppercase and punctuation frequencies, lexical features such as spelling, profanity and readability by applying applied linear and tree based classifier BIBREF7 . FastText, developed by the Facebook AI research (FAIR) team, is a text classification tool suitable to model text involving out-of-vocabulary (OOV) words BIBREF8 BIBREF9 . Zhang et al shown that character level CNN works well for text classification without the need for words BIBREF10 .
Question: Do the authors offer any hypothesis as to why the transformations sometimes disimproved performance?
Paper: Is preprocessing of text really worth your time for online comment classification?
Toxic comment classification is relatively new field and in recent years, different studies have been carried out to automatically classify toxic comments.Yin et.al. proposed a supervised classification method with n-grams and manually developed regular expressions patterns to detect abusive language BIBREF11 . Sood et. al. used predefined blacklist words and edit distance metric to detect profanity which allowed them to catch words such as sh!+ or @ss as profane BIBREF12 . Warner and Hirschberg detected hate speech by annotating corpus of websites and user comments geared towards detecting anti-semitic hate BIBREF13 . Nobata et. al. used manually labeled online user comments from Yahoo! Finance and news website for detecting hate speech BIBREF5 . Chen et. al. performed feature engineering for classification of comments into abusive, non-abusive and undecided BIBREF14 . Georgakopoulos and Plagianakos compared performance of five different classifiers namely; Word embeddings and CNN, BoW approach SVM, NB, k-Nearest Neighbor (kNN) and Linear Discriminated Analysis (LDA) and found that CNN outperform all other methods in classifying toxic comments BIBREF15 .
Question: What preprocessing techniques are used in the experiments?
Paper: Is preprocessing of text really worth your time for online comment classification?
We found few dedicated papers that address the effect of incorporating different text transformations on the model accuracy for sentiment classification. Uysal and Gunal shown the impact of transformation on text classification by taking into account four transformations and their all possible combination on news and email domain to observe the classification accuracy. Their experimental analyses shown that choosing appropriate combination may result in significant improvement on classification accuracy BIBREF16 . Nobata et. al. used normalization of numbers, replacing very long unknown words and repeated punctuations with the same token BIBREF5 . Haddi et. al. explained the role of transformation in sentiment analyses and demonstrated with the help of SVM on movie review database that the accuracies improve significantly with the appropriate transformation and feature selection. They used transformation methods such as white space removal, expanding abbreviation, stemming, stop words removal and negation handling BIBREF17 .
Question: What state of the art models are used in the experiments?
Paper: Same Representation, Different Attentions: Shareable Sentence Representation Learning from Multiple Tasks
The distributed representation plays an important role in deep learning based natural language processing (NLP) BIBREF0 , BIBREF1 , BIBREF2 . On word level, many successful methods have been proposed to learn a good representation for single word, which is also called word embedding, such as skip-gram BIBREF3 , GloVe BIBREF4 , etc. There are also pre-trained word embeddings, which can easily used in downstream tasks. However, on sentence level, there is still no generic sentence representation which is suitable for various NLP tasks.
Question: What evaluation metrics are used?
Answer: Accuracy on each dataset and the average accuracy on all datasets.
Paper: Same Representation, Different Attentions: Shareable Sentence Representation Learning from Multiple Tasks
The primary role of sentence encoding models is to represent the variable-length sentence or paragraphs as fixed-length dense vector (distributed representation). Currently, the effective neural sentence encoding models include neural Bag-of-words (NBOW), recurrent neural networks (RNN) BIBREF2 , BIBREF6 , convolutional neural networks (CNN) BIBREF1 , BIBREF7 , BIBREF8 , and syntactic-based compositional model BIBREF9 , BIBREF10 , BIBREF11 .
Question: What dataset did they use?
Paper: Same Representation, Different Attentions: Shareable Sentence Representation Learning from Multiple Tasks
Multi-task Learning BIBREF5 utilizes the correlation between related tasks to improve classification by learning tasks in parallel, which has been widely used in various natural language processing tasks, such as text classification BIBREF12 , semantic role labeling BIBREF13 , machine translation BIBREF14 , and so on.
Question: What tasks did they experiment with?
Paper: Zero-Shot Paraphrase Generation with Multilingual Language Models
Paraphrasing is to express the same meaning using different expressions. Paraphrase generation plays an important role in various natural language processing (NLP) tasks such as response diversification in dialogue system, query reformulation in information retrieval, and data augmentation in machine translation. Recently, models based on Seq2Seq learning BIBREF1 have achieved the state-of-the-art results on paraphrase generation. Most of these models BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6 focus on training the paraphrasing models based on a paraphrase corpus, which contains a number of pairs of paraphrases. However, high-quality paraphrases are usually difficult to acquire in practice, which becomes the major limitation of these methods. Therefore, we focus on zero-shot paraphrase generation approach in this paper, which aims to generate paraphrases without requiring a paraphrase corpus.
Question: What multilingual parallel data is used for training proposed model?
Paper: Zero-Shot Paraphrase Generation with Multilingual Language Models
Transformer-based language model (TLM) is a neural language model constructed with a stack of Transformer decoder layers BIBREF8. Given a sequence of tokens, TLM is trained with maximizing the likelihood:
Question: How much better are results of proposed model compared to pivoting method?
Paper: WiSeBE: Window-based Sentence Boundary Evaluation
The goal of Automatic Speech Recognition (ASR) is to transform spoken data into a written representation, thus enabling natural human-machine interaction BIBREF0 with further Natural Language Processing (NLP) tasks. Machine translation, question answering, semantic parsing, POS tagging, sentiment analysis and automatic text summarization; originally developed to work with formal written texts, can be applied over the transcripts made by ASR systems BIBREF1 , BIBREF2 , BIBREF3 . However, before applying any of these NLP tasks a segmentation process called Sentence Boundary Detection (SBD) should be performed over ASR transcripts to reach a minimal syntactic information in the text.
Question: What kind of Youtube video transcripts did they use?
Answer: youtube video transcripts on news covering different topics like technology, human rights, terrorism and politics
Paper: WiSeBE: Window-based Sentence Boundary Evaluation
Sentence Boundary Detection (SBD) has been a major research topic science ASR moved to more general domains as conversational speech BIBREF4 , BIBREF5 , BIBREF6 . Performance of ASR systems has improved over the years with the inclusion and combination of new Deep Neural Networks methods BIBREF7 , BIBREF8 , BIBREF0 . As a general rule, the output of ASR systems lacks of any syntactic information such as capitalization and sentence boundaries, showing the interst of ASR systems to obtain the correct sequence of words with almost no concern of the overall structure of the document BIBREF9 .
Question: Which SBD systems did they compare?
Paper: WiSeBE: Window-based Sentence Boundary Evaluation
SBD research has been focused on two different aspects; features and methods. Regarding the features, some work focused on acoustic elements like pauses duration, fundamental frequencies, energy, rate of speech, volume change and speaker turn BIBREF17 , BIBREF18 , BIBREF19 .
Question: What makes it a more reliable metric?
Answer: It takes into account the agreement between different systems