z-uo/qasper-squad huggingface dataset:
Paper: Characterizing Political Fake News in Twitter by its Meta-Data
Previous works on the area (presented in the section above) suggest that there may be important determinants for the adoption and diffusion of fake news. Our hypotheses builds on them and identifies three important dimensions that may help distinguishing fake news from legit information:
Question: What are the characteristics of the accounts that spread fake news?
Answer: Accounts that spread fake news are mostly unverified, recently created and have on average high friends/followers ratio
Paper: Characterizing Political Fake News in Twitter by its Meta-Data
For this study, we collected publicly available tweets using Twitter's public API. Given the nature of the data, it is important to emphasize that such tweets are subject to Twitter's terms and conditions which indicate that users consent to the collection, transfer, manipulation, storage, and disclosure of data. Therefore, we do not expect ethical, legal, or social implications from the usage of the tweets. Our data was collected using search terms related to the presidential election held in the United States on November 8th 2016. Particularly, we queried Twitter's streaming API, more precisely the filter endpoint of the streaming API, using the following hashtags and user handles: #MyVote2016, #ElectionDay, #electionnight, @realDonaldTrump and @HillaryClinton. The data collection ran for just one day (Nov 8th 2016).
Question: What is the threshold for determining that a tweet has gone viral?
Paper: Characterizing Political Fake News in Twitter by its Meta-Data
The sample collected consisted on 1 785 855 tweets published by 848 196 different users. Within our sample, we identified 1327 tweets that went viral (retweeted more than 1000 times by the 8th of November 2016) produced by 643 users. Such small subset of viral tweets were retweeted on 290 841 occasions in the observed time-window.
Question: How is the ground truth for fake news established?
Answer: Ground truth is not established in the paper
Paper: Ensemble based discriminative models for Visual Dialog Challenge 2018
Visual dialog BIBREF0 is an interesting new task combining the research efforts from Computer Vision, Natural Language Processing and Information Retrieval. While BIBREF1 presents some tips and tricks for VQA 2.0 Challenge, we follow their guidelines for the Visual Dialog challenge 2018. Our models use attention similar to BIBREF2 to get object level image representations from Faster R-CNN model BIBREF3. We experiment with different encoder mechanisms to get representations of conversational history.
Question: What was the baseline?
Paper: Ensemble based discriminative models for Visual Dialog Challenge 2018
Common to all the models, we initialize our embedding matrix with pre-trained Glove word vectors of 300 dimensions using 6B tokens . Out of 11319 tokens present in the dataset, we found 188 tokens missing from the pre-trained Glove embeddings, so we manually map these tokens to words conveying semantically similar meaning, e.g. we map over ten variations of the word “yes” - misspelled or not picked up by tokenizer - “*yes", “yesa", “yess", “ytes", “yes-", “yes3", “yyes", “yees", etc.
Question: Which three discriminative models did they use?
Paper: Ancient-Modern Chinese Translation with a Large Training Dataset
Ancient Chinese is the writing language in ancient China. It is a treasure of Chinese culture which brings together the wisdom and ideas of the Chinese nation and chronicles the ancient cultural heritage of China. Learning ancient Chinese not only helps people to understand and inherit the wisdom of the ancients, but also promotes people to absorb and develop Chinese culture.
Question: what NMT models did they compare with?
Paper: Ancient-Modern Chinese Translation with a Large Training Dataset
There are four steps to build the ancient-modern Chinese translation dataset: (i) The parallel corpus crawling and cleaning. (ii) The paragraph alignment. (iii) The clause alignment based on aligned paragraphs. (iv) Augmenting data by merging aligned adjacent clauses. The most critical step is the third step.
Question: Where does the ancient Chinese dataset come from?
Answer: Ancient Chinese history records in several dynasties and articles written by celebrities during 1000BC-200BC collected from the internet
Paper: Follow Alice into the Rabbit Hole: Giving Dialogue Agents Understanding of Human Level Attributes.
Attempts toward constructing human-like dialogue agents have met significant difficulties, such as maintaining conversation consistency BIBREF0. This is largely due to inabilities of dialogue agents to engage the user emotionally because of an inconsistent personality BIBREF1. Many agents use personality models that attempt to map personality attributes into lower dimensional spaces (e.g. the Big Five BIBREF2). However, these represent human personality at a very high-level and lack depth. They prohibit the ability to link specific and detailed personality traits to characters, and to construct large datasets where dialogue is traceable back to these traits.
Question: How many different characters were in dataset?
Paper: Follow Alice into the Rabbit Hole: Giving Dialogue Agents Understanding of Human Level Attributes.
Task completion chatbots (TCC), or task-oriented chatbots, are dialogue agents used to fulfill specific purposes, such as helping customers book airline tickets, or a government inquiry system. Examples include the AIML based chatbot BIBREF5 and DIVA Framework BIBREF6. While TCC are low cost, easily configurable, and readily available, they are restricted to working well for particular domains and tasks.
Question: How does dataset model character's profiles?
Paper: Follow Alice into the Rabbit Hole: Giving Dialogue Agents Understanding of Human Level Attributes.
We collect HLA data from TV Tropes BIBREF3, a knowledge-based website dedicated to pop culture, containing information on a plethora of characters from a variety of sources. Similar to Wikipedia, its content is provided and edited collaboratively by a massive user-base. These attributes are determined by human viewers and their impressions of the characters, and are correlated with human-like characteristics. We believe that TV Tropes is better for our purpose of fictional character modeling than data sources used in works such as BIBREF25 shuster2019engaging because TV Tropes' content providers are rewarded for correctly providing content through community acknowledgement.
Question: How big is the difference in performance between proposed model and baselines?
Answer: Metric difference between Aloha and best baseline score:
Hits@1/20: +0.061 (0.3642 vs 0.3032)
MRR: +0.0572(0.5114 vs 0.4542)
F1: -0.0484 (0.3901 vs 0.4385)
BLEU: +0.0474 (0.2867 vs 0.2393)
Paper: Follow Alice into the Rabbit Hole: Giving Dialogue Agents Understanding of Human Level Attributes.
Our task is the following, where $t$ denotes “target":
Question: What baseline models are used?
Paper: PolyResponse: A Rank-based Approach to Task-Oriented Dialogue with Application in Restaurant Search and Booking
Task-oriented dialogue systems are primarily designed to search and interact with large databases which contain information pertaining to a certain dialogue domain: the main purpose of such systems is to assist the users in accomplishing a well-defined task such as flight booking BIBREF0, tourist information BIBREF1, restaurant search BIBREF2, or booking a taxi BIBREF3. These systems are typically constructed around rigid task-specific ontologies BIBREF1, BIBREF4 which enumerate the constraints the users can express using a collection of slots (e.g., price range for restaurant search) and their slot values (e.g., cheap, expensive for the aforementioned slots). Conversations are then modelled as a sequence of actions that constrain slots to particular values. This explicit semantic space is manually engineered by the system designer. It serves as the output of the natural language understanding component as well as the input to the language generation component both in traditional modular systems BIBREF5, BIBREF6 and in more recent end-to-end task-oriented dialogue systems BIBREF7, BIBREF8, BIBREF9, BIBREF3.
Question: Was PolyReponse evaluated against some baseline?
Paper: PolyResponse: A Rank-based Approach to Task-Oriented Dialogue with Application in Restaurant Search and Booking
The PolyResponse system is powered by a single large conversational search engine, trained on a large amount of conversational and image data, as shown in Figure FIGREF2. In simple words, it is a ranking model that learns to score conversational replies and images in a given conversational context. The highest-scoring responses are then retrieved as system outputs. The system computes two sets of similarity scores: 1) $S(r,c)$ is the score of a candidate reply $r$ given a conversational context $c$, and 2) $S(p,c)$ is the score of a candidate photo $p$ given a conversational context $c$. These scores are computed as a scaled cosine similarity of a vector that represents the context ($h_c$), and a vector that represents the candidate response: a text reply ($h_r$) or a photo ($h_p$). For instance, $S(r,c)$ is computed as $S(r,c)=C cos(h_r,h_c)$, where $C$ is a learned constant. The part of the model dealing with text input (i.e., obtaining the encodings $h_c$ and $h_r$) follows the architecture introduced recently by Henderson:2019acl. We provide only a brief recap here; see the original paper for further details.
Question: What metric is used to evaluate PolyReponse system?
Paper: PolyResponse: A Rank-based Approach to Task-Oriented Dialogue with Application in Restaurant Search and Booking
The model, implemented as a deep neural network, learns to respond by training on hundreds of millions context-reply $(c,r)$ pairs. First, similar to Henderson:2017arxiv, raw text from both $c$ and $r$ is converted to unigrams and bigrams. All input text is first lower-cased and tokenised, numbers with 5 or more digits get their digits replaced by a wildcard symbol #, while words longer than 16 characters are replaced by a wildcard token LONGWORD. Sentence boundary tokens are added to each sentence. The vocabulary consists of the unigrams that occur at least 10 times in a random 10M subset of the Reddit training set (see Figure FIGREF2) plus the 200K most frequent bigrams in the same random subset.
Question: How does PolyResponse architecture look like?
Paper: PolyResponse: A Rank-based Approach to Task-Oriented Dialogue with Application in Restaurant Search and Booking
Photos are represented using convolutional neural net (CNN) models pretrained on ImageNet BIBREF17. We use a MobileNet model with a depth multiplier of 1.4, and an input dimension of $224 \times 224$ pixels as in BIBREF18. This provides a $1,280 \times 1.4 = 1,792$-dimensional representation of a photo, which is then passed through a single hidden layer of dimensionality $1,024$ with ReLU activation, before being passed to a hidden layer of dimensionality 512 with no activation to provide the final representation $h_p$.
Question: In what 8 languages is PolyResponse engine used for restourant search and booking system?
Answer: English, German, Spanish, Mandarin, Polish, Russian, Korean and Serbian
Paper: Pretraining-Based Natural Language Generation for Text Summarization
Text summarization generates summaries from input documents while keeping salient information. It is an important task and can be applied to several real-world applications. Many methods have been proposed to solve the text summarization problem BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 . There are two main text summarization techniques: extractive and abstractive. Extractive summarization generates summary by selecting salient sentences or phrases from the source text, while abstractive methods paraphrase and restructure sentences to compose the summary. We focus on abstractive summarization in this work as it is more flexible and thus can generate more diverse summaries.
Question: Why masking words in the decoder is helpful?
Paper: Pretraining-Based Natural Language Generation for Text Summarization
In this paper, we focus on single-document multi-sentence summarization and propose a supervised abstractive model based on the neural attentive sequence-to-sequence framework which consists of two parts: a neural network for the encoder and another network for the decoder. The encoder encodes the input sequence to intermediate representation and the decoder predicts one word at a time step given the input sequence representation vector and previous decoded output. The goal of the model is to maximize the probability of generating the correct target sequences. In the encoding and generation process, the attention mechanism is used to concentrate on the most important positions of text. The learning objective of most sequence-to-sequence models is to minimize the negative log likelihood of the generated sequence as following equation shows, where $y^*_i$ is the i-th ground-truth summary token. 
Question: What is the ROUGE score of the highest performing model?
Paper: Pretraining-Based Natural Language Generation for Text Summarization
Recently, context encoders such as ELMo, GPT, and BERT have been widely used in many NLP tasks. These models are pre-trained on a huge unlabeled corpus and can generate better contextualized token embeddings, thus the approaches built on top of them can achieve better performance.
Question: How are the different components of the model trained? Is it trained end-to-end?
Paper: Pretraining-Based Natural Language Generation for Text Summarization
In this section, we describe the structure of our model, which learns to generate an abstractive multi-sentence summary from a given source document.
Question: When is this paper published?
Paper: Analysis of Wikipedia-based Corpora for Question Answering
Question answering (QA) has been a blooming research field for the last decade. Selection-based QA implies a family of tasks that find answer contexts from large data given questions in natural language. Three tasks have been proposed for selection-based QA. Given a document, answer extraction BIBREF0 , BIBREF1 finds answer phrases whereas answer selection BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 and answer triggering BIBREF6 , BIBREF7 find answer sentences instead, although the presence of the answer context is not assumed within the provided document for answer triggering but it is for the other two tasks. Recently, various QA tasks that are not selection-based have been proposed BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 ; however, selection-based QA remains still important because of its practical value to real applications (e.g., IBM Watson, MIT Start).
Question: Can their indexing-based method be applied to create other QA datasets in other domains, and not just Wikipedia?
Paper: Analysis of Wikipedia-based Corpora for Question Answering
Four publicly available corpora are selected for our analysis. These corpora are based on Wikipedia, so more comparable than the others, and have already been used for the evaluation of several QA systems.
Question: Do they employ their indexing-based method to create a sample of a QA Wikipedia dataset?
Paper: Analysis of Wikipedia-based Corpora for Question Answering
All corpora provide datasets/splits for answer selection, whereas only (WikiQA, SQuAD) and (WikiQA, SelQA) provide datasets for answer extraction and answer triggering, respectively. SQuAD is much larger in size although questions in this corpus are often paraphrased multiple times. On the contrary, SQuAD's average candidates per question ( INLINEFORM0 ) is the smallest because SQuAD extracts answer candidates from paragraphs whereas the others extract them from sections or infoboxes that consist of bigger contexts. Although InfoboxQA is larger than WikiQA or SelQA, the number of token types ( INLINEFORM1 ) in InfoboxQA is smaller than those two, due to the repetitive nature of infoboxes.
Question: How many question types do they find in the datasets analyzed?
Answer: 7
Paper: Analysis of Wikipedia-based Corpora for Question Answering
This section describes another selection-based QA task, called answer retrieval, that finds the answer context from a larger dataset, the entire Wikipedia. SQuAD provides no mapping of the answer contexts to Wikipedia, whereas WikiQA and SelQA provide mappings; however, their data do not come from the same version of Wikipedia. We propose an automatic way of mapping the answer contexts from all corpora to the same version of Wikipeda so they can be coherently used for answer retrieval.
Question: How do they analyze contextual similaries across datasets?
Answer: They compare the tasks that the datasets are suitable for, average number of answer candidates per question, number of token types, average answer candidate lengths, average question lengths, question-answer word overlap.
Paper: Deep Learning for Detecting Cyberbullying Across Multiple Social Media Platforms
Cyberbullying has been defined by the National Crime Prevention Council as the use of the Internet, cell phones or other devices to send or post text or images intended to hurt or embarrass another person. Various studies have estimated that between to 10% to 40% of internet users are victims of cyberbullying BIBREF0 . Effects of cyberbullying can range from temporary anxiety to suicide BIBREF1 . Many high profile incidents have emphasized the prevalence of cyberbullying on social media. Most recently in October 2017, a Swedish model Arvida Byström was cyberbullied to the extent of receiving rape threats after she appeared in an advertisement with hairy legs.
Question: What were their performance results?
Answer: best model achieves 0.94 F1 score for Wikipedia and Twitter datasets and 0.95 F1 on Formspring dataset
Paper: Deep Learning for Detecting Cyberbullying Across Multiple Social Media Platforms
Please refer to Table TABREF7 for summary of datasets used. We performed experiments using large, diverse, manually annotated, and publicly available datasets for cyberbullying detection in social media. We cover three different types of social networks: teen oriented Q&A forum (Formspring), large microblogging platform (Twitter), and collaborative knowledge repository (Wikipedia talk pages). Each dataset addresses a different topic of cyberbullying. Twitter dataset contains examples of racism and sexism. Wikipedia dataset contains examples of personal attack. However, Formspring dataset is not specifically about any single topic. All three datasets have the problem of class imbalance where posts labeled as cyberbullying are in the minority as compared to neutral posts. Variation in the number of posts across datasets also affects vocabulary size that represents the number of distinct words encountered in the dataset. We measure the size of a post in terms of the number of words in the post. For each dataset, there are only a few posts with large size. We truncate such large posts to the size of post ranked at 95 percentile in that dataset. For example, in Wikipedia dataset, the largest post has 2846 words. However, size of post ranked at 95 percentile in that dataset is only 231. Any post larger than size 231 in Wikipedia dataset will be truncated by considering only first 231 words. This truncation affects only a small minority of posts in each dataset. However, it is required for efficiently training various models in our experiments. Details of each dataset are as follows.
Question: What cyberbulling topics did they address?
Paper: From Textual Information Sources to Linked Data in the Agatha Project
The automatic identification, extraction and representation of the information conveyed in texts is a key task nowadays. In fact, this research topic is increasing its relevance with the exponential growth of social networks and the need to have tools that are able to automatically process them BIBREF0.
Question: Were any of the pipeline components based on deep learning models?
Paper: From Textual Information Sources to Linked Data in the Agatha Project
The framework for processing Portuguese texts is depicted in Fig. FIGREF2, which illustrates how relevant pieces of information are extracted from the text. Namely, input files (Portuguese texts) go through a series of modules: part-of-speech tagging, named entity recognition, dependency parsing, semantic role labeling, subject-verb-object triple extraction, and lexicon matching.
Question: How is the effectiveness of this pipeline approach evaluated?
Paper: Prepositional Attachment Disambiguation Using Bilingual Parsing and Alignments
Prepositional Phrase (PP) attachment disambiguation is an important problem in NLP, for it often gives rise to incorrect parse trees . Statistical parsers often predict incorrect attachment for prepositional phrases. For applications like Machine Translation, incorrect PP-attachment leads to serious errors in translation. Several approaches have been proposed to solve this problem. We attempt to tackle this problem for English. English is a syntactically ambiguous language with respect to PP attachments. For example, consider the following sentence where the prepositional phrase with pockets may attach either to the verb washed or to the noun jeans.
Question: What is the size of the parallel corpus used to train the model constraints?
Paper: Prepositional Attachment Disambiguation Using Bilingual Parsing and Alignments
A number of supervised and unsupervised approaches for solving the PP-attachment problem have been proposed in the literature. Ratnaparkhi:94 use a Maximum Entropy Model for solving the PP-attachment decision. Schwartz:03 propose an unsupervised approach for solving PP attachment using multilingual aligned data. They transform the data into high-level linguistic representations and use it make reattachment decisions. The intuition is similar to our work, but the approach is entirely different. Brill:94 discuss a transformation-based rule derivation method for PP-attachment disambiguation. It is a simple learning algorithm which derives a set of transformation rules from training corpus, which are then used for solving the PP-attachment problem. Stetina:97 make use of the semantic dictionary to solve the problem of disambiguating PP attachments. Their work describes use of word sense disambiguation (WSD) for both supervised and unsupervised techniques. Agirre Agirre:08 and Medimi Medimi:07 have used WSD-based strategies in different capacities to solve the problem of PP-attachment. Olteanu:05 have attempted to solve the pp-attachment problem as a classification problem of attachment either to the preceding verb or the noun, and have used Support Vector Machines (SVMs) that use complex syntactic and semantic features.
Question: How does enforcing agreement between parse trees work across different languages?
Paper: Topic Identification for Speech without ASR
Topic identification (topic ID) on speech aims to identify the topic(s) for given speech recordings, referred to as spoken documents, where the topics are a predefined set of classes or labels. This task is typically formulated as a three-step process. First, speech is tokenized into words or phones by automatic speech recognition (ASR) systems BIBREF0 , or by limited-vocabulary keyword spotting BIBREF1 . Second, standard text-based processing techniques are applied to the resulting tokenizations, and produce a vector representation for each spoken document, typically a bag-of-words multinomial representation, or a more compact vector given by probabilistic topic models BIBREF2 , BIBREF3 . Finally, topic ID is performed on the spoken document representations by supervised training of classifiers, such as Bayesian classifiers and support vector machines (SVMs).
Question: What datasets are used to assess the performance of the system?
Answer: LORELEI datasets of Uzbek, Mandarin and Turkish
Paper: Topic Identification for Speech without ASR
Question: How is the vocabulary of word-like or phoneme-like units automatically discovered?