z-uo/qasper-squad huggingface dataset:
Paper: Generating Word and Document Embeddings for Sentiment Analysis
Sentiment analysis has recently been one of the hottest topics in natural language processing (NLP). It is used to identify and categorise opinions expressed by reviewers on a topic or an entity. Sentiment analysis can be leveraged in marketing, social media analysis, and customer service. Although many studies have been conducted for sentiment analysis in widely spoken languages, this topic is still immature for Turkish and many other languages.
Question: What baseline method is used?
Answer: using word2vec to create features that are used as input to the SVM
Paper: Generating Word and Document Embeddings for Sentiment Analysis
In the literature, the main consensus is that the use of dense word embeddings outperform the sparse embeddings in many tasks. Latent semantic analysis (LSA) used to be the most popular method in generating word embeddings before the invention of the word2vec and other word vector algorithms which are mostly created by shallow neural network models. Although many studies have been employed on generating word vectors including both semantic and sentimental components, generating and analysing the effects of different types of embeddings on different tasks is an emerging field for Turkish.
Question: What details are given about the Twitter dataset?
Answer: one of the Twitter datasets is about Turkish mobile network operators, there are positive, neutral and negative labels and provide the total amount plus the distribution of labels
Paper: Generating Word and Document Embeddings for Sentiment Analysis
We generate several word vectors, which capture the sentimental, lexical, and contextual characteristics of words. In addition to these mostly original vectors, we also create word2vec embeddings to represent the corpus words by training the embedding model on these datasets. After generating these, we combine them with hand-crafted features to create document vectors and perform classification, as will be explained in Section 3.5.
Question: What details are given about the movie domain dataset?
Answer: there are 20,244 reviews divided into positive and negative with an average 39 words per review, each one having a star-rating score
Paper: Generating Word and Document Embeddings for Sentiment Analysis
Contextual information is informative in the sense that, in general, similar words tend to appear in the same contexts. For example, the word smart is more likely to cooccur with the word hardworking than with the word lazy. This similarity can be defined semantically and sentimentally. In the corpus-based approach, we capture both of these characteristics and generate word embeddings specific to a domain.
Question: Which hand-crafted features are combined with word2vec?
Paper: Generating Word and Document Embeddings for Sentiment Analysis
In Turkish, there do not exist well-established sentiment lexicons as in English. In this approach, we made use of the TDK (Türk Dil Kurumu - “Turkish Language Institution”) dictionary to obtain word polarities. Although it is not a sentiment lexicon, combining it with domain-specific polarity scores obtained from the corpus led us to have state-of-the-art results.
Question: What word-based and dictionary-based feature are used?
Paper: Generating Word and Document Embeddings for Sentiment Analysis
Our last component is a simple metric that uses four supervised scores for each word in the corpus. We extract these scores as follows. For a target word in the corpus, we scan through all of its contexts. In addition to the target word's polarity score (the self score), out of all the polarity scores of words occurring in the same contexts as the target word, minimum, maximum, and average scores are taken into consideration. The word polarity scores are computed using (DISPLAY_FORM4). Here, we obtain those scores from the training data.
Question: How are the supervised scores of the words calculated?
Paper: Seernet at EmoInt-2017: Tweet Emotion Intensity Estimator
Twitter, a micro-blogging and social networking site has emerged as a platform where people express themselves and react to events in real-time. It is estimated that nearly 500 million tweets are sent per day . Twitter data is particularly interesting because of its peculiar nature where people convey messages in short sentences using hashtags, emoticons, emojis etc. In addition, each tweet has meta data like location and language used by the sender. It's challenging to analyze this data because the tweets might not be grammatically correct and the users tend to use informal and slang words all the time. Hence, this poses an interesting problem for NLP researchers. Any advances in using this abundant and diverse data can help understand and analyze information about a person, an event, a product, an organization or a country as a whole. Many notable use cases of the twitter can be found here.
Question: what dataset was used?
Paper: Seernet at EmoInt-2017: Tweet Emotion Intensity Estimator
The preprocessing step modifies the raw tweets before they are passed to feature extraction. Tweets are processed using tweetokenize tool. Twitter specific features are replaced as follows: username handles to USERNAME, phone numbers to PHONENUMBER, numbers to NUMBER, URLs to URL and times to TIME. A continuous sequence of emojis is broken into individual tokens. Finally, all tokens are converted to lowercase.
Question: how many total combined features were there?
Paper: Seernet at EmoInt-2017: Tweet Emotion Intensity Estimator
Many tasks related to sentiment or emotion analysis depend upon affect, opinion, sentiment, sense and emotion lexicons. These lexicons associate words to corresponding sentiment or emotion metrics. On the other hand, the semantic meaning of words, sentences, and documents are preserved and compactly represented using low dimensional vectors BIBREF1 instead of one hot encoding vectors which are sparse and high dimensional. Finally, there are traditional NLP features like word N-grams, character N-grams, Part-Of-Speech N-grams and word clusters which are known to perform well on various tasks.
Question: what pretrained word embeddings were used?
Answer: Pretrained word embeddings  were not used
Paper: A Hybrid Architecture for Multi-Party Conversational Systems
Back to 42 BC, the philosopher Cicero has raised the issue that although there were many Oratory classes, there were none for Conversational skills BIBREF0 . He highlighted how important they were not only for politics, but also for educational purpose. Among other conversational norms, he claimed that people should be able to know when to talk in a conversation, what to talk depending on the subject of the conversation, and that they should not talk about themselves.
Question: What evaluation metrics did look at?
Answer: precision, recall, F1 and accuracy
Response time, resource consumption (memory, CPU, network bandwidth), precision, recall, F1, accuracy.
Paper: A Hybrid Architecture for Multi-Party Conversational Systems
There are plenty of challenges in conversation contexts, and even bigger ones when people and machines participate in those contexts. Conversation is a specialized form of interaction, which follows social conventions. Social interaction makes it possible to inform, context, create, ratify, refute, and ascribe, among other things, power, class, gender, ethnicity, and culture BIBREF2 . Social structures are the norms that emerge from the contact people have with others BIBREF7 , for example, the communicative norms of a negotiation, taking turns in a group, the cultural identity of a person, or power relationships in a work context.
Question: What datasets are used?
Answer: Custom dataset with user questions; set of documents, twitter posts and news articles, all related to finance.
a self-collected financial intents dataset in Portuguese
Paper: A Hybrid Architecture for Multi-Party Conversational Systems
In this section we discuss the state of the art on conversational systems in three perspectives: types of interactions, types of architecture, and types of context reasoning. Then we present a table that consolidates and compares all of them.
Question: What is the state of the art described in the paper?
Paper: ARAML: A Stable Adversarial Training Framework for Text Generation
Natural text generation, as a key task in NLP, has been advanced substantially thanks to the flourish of neural models BIBREF0 , BIBREF1 . Typical frameworks such as sequence-to-sequence (seq2seq) have been applied to various generation tasks, including machine translation BIBREF2 and dialogue generation BIBREF3 . The standard paradigm to train such neural models is maximum likelihood estimation (MLE), which maximizes the log-likelihood of observing each word in the text given the ground-truth proceeding context BIBREF4 .
Question: What GAN models were used as baselines to compare against?
Paper: ARAML: A Stable Adversarial Training Framework for Text Generation
Recently, text generation has been widely studied with neural models trained with maximum likelihood estimation BIBREF4 . However, MLE tends to generate universal text BIBREF18 . Various methods have been proposed to enhance the generation quality by refining the objective function BIBREF18 , BIBREF19 or modifying the generation distribution with external information like topic BIBREF20 , sentence type BIBREF21 , emotion BIBREF22 and knowledge BIBREF23 .
Question: How much improvement is gained from Adversarial Reward Augmented Maximum Likelihood (ARAML)?
Answer: ARAM has achieved improvement over all baseline methods using reverese perplexity and  slef-BLEU metric.  The maximum reverse perplexity improvement 936,16 is gained for EMNLP2017  WMT  dataset and  48,44 for COCO dataset.
Compared to the baselines, ARAML does not do better in terms of perplexity on COCO and EMNLP 2017 WMT datasets, but it does by up to 0.27 Self-BLEU points on COCO and 0.35 Self-BLEU on EMNLP 2017 WMT. In terms of Grammaticality and Relevance, it scores better than the baselines on up to 75.5% and 73% of the cases respectively.
Paper: ARAML: A Stable Adversarial Training Framework for Text Generation
Text generation can be formulated as follows: given the real data distribution INLINEFORM0 , the task is to train a generative model INLINEFORM1 where INLINEFORM2 can fit INLINEFORM3 well. In this formulation, INLINEFORM4 and INLINEFORM5 denotes a word in the vocabulary INLINEFORM6 .
Question: Is the discriminator's reward made available at each step to the generator?
Paper: Shared Task: Lexical Semantic Change Detection in German
Natural languages evolve and words have always been subject to semantic change over time BIBREF1. With the rise of large digitized text resources recent NLP technologies have made it possible to capture such change with vector space models BIBREF2, BIBREF3, BIBREF4, BIBREF5, topic models BIBREF6, BIBREF7, BIBREF8, and sense clustering models BIBREF9. However, many approaches for detecting LSC differ profoundly from each other and therefore drawing comparisons between them can be challenging BIBREF10. Not only do architectures for detecting LSC vary, their performance is also often evaluated without access to evaluation data or too sparse data sets. In cases where evaluation data is available, oftentimes LSCD systems are not evaluated on the same data set which hinders the research community to draw comparisons.
Question: What is the algorithm used to create word embeddings?
Paper: Shared Task: Lexical Semantic Change Detection in German
The goal of the shared task was to create an architecture to detect semantic change and to rank words according to their degree of change between two different time periods. Given two corpora Ca and Cb, the target words had to be ranked according to their degree of lexical semantic change between Ca and Cb as annotated by human judges. A competition was set up on Codalab and teams mostly consisting of 2 people were formed to take part in the task. There was one group consisting of 3 team members and two individuals who entered the task on their own. In total there were 12 LSCD systems participating in the shared task.
Question: What is the corpus used for the task?
Paper: Shared Task: Lexical Semantic Change Detection in German
The task, as framed above, requires to detect the semantic change between two corpora. The two corpora used in the shared task correspond to the diachronic corpus pair from BIBREF0: DTA18 and DTA19. They consist of subparts of DTA corpus BIBREF11 which is a freely available lemmatized, POS-tagged and spelling-normalized diachronic corpus of German containing texts from the 16th to the 20th century. DTA18 contains 26 million sentences published between 1750-1799 and DTA19 40 million between 1850-1899. The corpus version used in the task has the following format: "year [tab] lemma1 lemma2 lemma3 ...".
Question: How is evaluation performed?
Paper: ZuCo 2.0: A Dataset of Physiological Recordings During Natural Reading and Annotation
How humans process language has become increasingly relevant in natural language processing since physiological data during language understanding is more accessible and recorded with less effort. In this work, we focus on eye-tracking and electroencephalography (EEG) recordings to capture the reading process. On one hand, eye movement data provides millisecond-accurate records about where humans look when they are reading, and is highly correlated with the cognitive load associated with different stages of text processing. On the other hand, EEG records electrical brain activity across the scalp and is a direct measure of physiological processes, including language processing. The combination of both measurement methods enables us to study the language understanding process in a more natural setting, where participants read full sentences at a time, in their own speed. Eye-tracking then permits us to define exact word boundaries in the timeline of a subject reading a sentence, allowing the extraction of brain activity signals for each word.
Question: What is a normal reading paradigm?
Paper: ZuCo 2.0: A Dataset of Physiological Recordings During Natural Reading and Annotation
Some eye-tracking corpora of natural reading (e.g. the Dundee BIBREF2, Provo BIBREF3 and GECO corpus BIBREF4), and a few EEG corpora (for example, the UCL corpus BIBREF5) are available. It has been shown that this type of cognitive processing data is useful for improving and evaluating NLP methods (e.g. barrett2018sequence,hollenstein2019cognival, hale2018finding). However, before the Zurich Cognitive Language Processing Corpus (ZuCo 1.0), there was no available data for simultaneous eye-tracking and EEG recordings of natural reading. dimigen2011coregistration studied the linguistic effects of eye movements and EEG co-registration in natural reading and showed that they accurately represent lexical processing. Moreover, the simultaneous recordings are crucial to extract word-level brain activity signals.
Question: Did they experiment with this new dataset?
Paper: ZuCo 2.0: A Dataset of Physiological Recordings During Natural Reading and Annotation
In previous work, we recorded a first dataset of simultaneous eye-tracking and EEG during natural reading BIBREF1. ZuCo 1.0 consists of three reading tasks, two of which contain very similar reading material and experiments as presented in the current work. However, the main difference and reason for recording ZuCo 2.0, consists in the experiment procedure. For ZuCo 1.0 the normal reading and task-specific reading paradigms were recorded in different sessions on different days. Therefore, the recorded data is not appropriate as a means of comparison between natural reading and annotation, since the differences in the brain activity data might result mostly from the different sessions due to the sensitivity of EEG. This, and extending the dataset with more sentences and more subjects, were the main factors for recording the current corpus. We purposefully maintained an overlap of some sentences between both datasets to allow additional analyses (details are described in Section SECREF7).
Question: What kind of sentences were read?
Paper: Using Monolingual Data in Neural Machine Translation: a Systematic Study
The new generation of Neural Machine Translation (NMT) systems is known to be extremely data hungry BIBREF0 . Yet, most existing NMT training pipelines fail to fully take advantage of the very large volume of monolingual source and/or parallel data that is often available. Making a better use of data is particularly critical in domain adaptation scenarios, where parallel adaptation data is usually assumed to be small in comparison to out-of-domain parallel data, or to in-domain monolingual texts. This situation sharply contrasts with the previous generation of statistical MT engines BIBREF1 , which could seamlessly integrate very large amounts of non-parallel documents, usually with a large positive effect on translation quality.
Question: why are their techniques cheaper to implement?
Answer: They use a slightly modified copy of the target to create the pseudo-text instead of full BT to make their technique cheaper
They do not require the availability of a backward translation engine.
Paper: Using Monolingual Data in Neural Machine Translation: a Systematic Study
We are mostly interested with the following training scenario: a large out-of-domain parallel corpus, and limited monolingual in-domain data. We focus here on the Europarl domain, for which we have ample data in several languages, and use as in-domain training data the Europarl corpus BIBREF5 for two translation directions: English INLINEFORM0 German and English INLINEFORM1 French. As we study the benefits of monolingual data, most of our experiments only use the target side of this corpus. The rationale for choosing this domain is to (i) to perform large scale comparisons of synthetic and natural parallel corpora; (ii) to study the effect of BT in a well-defined domain-adaptation scenario. For both language pairs, we use the Europarl tests from 2007 and 2008 for evaluation purposes, keeping test 2006 for development. When measuring out-of-domain performance, we will use the WMT newstest 2014.
Question: what data simulation techniques were introduced?
Paper: Using Monolingual Data in Neural Machine Translation: a Systematic Study
Our baseline NMT system implements the attentional encoder-decoder approach BIBREF6 , BIBREF7 as implemented in Nematus BIBREF8 on 4 million out-of-domain parallel sentences. For French we use samples from News-Commentary-11 and Wikipedia from WMT 2014 shared translation task, as well as the Multi-UN BIBREF9 and EU-Bookshop BIBREF10 corpora. For German, we use samples from News-Commentary-11, Rapid, Common-Crawl (WMT 2017) and Multi-UN (see table TABREF5 ). Bilingual BPE units BIBREF11 are learned with 50k merge operations, yielding vocabularies of about respectively 32k and 36k for English INLINEFORM0 French and 32k and 44k for English INLINEFORM1 German.
Question: what is their explanation for the effectiveness of back-translation?
Paper: Using Monolingual Data in Neural Machine Translation: a Systematic Study
A simple way to use monolingual data in MT is to turn it into synthetic parallel data and let the training procedure run as usual BIBREF16 . In this section, we explore various ways to implement this strategy. We first reproduce results of BIBREF2 with BT of various qualities, that we then analyze thoroughly.
Question: what dataset is used?
Answer: Europarl tests from 2006, 2007, 2008; WMT newstest 2014.
Paper: Using Monolingual Data in Neural Machine Translation: a Systematic Study
BT requires the availability of an MT system in the reverse translation direction. We consider here three MT systems of increasing quality:
Question: what language pairs are explored?
Answer: English-German, English-French.
English-German, English-French
Paper: Using Monolingual Data in Neural Machine Translation: a Systematic Study
Comparing the natural and artificial sources of our parallel data wrt. several linguistic and distributional properties, we observe that (see Fig. FIGREF21 - FIGREF22 ):
Question: what language is the data in?
Paper: Language Use Matters: Analysis of the Linguistic Structure of Question Texts Can Characterize Answerability in Quora
From a group of small users at the time of its inception in 2009, Quora has evolved in the last few years into one of the largest community driven Q&A sites with diverse user communities. With the help of efficient content moderation/review policies and active in-house review team, efficient Quora bots, this site has emerged into one of the largest and reliable sources of Q&A on the Internet. On Quora, users can post questions, follow questions, share questions, tag them with relevant topics, follow topics, follow users apart from answering, commenting, upvoting/downvoting etc. The integrated social structure at the backbone of it and the topical organization of its rich content have made Quora unique with respect to other Q&A sites like Stack Overflow, Yahoo! Answers etc. and these are some of the prime reasons behind its popularity in recent times. Quality question posting and getting them answered are the key objectives of any Q&A site. In this study we focus on the answerability of questions on Quora, i.e., whether a posted question shall eventually get answered. In Quora, the questions with no answers are referred to as “open questions”. These open questions need to be studied separately to understand the reason behind their not being answered or to be precise, are there any characteristic differences between `open' questions and the answered ones. For example, the question “What are the most promising advances in the treatment of traumatic brain injuries?” was posted on Quora on 23rd June, 2011 and got its first answer after almost 2 years on 22nd April, 2013. The reason that this question remained open so long might be the hardness of answering it and the lack of visibility and experts in the domain. Therefore, it is important to identify the open questions and take measures based on the types - poor quality questions can be removed from Quora and the good quality questions can be promoted so that they get more visibility and are eventually routed to topical experts for better answers.
Question: Does the experiments focus on a specific domain?
Paper: Language Use Matters: Analysis of the Linguistic Structure of Question Texts Can Characterize Answerability in Quora
We obtained our Quora dataset BIBREF7 through web-based crawls between June 2014 to August 2014. This crawling exercise has resulted in the accumulation of a massive Q&A dataset spanning over a period of over four years starting from January 2010 to May 2014. We initiated crawling with 100 questions randomly selected from different topics so that different genre of questions can be covered. The crawling of the questions follow a BFS pattern through the related question links. We obtained 822,040 unique questions across 80,253 different topics with a total of 1,833,125 answers to these questions. For each question, we separately crawl their revision logs that contain different types of edit information for the question and the activity log of the question asker.
Question: how many training samples do you have for training?
Paper: Language Use Matters: Analysis of the Linguistic Structure of Question Texts Can Characterize Answerability in Quora
In this section, we identify various linguistic activities on Quora and propose quantifications of the language usage patterns in this Q&A site. In particular, we show that there exists significant differences in the linguistic structure of the open and the answered questions. Note that most of the measures that we define are simple, intuitive and can be easily obtained automatically from the data (without manual intervention). Therefore the framework is practical, inexpensive and highly scalable.
Question: Do the answered questions measure for the usefulness of the answer?