z-uo/qasper-squad huggingface dataset:
Paper: AvgOut: A Simple Output-Probability Measure to Eliminate Dull Responses
By only keeping a static shortlist of boring responses or tokens, one basically assumes that we humans should decide which tokens are dull. However, we argue that we should instead look from the model's perspective to identify dull tokens, because even if the model outputs a word that we consider rare, including it in too many responses is still considered a dull behavior. Motivated by this thought experiment, we propose a novel metric, Average Output Probability Distribution (AvgOut), that dynamically keeps track of which tokens the model is biased toward. To calculate this, during training, we average out all the output probability distributions for each time step of the decoder for the whole mini-batch. The resulting vector $D^{\prime }$ will reflect each token's probability of being generated from the model's perspective. Note that we do not use discrete ground-truth tokens to evaluate the model's bias, because there is a fine distinction between the two: a statistics of frequency on ground-truth tokens is an evaluation of the corpus's bias, while AvgOut is an evaluation of what bias the model has learned because by generating dull responses more frequently than the training corpus has, it is the model itself that we should adjust. Also note that the reason we take the average is that a single output distribution will largely depend on the context and the previous target tokens (which are fed as inputs to the decoder during training), but on average the distribution should be a faithful evaluation on which words are more likely to be generated from model's perspective.
Question: How is human evaluation performed, what was the criteria?
Answer: Through Amazon MTurk annotators to determine plausibility and content richness of the response
Paper: AvgOut: A Simple Output-Probability Measure to Eliminate Dull Responses
AvgOut can play at least three roles. First, it can be used to directly supervise output distribution during training; second, it can be used as a prior in labeled sequence transduction methods to control diversity of the generated response; and third, it can be used as a reward signal for Reinforcement Learning to encourage diverse sampled responses. In this section, we begin with a base vanilla seq2seq model, and next present our three models to diversify responses based on AvgOut.
Question: How much better were results of the proposed models than base LSTM-RNN model?
Answer: on diversity 6.87 and on relevance 4.6 points higher
Paper: AvgOut: A Simple Output-Probability Measure to Eliminate Dull Responses
Our MinAvgOut model (Figure FIGREF3) directly integrates AvgOut into the loss function by summarizing it into a single numerical value named Continuous-AvgOut. We do this by taking the dot-product of $D$ and $D^{\prime }$ (Figure FIGREF6). The intuition behind this simple calculation is that $D$ can also be viewed as a set of weights which add up to $1.0$, since it is a probability vector. By taking the dot product, we are actually calculating a weighted average of each probability in $D^{\prime }$. To evaluate how diverse the model currently is, the duller tokens should obviously carry higher weights since they contribute more to the "dullness" of the whole utterance. Assuming that $D$ is a column vector, the continuous diversity score is $B_c$, and the resulting extra loss term is $L_B$, the total loss $L$ is given by:
Question: Which one of the four proposed models performed best?
Answer: the hybrid model MinAvgOut + RL
Paper: Generative Dialog Policy for Task-oriented Dialog Systems
Task-oriented dialogue system is an important tool to build personal virtual assistants, which can help users to complete most of the daily tasks by interacting with devices via natural language. It's attracting increasing attention of researchers, and lots of works have been proposed in this area BIBREF0, BIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6, BIBREF7.
Question: What metrics are used to measure performance of models?
Paper: Generative Dialog Policy for Task-oriented Dialog Systems
Usually, the existing task-oriented dialogue systems use a pipeline of four separate modules: natural language understanding, dialogue belief tracker, dialogue policy and natural language generator. Among these four modules, dialogue policy maker plays a key role in task-oriented dialogue systems, which generates the next dialogue action.
Question: How much is proposed model better than baselines in performed experiments?
Answer: most of the models have similar performance on BPRA: DSTC2 (+0.0015), Maluuba (+0.0729)
GDP achieves the best performance in APRA: DSTC2 (+0.2893), Maluuba (+0.2896)
GDP significantly outperforms the baselines on BLEU: DSTC2 (+0.0791), Maluuba (+0.0492)
Paper: Generative Dialog Policy for Task-oriented Dialog Systems
Seq2Seq model was first introduced by BIBREF15 for statistical machine translation. It uses two recurrent neural networks (RNN) to solve the sequence-to-sequence mapping problem. One called encoder encodes the user utterance into a dense vector representing its semantics, the other called decoder decodes this vector to the target sentence. Now Seq2Seq framework has already been used in task-oriented dialog systems such as BIBREF4 and BIBREF1, and shows the challenging performance. In the Seq2Seq model, given the user utterance $Q=(x_1, x_2, ..., x_n)$, the encoder squeezes it into a context vector $C$ and then used by decoder to generate the response $R=(y_1, y_2, ..., y_m)$ word by word by maximizing the generation probability of $R$ conditioned on $Q$. The objective function of Seq2Seq can be written as:
Question: What are state-of-the-art baselines?
Paper: Generative Dialog Policy for Task-oriented Dialog Systems
Attention mechanisms BIBREF17 have been proved to improved effectively the generation quality for the Seq2Seq framework. In Seq2Seq with attention, each $y_i$ corresponds to a context vector $C_i$ which is calculated dynamically. It is a weighted average of all hidden states of the encoder RNN. Formally, $C_i$ is defined as $C_i=\sum _{j=1}^{n} \alpha _{ij}h_j$, where $\alpha _{ij}$ is given by:
Question: What two benchmark datasets are used?
Paper: Bidirectional Context-Aware Hierarchical Attention Network for Document Understanding
Recently, hierarchical architectures have become ubiquitous in NLP. They have been applied to a wide variety of tasks such as language modeling and generation BIBREF0, BIBREF1, neural machine translation (NMT) BIBREF2, summarization BIBREF3, sentiment and topic classification BIBREF4, BIBREF5, and spoken language understanding BIBREF6, BIBREF7, to cite only a few examples. All hierarchical architectures capitalize on the same intuitive idea that the representation of the input text should be learned in a bottom-up fashion by using a different encoder at each granularity level (e.g., words, sentences, paragraphs), where the encoder at level $l+1$ takes as input the output of the encoder at level $l$.
Question: What languages are the model evaluated on?
Paper: Bidirectional Context-Aware Hierarchical Attention Network for Document Understanding
HAN was highly successful and established new state of the art on six large-scale sentiment and topic classification datasets. However, it has a major weakness: at level 1, each sentence is encoded in isolation. That is, while producing the representation of a given sentence in the document, HAN completely ignores the other sentences. This lack of communication is obviously suboptimal. For example, in Fig. FIGREF2, the same highly negative feature (“terrible value”) has been repeated at the beginning of each sentence in the document. Because it encodes each sentence independently, HAN has no choice but to spend most of its attentional budget on the most salient feature every time. As a result, HAN neglects the other aspects of the document. On the other hand, CAHAN is informed about the context, and thus quickly stops spending attention weight on the same highly negative pattern, knowing that is has already been covered. CAHAN is then able to cover the other topics in the document (“seafood”,“scallops” and “mussels”; “entree” and “appetizer”; triple negation in the fourth sentence).
Question: Do they compare to other models appart from HAN?
Paper: Bidirectional Context-Aware Hierarchical Attention Network for Document Understanding
In this work, we propose and evaluate several modifications of the HAN architecture that allow the sentence encoder at level 1 to make its attentional decisions based on contextual information, allowing it to learn richer document representations. Another significant contribution is the introduction of a bidirectional version of the document encoder, where one RNN processes the document forwards, using the preceding sentences as context, and another one processes it backwards, using the following sentences as context.
Question: What are the datasets used
Paper: Twitter Sentiment Analysis via Bi-sense Emoji Embedding and Attention-based LSTM
The rapid growth of social media platforms such as Twitter provides rich multimedia data in large scales for various research opportunities, such as sentiment analysis which focuses on automatically sentiment (positive and negative) prediction on given contents. Sentiment analysis has been widely used in real world applications by analyzing the online user-generated data, such as election prediction, opinion mining and business-related activity analysis. Emojis, which consist of various symbols ranging from cartoon facial expressions to figures such as flags and sports, are widely used in daily communications to express people's feelings . Since their first release in 2010, emojis have taken the place of emoticons (such as “:- INLINEFORM0 ” and “:-P”) BIBREF0 to create a new form of language for social media users BIBREF1 . According to recent science reports, there are 2,823 emojis in unicode standard in Emoji 11.0 , with over 50% of the Instagram posts containing one or more emojis BIBREF2 and 92% of the online population using emojis BIBREF3 .
Question: Do they evaluate only on English datasets?
Paper: Twitter Sentiment Analysis via Bi-sense Emoji Embedding and Attention-based LSTM
Sentiment analysis is to extract and quantify subjective information including the status of attitudes, emotions and opinions from a variety of contents such as texts, images and audios BIBREF18 . Sentiment analysis has been drawing great attentions because of its wide applications in business and government intelligence, political science, sociology and psychology BIBREF19 , BIBREF20 , BIBREF21 , BIBREF22 . From a technical perspective, textual sentiment analysis is first explored by researchers as an NLP task. Methods range from lexical-based approaches using features including keywords BIBREF23 , BIBREF24 where each word corresponds to a sentiment vector with entries representing the possibility of the word and each sentiment and phase-level features (n-grams and unigrams) BIBREF25 , BIBREF26 , to deep neural network based embedding approaches including skip-grams, continuous bag-of-words (CBoW) and skip-thoughts BIBREF27 , BIBREF28 , BIBREF16 , BIBREF29 . It was until recent years when researchers start focusing on image and multimodal sentiments BIBREF30 , BIBREF31 and analyzing how to take advantage of the cross-modality resources BIBREF10 , BIBREF32 . For multimodal sentiment analysis, an underlying assumption is that both modalities express similar sentiment and such similarity is enforced in order to train a robust sentiment inference model BIBREF10 . However, the same assumption does not stand in modeling textual tweets and emojis because the complexities of natural language exist extensively, such as the use of irony, jokes, sarcasm, etc. BIBREF9 .
Question: What evidence does visualizing the attention give to show that it helps to obtain a more robust understanding of semantics and sentiments?
Paper: Twitter Sentiment Analysis via Bi-sense Emoji Embedding and Attention-based LSTM
With the overwhelming development of Internet of Things (IOT), the growing accessibility and popularity of subjective contents have provided new opportunities and challenges for sentiment analysis BIBREF33 . For example, social medias such as Twitter and Instagram have been explored because the massive user-generated contents with rich user sentiments BIBREF25 , BIBREF34 , BIBREF35 where emojis (and emoticons) are extensively used. Non-verbal cues of sentiment, such as emoticon which is considered as the previous generation of emoji, has been studied for their sentiment effect before emojis take over BIBREF36 , BIBREF37 , BIBREF38 . For instance, BIBREF36 , BIBREF38 pre-define sentiment labels to emoticons and construct a emoticon-sentiment dictionary. BIBREF37 applies emoticons for smoothing noisy sentiment labels. Similar work from BIBREF39 first considers emoji as a component in extracting the lexical feature for further sentiment analysis. BIBREF40 constructs an emoji sentiment ranking based on the occurrences of emojis and the human-annotated sentiments of the corresponding tweets where each emoji is assigned with a sentiment score from negative to positive , similar to the SentiWordNet BIBREF41 . However, the relatively intuitive use of emojis by lexical- and dictionary-based approaches lacks insightful understanding of the complexed semantics of emojis. Therefore, inspired by the success of word semantic embedding algorithms such as BIBREF28 , BIBREF16 , BIBREF7 obtains semantic embeddings of each emoji by averaging the words from its descriptions and shows it is effective to take advantage of the emoji embedding for the task of Twitter sentiment analysis. BIBREF8 proposes a convoluntional neural network to predict the emoji occurrence and jointly learns the emoji embedding via a matching layer based on cosine similarities. Despite the growing popularity of Twitter sentiment analysis, there is a limited number of emoji datasets with sentiment labels available because previous studies usually filter out urls, emojis and sometimes emoticons. However, BIBREF9 shows that it is effective to extract sentiment information from emojis for emotion classification and sarcasm detection tasks in the absence of learning vector-based emoji representations by pre-training a deep neural network to predict the emoji occurrence.
Question: Which SOTA models are outperformed?
Paper: Twitter Sentiment Analysis via Bi-sense Emoji Embedding and Attention-based LSTM
We propose two mechanisms, namely Word-guide Attention-based LSTM and Multi-level Attention-based LSTM, to take advantage of bi-sense emoji embedding for the sentiment analysis task. The frameworks of these two methods are shown in Figure FIGREF10 and Figure FIGREF19 , respectively. Our workflow includes the following steps: initialization of bi-sense emoji embedding, generating senti-emoji embedding based on self-selected attention, and sentiment classification via the proposed attention-based LSTM networks.
Question: What is the baseline for experiments?
Paper: Twitter Sentiment Analysis via Bi-sense Emoji Embedding and Attention-based LSTM
Recent research shows great success in word embedding task such as word2vec and fasttext BIBREF27 , BIBREF16 . We use fasttext to initialize emoji embeddings by considering each emoji as a special word, together with word embeddings. The catch is, different from conventional approaches where each emoji responds to one embedding vector (as we call word-emoji embedding), we embed each emoji into two distinct vectors (bi-sense emoji embedding): we first assign two distinct tokens to each emoji, of which one is for the particular emoji used in positive sentimental contexts and the other one is for this emoji used in negative sentimental contexts (text sentiment initialized by Vader BIBREF17 , details will be discussed in Section SECREF23 ), respectively; the same fasttext training process is used to embed each token into a distinct vector, and we thus obtain the positive-sense and negative-sense embeddings for each emoji.
Question: What is the motivation for training bi-sense embeddings?
Paper: Bridging the Gap for Tokenizer-Free Language Models
There has been a recent surge of improvements in language modeling, powered by the introduction of the transformer architecture BIBREF0. These gains stem from the ability of the transformer self-attention mechanism to better model long context (as compared to RNN networks), spanning hundreds of characters BIBREF1 or words BIBREF2, BIBREF3. These approaches consider language modeling as a classification problem with the aim of predicting the next token given a fixed-size preceding context. To support variable-length context, BIBREF4 adds recurrence to a transformer model, improving the state-of-the-art further.
Question: How many parameters does the model have?
Paper: Bridging the Gap for Tokenizer-Free Language Models
Language models (LMs) assign a probability distribution over a sequence $x_{0:t}$ by factoring out the joint probability from left to right as follows
Question: How many characters are accepted as input of the language model?
Paper: Features in Extractive Supervised Single-document Summarization: Case of Persian News
From the early days of artificial intelligence, automatically summarizing a text was an interesting task for many researchers. Followed by the advance of the World Wide Web and the advent of concepts such as Social networks, Big Data, and Cloud computing among others, text summarization became a crucial task in many applications BIBREF0, BIBREF1, BIBREF2. For example, it is essential, in many search engines and text retrieval systems to display a portion of each result entry which is representative of the whole text BIBREF3, BIBREF4. It is also becoming essential for managers and the general public to gain the gist of news and articles immediately, in order to save time, while being inundated with information on social media BIBREF5.
Question: What dataset is used for this task?
Paper: Features in Extractive Supervised Single-document Summarization: Case of Persian News
Text summarization has been widely studied by both academic and enterprise disciplines. Text summarization methods may be classified into different types. Based on input type, there are single-document BIBREF11, BIBREF12 vs multi-document summarization methods BIBREF13, BIBREF14, BIBREF15. Based on language, there are mono-lingual, bilingual and multi-lingual methods BIBREF16. There are also “query focused” methods in which a summary relevant to a given query is produced BIBREF17. From the perspective of procedure, however, there are two main approaches: abstractive vs extractive BIBREF18.
Question: What features of the document are integrated into vectors of every sentence?
Paper: Features in Extractive Supervised Single-document Summarization: Case of Persian News
As a way to investigate the need for document features in sentence ranking (as explained in the introduction and related works), we introduced several document-level features and incorporated them in the summarization process. These features are listed under subsection (SECREF4). Although stages of our method do not differ from general supervised extractive summarization, the whole process is explained in order to clarify the method of investigation.
Question: By how much is precission increased?
Answer: ROUGE-1 increases by 0.05, ROUGE-2 by 0.06 and ROUGE-L by 0.09
Paper: Features in Extractive Supervised Single-document Summarization: Case of Persian News
The input to this phase is a dataset of documents, each of which is associated with several human-written summaries. The output is a learned model with a good level of accuracy that is able to reliably predict the rank of sentences, in almost the same way that a human may rank them. To accomplish this, it is necessary to first perform normalization and transform various forms of phrases into their canonical form. Then, every text should be tokenized to sentences, and further tokenized to words. Another prerequisite is to remove stop words. The following subtasks should be carried out next.
Question: Is new approach tested against state of the art?
Paper: Dreaddit: A Reddit Dataset for Stress Analysis in Social Media
In our online world, social media users tweet, post, and message an incredible number of times each day, and the interconnected, information-heavy nature of our lives makes stress more prominent and easily observable than ever before. With many platforms such as Twitter, Reddit, and Facebook, the scientific community has access to a massive amount of data to study the daily worries and stresses of people across the world.
Question: Is the dataset balanced across categories?
Paper: Dreaddit: A Reddit Dataset for Stress Analysis in Social Media
Because of the subjective nature of stress, relevant research tends to focus on physical signals, such as cortisol levels in saliva BIBREF2, electroencephalogram (EEG) readings BIBREF3, or speech data BIBREF4. This work captures important aspects of the human reaction to stress, but has the disadvantage that hardware or physical presence is required. However, because of the aforementioned proliferation of stress on social media, we believe that stress can be observed and studied purely from text.
Question: What supervised methods are used?
Paper: Dreaddit: A Reddit Dataset for Stress Analysis in Social Media
Reddit is a social media website where users post in topic-specific communities called subreddits, and other users comment and vote on these posts. The lengthy nature of these posts makes Reddit an ideal source of data for studying the nuances of phenomena like stress. To collect expressions of stress, we select categories of subreddits where members are likely to discuss stressful topics:
Question: What labels are in the dataset?
Answer: binary label of stress or not stress
Paper: Dreaddit: A Reddit Dataset for Stress Analysis in Social Media
We annotate a subset of the data using Amazon Mechanical Turk in order to begin exploring the characteristics of stress. We partition the posts into contiguous five-sentence chunks for labeling; we wish to annotate segments of the posts because we are ultimately interested in what parts of the post depict stress, but we find through manual inspection that some amount of context is important. Our posts, however, are quite long, and it would be difficult for annotators to read and annotate entire posts. This type of data will allow us in the future not only to classify the presence of stress, but also to locate its expressions in the text, even if they are diffused throughout the post.
Question: What categories does the dataset come from?
Paper: Corporate IT-Support Help-Desk Process Hybrid-Automation Solution with Machine Learning Approach
In an organization, the Information Technology (IT) support help desk operation is an important unit which handles the IT services of a business. Many large scale organizations would have a comprehensive IT support team to handle engagement and requests with employees on a 24$\times $7 basis. As any routinized tasks, most processes of the support help desk unit are considered repetitive in nature BIBREF0. Some may occur on a daily basis and others may occur more frequently. Many support engineers and agent would spend time on these repetitive task such as entering information to an application, resetting passwords, unlocking applications, creating credentials, activating services, preparing documentation, etc.
Question: What are all machine learning approaches compared in this work?