z-uo/qasper-squad huggingface dataset:
Paper: BERT has a Moral Compass: Improvements of ethical and moral values of machines
This section investigates empirically whether text corpora contain recoverable and accurate imprints of our moral choices. Specifically, we move beyond BIBREF0, by showing that BERT has a more accurate moral representation than that of the Universal Sentence Encoder.
Question: How do the authors define deontological ethical reasoning?
Paper: Learning from Easy to Complex: Adaptive Multi-curricula Learning for Neural Dialogue Generation
Teaching machines to converse with humans naturally and engagingly is a fundamentally interesting and challenging problem in AI research. Many contemporary state-of-the-art approaches BIBREF0, BIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6 for dialogue generation follow the data-driven paradigm: trained on a plethora of query-response pairs, the model attempts to mimic human conversations. As a data-driven approach, the quality of generated responses in neural dialogue generation heavily depends on the training data. As such, in order to train a robust and well-behaved model, most works obtain large-scale query-response pairs by crawling human-generated conversations from publicly available sources such as OpenSubtitles BIBREF7.
Question: How does framework automatically chooses different curricula at the evolving learning process according to the learning status of the neural dialogue generation model?
Paper: Learning from Easy to Complex: Adaptive Multi-curricula Learning for Neural Dialogue Generation
Intuitively, a well-organized curriculum should provide the model learning with easy dialogues first, and then gradually increase the curriculum difficulty. However, currently, there is no unified approach for dialogue complexity evaluation, where the complexity involves multiple aspects of attributes. In this paper, we prepare the syllabus for dialogue learning with respect to five dialogue attributes. To ensure the universality and general applicability of the curriculum, we perform an in-depth investigation on three publicly available conversation corpora, PersonaChat BIBREF12, DailyDialog BIBREF13 and OpenSubtitles BIBREF7, consisting of 140 248, 66 594 and 358 668 real-life conversation samples, respectively.
Question: What human judgement metrics are used?
Paper: Learning from Easy to Complex: Adaptive Multi-curricula Learning for Neural Dialogue Generation
A notorious problem for neural dialogue generation model is that the model is prone to generate generic responses. The most unspecific responses are easy to learn, but are short and meaningless, while the most specific responses, consisting of too many rare words, are too difficult to learn, especially at the initial learning stage. Following BIBREF11, we measure the specificity of the response in terms of each word $w$ using Normalized Inverse Document Frequency (NIDF, ranging from 0 to 1):
Question: What automatic evaluation metrics are used?
Paper: Learning from Easy to Complex: Adaptive Multi-curricula Learning for Neural Dialogue Generation
Repetitive responses are easy to generate in current auto-regressive response decoding, where response generation loops frequently, whereas diverse and informative responses are much more complicated for neural dialogue generation. We measure the repetitiveness of a response $r$ as:
Question: What state of the art models were used in experiments?
Paper: Learning from Easy to Complex: Adaptive Multi-curricula Learning for Neural Dialogue Generation
A conversation is considered to be coherent if the response correlates well with the given query. For example, given a query “I like to paint”, the response “What kind of things do you paint?” is more relevant and easier to learn than another loosely-coupled response “Do you have any pets?”. Following previous work BIBREF14, we measure the query-relatedness using the cosine similarities between the query and its corresponding response in the embedding space: $\textit {cos\_sim}(\textit {sent\_emb}(c), \textit {sent\_emb}(r))$, where $c$ is the query and $r$ is the response. The sentence embedding is computed by taking the average word embedding weighted by the smooth inverse frequency $\textit {sent\_emb}(e)=\frac{1}{|e|}\sum _{w\in {}e}\frac{0.001}{0.001 + p(w)}emb(w)$ of words BIBREF15, where $emb(w)$ and $p(w)$ are the embedding and the probability of word $w$ respectively.
Question: What five dialogue attributes were analyzed?
Paper: Learning from Easy to Complex: Adaptive Multi-curricula Learning for Neural Dialogue Generation
A coherent response not only responds to the given query, but also triggers the next utterance. An interactive conversation is carried out for multiple rounds and a response in the current turn also acts as the query in the next turn. As such, we introduce the continuity metric, which is similar to the query-relatedness metric, to assess the continuity of a response $r$ with respect to the subsequent utterance $u$, by measuring the cosine similarities between them.
Question: What three publicly available coropora are used?
Paper: Incorporating Priors with Feature Attribution on Text Classification
One of the recent challenges in machine learning (ML) is interpreting the predictions made by models, especially deep neural networks. Understanding models is not only beneficial, but necessary for wide-spread adoption of more complex (and potentially more accurate) ML models. From healthcare to financial domains, regulatory agencies mandate entities to provide explanations for their decisions BIBREF0 . Hence, most machine learning progress made in those areas is hindered by a lack of model explainability – causing practitioners to resort to simpler, potentially low-performance models. To supply for this demand, there has been many attempts for model interpretation in recent years for tree-based algorithms BIBREF1 and deep learning algorithms BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 . On the other hand, the amount of research focusing on explainable natural language processing (NLP) models BIBREF8 , BIBREF9 , BIBREF10 is modest as opposed to image explanation techniques.
Question: Which datasets do they use?
Paper: A Density Ratio Approach to Language Model Fusion in End-to-End Automatic Speech Recognition
End-to-end models such as Listen, Attend & Spell (LAS) BIBREF0 or the Recurrent Neural Network Transducer (RNN-T) BIBREF1 are sequence models that directly define $P(W | X)$, the posterior probability of the word or subword sequence $W$ given an audio frame sequence $X$, with no chaining of sub-module probabilities. State-of-the-art, or near state-of-the-art results have been reported for these models on challenging tasks BIBREF2, BIBREF3.
Question: What metrics are used for evaluation?
Answer: word error rate
Paper: A Density Ratio Approach to Language Model Fusion in End-to-End Automatic Speech Recognition
Generative models and Bayes' rule. The Noisy Channel Model underlying the origins of statistical ASR BIBREF12 used Bayes' rule to combine generative models of both the acoustics $p(X|W)$ and the symbol sequence $P(W)$:
Question: How much training data is used?
Answer: 163,110,000 utterances
Paper: A Density Ratio Approach to Language Model Fusion in End-to-End Automatic Speech Recognition
The model makes the following assumptions:
Question: How is the training data collected?
Paper: A Density Ratio Approach to Language Model Fusion in End-to-End Automatic Speech Recognition
The RNN Transducer (RNN-T) BIBREF1 defines a sequence-level posterior $P(W|X)$ for a given acoustic feature vector sequence $X = {\mbox{\bf x}}_1, ..., {\mbox{\bf x}}_T$ and a given word or sub-word sequence $W = s_1, ..., s_U$ in terms of possible alignments $S_W = \lbrace ..., ({\bf s}, {\bf t}), ... \rbrace $ of $W$ to $X$. The tuple $({\bf s}, {\bf t})$ denotes a specific alignment sequence, a symbol sequence and corresponding sequence of time indices, consistent with the sequence $W$ and utterance $X$. The symbols in ${\bf s}$ are elements of an expanded symbol space that includes optional, repeatable blank symbols used to represent acoustics-only path extensions, where the time index is incremented, but no non-blank symbols are added. Conversely, non-blank symbols are only added to a partial path time-synchronously. (I.e., using $i$ to index elements of ${\bf s}$ and ${\bf t}$, $t_{i+1} = t_i + 1$ if $s_{i+1}$ is blank, and $t_{i + 1} = t_i$ if $s_{i+1}$ is non-blank). $P(W|X)$ is defined by summing over alignment posteriors:
Question: What language(s) is the model trained/tested on?
Paper: Measuring Conversational Fluidity in Automated Dialogue Agents
Conversational interactions between humans and Artificial Intelligence (AI) agents could amount to as much as thousands of interactions a day given recent developments BIBREF0. This surge in human-AI interactions has led to an interest in developing more fluid interactions between agent and human. The term `fluidity', when we refer to dialogue systems, tries to measure the concept of how humanlike communication is between a human and an AI entity. Conversational fluidity has historically been measured using metrics such as perplexity, recall, and F1-scores. However, one finds various drawbacks using these metrics. During the automatic evaluation stage of the second Conversational Intelligence Challenge (ConvAI2) BIBREF1 competition, it was noted that consistently replying with “I am you to do and your is like” would outperform the F1-score of all the models in the competition. This nonsensical phrase was constructed simply by picking several frequent words from the training set. Also, Precision at K, or the more specific Hits@1 metric has been used historically in assessing retrieval based aspects of the agent. This is defined as the accuracy of the next dialogue utterance when choosing between the gold response and N–1 distractor responses. Since these metrics are somewhat flawed, human evaluations were used in conjunction. Multiple attempts have been made historically to try to develop automatic metrics to assess dialogue fluidity. One of the earliest Eckert et al. (1997), used a stochastic system which regulated user-generated dialogues to debug and evaluate chatbots BIBREF2. In the same year, Marilyn et al. (1997) proposed the PARADISE BIBREF3 model. This framework was developed to evaluate dialogue agents in spoken conversations. A few years later the BLEU BIBREF4 metric was proposed. Subsequently, for almost two decades, this metric has been one of the few to be widely adopted by the research community. The method, which compares the matches in n-grams from the translated outputted text and the input text proved to be quick, inexpensive and has therefore been widely used. Therefore, we use the BLEU metric as a baseline to compare the quality of our proposed model.
Question: was bert used?
Paper: Measuring Conversational Fluidity in Automated Dialogue Agents
For this study, we use two types of data namely single-turn and multi-turn. The first type, single-turn, is defined such that each instance is made up of one statement and one response. This pair is usually a fragment of a larger dialogue. When given to humans for evaluation of fluidity, we ask to give a score on characteristics such as “How related is the response to the statement?” or “Does the response contain repeated text from the user's statement?”. These are all things that should not be affected by the fact that no history or context is provided and therefore, can still be classified reasonably. Contrary to the single turn datasets, the second type is the multi-turn dataset. This contains multiple instances of statements and responses, building on each other to create a fuller conversation. With these kinds of datasets, one can also evaluate and classify the data on various other attributes. An example of such evaluations would be something like “Does this response continue on the flow of the conversation?” or “Is the chatbot using repetitive text from previous responses?”. The details of how we collected each dataset are detailed below.
Question: what datasets did they use?
Paper: Measuring Conversational Fluidity in Automated Dialogue Agents
This section discusses the methods and used to develop our attributes and the technical details of how they are combined to create a final classification layer.
Question: which existing metrics do they compare with?
Paper: Attention Is (not) All You Need for Commonsense Reasoning
Recently, neural models pre-trained on a language modeling task, such as ELMo BIBREF0 , OpenAI GPT BIBREF1 , and BERT BIBREF2 , have achieved impressive results on various natural language processing tasks such as question-answering and natural language inference. The success of BERT can largely be associated to the notion of context-aware word embeddings, which differentiate it from common approaches such as word2vec BIBREF3 that establish a static semantic embedding. Since the introduction of BERT, the NLP community continues to be impressed by the amount of ideas produced on top of this powerful language representation model. However, despite its success, it remains unclear whether the representations produced by BERT can be utilized for tasks such as commonsense reasoning. Particularly, it is not clear whether BERT shed light on solving tasks such as the Pronoun Disambiguation Problem (PDP) and Winograd Schema Challenge (WSC). These tasks have been proposed as potential alternatives to the Turing Test, because they are formulated to be robust to statistics of word co-occurrence BIBREF4 .
Question: Which datasets do they evaluate on?
Paper: Attention Is (not) All You Need for Commonsense Reasoning
In this section we first review the main aspects of the BERT approach, which are important to understand our proposal and we introduce notations used in the rest of the paper. Then, we introduce Maximum Attention Score (MAS), and explain how it can be utilized for commonsense reasoning.
Question: How does their model differ from BERT?
Answer: Their model does not differ from BERT.
Paper: Incorporating Structured Commonsense Knowledge in Story Completion
Narrative is a fundamental form of representation in human language and culture. Stories connect individuals and deliver experience, emotions and knowledge. Narrative comprehension has attracted long-standing interests in natural language processing (NLP) BIBREF1 , and is widely applicable to areas such as content creation. Enabling machines to understand narrative is also an important first step towards real intelligence. Previous studies on narrative comprehension include character roles identification BIBREF2 , narratives schema construction BIBREF3 , and plot pattern identification BIBREF4 . However, their main focus is on analyzing the stories themselves. In contrast, we concentrate on training machines to predict the end of the stories. Story completion tasks rely not only on the logic of the story itself, but also requires implicit commonsense knowledge outside the story. To understand stories, human can use the information from both the story itself and other implicit sources such as commonsense knowledge and normative social behaviors BIBREF5 . In this paper, we propose to imitate such behaviors to incorporate structured commonsense knowledge to aid the story ending prediction.
Question: Which metrics are they evaluating with?
Paper: On the Importance of the Kullback-Leibler Divergence Term in Variational Autoencoders for Text Generation
Despite the recent success of deep generative models such as Variational Autoencoders (VAEs) BIBREF0 and Generative Adversarial Networks (GANs) BIBREF1 in different areas of Machine Learning, they have failed to produce similar generative quality in NLP. In this paper we focus on VAEs and their mathematical underpinning to explain their behaviors in the context of text generation.
Question: What different properties of the posterior distribution are explored in the paper?
Paper: On the Importance of the Kullback-Leibler Divergence Term in Variational Autoencoders for Text Generation
We take the encoder-decoder of VAEs as the sender-receiver in a communication network. Given an input message $x$, a sender generates a compressed encoding of $x$ denoted by $z$, while the receiver aims to fully decode $z$ back into $x$. The quality of this communication can be explained in terms of rate (R) which measures the compression level of $z$ as compared to the original message $x$, and distortion (D) which quantities the overall performance of the communication in encoding a message at sender and successfully decoding it at the receiver. Additionally, the capacity of the encoder channel can be measured in terms of the amount of mutual information between $x$ and $z$, denoted by $\text{I}({x};{z})$ BIBREF17.
Question: Why does proposed term help to avoid posterior collapse?
Paper: On the Importance of the Kullback-Leibler Divergence Term in Variational Autoencoders for Text Generation
The reconstruction loss can naturally measure distortion ($D := - \big \langle \log p_\theta ({x}|{z}) \big \rangle $), while the KL term quantifies the amount of compression (rate; $R := D_{KL}[q_\phi ({z}|{x})|| p({z})]$) by measuring the divergence between a channel that transmits zero bit of information about $x$, denoted by $p(z)$, and the encoder channel of VAEs, $q_\phi (z|x)$.
Question: How does explicit constraint on the KL divergence term that authors propose looks like?
Answer: Answer with content missing: (Formula 2) Formula 2 is an answer: 
\big \langle\! \log p_\theta({x}|{z}) \big \rangle_{q_\phi({z}|{x})}  -  \beta |D_{KL}\big(q_\phi({z}|{x}) || p({z})\big)-C|
Paper: Seshat: A tool for managing and verifying annotation campaigns of audio data
Large corpora of speech, obtained in the laboratory and in naturalistic conditions, become easier to collect. This new trend broadens the scope of scientific questions on speech and language that can be answered. However, this poses an important challenge for the construction of reliable and usable annotations. Managing annotators and ensuring the quality of their annotations are highly demanding tasks for research endeavours and industrial projects BIBREF0. When organised manually, the manager of annotation campaigns usually faces three major problems: the mishandling of files (e.g., character-encoding problems, incorrect naming of files), the non-conformity of the annotations BIBREF1, and the inconsistency of the annotations BIBREF2.
Question: Did they experiment with the tool?
Paper: Seshat: A tool for managing and verifying annotation campaigns of audio data
Self-hosted annotation systems. There are many standalone solutions for the transcription of speech data that are already used by researchers: Transcriber BIBREF3, Wavesurfer BIBREF4, Praat BIBREF5, ELAN BIBREF6, XTrans BIBREF7. These systems allow the playback of sound data and the construction of different layers of annotations with various specifications, with some advanced capabilities (such as annotations with hierarchical or no relationship between layers, number of audio channels, video support). Yet, these solutions lack a management system: each researcher must track the files assigned to annotators and build a pipeline to parse (and eventually check) the output annotation files. Moreover, checking can only be done once the annotations have been submitted to the researchers. This task becomes quickly untraceable as the number of files and annotators grow. In addition, most of these transcription systems do not provide a way to evaluate consistency (intra- and inter-annotator agreement) that would be appropriate for speech data BIBREF8.
Question: Can it be used for any language?
Paper: Seshat: A tool for managing and verifying annotation campaigns of audio data
Seshat is a user-friendly web-based interface whose objective is to smoothly manage large campaigns of audio data annotation, see Figure FIGREF8. Below, we describe the several terms used in Seshat's workflow:
Question: Is this software available to the public?
Paper: Discriminating between similar languages in Twitter using label propagation
Language identification is a crucial first step in textual data processing and is considered feasible over formal texts BIBREF0 . The task is harder for social media (e.g. Twitter) where text is less formal, noisier and can be written in wide range of languages. We focus on identifying similar languages, where surface-level content alone may not be sufficient. Our approach combines a content model with evidence propagated over the social network of the authors. For example, a user well-connected to users posting in a language is more likely to post in that language. Our system scores 76.63%, 1.4% higher than the top submission to the tweetLID workshop.
Question: What shared task does this system achieve SOTA in?
Paper: Discriminating between similar languages in Twitter using label propagation
Traditional language identification compares a document with a language fingerprint built from n-gram bag-of-words (character or word level). Tweets carry additional metadata useful for identifying language, such as geolocation BIBREF1 , username BIBREF2 , BIBREF1 and urls mentioned in the tweet BIBREF2 . Other methods expand beyond the tweet itself to use a histogram of previously predicted languages, those of users @-mentioned and lexical content of other tweets in a discussion BIBREF1 . Discriminating between similar languages was the focus of the VarDial workshop BIBREF3 , and most submissions used content analysis. These methods make limited use of the social context in which the authors are tweeting – our research question is “Can we identify the language of a tweet using the social graph of the tweeter?”.
Question: How are labels propagated using this approach?
Paper: Discriminating between similar languages in Twitter using label propagation
Our method predicts the language INLINEFORM0 for a tweet INLINEFORM1 by combining scores from a content model and a graph model that takes social context into account, as per Equation EQREF2 :
Question: What information is contained in the social graph of tweet authors?
Paper: BB_twtr at SemEval-2017 Task 4: Twitter Sentiment Analysis with CNNs and LSTMs
Determining the sentiment polarity of tweets has become a landmark homework exercise in natural language processing (NLP) and data science classes. This is perhaps because the task is easy to understand and it is also easy to get good results with very simple methods (e.g. positive - negative words counting). The practical applications of this task are wide, from monitoring popular events (e.g. Presidential debates, Oscars, etc.) to extracting trading signals by monitoring tweets about public companies. These applications often benefit greatly from the best possible accuracy, which is why the SemEval-2017 Twitter competition promotes research in this area. The competition is divided into five subtasks which involve standard classification, ordinal classification and distributional estimation. For a more detailed description see BIBREF0 .
Question: What were the five English subtasks?
Paper: BB_twtr at SemEval-2017 Task 4: Twitter Sentiment Analysis with CNNs and LSTMs
Let us now describe the architecture of the CNN we worked with. Its architecture is almost identical to the CNN of BIBREF8 . A smaller version of our model is illustrated on Fig. FIGREF2 . The input of the network are the tweets, which are tokenized into words. Each word is mapped to a word vector representation, i.e. a word embedding, such that an entire tweet can be mapped to a matrix of size INLINEFORM0 , where INLINEFORM1 is the number of words in the tweet and INLINEFORM2 is the dimension of the embedding space (we chose INLINEFORM3 ). We follow BIBREF8 zero-padding strategy such that all tweets have the same matrix dimension INLINEFORM4 , where we chose INLINEFORM5 . We then apply several convolution operations of various sizes to this matrix. A single convolution involves a filtering matrix INLINEFORM6 where INLINEFORM7 is the size of the convolution, meaning the number of words it spans. The convolution operation is defined as DISPLAYFORM0 
Question: How many CNNs and LSTMs were ensembled?