z-uo/qasper-squad huggingface dataset:
Paper: Contextual Out-of-Domain Utterance Handling With Counterfeit Data Augmentation
Recently, there has been a surge of excitement in developing chatbots for various purposes in research and enterprise. Data-driven approaches offered by common bot building platforms (e.g. Google Dialogflow, Amazon Alexa Skills Kit, Microsoft Bot Framework) make it possible for a wide range of users to easily create dialog systems with a limited amount of data in their domain of interest. Although most task-oriented dialog systems are built for a closed set of target domains, any failure to detect out-of-domain (OOD) utterances and respond with an appropriate fallback action can lead to frustrating user experience. There have been a set of prior approaches for OOD detection which require both in-domain (IND) and OOD data BIBREF0 , BIBREF1 . However, it is a formidable task to collect sufficient data to cover in theory unbounded variety of OOD utterances. In contrast, BIBREF2 introduced an in-domain verification method that requires only IND utterances. Later, with the rise of deep neural networks, BIBREF3 proposed an autoencoder-based OOD detection method which surpasses prior approaches without access to OOD data. However, those approaches still have some restrictions such that there must be multiple sub-domains to learn utterance representation and one must set a decision threshold for OOD detection. This can prohibit these methods from being used for most bots that focus on a single task.
Question: By how much does their method outperform state-of-the-art OOD detection?
Answer: AE-HCN outperforms by 17%, AE-HCN-CNN outperforms by 20% on average
Paper: Efficient keyword spotting using dilated convolutions and gating
Keyword spotting (KWS) aims at detecting a pre-defined keyword or set of keywords in a continuous stream of audio. In particular, wake-word detection is an increasingly important application of KWS, used to initiate an interaction with a voice interface. In practice, such systems run on low-resource devices and listen continuously for a specific wake word. An effective on-device KWS therefore requires real-time response and high accuracy for a good user experience, while limiting memory footprint and computational cost.
Question: What are dilated convolutions?
Answer: Similar to standard convolutional networks but instead they skip some input values effectively operating on a broader scale.
Paper: An Open-World Extension to Knowledge Graph Completion Models
Knowledge graphs are a vital source for disambiguation and discovery in various tasks such as question answering BIBREF0 , information extraction BIBREF1 and search BIBREF2 . They are, however, known to suffer from data quality issues BIBREF3 . Most prominently, since formal knowledge is inherently sparse, relevant facts are often missing from the graph.
Question: what was the evaluation metrics studied in this work?
Paper: Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks
Parameters of the encoder-decoder were tuned on a dedicated validation set. We experienced with different learning rates (0.1, 0.01, 0.001), dropout-rates (0.1, 0.2, 0.3, 0.5) BIBREF11 and optimization techniques (AdaGrad BIBREF6 , AdaDelta BIBREF30 , Adam BIBREF15 and RMSprop BIBREF29 ). We also experimented with different batch sizes (8, 16, 32), and found improvement in runtime but no significant improvement in performance.
Question: Do they analyze ELMo?
Paper: Red Dragon AI at TextGraphs 2019 Shared Task: Language Model Assisted Explanation Generation
The Explanation Regeneration shared task asked participants to develop methods to reconstruct gold explanations for elementary science questions BIBREF1, using a new corpus of gold explanations BIBREF2 that provides supervision and instrumentation for this multi-hop inference task.
Question: what are the three methods presented in the paper?
Answer: Optimized TF-IDF, iterated TF-IDF, BERT re-ranking.
Paper: Impact of Sentiment Detection to Recognize Toxic and Subversive Online Comments
Online communities abound today, forming on social networks, on webforums, within videogames, and even in the comments sections of articles and videos. While this increased international contact and exchange of ideas has been a net positive, it has also been matched with an increase in the spread of high-risk and toxic content, a category which includes cyberbullying, racism, sexual predation, and other negative behaviors that are not tolerated in society. The two main strategies used by online communities to moderate themselves and stop the spread of toxic comments are automated filtering and human surveillance. However, given the sheer number of messages sent online every day, human moderation simply cannot keep up, and either leads to a severe slowdown of the conversation (if messages are pre-moderated before posting) or allows toxic messages to be seen and shared thousands of times before they are deleted (if they are post-moderated after being posted and reported). In addition, human moderation cannot scale up easily to the number of messages to monitor; for example, Facebook has a team of 20,000 human moderators, which is both massive compared to the total of 25,000 other employees in the company, and minuscule compared to the fact its automated algorithms flagged messages that would require 180,000 human moderators to review. Keyword detection, on the other hand, is instantaneous, scales up to the number of messages, and prevents toxic messages from being posted at all, but it can only stop messages that use one of a small set of denied words, and, are thus fairly easy to circumvent by introducing minor misspellings (i.e. writing "kl urself" instead of "kill yourself"). In BIBREF0 , the authors show how minor changes can elude even complex systems. These attempts to bypass the toxicity detection system are called subverting the system, and toxic users doing it are referred to as subversive users.
Question: what datasets did the authors use?
Answer: Kaggle
Subversive Kaggle
Wikipedia
Subversive Wikipedia
Reddit
Subversive Reddit
Paper: Phase transitions in a decentralized graph-based approach to human language
This letter arises from two intriguing questions about human language. The first question is: To what extent language, and also language evolution, can be viewed as a graph-theoretical problem? Language is an amazing example of a system of interrelated units at different organization scales. Several recent works have stressed indeed the fact that human languages can be viewed language as a (complex) network of interacting parts BIBREF0, BIBREF1, BIBREF2, BIBREF3. Within the graph-based approach to human language, one may think word-meaning mappings (that is, vocabularies) as bipartite graphs, formed by two sets: words and meanings BIBREF2.
Question: What are three possible phases for language formation?
Paper: Improving Pre-Trained Multilingual Models with Vocabulary Expansion
Comments: An approach to handle the OOV issue in multilingual BERT is proposed. A great deal of nice experiments were done but ultimately (and in message board discussions) the reviewers agreed there wasn't enough novelty or result here to justify acceptance.
Question: How many parameters does the model have?
Paper: Explaining Predictions of Non-Linear Classifiers in NLP
Following seminal work by Bengio and Collobert, the use of deep learning models for natural language processing (NLP) applications received an increasing attention in recent years. In parallel, initiated by the computer vision domain, there is also a trend toward understanding deep learning models through visualization techniques BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 or through decision tree extraction BIBREF6 . Most work dedicated to understanding neural network classifiers for NLP tasks BIBREF7 , BIBREF8 use gradient-based approaches. Recently, a technique called layer-wise relevance propagation (LRP) BIBREF4 has been shown to produce more meaningful explanations in the context of image classifications BIBREF9 . In this paper, we apply the same LRP technique to a NLP task, where a neural network maps a sequence of word2vec vectors representing a text document to its category, and evaluate whether similar benefits in terms of explanation quality are observed.
Question: Do the experiments explore how various architectures and layers contribute towards certain decisions?
Paper: Hate Speech Detection on Vietnamese Social Media Text using the Bidirectional-LSTM Model
In recent years, social networking has grown and become prevalent with every people, it makes easy for people to interact and share with each other. However, every problem has two sides. It also has some negative issues, hate speech is a hot topic in the domain of social media. With the freedom of speech on social networks and anonymity on the internet, some people are free to comment on hate and insults. Hate speech can have an adverse effect on human behavior as well as directly affect society. We don't manually delete each of those comments, which is time-consuming and boring. This spurs research to build an automated system that detects hate speech and eliminates them. With that system, we can detect and eliminate hate speech and thus reduce their spread on social media. With Vietnamese, we can use methods to apply specific extraction techniques manually and in combination with string labeling algorithms such as Conditional Random Field (CRF)[1], Model Hidden Markov (HMM)[2] or Entropy[3]. However, we have to choose the features manually to bring the model with high accuracy. Deep Neural Network architectures can handle the weaknesses of the above methods. In this report we apply Bidirectional Long Short-Term Memory (Bi-LSTM) to build the model. Also combined with the word embedding matrix to increase the accuracy of the model.
Question: What social media platform does the data come from?
Paper: Stochastic Answer Networks for Machine Reading Comprehension
Machine reading comprehension (MRC) is a challenging task: the goal is to have machines read a text passage and then answer any question about the passage. This task is an useful benchmark to demonstrate natural language understanding, and also has important applications in e.g. conversational agents and customer service support. It has been hypothesized that difficult MRC problems require some form of multi-step synthesis and reasoning. For instance, the following example from the MRC dataset SQuAD BIBREF0 illustrates the need for synthesis of information across sentences and multiple steps of reasoning:
Question: How much performance improvements they achieve on SQuAD?
Answer: Compared to baselines SAN (Table 1) shows  improvement of 1.096% on EM and 0.689% F1. Compared to other published SQuAD results (Table 2) SAN is ranked second.
Paper: Grounded Agreement Games: Emphasizing Conversational Grounding in Visual Dialogue Settings
If you're good at replying to a single request, are you also likely to be good at doing dialogue? Much current work seems to assume that the answer to this question is yes, in that it attempts a scaling up from single pairs of utterance plus response to longer dialogues: See, e.g., the work on neural chatbots following on from BIBREF0, where the main evaluation metric is “next utterance retrieval”; and on visual dialogue BIBREF1, which views itself as a natural extension of visual question answering BIBREF2.
Question: Do the authors perform experiments using their proposed method?
Paper: Natural Language Processing with Small Feed-Forward Networks
Deep and recurrent neural networks with large network capacity have become increasingly accurate for challenging language processing tasks. For example, machine translation models have been able to attain impressive accuracies, with models that use hundreds of millions BIBREF0 , BIBREF1 or billions BIBREF2 of parameters. These models, however, may not be feasible in all computational settings. In particular, models running on mobile devices are often constrained in terms of memory and computation.
Question: What NLP tasks do the authors evaluate feed-forward networks on?
Paper: Back to the Future -- Sequential Alignment of Text Representations
As time passes, language usage changes. For example, the names `Bert' and `Elmo' would only rarely make an appearance prior to 2018 in the context of scientific writing. After the publication of BERT BIBREF0 and ELMo BIBREF1, however, usage has increased in frequency. In the context of named entities on Twitter, it is also likely that these names would be tagged as person prior to 2018, and are now more likely to refer to an artefact. As such, their part-of-speech tags will also differ. Evidently, evolution of language usage affects multiple natural language processing (NLP) tasks and models based on data from one point in time cannot be expected to operate for an extended period of time.
Question: What are three challenging tasks authors evaluated their sequentially aligned representations?
Paper: Analyzing Language Learned by an Active Question Answering Agent
 BIBREF0 propose a reinforcement learning framework for question answering, called active question answering (ActiveQA), that aims to improve answering by systematically perturbing input questions (cf. BIBREF1 ). Figure 1 depicts the generic agent-environment framework. The agent (AQA) interacts with the environment (E) in order to answer a question ( $q_0$ ). The environment includes a question answering system (Q&A), and emits observations and rewards. A state $s_t$ at time $t$ is the sequence of observations and previous actions generated starting from $q_0$ : $s_t=x_0,u_0,x_1,\ldots ,u_{t-1},x_t$ , where $x_i$ includes the question asked ( $q_{i}$ ), the corresponding answer returned by the QA system ( $a_i$ ), and possibly additional information such as features or auxiliary tasks. The agent includes an action scoring component (U), which produced and action $u_t$ by deciding whether to submit a new question to the environment or to return a final answer. Formally, $u_t\in \mathcal {Q}\cup \mathcal {A}$ , where $s_t$0 is the set of all possible questions, and $s_t$1 is the set of all possible answers. The agent relies on a question reformulation system (QR), that provides candidate follow up questions, and on an answer ranking system (AR), which scores the answers contained in $s_t$2 . Each answer returned is assigned a reward. The objective is to maximize the expected reward over a set of questions.
Question: What is the difference in findings of Buck et al? It looks like the same conclusion was mentioned in Buck et al..
Paper: Deep Semi-Supervised Learning with Linguistically Motivated Sequence Labeling Task Hierarchies
It is natural to think of NLP tasks existing in a hierarchy, with each task building upon the previous tasks. For example, Part of Speech (POS) is known to be an extremely strong feature for Noun Phrase Chunking, and downstream tasks such as greedy Language Modeling (LM) can make use of information about the syntactic and semantic structure recovered from junior tasks in making predictions.
Question: What is the baseline?
Answer: The baseline is a multi-task architecture inspired by another paper.
Paper: Deep Semi-Supervised Learning with Linguistically Motivated Sequence Labeling Task Hierarchies
When we speak and understand language we are arguably performing many different linguistic tasks at once. At the top level we might be trying to formulate the best possible sequence of words given all of the contextual and prior information, but this requires us to do lower-level tasks like understanding the syntactic and semantic roles of the words we choose in a specific context.
Question: What is the unsupervised task in the final layer?
Paper: Deep Semi-Supervised Learning with Linguistically Motivated Sequence Labeling Task Hierarchies
In the original introductory paper to Noun Phrase Chunking, abney1991parsing, Chunking is motivated by describing a three-phase process - first, you read the words and assign a Part of Speech tag, you then use a ‘Chunker’ to group these words together into chunks depending on the context and the Parts of Speech, and finally you build a parse tree on top of the chunks.
Question: How many supervised tasks are used?
Paper: Deep Semi-Supervised Learning with Linguistically Motivated Sequence Labeling Task Hierarchies
In our model we represent linguistically motivated hierarchies in a multi-task Bi-Directional Recurrent Neural Network where junior tasks in the hierarchy are supervised at lower layers.This architecture builds upon sogaard2016deep, but is adapted in two ways: first, we add an unsupervised sequence labeling task (Language Modeling), second, we add a low-dimensional embedding layer between tasks in the hierarchy to learn dense representations of label tags. In addition to sogaard2016deep.
Question: What is the network architecture?
Answer: The network architecture has a multi-task Bi-Directional Recurrent Neural Network, with an unsupervised sequence labeling task and a low-dimensional embedding layer between tasks. There is a hidden layer after each successive task with skip connections to the senior supervised layers.
Paper: Context-Aware Learning for Neural Machine Translation
Despite its rapid adoption by academia and industry and its recent success BIBREF0 , neural machine translation has been found largely incapable of exploiting additional context other than the current source sentence. This incapability stems from the fact that larger-context machine translation systems tend to ignore additional context, such as previous sentences and associated images. Much of recent efforts have gone into building a novel network architecture that can better exploit additional context however without much success BIBREF1 , BIBREF2 , BIBREF3 .
Question: Is the proposed model more sensitive than previous context-aware models too?
Paper: Context-Aware Learning for Neural Machine Translation
A larger-context neural machine translation system extends upon the conventional neural machine translation system by incorporating the context $C$ , beyond a source sentence $X$ , when translating into a sentence $Y$ in the target language. In the case of multimodal machine translation, this additional context is an image which the source sentence $X$ describes. In the case of document-level machine translation, the additional context $C$ may include other sentences in a document in which the source sentence $X$ appears. Such a larger-context neural machine translation system consists of an encoder $f^C$ that encodes the additional context $C$ into a set of vector representations that are combined with those extracted from the source sentence $X$ by the original encoder $f^X$ . These vectors are then used by the decoder $X$0 to compute the conditional distribution over the target sequences $X$1 in the autoregressive paradigm, i.e., $X$2 
Question: In what ways the larger context is ignored for the models that do consider larger context?
Paper: Recurrent Deep Stacking Networks for Speech Recognition
Ever since the introduction of Deep Neural Networks (DNNs) to Automatic Speech Recognition (ASR) tasks BIBREF0 , researchers had been trying to use additional inputs to the raw input features. We extracted features that are more representative using the first and second order differentiates of the raw input features. And we utilized features in multiple neighboring frames to make use of the context information.
Question: What does recurrent deep stacking network do?
Answer: Stacks and joins outputs of previous frames with inputs of the current frame
Paper: Hybrid Code Networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning
Task-oriented dialog systems help a user to accomplish some goal using natural language, such as making a restaurant reservation, getting technical support, or placing a phonecall. Historically, these dialog systems have been built as a pipeline, with modules for language understanding, state tracking, action selection, and language generation. However, dependencies between modules introduce considerable complexity – for example, it is often unclear how to define the dialog state and what history to maintain, yet action selection relies exclusively on the state for input. Moreover, training each module requires specialized labels.
Question: Does the latent dialogue state heklp their model?
Paper: Hybrid Code Networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning
At a high level, the four components of a Hybrid Code Network are a recurrent neural network; domain-specific software; domain-specific action templates; and a conventional entity extraction module for identifying entity mentions in text. Both the RNN and the developer code maintain state. Each action template can be a textual communicative action or an API call. The HCN model is summarized in Figure 1 .
Question: Do the authors test on datasets other than bAbl?
Paper: Hybrid Code Networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning
Broadly there are two lines of work applying machine learning to dialog control. The first decomposes a dialog system into a pipeline, typically including language understanding, dialog state tracking, action selection policy, and language generation BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 . Specifically related to HCNs, past work has implemented the policy as feed-forward neural networks BIBREF12 , trained with supervised learning followed by reinforcement learning BIBREF13 . In these works, the policy has not been recurrent – i.e., the policy depends on the state tracker to summarize observable dialog history into state features, which requires design and specialized labeling. By contrast, HCNs use an RNN which automatically infers a representation of state. For learning efficiency, HCNs use an external light-weight process for tracking entity values, but the policy is not strictly dependent on it: as an illustration, in Section "Supervised learning evaluation II" below, we demonstrate an HCN-based dialog system which has no external state tracker. If there is context which is not apparent in the text in the dialog, such as database status, this can be encoded as a context feature to the RNN.
Question: What is the reward model for the reinforcement learning appraoch?
Answer: reward 1 for successfully completing the task, with a discount by the number of turns, and reward 0 when fail