z-uo/qasper-squad huggingface dataset:
Paper: Paying Attention to Attention: Highlighting Influential Samples in Sequential Analysis
Neural networks are powerful learning algorithms, but are also some of the most complex. This is made worse by the non-deterministic nature of neural network training; a small change in a learning parameter can drastically affect the network's learning ability. This has led to the development of methodologies for understanding and uncovering not just neural networks, but black box models in general. The interpretation of deep networks is a young field of research. We refer readers to BIBREF4 for a comprehensive overview of different methods for understanding and visualizing deep neural networks. More recent developments include DeepLIFT BIBREF5 (not yet applicable to RNNs), layerwise relevance propagation BIBREF6 (only very recently adapted to textual input and LSTMs BIBREF7 , BIBREF8 ), and LIME BIBREF9 .
Question: Do they explain model predictions solely on attention weights?
Paper: Paying Attention to Attention: Highlighting Influential Samples in Sequential Analysis
In Table TABREF3 , we see the bottom visualization where the weights are uniform at the point of escalation. However, on the 2nd turn, the Han had produced more distinct weights. It is clear from this example that the importance of a single sample can change drastically as a sequence progresses. Using these changes in attention over the sequence, we formalized a set of rules to create an alternative visualization for the entire sequence to be applied in cases where the attention weights are uniform over all samples at the stopping point.
Question: Can their method of creating more informative visuals be applied to tasks other than turn taking in conversations?
Paper: Video Highlight Prediction Using Audience Chat Reactions
On-line eSports events provide a new setting for observing large-scale social interaction focused on a visual story that evolves over time—a video game. While watching sporting competitions has been a major source of entertainment for millennia, and is a significant part of today's culture, eSports brings this to a new level on several fronts. One is the global reach, the same games are played around the world and across cultures by speakers of several languages. Another is the scale of on-line text-based discourse during matches that is public and amendable to analysis. One of the most popular games, League of Legends, drew 43 million views for the 2016 world series final matches (broadcast in 18 languages) and a peak concurrent viewership of 14.7 million. Finally, players interact through what they see on screen while fans (and researchers) can see exactly the same views.
Question: What was the baseline?
Paper: Video Highlight Prediction Using Audience Chat Reactions
We briefly discuss a small sample of the related work on language and vision datasets, summarization, and highlight prediction. There has been a surge of vision and language datasets focusing on captions over the last few years, BIBREF0 , BIBREF1 , BIBREF2 , followed by efforts to focus on more specific parts of images BIBREF3 , or referring expressions BIBREF4 , or on the broader context BIBREF5 . For video, similar efforts have collected descriptions BIBREF6 , while others use existing descriptive video service (DVS) sources BIBREF7 , BIBREF8 . Beyond descriptions, other datasets use questions to relate images and language BIBREF9 , BIBREF10 . This approach is extended to movies in MovieQA.
Question: What is the average length of the recordings?
Answer: 40 minutes
Paper: Video Highlight Prediction Using Audience Chat Reactions
Our dataset covers 218 videos from NALCS and 103 from LMS for a total of 321 videos from week 1 to week 9 in 2017 spring series from each tournament. Each week there are 10 matches for NALCS and 6 matches for LMS. Matches are best of 3, so consist of two games or three games. The first and third games are used for training. The second games in the first 4 weeks are used as validation and the remainder of second games are used as test. Table TABREF3 lists the numbers of videos in train, validation, and test subsets.
Question: How big was the dataset presented?
Paper: Video Highlight Prediction Using Audience Chat Reactions
In this section, we explain the proposed models and components. We first describe the notation and definition of the problem, plus the evaluation metric used. Next, we explain our vision model V-CNN-LSTM and language model L-Char-LSTM. Finally, we describe the joint multimodal model INLINEFORM0 -LSTM.
Question: What were their results?
Answer: Best model achieved F-score 74.7 on NALCS and F-score of 70.0 on LMS on test set
Paper: Neural Multi-Step Reasoning for Question Answering on Semi-Structured Tables
Teaching computers to answer complex natural language questions requires sophisticated reasoning and human language understanding. We investigate generic natural language interfaces for simple arithmetic questions on semi-structured tables. Typical questions for this task are topic independent and may require performing multiple discrete operations such as aggregation, comparison, superlatives or arithmetics.
Question: Does a neural scoring function take both the question and the logical form as inputs?
Paper: Neural Multi-Step Reasoning for Question Answering on Semi-Structured Tables
We briefly mention here two main types of QA systems related to our task: semantic parsing-based and embedding-based. Semantic parsing-based methods perform a functional parse of the question that is further converted to a machine understandable program and executed on a knowledgebase or database. For QA on semi-structured tables with multi-compositional queries, BIBREF0 generate and rank candidate logical forms with a log-linear model, resorting to hand-crafted features for scoring. As opposed, we learn neural features for each question and the paraphrase of each candidate logical form. Paraphrases and hand-crafted features have successfully facilitated semantic parsers targeting simple factoid BIBREF1 and compositional questions BIBREF2 . Compositional questions are also the focus of BIBREF3 that construct logical forms from the question embedding through operations parametrized by RNNs, thus losing interpretability. A similar fully neural, end-to-end differentiable network was proposed by BIBREF4 .
Question: What is the source of the paraphrases of the questions?
Paper: Neural Multi-Step Reasoning for Question Answering on Semi-Structured Tables
We describe our QA system. For every question $q$ : i) a set of candidate logical forms $\lbrace z_i\rbrace _{i = 1, \ldots , n_q}$ is generated using the method of BIBREF0 ; ii) each such candidate program $z_i$ is transformed in an interpretable textual representation $t_i$ ; iii) all $t_i$ 's are jointly embedded with $q$ in the same vector space and scored using a neural similarity function; iv) the logical form $z_i^*$ corresponding to the highest ranked $t_i^*$ is selected as the machine-understandable translation of question $q$ and executed on the input table to retrieve the final answer. Our contributions are the novel models that perform steps ii) and iii), while for step i) we rely on the work of BIBREF0 (henceforth: PL2015).
Question: Does the dataset they use differ from the one used by Pasupat and Liang, 2015?
Paper: Creation of an Annotated Corpus of Spanish Radiology Reports
The availability of annotated corpora from the biomedical domain, in particular for non-English texts, is scarce. There are two main reasons for that: the generation of new annotated data is expensive due to the need of expert knowledge and to privacy issues: the patient and the physician should not be identified from the texts. So, although the availability of annotated data is a highly valuable asset for the research community, it is very difficult to access it.
Question: Did they experiment on this corpus?
Paper: DP-LSTM: Differential Privacy-inspired LSTM for Stock Prediction Using Financial News
Stock prediction is crucial for quantitative analysts and investment companies. Stocks' trends, however, are affected by a lot of factors such as interest rates, inflation rates and financial news [12]. To predict stock prices accurately, one must use these variable information. In particular, in the banking industry and financial services, analysts' armies are dedicated to pouring over, analyzing, and attempting to quantify qualitative data from news. A large amount of stock trend information is extracted from the large amount of text and quantitative information that is involved in the analysis.
Question: Is the model compared against a linear regression baseline?
Paper: DP-LSTM: Differential Privacy-inspired LSTM for Stock Prediction Using Financial News
In this section, we first introduce the background of the stock price model, which is based on the autoregressive moving average (ARMA) model. Then, we present the sentiment analysis details of the financial news and introduce how to use them to improve prediction performance. At last, we introduce the differential privacy framework and the loss function.
Question: What is the prediction accuracy of the model?
Answer: mean prediction accuracy 0.99582651
S&P 500 Accuracy 0.99582651
Paper: DP-LSTM: Differential Privacy-inspired LSTM for Stock Prediction Using Financial News
The ARMA model, which is one of the most widely used linear models in time series prediction [17], where the future value is assumed as a linear combination of the past errors and past values. ARMA is used to set the stock midterm prediction problem up. Let ${X}_t^\text{A}$ be the variable based on ARMA at time $t$, then we have
Question: What is the dataset used in the paper?
Answer: historical S&P 500 component stocks
 306242 news articles
Paper: DP-LSTM: Differential Privacy-inspired LSTM for Stock Prediction Using Financial News
Another variable highly related to stock price is the textual information from news, whose changes may be a precursor to price changes. In our paper, news refers to a news article's title on a given trading day. It has been used to infer whether an event had informational content and whether investors' interpretations of the information were positive, negative or neutral. We hence use sentiment analysis to identify and extract opinions within a given text. Sentiment analysis aims at gauging the attitude, sentiments, evaluations and emotions of a speaker or writer based on subjectivity's computational treatment in a text [19]-[20].
Question: How does the differential privacy mechanism work?
Paper: Compositional generalization in a deep seq2seq model by separating syntax and semantics
A crucial property underlying the expressive power of human language is its systematicity BIBREF0 , BIBREF1 : syntactic or grammatical rules allow arbitrary elements to be combined in novel ways, making the number of sentences possible in a language to be exponential in the number of its basic elements. Recent work has shown that standard deep learning methods in natural language processing fail to capture this important property: when tested on unseen combinations of known elements, state-of-the-art models fail to generalize BIBREF2 , BIBREF3 , BIBREF4 . It has been suggested that this failure represents a major deficiency of current deep learning models, especially when they are compared to human learners BIBREF5 , BIBREF0 .
Question: How does the SCAN dataset evaluate compositional generalization?
Answer: it systematically holds out inputs in the training set containing basic primitive verb, "jump", and tests on sequences containing that verb.
Paper: Effectiveness of self-supervised pre-training for speech recognition
Representation learning has been an active research area for more than 30 years BIBREF1, with the goal of learning high level representations which separates different explanatory factors of the phenomena represented by the input data BIBREF2, BIBREF3. Disentangled representations provide models with exponentially higher ability to generalize, using little amount of labels, to new conditions by combining multiple sources of variations.
Question: Is their model fine-tuned also on all available data, what are results?
Paper: Cooperative Learning of Disjoint Syntax and Semantics
This document has been adapted from the instructions for earlier ACL and NAACL proceedings, including those for ACL 2018 by Shay Cohen, Kevin Gimpel, and Wei Lu, NAACL 2018 by Margaret Michell and Stephanie Lukin, 2017/2018 (NA)ACL bibtex suggestions from Jason Eisner, ACL 2017 by Dan Gildea and Min-Yen Kan, NAACL 2017 by Margaret Mitchell, ACL 2012 by Maggie Li and Michael White, those from ACL 2010 by Jing-Shing Chang and Philipp Koehn, those for ACL 2008 by JohannaD. Moore, Simone Teufel, James Allan, and Sadaoki Furui, those for ACL 2005 by Hwee Tou Ng and Kemal Oflazer, those for ACL 2002 by Eugene Charniak and Dekang Lin, and earlier ACL and EACL formats. Those versions were written by several people, including John Chen, Henry S. Thompson and Donald Walker. Additional elements were taken from the formatting instructions of the International Joint Conference on Artificial Intelligence and the Conference on Computer Vision and Pattern Recognition.
Question: How much does this system outperform prior work?
Answer: The system outperforms by 27.7% the LSTM model, 38.5% the RL-SPINN model and 41.6% the Gumbel Tree-LSTM
Paper: Cooperative Learning of Disjoint Syntax and Semantics
The following instructions are directed to authors of papers submitted to NAACL-HLT 2019 or accepted for publication in its proceedings. All authors are required to adhere to these specifications. Authors are required to provide a Portable Document Format (PDF) version of their papers. The proceedings are designed for printing on A4 paper.
Question: What are the baseline systems that are compared against?
Answer: The system is compared to baseline models: LSTM, RL-SPINN and Gumbel Tree-LSTM
Paper: Non-native Speaker Verification for Spoken Language Assessment
Automatic spoken assessment systems are becoming increasingly popular, especially for English with the high demand around the world for learning of English as a second language BIBREF0, BIBREF1, BIBREF2, BIBREF3. In addition to assessing a candidate's English ability such as fluency and pronunciation and giving feedback to the candidate, these automatic systems also need to ensure the integrity of the candidate's score by detecting malpractice, as shown in Figure FIGREF1. Malpractice is the action by a candidate that breaks the assessment regulation and potentially threatens the reliability of the exam and associated certification. Malpractice can take a range of forms in spoken language assessment scenarios, such as using or trying to use unauthorised materials, impersonation, speaking irrelevant to prompts/questions, speaking in his/her first language (L1) instead of the target language for spoken tests, etc. This work aims to investigate the problem of automatically detecting impersonation, in which a candidate attempts to impersonate another in a speaking test. This is closely related to speaker verification.
Question: What standard large speaker verification corpora is used for evaluation?
Paper: Non-native Speaker Verification for Spoken Language Assessment
In this work both i-vector and x-vector representations are used. For the i-vector speaker representation the form described in BIBREF4, BIBREF19 is used. This section will just discuss the x-vector speaker representation as this is the form that is adapted to the non-native verification task.
Question: What systems are tested?
Answer: BULATS i-vector/PLDA
BULATS x-vector/PLDA
VoxCeleb x-vector/PLDA
PLDA adaptation (X1)
 Extractor fine-tuning (X2)
Paper: Domain Adaptation for Neural Networks by Parameter Augmentation
Domain adaptation is a machine learning paradigm that aims at improving the generalization performance of a new (target) domain by using a dataset from the original (source) domain. Suppose that, as the source domain dataset, we have a captioning corpus, consisting of images of daily lives and each image has captions. Suppose also that we would like to generate captions for exotic cuisine, which are rare in the corpus. It is usually very costly to make a new corpus for the target domain, i.e., taking and captioning those images. The research question here is how we can leverage the source domain dataset to improve the performance on the target domain.
Question: How many examples are there in the source domain?
Paper: Domain Adaptation for Neural Networks by Parameter Augmentation
There are several recent studies applying domain adaptation methods to deep neural networks. However, few studies have focused on improving the fine tuning and dual outputs methods in the supervised setting.
Question: How many examples are there in the target domain?
Paper: Domain Adaptation for Neural Networks by Parameter Augmentation
We start with the basic notations and formalization for domain adaptation. Let $\mathcal {X}$ be the set of inputs and $\mathcal {Y}$ be the outputs. We have a source domain dataset $D^s$ , which is sampled from some distribution $\mathcal {D}^s$ . Also, we have a target domain dataset $D^t$ , which is sampled from another distribution $\mathcal {D}^t$ . Since we are considering supervised settings, each element of the datasets has a form of input output pair $(x,y)$ . The goal of domain adaptation is to learn a function $f : \mathcal {X} \rightarrow \mathcal {Y}$ that models the input-output relation of $D^t$ . We implicitly assume that there is a connection between the source and target distributions and thus can leverage the information of the source domain dataset. In the case of image caption generation, the input $x$ is an image (or the feature vector of an image) and $\mathcal {Y}$0 is the caption (a sequence of words).
Question: Did they only experiment with captioning task?
Paper: Crowd Sourced Data Analysis: Mapping of Programming Concepts to Syntactical Patterns
1.20pt
Question: how well this method is compared to other method?
Paper: Learning to Compose Neural Networks for Question Answering
This paper presents a compositional, attentional model for answering questions about a variety of world representations, including images and structured knowledge bases. The model translates from questions to dynamically assembled neural networks, then applies these networks to world representations (images or knowledge bases) to produce answers. We take advantage of two largely independent lines of work: on one hand, an extensive literature on answering questions by mapping from strings to logical representations of meaning; on the other, a series of recent successes in deep neural models for image recognition and captioning. By constructing neural networks instead of logical forms, our model leverages the best aspects of both linguistic compositionality and continuous representations.
Question: What benchmark datasets they use?
Answer: VQA and GeoQA
Paper: The Evolved Transformer
Over the past few years, impressive advances have been made in the field of neural architecture search. Reinforcement learning and evolution have both proven their capacity to produce models that exceed the performance of those designed by humans BIBREF0 , BIBREF1 . These advances have mostly focused on improving image models, although some effort has also been invested in searching for sequence models BIBREF2 , BIBREF3 . In these cases, it has always been to find improved recurrent neural networks (RNNs), which were long established as the de facto neural model for sequence problems BIBREF4 , BIBREF5 .
Question: what is the proposed Progressive Dynamic Hurdles method?
Paper: The Evolved Transformer
RNNs have long been used as the default option for applying neural networks to sequence modeling BIBREF4 , BIBREF5 , with LSTM BIBREF8 and GRU BIBREF9 architectures being the most popular. However, recent work has shown that RNNs are not necessary to build state-of-the-art sequence models. For example, many high performance convolutional models have been designed, such as WaveNet BIBREF10 , Gated Convolution Networks BIBREF11 , Conv Seq2Seq BIBREF6 and Dynamic Lightweight Convolution model BIBREF12 . Perhaps the most promising architecture in this direction is the Transformer architecture BIBREF7 , which relies only on multi-head attention to convey spatial information. In this work, we use both convolutions and attention in our search space to leverage the strengths of both layer types.
Question: What is in the model search space?
Paper: The Evolved Transformer
We employ evolution-based architecture search because it is simple and has been shown to be more efficient than reinforcement learning when resources are limited BIBREF0 . We use the same tournament selection BIBREF28 algorithm as Real et al. real19, with the aging regularization omitted, and so encourage the reader to view their in-depth description of the method. In the interest of saving space, we will only give a brief overview of the algorithm here.
Question: How much energy did the NAS consume?
Paper: The Evolved Transformer
Our encoding search space is inspired by the NASNet search space BIBREF1 , but is altered to allow it to express architecture characteristics found in recent state-of-the-art feed-forward seq2seq networks. Crucially, we ensured that the search space can represent the Transformer, so that we can seed the search process with the Transformer itself.
Question: How does Progressive Dynamic Hurdles work?
Paper: MonaLog: a Lightweight System for Natural Language Inference Based on Monotonicity
There has been rapid progress on natural language inference (NLI) in the last several years, due in large part to recent advances in neural modeling BIBREF0 and the introduction of several new large-scale inference datasets BIBREF1, BIBREF2, BIBREF3, BIBREF4. Given the high performance of current state-of-the-art models, there has also been interest in understanding the limitations of these models (given their uninterpretability) BIBREF5, BIBREF6, as well as finding systematic biases in benchmark datasets BIBREF7, BIBREF8. In parallel to these efforts, there have also been recent logic-based approaches to NLI BIBREF9, BIBREF10, BIBREF11, BIBREF12, BIBREF13, which take inspiration from linguistics. In contrast to early attempts at using logic BIBREF14, these approaches have proven to be more robust. However they tend to use many rules and their output can be hard to interpret. It is sometimes unclear whether the attendant complexity is justified, especially given that such models are currently far outpaced by data-driven models and are generally hard to hybridize with data-driven techniques.
Question: Do they beat current state-of-the-art on SICK?