z-uo/qasper-squad huggingface dataset:
Paper: A Multi-level Neural Network for Implicit Causality Detection in Web Texts
Causality mining is a fundamental task with abundant upstream applications. Early works utilize Bayesian network BIBREF14, BIBREF15, syntactic constraint BIBREF16, and dependency structure BIBREF17 to extract cause-effect pairs. Nevertheless, they could hardly summarize moderate patterns and rules to avoid overfitting. Further studies incorporate world knowledge that provides a supplement to lexico-syntax analysis. Generalizing nouns to their hypernyms in WordNet and each verb to its class in VerbNet BIBREF18, BIBREF19 eliminates the negative effect of lexical variations and discover frequent patterns of cause-effect pairs. As is well known, the implicit expressions of causality are more frequent. J.-H. Oh et al. BIBREF20 exploited cue words and sequence labeling by CRFs and selected the most relevant causality expressions as complements to implicitly expressed causality. However, the method requires retrieval and ranking from enormous web texts. From natural properties perspective, causality describes relations between regularly correlated events or phenomena. Constructing a cause-effect network or graph could help discover co-occurrence patterns and evolution rules of causation BIBREF3, BIBREF19. Therefore, Zhao et al. BIBREF21 conducted causality reasoning on the heterogeneous network to extract implicit relations cross sentences and find new causal relations.
Question: What performance did proposed method achieve, how much better is than previous state-of-the-art?
Paper: A Multi-level Neural Network for Implicit Causality Detection in Web Texts
Relation Networks (RNs) is initially a simple plug-and-play module to solve Visual-QA problems that fundamentally hinge on relational reasoning BIBREF22. RNs can effectively couple with convolutional neural networks (CNNs), long short-term memory networks (LSTMs), and multi-layer perceptrons (MLPs) to reduce overall network complexity. We gain a general ability to reason about the relations between entities and their properties. Original RNs can only perform single step inference such as $A \rightarrow B$ rather than $A \rightarrow B \rightarrow C$. For tasks that require complex multi-step of relational reasoning, Palm et al. BIBREF23 introduced the recurrent relational network that operates on a graph representation of objects. Pavez et al. BIBREF24 added complex reasoning ability to Memory Networks with RNs, which reduced its computational complexity from quadratic to linear. However, their tasks remain text QA and visual QA. In this paper, it's the first time that RNs is applied to relation extraction as proposed SCRN.
Question: What was previous state-of-the-art approach?
Paper: A Multi-level Neural Network for Implicit Causality Detection in Web Texts
This section describes the linguistic background of causal relation and the AltLexes dataset, which we used. It's a commonly held belief that causality can be expressed explicitly and implicitly using various propositions. In the Penn Discourse Treebank (PDTB) BIBREF25, over $12\%$ of explicit discourse connectives are marked as causal such as "hence", "as a result" and "consequently", as are nearly $26\%$ of implicit discourse relationships. In addition to these, there exists a type of implicit connectives in PDTB named AltLex (Alternative lexicalization) has been capable of indicating causal relations, which is an open class of markers and potentially infinite.
Question: How is Relation network used to infer causality at segment level?
Paper: Passage Re-ranking with BERT
We have seen rapid progress in machine reading compression in recent years with the introduction of large-scale datasets, such as SQuAD BIBREF3 , MS MARCO BIBREF4 , SearchQA BIBREF5 , TriviaQA BIBREF6 , and QUASAR-T BIBREF7 , and the broad adoption of neural models, such as BiDAF BIBREF8 , DrQA BIBREF9 , DocumentQA BIBREF10 , and QAnet BIBREF11 .
Question: What is the TREC-CAR dataset?
Paper: Improving Slot Filling by Utilizing Contextual Information
Slot Filling (SF) is the task of identifying the semantic concept expressed in natural language utterance. For instance, consider a request to edit an image expressed in natural language: “Remove the blue ball on the table and change the color of the wall to brown”. Here, the user asks for an "Action" (i.e., removing) on one “Object” (blue ball on the table) in the image and changing an “Attribute” (i.e., color) of the image to new “Value” (i.e., brown). Our goal in SF is to provide a sequence of labels for the given sentence to identify the semantic concept expressed in the given sentence.
Question: How does their model utilize contextual information for each work in the given sentence in a multi-task setting? setting?
Paper: Improving Slot Filling by Utilizing Contextual Information
The task of Slot Filling is formulated as a sequence labeling problem. Deep learning has been extensively employed for this task (BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10, BIBREF11). The prior work has mainly utilized the recurrent neural network as the encoder to extract features per word and Conditional Random Field (CRF) BIBREF12 as the decoder to generate the labels per word. Recently the work BIBREF1 shows that the global context of the sentence could be useful to enhance the performance of neural sequence labeling. In their approach, they use a separate sequential model to extract word features. Afterwards, using max pooling over the representations of the words, they obtain the sentence representations and concatenate it to the word embedding as the input to the main task encoder (i.e. the RNN model to perform sequence labeling). The benefit of using the global context along the word representation is 2-fold: 1) it enhance the representations of the word by the semantics of the entire sentence thus the word representation are more contextualized 2) The global view of the sentence would increase the model performance as it contains information about the entire sentence and this information might not be encoded in word representations due to long decencies.
Question: What metris are used for evaluation?
Paper: Improving Slot Filling by Utilizing Contextual Information
Our model is trained in a multi-task setting in which the main task is slot filling to identify the best possible sequence of labels for the given sentence. In the first auxiliary task we aim to increase consistency between the word representation and its context. The second auxiliary task is to enhance task specific information in contextual information. In this section, we explain each of these tasks in more details.
Question: How better is proposed model compared to baselines?
Paper: Improving Slot Filling by Utilizing Contextual Information
The input to the model is a sequence of words $x_1,x_2,...,x_N$. The goal is to assign each word one of the labels action, object, attribute, value or other. Following other methods for sequence labelling, we use the BIO encoding schema. In addition to the sequence of words, the part-of-speech (POS) tags and the dependency parse tree of the input are given to the model.
Question: What are the baselines?
Paper: Improving Slot Filling by Utilizing Contextual Information
In this sub-task we aim to increase the consistency of the word representation and its context. To obtain the context of each word we perform max pooling over the all words of the sentence excluding the word itself:
Question: How big is slot filing dataset?
Answer: Dataset has 1737 train, 497 dev and 559 test sentences.
Paper: How do you correct run-on sentences it's not as easy as it seems
A run-on sentence is defined as having at least two main or independent clauses that lack either a conjunction to connect them or a punctuation mark to separate them. Run-ons are problematic because they not only make the sentence unfriendly to the reader but potentially also to the local discourse. Consider the example in Table TABREF1 .
Question: Which machine learning models do they use to correct run-on sentences?
Paper: How do you correct run-on sentences it's not as easy as it seems
Early work in the field of GEC focused on correcting specific error types such as preposition and article errors BIBREF2 , BIBREF3 , BIBREF4 , but did not consider run-on sentences. The closest work to our own is BIBREF5 , who used Conditional Random Fields (CRFs) for correcting comma errors (excluding comma splices, a type of run-on sentence). BIBREF6 used a similar system based on CRFs but focused on comma splice correction. Recently, the field has focused on the task of whole-sentence correction, targeting all errors in a sentence in one pass. Whole-sentence correction methods borrow from advances in statistical machine translation BIBREF7 , BIBREF8 , BIBREF9 and, more recently, neural machine translation BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 .
Question: How large is the dataset they generate?
Answer: 4.756 million sentences
Paper: Component Analysis for Visual Question Answering Architectures
Recent research advances in Computer Vision (CV) and Natural Language Processing (NLP) introduced several tasks that are quite challenging to be solved, the so-called AI-complete problems. Most of those tasks require systems that understand information from multiple sources, i.e., semantics from visual and textual data, in order to provide some kind of reasoning. For instance, image captioning BIBREF0, BIBREF1, BIBREF2 presents itself as a hard task to solve, though it is actually challenging to quantitatively evaluate models on that task, and that recent studies BIBREF3 have raised questions on its AI-completeness.
Question: What are least important components identified in the the training of VQA models?
Paper: Component Analysis for Visual Question Answering Architectures
The task of VAQ has gained attention since Antol et al. BIBREF3 presented a large-scale dataset with open-ended questions. Many of the developed VQA models employ a very similar architecture BIBREF3, BIBREF22, BIBREF23, BIBREF24, BIBREF25, BIBREF26, BIBREF27: they represent images with features from pre-trained convolutional neural networks; they use word embeddings or recurrent neural networks to represent questions and/or answers; and they combine those features in a classification model over possible answers.
Question: What type of experiments are performed?
Paper: Component Analysis for Visual Question Answering Architectures
In this section we first introduce the baseline approach, with default image and text encoders, alongside a pre-defined fusion strategy. That base approach is inspired by the pioneer of Antol et al. on VQA BIBREF3. To understand the importance of each component, we update the base architecture according to each component we are investigating.
Question: What components are identified as core components for training VQA models?
Paper: Reference-less Quality Estimation of Text Simplification Systems
Text simplification (hereafter TS) has received increasing interest by the scientific community in recent years. It aims at producing a simpler version of a source text that is both easier to read and to understand, thus improving the accessibility of text for people suffering from a range of disabilities such as aphasia BIBREF0 or dyslexia BIBREF1 , as well as for second language learners BIBREF2 and people with low literacy BIBREF3 . This topic has been researched for a variety of languages such as English BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 , French BIBREF8 , Spanish BIBREF9 , Portuguese BIBREF10 , Italian BIBREF11 and Japanese BIBREF12 .
Question: what approaches are compared?
Paper: Improving Interaction Quality Estimation with BiLSTMs and the Impact on Dialogue Policy Learning
One prominent way of modelling the decision-making component of a spoken dialogue system (SDS) is to use (partially observable) Markov decision processes ((PO)MDPs) BIBREF0, BIBREF1. There, reinforcement learning (RL) BIBREF2 is applied to find the optimal system behaviour represented by the policy $\pi $. Task-oriented dialogue systems model the reward $r$, used to guide the learning process, traditionally with task success as the principal reward component BIBREF3, BIBREF4, BIBREF5, BIBREF6, BIBREF1, BIBREF7, BIBREF8.
Question: What model do they use a baseline to estimate satisfaction?
Paper: Retrieval-based Goal-Oriented Dialogue Generation
Dialogue systems have become a very popular research topic in recent years with the rapid improvement of personal assistants and the growing demand for online customer support. However, research has been split in two subfields BIBREF2: models presented for generation of open-ended conversations BIBREF3, BIBREF4, BIBREF5, BIBREF6, BIBREF7 and work on solving goal-oriented dialogue through dialogue management pipelines that include dialogue state tracking and dialogue policy BIBREF8, BIBREF9, BIBREF10, BIBREF11, BIBREF12, BIBREF13, BIBREF14.
Question: what semantically conditioned models did they compare with?
Paper: INFODENS: An Open-source Framework for Learning Text Representations
Linear classifiers in combination with the right features achieve good performance on text classification tasks BIBREF0 . Those hand-crafted features provide baselines for evaluating deep learning methods and are sometimes difficult to beat BIBREF1 , BIBREF2 . In some cases, hand-crafted features can even be combined with learned features to improve performance on a given task BIBREF3 , BIBREF4 highlighting some complementarity in the information captured by each approach. Conducting empirical experiments to study such complementarity would be beneficial, and the reasons are threefold: Firstly, this enables us to compare the performance of both hand crafted and learned representations and make design decisions regarding the trade-offs between speed and accuracy on a specific dataset. Secondly, it helps in investigating where the performance gaps are and whether these methods can complement each other and how they can be combined to improve performance. Finally, it allows us to derive new linguistic hypotheses as in many cases, deep learning methods are great engineering tools but they operate as black box methods and it is difficult to extract from them linguistic insights.
Question: Do they differentiate insights where they are dealing with learned or engineered representations?
Paper: INFODENS: An Open-source Framework for Learning Text Representations
The framework is designed in a modular and developer-friendly manner to encourage changes and extensions. The source code is accompanied by a user and a developer guide, and we give a brief overview of the architecture in this section, summarized in Figure FIGREF2 . The framework consists of the following frozen and hot spots:
Question: Do they show an example of usage for INFODENS?
Paper: INFODENS: An Open-source Framework for Learning Text Representations
These are the modules of the framework that need not be changed for extending the functionality in typical use cases.
Question: What kind of representation exploration does INFODENS provide?
Paper: AttSum: Joint Learning of Focusing and Summarization with Neural Attention
Query-focused summarization BIBREF0 aims to create a brief, well-organized and fluent summary that answers the need of the query. It is useful in many scenarios like news services and search engines, etc. Nowadays, most summarization systems are under the extractive framework which directly selects existing sentences to form the summary. Basically, there are two major tasks in extractive query-focused summarization, i.e., to measure the saliency of a sentence and its relevance to a user's query.
Question: What models do they compare to?
Paper: Trading the Twitter Sentiment with Reinforcement Learning
In a world where traditional financial information is ubiquitous and the financial models are largely homogeneous, finding hidden information that has not been priced in from alternative data is critical. The recent development in Natural Language Processing provides such opportunities to look into text data in addition to numerical data. When the market sets the stock price, it is not uncommon that the expectation of the company growth outweighs the company fundamentals. Twitter, a online news and social network where the users post and interact with messages to express views about certain topics, contains valuable information on the public mood and sentiment. A collection of research BIBREF0 BIBREF1 have shown that there is a positive correlation between the "public mood" and the "market mood". Other research BIBREF2 also shows that significant correlation exists between the Twitter sentiment and the abnormal return during the peaks of the Twitter volume during a major event.
Question: What is the optimal trading strategy based on reinforcement learning?
Paper: Trading the Twitter Sentiment with Reinforcement Learning
There are two options of getting the Tweets. First, Twitter provides an API to download the Tweets. However, rate limit and history limit make it not an option for this paper. Second, scrapping Tweets directly from Twitter website. Using the second option, the daily Tweets for stocks of interest from 2015 January to 2017 June were downloaded.
Question: Do the authors give any examples of major events which draw the public's attention and the impact they have on stock price?
Paper: Trading the Twitter Sentiment with Reinforcement Learning
To translate each tweet into a sentiment score, the Stanford coreNLP software was used. Stanford CoreNLP is designed to make linguistic analysis accessible to the general public. It provides named Entity Recognition, co-reference and basic dependencies and many other text understanding applications. An example that illustrate the basic functionality of Stanford coreNLP is shown in Figure. FIGREF5 
Question: Which tweets are used to output the daily sentiment signal?
Paper: Trading the Twitter Sentiment with Reinforcement Learning
Feature engineering is the process to extract meaningful information from the raw data in order to improve the performance of machine learning mode. Domain knowledge and intuition are often applied to keep the number of the features reasonable relative to the training data size. Two categories of features are defines: technical features and sentiment features. The technical features include previous day's return and volume, price momentum and volatility. The sentiment features include number of Tweets, daily average sentiment score, cross-section sentiment volatility, sentiment momentum and reversal.
Question: What is the baseline machine learning prediction approach?
Paper: Semantic Structure and Interpretability of Word Embeddings
Words are the smallest elements of a language with a practical meaning. Researchers from diverse fields including linguistics BIBREF0 , computer science BIBREF1 and statistics BIBREF2 have developed models that seek to capture “word meaning" so that these models can accomplish various NLP tasks such as parsing, word sense disambiguation and machine translation. Most of the effort in this field is based on the distributional hypothesis BIBREF3 which claims that a word is characterized by the company it keeps BIBREF4 . Building on this idea, several vector space models such as well known Latent Semantic Analysis (LSA) BIBREF5 and Latent Dirichlet Allocation (LDA) BIBREF6 that make use of word distribution statistics have been proposed in distributional semantics. Although these methods have been commonly used in NLP, more recent techniques that generate dense, continuous valued vectors, called embeddings, have been receiving increasing interest in NLP research. Approaches that learn embeddings include neural network based predictive methods BIBREF1 , BIBREF7 and count-based matrix-factorization methods BIBREF8 . Word embeddings brought about significant performance improvements in many intrinsic NLP tasks such as analogy or semantic textual similarity tasks, as well as downstream NLP tasks such as part-of-speech (POS) tagging BIBREF9 , named entity recognition BIBREF10 , word sense disambiguation BIBREF11 , sentiment analysis BIBREF12 and cross-lingual studies BIBREF13 .
Question: What are the weaknesses of their proposed interpretability quantification method?
Answer: can be biased by dataset used and may generate categories which are suboptimal compared to human designed categories
Paper: Semantic Structure and Interpretability of Word Embeddings
In the word embedding literature, the problem of interpretability has been approached via several different routes. For learning sparse, interpretable word representations from co-occurrence variant matrices, BIBREF21 suggested algorithms based on non-negative matrix factorization (NMF) and the resulting representations are called non-negative sparse embeddings (NNSE). To address memory and scale issues of the algorithms in BIBREF21 , BIBREF22 proposed an online method of learning interpretable word embeddings. In both studies, interpretability was evaluated using a word intrusion test introduced in BIBREF20 . The word intrusion test is expensive to apply since it requires manual evaluations by human observers separately for each embedding dimension. As an alternative method to incorporate human judgement, BIBREF23 proposed joint non-negative sparse embedding (JNNSE), where the aim is to combine text-based similarity information among words with brain activity based similarity information to improve interpretability. Yet, this approach still requires labor-intensive collection of neuroimaging data from multiple subjects.
Question: What advantages does their proposed method of quantifying interpretability have over the human-in-the-loop evaluation they compare to?
Answer: it is less expensive and quantifies interpretability using continuous values rather than binary evaluations
Paper: Explaining Queries over Web Tables to Non-Experts
Natural language interfaces have been gaining significant popularity, enabling ordinary users to write and execute complex queries. One of the prominent paradigms for developing NL interfaces is semantic parsing, which is the mapping of NL phrases into a formal language. As Machine Learning techniques are standardly used in semantic parsing, a training set of question-answer pairs is provided alongside a target database BIBREF0 , BIBREF1 , BIBREF2 . The parser is a parameterized function that is trained by updating its parameters such that questions from the training set are translated into queries that yield the correct answers.
Question: How do they generate a graphic representation of a query from a query?
Paper: Explaining Queries over Web Tables to Non-Experts
We review our system architecture from Figure FIGREF7 and describe its general workflow.
Question: How do they gather data for the query explanation problem?
Paper: Explaining Queries over Web Tables to Non-Experts
We begin by formally defining our task of querying tables. Afterwards, we discuss the formal query language and show how lambda DCS queries can be translated directly into SQL.
Question: Which query explanation method was preffered by the users in terms of correctness?
Paper: Explaining Queries over Web Tables to Non-Experts
An NL interface for querying tables receives a question INLINEFORM0 on a table INLINEFORM1 and outputs a set of values INLINEFORM2 as the answer (where each value is either the content of a cell, or the result of an aggregate function on cells). As discussed in the introduction, we make the assumption that a query concerns a single table.
Question: Do they conduct a user study where they show an NL interface with and without their explanation?
Paper: Explaining Queries over Web Tables to Non-Experts
Following the definition of our data model we introduce our formal query language, lambda dependency-based compositional semantics (lambda DCS) BIBREF6 , BIBREF0 , which is a language inspired by lambda calculus, that revolves around sets. Lambda DCS was originally designed for building an NL interface over Freebase BIBREF9 .
Question: How do the users in the user studies evaluate reliability of a NL interface?