z-uo/qasper-squad huggingface dataset:
Paper: Crowdsourcing a High-Quality Gold Standard for QA-SRL
Evaluation in QA-SRL involves aligning predicted and ground truth argument spans and evaluating role label equivalence. Since detecting question paraphrases is still an open challenge, we propose both unlabeled and labeled evaluation metrics.
Question: How big is the dataset?
Paper: Zero-Shot Cross-lingual Classification Using Multilingual Neural Machine Translation
Transfer learning has been shown to work well in Computer Vision where pre-trained components from a model trained on ImageNet BIBREF0 are used to initialize models for other tasks BIBREF1 . In most cases, the other tasks are related to and share architectural components with the ImageNet task, enabling the use of such pre-trained models for feature extraction. With this transfer capability, improvements have been obtained on other image classification datasets, and on other tasks such as object detection, action recognition, image segmentation, etc BIBREF2 . Analogously, we propose a method to transfer a pre-trained component - the multilingual encoder from an NMT system - to other NLP tasks.
Question: Do the other multilingual baselines make use of the same amount of training data?
Paper: Zero-Shot Cross-lingual Classification Using Multilingual Neural Machine Translation
We propose an Encoder-Classifier model, where the Encoder, leveraging the representations learned by a multilingual NMT model, converts an input sequence ${\mathbf {x}}$ into a set of vectors C, and the Classifier predicts a class label $y$ given the encoding of the input sequence, C.
Question: How big is the impact of training data size on the performance of the multilingual encoder?
Paper: Zero-Shot Cross-lingual Classification Using Multilingual Neural Machine Translation
Although there has been a large body of work in building multilingual NMT models which can translate between multiple languages at the same time BIBREF29 , BIBREF30 , BIBREF31 , BIBREF8 , zero-shot capabilities of such multilingual representations have only been tested for MT BIBREF8 . We propose a simple yet effective solution - reuse the encoder of a multilingual NMT model to initialize the encoder for other NLP tasks. To be able to achieve promising zero-shot classification performance, we consider two factors: (1) The ability to encode multiple source languages with the same encoder and (2) The ability to learn language agnostic representations of the source sequence. Based on the literature, both requirements can be satisfied by training a multilingual NMT model having a shared encoder BIBREF32 , BIBREF8 , and a separate decoder and attention mechanism for each target language BIBREF30 . After training such a multilingual NMT model, the decoder and the corresponding attention mechanisms (which are target-language specific) are discarded, while the multilingual encoder is used to initialize the encoder of our proposed Encoder-Classifier model.
Question: What data were they used to train the multilingual encoder?
Answer: WMT 2014 En-Fr parallel corpus
Paper: An Analysis of Visual Question Answering Algorithms
In open-ended visual question answering (VQA) an algorithm must produce answers to arbitrary text-based questions about images BIBREF0 , BIBREF1 . VQA is an exciting computer vision problem that requires a system to be capable of many tasks. Truly solving VQA would be a milestone in artificial intelligence, and would significantly advance human computer interaction. However, VQA datasets must test a wide range of abilities for progress to be adequately measured.
Question: From when are many VQA datasets collected?
Answer: late 2014
Paper: Imitation Learning of Robot Policies by Combining Language, Vision and Demonstration
A significant challenge when designing robots to operate in the real world lies in the generation of control policies that can adapt to changing environments. Programming such policies is a labor and time-consuming process which requires substantial technical expertise. Imitation learning BIBREF0, is an appealing methodology that aims at overcoming this challenge – instead of complex programming, the user only provides a set of demonstrations of the intended behavior. These demonstrations are consequently distilled into a robot control policy by learning appropriate parameter settings of the controller. Popular approaches to imitation, such as Dynamic Motor Primitives (DMPs) BIBREF1 or Gaussian Mixture Regression (GMR) BIBREF2 largely focus on motion as the sole input and output modality, i.e., joint angles, forces or positions. Critical semantic and visual information regarding the task, such as the appearance of the target object or the type of task performed, is not taken into account during training and reproduction. The result is often a limited generalization capability which largely revolves around adaptation to changes in the object position. While imitation learning has been successfully applied to a wide range of tasks including table-tennis BIBREF3, locomotion BIBREF4, and human-robot interaction BIBREF5 an important question is how to incorporate language and vision into a differentiable end-to-end system for complex robot control.
Question: What is task success rate achieved? 
Answer: 96-97.6% using the objects color or shape and 79% using shape alone
Paper: Imitation Learning of Robot Policies by Combining Language, Vision and Demonstration
In order to outline our problem statement, we contrast our approach to Imitation learning BIBREF0 which considers the problem of learning a policy $\mathbf {\pi }$ from a given set of demonstrations ${\cal D}=\lbrace \mathbf {d}^0,.., \mathbf {d}^m\rbrace $. Each demonstration spans a time horizon $T$ and contains information about the robot's states and actions, e.g., demonstrated sensor values and control inputs at each time step. Robot states at each time step within a demonstration are denoted by $\mathbf {x}_t$. In contrast to other imitation learning approaches, we assume that we have access to the raw camera images of the robot $_t$ at teach time step, as well as access to a verbal description of the task in natural language. This description may provide critical information about the context, goals or objects involved in the task and is denoted as $\mathbf {s}$. Given this information, our overall objective is to learn a policy $\mathbf {\pi }$ which imitates the demonstrated behavior, while also capturing semantics and important visual features. After training, we can provide the policy $\mathbf {\pi }(\mathbf {s},)$ with a different, new state of the robot and a new verbal description (instruction) as parameters. The policy will then generate the control signals needed to perform the task which takes the new visual input and semantic context int o account.
Question: What simulations are performed by the authors to validate their approach?
Paper: Imitation Learning of Robot Policies by Combining Language, Vision and Demonstration
A fundamental challenge in imitation learning is the extraction of policies that do not only cover the trained scenarios, but also generalize to a wide range of other situations. A large body of literature has addressed the problem of learning robot motor skills by imitation BIBREF6, learning functional BIBREF1 or probabilistic BIBREF7 representations. However, in most of these approaches, the state vector has to be carefully designed in order to ensure that all necessary information for adaptation is available. Neural approaches to imitation learning BIBREF8 circumvent this problem by learning suitable feature representations from rich data sources for each task or for a sequence of tasks BIBREF9, BIBREF10, BIBREF11. Many of these approaches assume that either a sufficiently large set of motion primitives is already available or that a taxonomy of the task is available, i.e., semantics and motions are not trained in conjunction. The importance of maintaining this connection has been shown in BIBREF12, allowing the robot to adapt to untrained variations of the same task. To learn entirely new tasks, meta-learning aims at learning policy parameters that can quickly be fine-tuned to new tasks BIBREF13. While very successful in dealing with visual and spatial information, these approaches do not incorporate any semantic or linguistic component into the learning process. Language has shown to successfully generate task descriptions in BIBREF14 and several works have investigated the idea of combining natural language and imitation learning: BIBREF15, BIBREF16, BIBREF17, BIBREF18, BIBREF19. However, most approaches do not utilize the inherent connection between semantic task descriptions and low-level motions to train a model.
Question: Does proposed end-to-end approach learn in reinforcement or supervised learning manner?
Answer: supervised learning
Paper: Overcoming the Rare Word Problem for Low-Resource Language Pairs in Neural Machine Translation
NMT systems have achieved better performance compared to statistical machine translation (SMT) systems in recent years not only on available data language pairs BIBREF1, BIBREF2, but also on low-resource language pairs BIBREF3, BIBREF4. Nevertheless, NMT still exists many challenges which have adverse effects on its effectiveness BIBREF0. One of these challenges is that NMT has biased tend in translating high-frequency words, thus words which have lower frequencies are often translated incorrectly. This challenge has also been confirmed again in BIBREF3, and they have proposed two strategies to tackle this problem with modifications on the model's output distribution: one for normalizing some matrices by fixing them to constants after several training epochs and another for adding a direct connection from source embeddings through a simple feed forward neural network (FFNN). These approaches increase the size and the training time of their NMT systems. In this work, we follow their second approach but simplify the computations by replacing FFNN with two single operations.
Question: Are synonymous relation taken into account in the Japanese-Vietnamese task?
Paper: Overcoming the Rare Word Problem for Low-Resource Language Pairs in Neural Machine Translation
Our NMT system use a bidirectional recurrent neural network (biRNN) as an encoder and a single-directional RNN as a decoder with input feeding of BIBREF11 and the attention mechanism of BIBREF5. The Encoder's biRNN are constructed by two RNNs with the hidden units in the LSTM cell, one for forward and the other for backward of the source sentence $\mathbf {x}=(x_1, ...,x_n)$. Every word $x_i$ in sentence is first encoded into a continuous representation $E_s(x_i)$, called the source embedding. Then $\mathbf {x}$ is transformed into a fixed-length hidden vector $\mathbf {h}_i$ representing the sentence at the time step $i$, which called the annotation vector, combined by the states of forward $\overrightarrow{\mathbf {h}}_i$ and backward $\overleftarrow{\mathbf {h}}_i$:
Question: Is the supervised morphological learner tested on Japanese?
Paper: A framework for anomaly detection using language modeling, and its applications to finance
The detection of anomalous trends in the financial domain has focused largely on fraud detection BIBREF0, risk modeling BIBREF1, and predictive analysis BIBREF2. The data used in the majority of such studies is of time-series, transactional, graph or generally quantitative or structured nature. This belies the critical importance of semi-structured or unstructured text corpora that practitioners in the finance domain derive insights from—corpora such as financial reports, press releases, earnings call transcripts, credit agreements, news articles, customer interaction logs, and social data.
Question: What is the dataset that is used in the paper?
Paper: A framework for anomaly detection using language modeling, and its applications to finance
Anomaly detection is a strategy that is often employed in contexts where a deviation from a certain norm is sought to be captured, especially when extreme class imbalance impedes the use of a supervised approach. The implementation of such methods allows for the unveiling of previously hidden or obstructed insights.
Question: What is the performance of the models discussed in the paper?
Paper: A framework for anomaly detection using language modeling, and its applications to finance
Previous studies have used anomaly detection to identify and correct errors in text BIBREF4, BIBREF5. These are often unintentional errors that occur as a result of some form of data transfer, e.g. from audio to text, from image to text, or from one language to another. Such studies have direct applicability to the error-prone process of earnings call or customer call transcription, where audio quality, accents, and domain-specific terms can lead to errors. Consider a scenario where the CEO of a company states in an audio conference, `Now investments will be made in Asia.' However, the system instead transcribes, `No investments will be made in Asia.' There is a meaningful difference in the implication of the two statements that could greatly influence the analysis and future direction of the company. Additionally, with regards to the second scenario, it is highly unlikely that the CEO would make such a strong and negative statement in a public setting thus supporting the use of anomaly detection for error correction.
Question: Does the paper consider the use of perplexity in order to identify text anomalies?
Paper: A framework for anomaly detection using language modeling, and its applications to finance
Anomaly in the semantic space might reflect irregularities that are intentional or emergent, signaling risky behavior or phenomena. A sudden change in the tone and vocabulary of a company's leadership in their earnings calls or financial reports can signal risk. News stories that have abnormal language, or irregular origination or propagation patterns might be unreliable or untrustworthy.
Question: Does the paper report a baseline for the task?
Paper: What Gets Echoed? Understanding the"Pointers"in Explanations of Persuasive Arguments
Explanations are essential for understanding and learning BIBREF0. They can take many forms, ranging from everyday explanations for questions such as why one likes Star Wars, to sophisticated formalization in the philosophy of science BIBREF1, to simply highlighting features in recent work on interpretable machine learning BIBREF2.
Question: What non-contextual properties do they refer to?
Paper: What Gets Echoed? Understanding the"Pointers"in Explanations of Persuasive Arguments
To provide background for our study, we first present a brief overview of explanations for the NLP community, and then discuss the connection of our study with pointer networks, linguistic accommodation, and argumentation mining.
Question: What is the baseline?
Paper: What Gets Echoed? Understanding the"Pointers"in Explanations of Persuasive Arguments
Our dataset is derived from the /r/ChangeMyView subreddit, which has more than 720K subscribers BIBREF31. /r/ChangeMyView hosts conversations where someone expresses a view and others then try to change that person's mind. Despite being fundamentally based on argument, /r/ChangeMyView has a reputation for being remarkably civil and productive BIBREF32, e.g., a journalist wrote “In a culture of brittle talking points that we guard with our lives, Change My View is a source of motion and surprise” BIBREF33.
Question: What are their proposed features?
Paper: What Gets Echoed? Understanding the"Pointers"in Explanations of Persuasive Arguments
To further investigate how explanations select words from the explanandum, we formulate a word-level prediction task to predict whether words in an OP or PC are echoed in its explanation. Formally, given a tuple of (OP, PC, explanation), we extract the unique stemmed words as $\mathcal {V}_{\text{OP}}, \mathcal {V}_{\text{PC}}, \mathcal {V}_{\text{EXP}}$. We then define the label for each word in the OP or PC, $w \in \mathcal {V}_{\text{OP}} \cup \mathcal {V}_{\text{PC}}$, based on the explanation as follows:
Question: What are overall baseline results on new this new task?
Paper: What Gets Echoed? Understanding the"Pointers"in Explanations of Persuasive Arguments
We further examine the effectiveness of our proposed features in a predictive setting. These features achieve strong performance in the word-level classification task, and can enhance neural models in both the word-level task and generating explanations. However, the word-level task remains challenging, especially for content words.
Question: What metrics are used in evaluation of this task?
Paper: What Gets Echoed? Understanding the"Pointers"in Explanations of Persuasive Arguments
We consider two classifiers for our word-level classification task: logistic regression and gradient boosting tree (XGBoost) BIBREF39. We hypothesized that XGBoost would outperform logistic regression because our problem is non-linear, as shown in Figure FIGREF8.
Question: Do authors provide any explanation for intriguing patterns of word being echoed?
Paper: What Gets Echoed? Understanding the"Pointers"in Explanations of Persuasive Arguments
Overall performance (Figure FIGREF28). Although our word-level task is heavily imbalanced, all of our models outperform the random baseline by a wide margin. As expected, content words are much more difficult to predict than stopwords, but the best F1 score in content words more than doubles that of the random baseline (0.286 vs. 0.116). Notably, although we strongly improve on our random baseline, even our best F1 scores are relatively low, and this holds true regardless of the model used. Despite involving more tokens than standard tagging tasks (e.g., BIBREF41 and BIBREF42), predicting whether a word is going to be echoed in explanations remains a challenging problem.
Question: What features are proposed?
Paper: Automating Reading Comprehension by Generating Question and Answer Pairs
Asking relevant and intelligent questions has always been an integral part of human learning, as it can help assess the user's understanding of a piece of text (an article, an essay etc.). However, forming questions manually can be sometimes arduous. Automated question generation (QG) systems can help alleviate this problem by learning to generate questions on a large scale and in lesser time. Such a system has applications in a myriad of areas such as FAQ generation, intelligent tutoring systems, and virtual assistants.
Question: Which datasets are used to train this model?
Paper: Automatic Reminiscence Therapy for Dementia.
Increases in life expectancy in the last century have resulted in a large number of people living to old ages and will result in a double number of dementia cases by the middle of the century BIBREF0BIBREF1. The most common form of dementia is Alzheimer disease which contributes to 60–70% of cases BIBREF2. Research focused on identifying treatments to slow down the evolution of Alzheimer's disease is a very active pursuit, but it has been only successful in terms of developing therapies that eases the symptoms without addressing the cause BIBREF3BIBREF4. Besides, people with dementia might have some barriers to access to the therapies, such as cost, availability and displacement to the care home or hospital, where the therapy takes place. We believe that Artificial Intelligence (AI) can contribute in innovative systems to give accessibility and offer new solutions to the patients needs, as well as help relatives and caregivers to understand the illness of their family member or patient and monitor the progress of the dementia.
Question: How is performance of this system measured?
Answer: using the BLEU score as a quantitative metric and human evaluation for quality
Paper: Automatic Reminiscence Therapy for Dementia.
The origin of chatbots goes back to 1966 with the creation of ELIZA BIBREF8 by Joseph Weizenbaum at MIT. Its implementation consisted in pattern matching and substitution methodology. Recently, data driven approaches have drawn significant attention. Existing work along this line includes retrieval-based methods BIBREF9BIBREF10 and generation-based methodsBIBREF11BIBREF12. In this work we focus on generative models, where sequence-to-sequence algorithm that uses RNNs to encode and decode inputs into responses is a current best practice.
Question: How many questions per image on average are available in dataset?
Paper: Automatic Reminiscence Therapy for Dementia.
In this section we explain the main two components of our model, as well as how the interaction with the model works. We named it Elisabot and its goal is to mantain a dialog with the patient about her user’s life experiences.
Question: Is machine learning system underneath similar to image caption ML systems?
Paper: Automatic Reminiscence Therapy for Dementia.
The algorithm behind VQG consists in an Encoder-Decoder architecture with attention. The Encoder takes as input one of the given photos $I$ from the user and learns its information using a CNN. CNNs have been widely studied for computer vision tasks. The CNN provides the image's learned features to the Decoder which generates the question $y$ word by word by using an attention mechanism with a Long Short-Term Memory (LSTM). The model is trained to maximize the likelihood $p(y|I)$ of producing a target sequence of words:
Question: How big dataset is used for training this system?
Answer: For the question generation model 15,000 images with 75,000 questions. For the chatbot model, around 460k utterances over 230k dialogues.
Paper: Lattice CNNs for Matching Based Chinese Question Answering
Short text matching plays a critical role in many natural language processing tasks, such as question answering, information retrieval, and so on. However, matching text sequences for Chinese or similar languages often suffers from word segmentation, where there are often no perfect Chinese word segmentation tools that suit every scenario. Text matching usually requires to capture the relatedness between two sequences in multiple granularities. For example, in Figure FIGREF4 , the example phrase is generally tokenized as “China – citizen – life – quality – high”, but when we plan to match it with “Chinese – live – well”, it would be more helpful to have the example segmented into “Chinese – livelihood – live” than its common segmentation. 
Question: How do they obtain word lattices from words?
Answer: By considering words as vertices and generating directed edges between neighboring words within a sentence
Paper: Lattice CNNs for Matching Based Chinese Question Answering
Our Lattice CNNs framework is built upon the siamese architecture BIBREF5 , one of the most successful frameworks in text matching, which takes the word lattice format of a pair of sentences as input, and outputs the matching score.
Question: Which metrics do they use to evaluate matching?
Paper: Lattice CNNs for Matching Based Chinese Question Answering
The siamese architecture and its variant have been widely adopted in sentence matching BIBREF6 , BIBREF3 and matching based question answering BIBREF7 , BIBREF0 , BIBREF8 , that has a symmetrical component to extract high level features from different input channels, which share parameters and map inputs to the same vector space. Then, the sentence representations are merged and compared to output the similarities.
Question: Which dataset(s) do they evaluate on?
Paper: On the coexistence of competing languages
The dynamics of language evolution is one of many interdisciplinary fields to which methods and insights from statistical physics have been successfully applied (see BIBREF0 for an overview, and BIBREF1 for a specific comprehensive review).
Question: What languages do they look at?