z-uo/qasper-squad huggingface dataset:
Paper: DisSim: A Discourse-Aware Syntactic Text Simplification Frameworkfor English and German
In a first step, source sentences that present a complex linguistic form are turned into clean, compact structures by decomposing clausal and phrasal components. For this purpose, the transformation rules encode both the splitting points and rephrasing procedure for reconstructing proper sentences.
Question: Is the model evaluated?
Answer: the English version is evaluated. The German version evaluation is in progress
Paper: Learning Word Embeddings from the Portuguese Twitter Stream: A Study of some Practical Aspects
Word embeddings have great practical importance since they can be used as pre-computed high-density features to ML models, significantly reducing the amount of training data required in a variety of NLP tasks. However, there are several inter-related challenges with computing and consistently distributing word embeddings concerning the:
Question: What new metrics are suggested to track progress?
Paper: Learning Word Embeddings from the Portuguese Twitter Stream: A Study of some Practical Aspects
There are several approaches to generating word embeddings. One can build models that explicitly aim at generating word embeddings, such as Word2Vec or GloVe BIBREF1 , BIBREF2 , or one can extract such embeddings as by-products of more general models, which implicitly compute such word embeddings in the process of solving other language tasks.
Question: What intrinsic evaluation metrics are used?
Paper: Learning Word Embeddings from the Portuguese Twitter Stream: A Study of some Practical Aspects
The neural word embedding model we use in our experiments is heavily inspired in the one described in BIBREF4 , but ours is one layer deeper and is set to solve a slightly different word prediction task. Given a sequence of 5 words - INLINEFORM0 INLINEFORM1 INLINEFORM2 INLINEFORM3 INLINEFORM4 , the task the model tries to perform is that of predicting the middle word, INLINEFORM5 , based on the two words on the left - INLINEFORM6 INLINEFORM7 - and the two words on the right - INLINEFORM8 INLINEFORM9 : INLINEFORM10 . This should produce embeddings that closely capture distributional similarity, so that words that belong to the same semantic class, or which are synonyms and antonyms of each other, will be embedded in “close” regions of the embedding hyper-space.
Question: What experimental results suggest that using less than 50% of the available training examples might result in overfitting?
Paper: Procedural Reasoning Networks for Understanding Multimodal Procedures
A great deal of commonsense knowledge about the world we live is procedural in nature and involves steps that show ways to achieve specific goals. Understanding and reasoning about procedural texts (e.g. cooking recipes, how-to guides, scientific processes) are very hard for machines as it demands modeling the intrinsic dynamics of the procedures BIBREF0, BIBREF1, BIBREF2. That is, one must be aware of the entities present in the text, infer relations among them and even anticipate changes in the states of the entities after each action. For example, consider the cheeseburger recipe presented in Fig. FIGREF2. The instruction “salt and pepper each patty and cook for 2 to 3 minutes on the first side” in Step 5 entails mixing three basic ingredients, the ground beef, salt and pepper, together and then applying heat to the mix, which in turn causes chemical changes that alter both the appearance and the taste. From a natural language understanding perspective, the main difficulty arises when a model sees the word patty again at a later stage of the recipe. It still corresponds to the same entity, but its form is totally different.
Question: What multimodality is available in the dataset?
Paper: Procedural Reasoning Networks for Understanding Multimodal Procedures
In our study, we particularly focus on the visual reasoning tasks of RecipeQA, namely visual cloze, visual coherence, and visual ordering tasks, each of which examines a different reasoning skill. We briefly describe these tasks below.
Question: What are previously reported models?
Paper: Procedural Reasoning Networks for Understanding Multimodal Procedures
In the following, we explain our Procedural Reasoning Networks model. Its architecture is based on a bi-directional attention flow (BiDAF) model BIBREF6, but also equipped with an explicit reasoning module that acts on entity-specific relational memory units. Fig. FIGREF4 shows an overview of the network architecture. It consists of five main modules: An input module, an attention module, a reasoning module, a modeling module, and an output module. Note that the question answering tasks we consider here are multimodal in that while the context is a procedural text, the question and the multiple choice answers are composed of images.
Question: How better is accuracy of new model compared to previously reported models?
Answer: Average accuracy of proposed model vs best prevous result:
Single-task Training: 57.57 vs 55.06
Multi-task Training: 50.17 vs 50.59
Paper: Active Learning for Chinese Word Segmentation in Medical Text
Electronic health records (EHRs) systematically collect patients' clinical information, such as health profiles, histories of present illness, past medical histories, examination results and treatment plans BIBREF0 . By analyzing EHRs, many useful information, closely related to patients, can be discovered BIBREF1 . Since Chinese EHRs are recorded without explicit word delimiters (e.g., “UTF8gkai糖尿病酮症酸中毒” (diabetic ketoacidosis)), Chinese word segmentation (CWS) is a prerequisite for processing EHRs. Currently, state-of-the-art CWS methods usually require large amounts of manually-labeled data to reach their full potential. However, there are many challenges inherent in labeling EHRs. First, EHRs have many medical terminologies, such as “UTF8gkai高血压性心脏病” (hypertensive heart disease) and “UTF8gkai罗氏芬” (Rocephin), so only annotators with medical backgrounds can be qualified to label EHRs. Second, EHRs may involve personal privacies of patients. Therefore, they cannot be openly published on a large scale for labeling. The above two problems lead to the high annotation cost and insufficient training corpus in the research of CWS in medical text.
Question: How does the scoring model work?
Paper: Active Learning for Chinese Word Segmentation in Medical Text
In past decades, researches on CWS have a long history and various methods have been proposed BIBREF13 , BIBREF14 , BIBREF15 , which is an important task for Chinese NLP BIBREF7 . These methods are mainly focus on two categories: supervised learning and deep learning BIBREF2 .
Question: How does the active learning model work?
Answer: Active learning methods has a learning engine (mainly used for training of classification problems) and the selection engine (which chooses samples that need to be relabeled by annotators from unlabeled data). Then, relabeled samples are added to training set for classifier to re-train, thus continuously improving the accuracy of the classifier. In this paper, CRF-based segmenter and a scoring model are employed as learning engine and selection engine, respectively.
Paper: Active Learning for Chinese Word Segmentation in Medical Text
Active learning BIBREF22 mainly aims to ease the data collection process by automatically deciding which instances should be labeled by annotators to train a model as quickly and effectively as possible BIBREF23 . The sampling strategy plays a key role in active learning. In the past decade, the rapid development of active learning has resulted in various sampling strategies, such as uncertainty sampling BIBREF24 , query-by-committee BIBREF25 and information gain BIBREF26 . Currently, the most mainstream sampling strategy is uncertainty sampling. It focuses its selection on samples closest to the decision boundary of the classifier and then chooses these samples for annotators to relabel BIBREF27 .
Question: Which neural network architectures are employed?
Paper: InScript: Narrative texts annotated with script information
A script is “a standardized sequence of events that describes some stereotypical human activity such as going to a restaurant or visiting a doctor” BIBREF0 . Script events describe an action/activity along with the involved participants. For example, in the script describing a visit to a restaurant, typical events are entering the restaurant, ordering food or eating. Participants in this scenario can include animate objects like the waiter and the customer, as well as inanimate objects such as cutlery or food.
Question: What are the key points in the role of script knowledge that can be studied?
Paper: InScript: Narrative texts annotated with script information
We selected 10 scenarios from different available scenario lists (e.g. Regneri:2010 , VanDerMeer2009, and the OMICS corpus BIBREF1 ), including scripts of different complexity (Taking a bath vs. Flying in an airplane) and specificity (Riding a public bus vs. Repairing a flat bicycle tire). For the full scenario list see Table 2 .
Question: Did the annotators agreed and how much?
Answer: For event types and participant types, there was a moderate to substantial level of agreement using the Fleiss' Kappa. For coreference chain annotation, there was average agreement of 90.5%.
Moderate agreement of 0.64-0.68 Fleiss’ Kappa over event type labels, 0.77 Fleiss’ Kappa over participant labels, and good agreement of 90.5% over coreference information.
Paper: InScript: Narrative texts annotated with script information
Statistics for the corpus are given in Table 2 . On average, each story has a length of 12 sentences and 217 words with 98 word types on average. Stories are coherent and concentrate mainly on the corresponding scenario. Neglecting auxiliaries, modals and copulas, on average each story has 32 verbs, out of which 58% denote events related to the respective scenario. As can be seen in Table 2 , there is some variation in stories across scenarios: The flying in an airplane scenario, for example, is most complex in terms of the number of sentences, tokens and word types that are used. This is probably due to the inherent complexity of the scenario: Taking a flight, for example, is more complicated and takes more steps than taking a bath. The average count of sentences, tokens and types is also very high for the baking a cake scenario. Stories from the scenario often resemble cake recipes, which usually contain very detailed steps, so people tend to give more detailed descriptions in the stories.
Question: How many subjects have been used to create the annotations?
Paper: Investigating Robustness and Interpretability of Link Prediction via Adversarial Modifications
Knowledge graphs (KG) play a critical role in many real-world applications such as search, structured data management, recommendations, and question answering. Since KGs often suffer from incompleteness and noise in their facts (links), a number of recent techniques have proposed models that embed each entity and relation into a vector space, and use these embeddings to predict facts. These dense representation models for link prediction include tensor factorization BIBREF0 , BIBREF1 , BIBREF2 , algebraic operations BIBREF3 , BIBREF4 , BIBREF5 , multiple embeddings BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 , and complex neural models BIBREF10 , BIBREF11 . However, there are only a few studies BIBREF12 , BIBREF13 that investigate the quality of the different KG models. There is a need to go beyond just the accuracy on link prediction, and instead focus on whether these representations are robust and stable, and what facts they make use of for their predictions. In this paper, our goal is to design approaches that minimally change the graph structure such that the prediction of a target fact changes the most after the embeddings are relearned, which we collectively call Completion Robustness and Interpretability via Adversarial Graph Edits (). First, we consider perturbations that red!50!blackremove a neighboring link for the target fact, thus identifying the most influential related fact, providing an explanation for the model's prediction. As an example, consider the excerpt from a KG in Figure 1 with two observed facts, and a target predicted fact that Princes Henriette is the parent of Violante Bavaria. Our proposed graph perturbation, shown in Figure 1 , identifies the existing fact that Ferdinal Maria is the father of Violante Bavaria as the one when removed and model retrained, will change the prediction of Princes Henriette's child. We also study attacks that green!50!blackadd a new, fake fact into the KG to evaluate the robustness and sensitivity of link prediction models to small additions to the graph. An example attack for the original graph in Figure 1 , is depicted in Figure 1 . Such perturbations to the the training data are from a family of adversarial modifications that have been applied to other machine learning tasks, known as poisoning BIBREF14 , BIBREF15 , BIBREF16 , BIBREF17 .
Question: What datasets are used to evaluate this approach?
Answer:  Kinship and Nations knowledge graphs, YAGO3-10 and WN18KGs knowledge graphs
Paper: Investigating Robustness and Interpretability of Link Prediction via Adversarial Modifications
In this section, we briefly introduce some notations, and existing relational embedding approaches that model knowledge graph completion using dense vectors. In KGs, facts are represented using triples of subject, relation, and object, $\langle s, r, o\rangle $ , where $s,o\in \xi $ , the set of entities, and $r\in $ , the set of relations. To model the KG, a scoring function $\psi :\xi \times \times \xi \rightarrow $ is learned to evaluate whether any given fact is true. In this work, we focus on multiplicative models of link prediction, specifically DistMult BIBREF2 because of its simplicity and popularity, and ConvE BIBREF10 because of its high accuracy. We can represent the scoring function of such methods as $\psi (s,r,o) = , ) \cdot $ , where $,,\in ^d$ are embeddings of the subject, relation, and object respectively. In DistMult, $, ) = \odot $ , where $\odot $ is element-wise multiplication operator. Similarly, in ConvE, $, )$ is computed by a convolution on the concatenation of $$ and $s,o\in \xi $0 .
Question: How is this approach used to detect incorrect facts?
Paper: Investigating Robustness and Interpretability of Link Prediction via Adversarial Modifications
For adversarial modifications on KGs, we first define the space of possible modifications. For a target triple $\langle s, r, o\rangle $ , we constrain the possible triples that we can remove (or inject) to be in the form of $\langle s^{\prime }, r^{\prime }, o\rangle $ i.e $s^{\prime }$ and $r^{\prime }$ may be different from the target, but the object is not. We analyze other forms of modifications such as $\langle s, r^{\prime }, o^{\prime }\rangle $ and $\langle s, r^{\prime }, o\rangle $ in appendices "Modifications of the Form 〈s,r ' ,o ' 〉\langle s, r^{\prime }, o^{\prime } \rangle " and "Modifications of the Form 〈s,r ' ,o〉\langle s, r^{\prime }, o \rangle " , and leave empirical evaluation of these modifications for future work.
Question: Can this adversarial approach be used to directly improve model accuracy?
Paper: Learning Supervised Topic Models for Classification and Regression from Crowds
Topic models, such as latent Dirichlet allocation (LDA), allow us to analyze large collections of documents by revealing their underlying themes, or topics, and how each document exhibits them BIBREF0 . Therefore, it is not surprising that topic models have become a standard tool in data analysis, with many applications that go even beyond their original purpose of modeling textual data, such as analyzing images BIBREF1 , BIBREF2 , videos BIBREF3 , survey data BIBREF4 or social networks data BIBREF5 .
Question: what are the advantages of the proposed model?
Paper: Learning Supervised Topic Models for Classification and Regression from Crowds
Latent Dirichlet allocation (LDA) soon proved to be a powerful tool for modeling documents BIBREF0 and images BIBREF1 by extracting their underlying topics, where topics are probability distributions across words, and each document is characterized by a probability distribution across topics. However, the need to model the relationship between documents and labels quickly gave rise to many supervised variants of LDA. One of the first notable works was that of supervised LDA (sLDA) BIBREF6 . By extending LDA through the inclusion of a response variable that is linearly dependent on the mean topic-assignments of the words in a document, sLDA is able to jointly model the documents and their responses, in order to find latent topics that will best predict the response variables for future unlabeled documents. Although initially developed for general continuous response variables, sLDA was later extended to classification problems BIBREF2 , by modeling the relationship between topic-assignments and labels with a softmax function as in logistic regression.
Question: what are the state of the art approaches?
Paper: Learning Supervised Topic Models for Classification and Regression from Crowds
Learning from multiple annotators is an increasingly important research topic. Since the early work of Dawid and Skeene BIBREF19 , who attempted to obtain point estimates of the error rates of patients given repeated but conflicting responses to various medical questions, many approaches have been proposed. These usually rely on latent variable models. For example, in BIBREF20 the authors propose a model to estimate the ground truth from the labels of multiple experts, which is then used to train a classifier.
Question: what datasets were used?
Paper: CrossWOZ: A Large-Scale Chinese Cross-Domain Task-Oriented Dialogue Dataset
Recently, there have been a variety of task-oriented dialogue models thanks to the prosperity of neural architectures BIBREF0, BIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5. However, the research is still largely limited by the availability of large-scale high-quality dialogue data. Many corpora have advanced the research of task-oriented dialogue systems, most of which are single domain conversations, including ATIS BIBREF6, DSTC 2 BIBREF7, Frames BIBREF8, KVRET BIBREF9, WOZ 2.0 BIBREF10 and M2M BIBREF11.
Question: How was the dataset collected?
Answer: They crawled travel information from the Web to build a database, created a multi-domain goal generator from the database, collected dialogue between workers an automatically annotated dialogue acts.
Paper: CrossWOZ: A Large-Scale Chinese Cross-Domain Task-Oriented Dialogue Dataset
According to whether the dialogue agent is human or machine, we can group the collection methods of existing task-oriented dialogue datasets into three categories. The first one is human-to-human dialogues. One of the earliest and well-known ATIS dataset BIBREF6 used this setting, followed by BIBREF8, BIBREF9, BIBREF10, BIBREF15, BIBREF16 and BIBREF12. Though this setting requires many human efforts, it can collect natural and diverse dialogues. The second one is human-to-machine dialogues, which need a ready dialogue system to converse with humans. The famous Dialogue State Tracking Challenges provided a set of human-to-machine dialogue data BIBREF17, BIBREF7. The performance of the dialogue system will largely influence the quality of dialogue data. The third one is machine-to-machine dialogues. It needs to build both user and system simulators to generate dialogue outlines, then use templates BIBREF3 to generate dialogues or further employ people to paraphrase the dialogues to make them more natural BIBREF11, BIBREF13. It needs much less human effort. However, the complexity and diversity of dialogue policy are limited by the simulators. To explore dialogue policy in multi-domain scenarios, and to collect natural and diverse dialogues, we resort to the human-to-human setting.
Question: What are the benchmark models?
Paper: CrossWOZ: A Large-Scale Chinese Cross-Domain Task-Oriented Dialogue Dataset
Our corpus is to simulate scenarios where a traveler seeks tourism information and plans her or his travel in Beijing. Domains include hotel, attraction, restaurant, metro, and taxi. The data collection process is summarized as below:
Question: How was the corpus annotated?
Paper: BERTRAM: Improved Word Embeddings Have Big Impact on Contextualized Model Performance
As traditional word embedding algorithms BIBREF1 are known to struggle with rare words, several techniques for improving their representations have been proposed over the last few years. These approaches exploit either the contexts in which rare words occur BIBREF2, BIBREF3, BIBREF4, BIBREF5, their surface-form BIBREF6, BIBREF7, BIBREF8, or both BIBREF9, BIBREF10. However, all of these approaches are designed for and evaluated on uncontextualized word embeddings.
Question: What models other than standalone BERT is new model compared to?
Answer: Only Bert base and Bert large are compared to proposed approach.
Paper: BERTRAM: Improved Word Embeddings Have Big Impact on Contextualized Model Performance
Incorporating surface-form information (e.g., morphemes, characters or character $n$-grams) is a commonly used technique for improving word representations. For context-independent word embeddings, this information can either be injected into a given embedding space BIBREF6, BIBREF8, or a model can directly be given access to it during training BIBREF7, BIBREF24, BIBREF25. In the area of contextualized representations, many architectures employ subword segmentation methods BIBREF12, BIBREF13, BIBREF26, BIBREF14, whereas others use convolutional neural networks to directly access character-level information BIBREF27, BIBREF11, BIBREF17.
Question: How much is representaton improved for rare/medum frequency words compared to standalone BERT and previous work?
Paper: BERTRAM: Improved Word Embeddings Have Big Impact on Contextualized Model Performance
We review the architecture of the form-context model (FCM) BIBREF9, which forms the basis for our model. Given a set of $d$-dimensional high-quality embeddings for frequent words, FCM can be used to induce embeddings for infrequent words that are appropriate for the given embedding space. This is done as follows: Given a word $w$ and a context $C$ in which it occurs, a surface-form embedding $v_{(w,{C})}^\text{form} \in \mathbb {R}^d$ is obtained similar to BIBREF7 by averaging over embeddings of all $n$-grams in $w$; these $n$-gram embeddings are learned during training. Similarly, a context embedding $v_{(w,{C})}^\text{context} \in \mathbb {R}^d$ is obtained by averaging over the embeddings of all words in $C$. The so-obtained form and context embeddings are then combined using a gate
Question: What are three downstream task datasets?
Paper: BERTRAM: Improved Word Embeddings Have Big Impact on Contextualized Model Performance
To overcome both limitations described above, we introduce Bertram, an approach that combines a pretrained BERT language model BIBREF13 with Attentive Mimicking BIBREF19. To this end, let $d_h$ be the hidden dimension size and $l_\text{max}$ be the number of layers for the BERT model being used. We denote with $e_{t}$ the (uncontextualized) embedding assigned to a token $t$ by BERT and, given a sequence of such uncontextualized embeddings $\mathbf {e} = e_1, \ldots , e_n$, we denote by $\textbf {h}_j^l(\textbf {e})$ the contextualized representation of the $j$-th token at layer $l$ when the model is given $\mathbf {e}$ as input.
Question: What is dataset for word probing task?
Paper: Joint Entity Linking with Deep Reinforcement Learning
Entity Linking (EL), which is also called Entity Disambiguation (ED), is the task of mapping mentions in text to corresponding entities in a given knowledge Base (KB). This task is an important and challenging stage in text understanding because mentions are usually ambiguous, i.e., different named entities may share the same surface form and the same entity may have multiple aliases. EL is key for information retrieval (IE) and has many applications, such as knowledge base population (KBP), question answering (QA), etc.
Question: How fast is the model compared to baselines?