z-uo/qasper-squad huggingface dataset:
Paper: Construction of a Japanese Word Similarity Dataset
Traditionally, a word is represented as a sparse vector indicating the word itself (one-hot vector) or the context of the word (distributional vector). However, both the one-hot notation and distributional notation suffer from data sparseness since dimensions of the word vector do not interact with each other. Distributed word representation addresses the data sparseness problem by constructing a dense vector of a fixed length, wherein contexts are shared (or distributed) across dimensions. Distributed word representation is known to improve the performance of many NLP applications such as machine translation BIBREF0 and sentiment analysis BIBREF1 to name a few. The task to learn a distributed representation is called representation learning.
Question: what is the size of this dataset?
Paper: Construction of a Japanese Word Similarity Dataset
In general, distributed word representations are evaluated using a word similarity task. For instance, WordSim353 2002:PSC:503104.503110, MC BIBREF2 , RG BIBREF3 , and SCWS Huang:2012:IWR:2390524.2390645 have been used to evaluate word similarities in English. Moreover, baker-reichart-korhonen:2014:EMNLP2014 built a verb similarity dataset (VSD) based on WordSim353 because there was no dataset of verbs in the word-similarity task. Recently, SimVerb-3500 was introduced to evaluate human understanding of verb meaning Gerz:2016:EMNLP. It provides human ratings for the similarity of 3,500 verb pairs so that it enables robust evaluation of distributed representation for verbs. However, most of these datasets include English words only. There has been no Japanese dataset for the word-similarity task.
Question: did they use a crowdsourcing platform for annotations?
Paper: Construction of a Japanese Word Similarity Dataset
What makes a pair of words similar? Most of the previous datasets do not concretely define the similarity of word pairs. The difference in the similarity of word pairs originates from each annotator's mind, resulting in different scales of a word. Thus, we propose to use an example-based approach (Table TABREF9 ) to control the variance of the similarity ratings. We remove the context of word when we extracted the word. So, we consider that an ambiguous word has high variance of the similarity, but we can get low variance of the similarity when the word is monosemous.
Question: where does the data come from?
Paper: Towards Neural Language Evaluators
Evaluation metrics play a central role in the machine learning community. They direct the efforts of the research community and are used to define the state of the art models. In machine translation and summarization, the two most common metrics used for evaluating similarity between candidate and reference texts are BLEU BIBREF0 and ROUGE BIBREF1. Both approaches rely on counting the matching n-grams in the candidates summary to n-grams in the reference text. BLEU is precision focused while ROUGE is recall focused. These metrics have posed serious limitations and have already been criticized by the academic community.In this work we formulate three criticisms of BLEU and ROUGE, establish criteria that a sound metric should have and propose concrete ways to use recent advances in NLP to design data-driven metric addressing the weaknesses found in BLEU and ROUGE.
Question: What is the criteria for a good metric?
Paper: Towards Neural Language Evaluators
BLEU (Bilingual Evaluation Understudy) BIBREF0 and ROUGE BIBREF1 have been used to evaluate many NLP tasks for almost two decades. The general acceptance of these methods depend on many factors including their simplicity and the intuitive interpretability. Yet the main factor is the claim that they highly correlate with human judgement BIBREF0. This has been criticised extensively by the literature and the shortcomings of these methods have been widely studied. Reiter BIBREF2 , in his structured review of BLEU, finds a low correlation between BLEU and human judgment. Callison et al BIBREF3 examines BLEU in the context of machine translation and find that BLEU does neither correlate with human judgment on adequacy(whether the hypothesis sentence adequately captures the meaning of the reference sentence) nor fluency(the quality of language in a sentence). Sulem et al BIBREF4 examines BLEU in the context of text simplification on grammaticality, meaning preservation and simplicity and report BLEU has very low or in some cases negative correlation with human judgment. Considering these results it is a natural step to pursue new avenues for natural language evaluation and with the advent of deep learning using neural networks for this task is a promising step forward.
Question: What are the three limitations?
Paper: Multi-domain Dialogue State Tracking as Dynamic Knowledge Graph Enhanced Question Answering
In a task-oriented dialogue system, the dialogue policy determines the next action to perform and next utterance to say based on the current dialogue state. A dialogue state defined by frame-and-slot semantics is a set of (key, value) pairs specified by the domain ontology BIBREF0. A key is a (domain, slot) pair and a value is a slot value provided by the user. Figure FIGREF1 shows a dialogue and state in three domain contexts. Dialogue state tracking (DST) in multiple domains is a challenging problem. First of all, in production environments, the domain ontology is being continuously updated such that the model must generalize to new values, new slots, or even new domains during inference. Second, the number of slots and values in the training data are usually quite large. For example, the MultiWOZ $2.0/2.1$ datasets BIBREF1, BIBREF2 have 30 (domain, slot) pairs and more than $4,500$ values BIBREF3. As the model must understand slot and value paraphrases, it is infeasible to train each slot or value independently. Third, multi-turn inferences are often required as shown in the underlined areas of Figure FIGREF1.
Question: What is current state-of-the-art model?
Paper: Improved Word Sense Disambiguation Using Pre-Trained Contextualized Word Representations
Word sense disambiguation (WSD) automatically assigns a pre-defined sense to a word in a text. Different senses of a word reflect different meanings a word has in different contexts. Identifying the correct word sense given a context is crucial in natural language processing (NLP). Unfortunately, while it is easy for a human to infer the correct sense of a word given a context, it is a challenge for NLP systems. As such, WSD is an important task and it has been shown that WSD helps downstream NLP tasks, such as machine translation BIBREF0 and information retrieval BIBREF1.
Question: Which language(s) are found in the WSD datasets?
Paper: Improved Word Sense Disambiguation Using Pre-Trained Contextualized Word Representations
Continuous word representations in real-valued vectors, or commonly known as word embeddings, have been shown to help improve NLP performance. Initially, exploiting continuous representations was achieved by adding real-valued vectors as classification features BIBREF14. BIBREF6 fine-tuned non-contextualized word embeddings by a feed-forward neural network such that those word embeddings were more suited for WSD. The fine-tuned embeddings were incorporated into an SVM classifier. BIBREF7 explored different strategies of incorporating word embeddings and found that their best strategy involved exponential decay that decreased the contribution of surrounding word features as their distances to the target word increased.
Question: What datasets are used for testing?
Paper: Natural- to formal-language generation using Tensor Product Representations
When people perform explicit reasoning, they can typically describe the way to the conclusion step by step via relational descriptions. There is ample evidence that relational representations are important for human cognition (e.g., BIBREF0, BIBREF1, BIBREF2, BIBREF3, BIBREF4). Although a rapidly growing number of researchers use deep learning to solve complex symbolic reasoning and language tasks (a recent review is BIBREF5), most existing deep learning models, including sequence models such as LSTMs, do not explicitly capture human-like relational structure information.
Question: How does TP-N2F compare to LSTM-based Seq2Seq in terms of training and inference speed?
Answer: Full Testing Set accuracy: 84.02
Cleaned Testing Set accuracy: 93.48
Paper: Natural- to formal-language generation using Tensor Product Representations
The TPR mechanism is a method to create a vector space embedding of complex symbolic structures. The type of a symbol structure is defined by a set of structural positions or roles, such as the left-child-of-root position in a tree, or the second-argument-of-$R$ position of a given relation $R$. In a particular instance of a structural type, each of these roles may be occupied by a particular filler, which can be an atomic symbol or a substructure (e.g., the entire left sub-tree of a binary tree can serve as the filler of the role left-child-of-root). For now, we assume the fillers to be atomic symbols.
Question: What is the performance proposed model achieved on AlgoList benchmark?
Answer: Full Testing Set Accuracy: 84.02
Cleaned Testing Set Accuracy: 93.48
Paper: Natural- to formal-language generation using Tensor Product Representations
We propose a general TP-N2F neural network architecture operating over TPRs to solve N2F tasks under a proposed role-level description of those tasks. In this description, natural-language input is represented as a straightforward order-2 role structure, and formal-language relational representations of outputs are represented with a new order-3 recursive role structure proposed here. Figure FIGREF3 shows an overview diagram of the TP-N2F model. It depicts the following high-level description.
Question: What is the performance proposed model achieved on MathQA?
Answer: Operation accuracy: 71.89
Execution accuracy: 55.95
Paper: Local Contextual Attention with Hierarchical Structure for Dialogue Act Recognition
Dialogue act (DA) characterizes the type of a speaker's intention in the course of producing an utterance and is approximately equivalent to the illocutionary act of BIBREF0 or the speech act of BIBREF1. The recognition of DA is essential for modeling and automatically detecting discourse structure, especially in developing a human-machine dialogue system. It is natural to predict the Answer acts following an utterance of type Question, and then match the Question utterance to each QA-pair in the knowledge base. The predicted DA can also guide the response generation process BIBREF2. For instance, system generates a Greeting type response to former Greeting type utterance. Moreover, DA is beneficial to other online dialogue strategies, such as conflict avoidance BIBREF3. In the offline system, DA also plays a significant role in summarizing and analyzing the collected utterances. For instance, recognizing DAs of a wholly online service record between customer and agent is beneficial to mine QA-pairs, which are selected and clustered then to expand the knowledge base. DA recognition is challenging due to the same utterance may have a different meaning in a different context. Table TABREF1 shows an example of some utterances together with their DAs from Switchboard dataset. In this example, utterance “Okay.” corresponds to two different DA labels within different semantic context.
Question: How do previous methods perform on the Switchboard Dialogue Act and DailyDialog datasets?
Paper: Local Contextual Attention with Hierarchical Structure for Dialogue Act Recognition
DA recognition is aimed to assign a label to each utterance in a conversation. It can be formulated as a supervised classification problem. There are two trends to solve this problem: 1) as a sequence labeling problem, it will predict the labels for all utterances in the whole dialogue history BIBREF13, BIBREF14, BIBREF9; 2) as a sentence classification problem, it will treat utterance independently without any context history BIBREF5, BIBREF15. Early studies rely heavily on handcrafted features such as lexical, syntactic, contextual, prosodic and speaker information and achieve good results BIBREF13, BIBREF4, BIBREF16.
Question: What previous methods is the proposed method compared against?
Answer: BLSTM+Attention+BLSTM
Hierarchical BLSTM-CRF
CRF-ASN
Hierarchical CNN (window 4)
mLSTM-RNN
DRLM-Conditional
LSTM-Softmax
RCNN
CNN
CRF
LSTM
BERT
Paper: Local Contextual Attention with Hierarchical Structure for Dialogue Act Recognition
Self-attention BIBREF11 achieves great success for its efficiently parallel computation and long-range dependency modeling.
Question: What is dialogue act recognition?
Paper: Local Contextual Attention with Hierarchical Structure for Dialogue Act Recognition
Before we describe the proposed model in detail, we first define the mathematical notation for the DA recognition task in this paper. Given the dataset, $X = (D_1,D_2,... D_L)$ with corresponding DA labels $(Y_1,Y_2,...Y_L)$. Each dialogue is a sequence of $ N_l $ utterances $ D_l = (u_1,u_2,...u_{N_l})$ with $ Y_l = (y_1,y_2,...y_{N_l}) $. Each utterance is padded or truncated to the length of $ M $ words, $u_j = (w_1,w_2,...w_{M})$.
Question: Which natural language(s) are studied?
Paper: Select and Attend: Towards Controllable Content Selection in Text Generation
Many text generation tasks, e.g., data-to-text, summarization and image captioning, can be naturally divided into two steps: content selection and surface realization. The generations are supposed to have two levels of diversity: (1) content-level diversity reflecting multiple possibilities of content selection (what to say) and (2) surface-level diversity reflecting the linguistic variations of verbalizing the selected contents (how to say) BIBREF0 , BIBREF1 . Recent neural network models handle these tasks with the encoder-decoder (Enc-Dec) framework BIBREF2 , BIBREF3 , which simultaneously performs selecting and verbalizing in a black-box way. Therefore, both levels of diversity are entangled within the generation. This entanglement, however, sacrifices the controllability and interpretability, making it diffifcult to specify the content to be conveyed in the generated text BIBREF4 , BIBREF5 .
Question: Does the performance necessarily drop when more control is desired?
Paper: Select and Attend: Towards Controllable Content Selection in Text Generation
Let INLINEFORM0 denote a source-target pair. INLINEFORM1 is a sequence of INLINEFORM2 and can be either some structured data or unstructured text/image depending on the task. INLINEFORM3 corresponds to INLINEFORM4 which is a text description of INLINEFORM5 . The goal of text generation is to learn a distribution INLINEFORM6 to automatically generate proper text.
Question: How does the model perform in comparison to end-to-end headline generation models?
Paper: Select and Attend: Towards Controllable Content Selection in Text Generation
Our goal is to decouple the content selection from the decoder by introducing an extra content selector. We hope the content-level diversity can be fully captured by the content selector for a more interpretable and controllable generation process. Following BIBREF6 , BIBREF16 , we define content selection as a sequence labeling task. Let INLINEFORM0 denote a sequence of binary selection masks. INLINEFORM1 if INLINEFORM2 is selected and 0 otherwise. INLINEFORM3 is assumed to be independent from each other and is sampled from a bernoulli distribution INLINEFORM4 . INLINEFORM6 is the bernoulli parameter, which we estimate using a two-layer feedforward network on top of the source encoder. Text are generated by first sampling INLINEFORM7 from INLINEFORM8 to decide which content to cover, then decode with the conditional distribution INLINEFORM9 . The text is expected to faithfully convey all selected contents and drop unselected ones. Fig. FIGREF8 depicts this generation process. Note that the selection is based on the token-level context-aware embeddings INLINEFORM10 and will maintain information from the surrounding contexts. It encourages the decoder to stay faithful to the original information instead of simply fabricating random sentences by connecting the selected tokens.
Question: How is the model trained to do only content selection?
Paper: Review Conversational Reading Comprehension
Seeking information to assess whether some products or services suit one's needs is a vital activity for consumer decision making. In online businesses, one major hindrance is that customers have limited access to answers to their specific questions or concerns about products and user experiences. Given the ever-changing environment of products and services, it is very hard, if not impossible, to pre-compile an up-to-date knowledge base to answer user questions as in KB-QA BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 . As a compromise, community question-answering (CQA) BIBREF4 is leveraged to enable existing customers or sellers to answer customer questions. However, one obvious drawback of this approach is that many questions are not answered, and even if they are answered, the answers and the following up questions are delayed, which is not suitable for interactive QA. Although existing studies have used information retrieval (IR) techniques BIBREF4 , BIBREF5 to identify a whole review as an answer to a question, it is time-consuming to read a whole review and the approach has difficulty to answer questions in multiple turns.
Question: What is the baseline model used?
Answer: The baseline models used are DrQA modified to support answering no answer questions, DrQA+CoQA which is pre-tuned on CoQA dataset, vanilla BERT, BERT+review tuned on domain reviews, BERT+CoQA tuned on the supervised CoQA data
Paper: Human Languages in Source Code: Auto-Translation for Localized Instruction
Reading and writing comments, method names and variable names is a crucial part of software engineering and as such, programs have both a human language, the language of identifiers and comments, in addition to the source-code language (eg Java or Python). This has meant that non-English speakers are often second class citizens when learning to program BIBREF0. In this paper we present a tool for translating a program from one human-language to another to assist in code education, which could reduce the barrier to computer science education for non-English speakers.
Question: Is this auto translation tool based on neural networks?
Paper: Human Languages in Source Code: Auto-Translation for Localized Instruction
To the best of our knowledge, automatic translation of code between human languages, did not appear in literature, making us hypothesize: it is either difficult, or had remained ignored. Nonetheless, we summarize related work that motivate our contribution.
Question: What are results of public code repository study?
Paper: Schema-Guided Dialogue State Tracking Task at DSTC8
Virtual assistants help users accomplish tasks including but not limited to finding flights, booking restaurants, by providing a natural language interface to services and APIs on the web. Large-scale assistants like Google Assistant, Amazon Alexa, Apple Siri, Microsoft Cortana etc. need to support a large and constantly increasing number of services, over a wide variety of domains. Consequently, recent work has focused on scalable dialogue systems that can handle tasks across multiple application domains. Data-driven deep learning based approaches for multi-domain modeling have shown promise, both for end-to-end and modular systems involving dialogue state tracking and policy learning. This line of work has been facilitated by the release of multi-domain dialogue corpora such as MultiWOZ BIBREF0, Taskmaster-1 BIBREF1, M2M BIBREF2 and FRAMES BIBREF3.
Question: Where is the dataset from?
Paper: Schema-Guided Dialogue State Tracking Task at DSTC8
Dialogue systems have constituted an active area of research for the past few decades. The advent of commercial personal assistants has provided further impetus to dialogue systems research. As virtual assistants incorporate diverse domains, zero-shot modeling BIBREF4, BIBREF5, BIBREF6, domain adaptation and transfer learning techniques BIBREF7, BIBREF8, BIBREF9 have been explored to support new domains in a data efficient manner.
Question: What data augmentation techniques are used?
Paper: Schema-Guided Dialogue State Tracking Task at DSTC8
The primary task of this challenge is to develop multi-domain models for DST suitable for the scale and complexity of large scale virtual assistants. Supporting a wide variety of APIs or services with possibly overlapping functionality is an important requirement of such assistants. A common approach to do this involves defining a large master schema that lists all intents and slots supported by the assistant. Each service either adopts this master schema for the representation of the underlying data, or provides logic to translate between its own schema and the master schema.
Question: Do all teams use neural networks for their models?
Paper: Schema-Guided Dialogue State Tracking Task at DSTC8
Under the Schema-Guided approach, each service provides a schema listing the supported slots and intents along with their natural language descriptions (Figure FIGREF2 shows an example). The dialogue annotations are guided by the schema of the underlying service or API, as shown in Figure FIGREF3. In this example, the departure and arrival cities are captured by analogously functioning but differently named slots in both schemas. Furthermore, values for the number_stops and direct_only slots highlight idiosyncrasies between services interpreting the same concept.
Question: How are the models evaluated?
Paper: Schema-Guided Dialogue State Tracking Task at DSTC8
As shown in Table TABREF9, our Schema-Guided Dialogue (SGD) dataset exceeds other datasets in most of the metrics at scale. The especially larger number of domains, slots, and slot values, and the presence of multiple services per domain, are representative of these scale-related challenges. Furthermore, our evaluation sets contain many services, and consequently slots, which are not present in the training set, to help evaluate model performance on unseen services.
Question: What is the baseline model?
Paper: Schema-Guided Dialogue State Tracking Task at DSTC8
The dataset consists of conversations between a virtual assistant and a user. Each conversation can span multiple services across various domains. The dialogue is represented as a sequence of turns, each containing a user or system utterance. The annotations for each turn are grouped into frames, where each frame corresponds to a single service. The annotations for user turns include the active intent, the dialogue state and slot spans for the different slots values mentioned in the turn. For system turns, we have the system actions representing the semantics of the system utterance. Each system action is represented using a dialogue act with optional parameters.
Question: What domains are present in the data?
Answer: Alarm, Banks, Buses, Calendar, Events, Flights, Homes, Hotels, Media, Messaging, Movies, Music, Payment, Rental Cars, Restaurants, Ride Sharing, Services, Train, Travel, Weather