z-uo/qasper-squad huggingface dataset:
Paper: Pre-training of Graph Augmented Transformers for Medication Recommendation
The availability of massive electronic health records (EHR) data and the advances of deep learning technologies have provided unprecedented resource and opportunity for predictive healthcare, including the computational medication recommendation task. A number of deep learning models were proposed to assist doctors in making medication recommendation BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 . They often learn representations for medical entities (e.g., patients, diagnosis, medications) from patient EHR data, and then use the learned representations to predict medications that are suited to the patient's health condition.
Question: IS the graph representation supervised?
Answer: The graph representation appears to be semi-supervised. It is included in the learning pipeline for the medical recommendation, where the attention model is learned. (There is some additional evidence that is unavailable in parsed text)
Paper: Pre-training of Graph Augmented Transformers for Medication Recommendation
Medication Recommendation Medication Recommendation can be categorized into instance-based and longitudinal recommendation methods BIBREF1 . Instance-based methods focus on current health conditions. Among them, Leap BIBREF9 formulates a multi-instance multi-label learning framework and proposes a variant of sequence-to-sequence model based on content-attention mechanism to predict combination of medicines given patient's diagnoses. Longitudinal-based methods leverage the temporal dependencies among clinical events, see BIBREF10 , BIBREF11 , BIBREF12 . Among them, RETAIN BIBREF10 uses a two-level neural attention model to detect influential past visits and significant clinical variables within those visits for improved medication recommendation.
Question: Is the G-BERT model useful beyond the task considered?
Answer: There is nothing specific about the approach that depends on medical recommendations. The approach combines graph data and text data into a single embedding.
It learns a representation of medical records. The learned representation (embeddings) can be used for other predictive tasks involving information from electronic health records.
Paper: RUN through the Streets: A New Dataset and Baseline Models for Realistic Urban Navigation
The task of interpreting and following natural language (NL) navigation instructions involves interleaving different signals, at the very least the linguistic utterance and the representation of the world. For example, in turn right on the first intersection, the instruction needs to be interpreted, and a specific object in the world (the intersection) needs to be located in order to execute the instruction. In NL navigation studies, the representation of the world may be provided via visual sensors BIBREF0, BIBREF1, BIBREF2, BIBREF3, BIBREF4 or as a symbolic world representation. This work focuses on navigation based on a symbolic world representation (referred to as a map).
Question: How well did the baseline perform?
Paper: RUN through the Streets: A New Dataset and Baseline Models for Realistic Urban Navigation
In this work we address the task of following a sequence of NL navigation instructions given in colloquial language based on a dense urban map.
Question: What is the baseline?
Paper: Combining Advanced Methods in Japanese-Vietnamese Neural Machine Translation
Neural machine translation (NMT) BIBREF0 , BIBREF1 is widely applied for machine translation (MT) in recent years and focuses on popular language pairs such as English INLINEFORM0 French, English INLINEFORM1 German, English INLINEFORM2 Chinese or English INLINEFORM3 Japanese. NMT has obtained state-of-the-art performance on those language pairs compared to the traditional statistical machine translation (SMT) when given enough data BIBREF2 , BIBREF3 . Furthermore, due to the ability of feature learning, NMT systems can be trained end-to-end with pure parallel texts and minimal linguistic knowledge of the languages involved. Thus it makes training NMT for a new language pair much easier, more scalable and robust. Nevertheless, NMT has not been employed in many low-resourced language pairs since in those scenarios, data scarcity often limits the learning ability of neural methods. In contrast, combinating complicated linguistic-driven features in a typical log-linear framework still keeps SMT the best approach in many translation directions but also hard to apply to new domains or to other language pairs.
Question: what methods were used to reduce data sparsity effects?
Paper: Combining Advanced Methods in Japanese-Vietnamese Neural Machine Translation
In this section, we will describe the general architecture of NMT as a kind of sequence-to-sequence modeling framework. In this kind of sequence-to-sequence modeling framework, often there is an encoder trying to encode context information from the input sequence and a decoder to generate one item of the output sequence at a time based on the context of both input and output sequences. Besides, an additional component, named attention, exists in between, deciding which parts of the input sequence the decoder should pay attention in order to choose which to output next. In other words, this attention component calculates the context relevant to the decision of the decoder at the considering time. Those components as a whole constitute a large trainable neural architecture called the famous attention-based encoder-decoder framework. This framework becomes popular in many sequence-to-sequence tasks.
Question: what was the baseline?
Paper: Combining Advanced Methods in Japanese-Vietnamese Neural Machine Translation
One of the most severe problems of NMT is dealing with the rare words, which are not in the short lists of the vocabularies, i.e. out-of-vocabulary (OOV) words, or do not appear in the training set at all. On one hand, we would like to have fewer OOV words by increasing the size of the short lists. On the other hand, we need our neural network to learn fast and has a good generalization capability on the unseen words as well.
Question: did they collect their own data?
Paper: Combining Advanced Methods in Japanese-Vietnamese Neural Machine Translation
Vietnamese From the linguistic point of view, each sequence of characters between two white spaces in Vietnamese texts cannot be considered as a word since it does not always have a full meaning to stand alone. For example, in the sentence “hôm nay là sinh nhật của tôi” (English equivalence: “Today is my birthday”), “hôm” and “nay” are not two words, they together form a word, which means “today”. Nevertheless, “hôm” and “nay” somehow still bear some meaning: “hôm”-“day”, “nay”-“now”. Similarly, “sinh”-“birth” and “nhật”-“date” also form the word “sinh nhật”-“birthday” but they are not two distinct words. We could also call them subwords.
Question: what japanese-vietnamese dataset do they use?
Paper: Grammatical Error Correction and Style Transfer via Zero-shot Monolingual Translation
Sequence-to-sequence (seq2seq) transformations have recently proven to be a successful framework for several natural language processing tasks, like: machine translation (MT) BIBREF0 , BIBREF1 , speech recognition BIBREF2 , speech synthesis BIBREF3 , natural language inference BIBREF4 and others. However, the success of these models depends on the availability of large amounts of directly annotated data for the task at hand (like translation examples, text segments and their speech recordings, etc.). This is a severe limitation for tasks where data is not abundantly available as well as for low-resource languages.
Question: How do they measure style transfer success?
Paper: Grammatical Error Correction and Style Transfer via Zero-shot Monolingual Translation
As mentioned in the introduction, our approach is based on the idea of zero-shot MT BIBREF11 . There the authors show that after training a single model to translate from Portuguese to English as well as from English to Spanish, it can also translate Portuguese into Spanish, without seeing any translation examples for this language pair. We use the zero-shot effect to achieve monolingual translation by training the model on bilingual examples in both directions, and then doing translation into the same language as the input: illustrated on Figure FIGREF1 .
Question: Do they introduce errors in the data or does the data already contain them?
Answer: Data already contain errors
Paper: Grammatical Error Correction and Style Transfer via Zero-shot Monolingual Translation
We use three languages in our experiments: English, Estonian and Latvian. All three have different characteristics, for example Latvian and (especially) Estonian are morphologically complex and have loose word order, while English has a strict word order and the morphology is much simpler. Most importantly, all three languages have error-corrected corpora for testing purposes, though work on their automatic grammatical error correction is extremely limited (see Section SECREF3 ).
Question: What error types is their model more reliable for?
Paper: Grammatical Error Correction and Style Transfer via Zero-shot Monolingual Translation
For Europarl, JRC-Acquis and EMEA we use all data available for English-Estonian, English-Latvian and Estonian-Latvian language pairs. From OpenSubtitles2018 we take a random subset of 3M sentence pairs for English-Estonian, which is still more than English-Latvian and Estonian-Latvian (below 1M; there we use the whole corpus). This is done to balance the corpora representation and to limit the size of training data.
Question: How does their parallel data differ in terms of style?
Paper: Query-based Attention CNN for Text Similarity Map
Many machine learning models in question answering tasks often involve matching mechanism. For example, in factoid question answering such as SQuAD BIBREF1 , one needs to match between query and corpus in order to find out the most possible fragment as answer. In multiple choice question answering, such as MC Test BIBREF2 , matching mechanism can also help make the correct decision.
Question: How do they split text to obtain sentence levels?
Paper: Query-based Attention CNN for Text Similarity Map
In this question answering task, a reading passage , a query and several answer choices are given. P denotes the passage, Q denotes query and C denotes one of the multiple choices. The target of the model is to choose a correct answer A from multiple choices based on informations of P and Q.
Question: Do they experiment with their proposed model on any other dataset other than MovieQA?
Paper: Natural Language Generation for Spoken Dialogue System using RNN Encoder-Decoder Networks
Natural Language Generation (NLG) plays a critical role in Spoken Dialogue Systems (SDS) with task is to convert a meaning representation produced by the Dialogue Manager into natural language utterances. Conventional approaches still rely on comprehensive hand-tuning templates and rules requiring expert knowledge of linguistic representation, including rule-based BIBREF0 , corpus-based n-gram models BIBREF1 , and a trainable generator BIBREF2 .
Question: What is the difference of the proposed model with a standard RNN encoder-decoder?
Answer: Introduce a "Refinement Adjustment LSTM-based component" to the decoder
Paper: Natural Language Generation for Spoken Dialogue System using RNN Encoder-Decoder Networks
Recently, RNNs-based models have shown promising performance in tackling the NLG problems. BIBREF16 proposed a generator using RNNs to create Chinese poetry. BIBREF11 , BIBREF17 , BIBREF18 also used RNNs in a multi-modal setting to solve image captioning tasks. The RNN-based Sequence to Sequence models have applied to solve variety of tasks: conversational modeling BIBREF6 , BIBREF7 , BIBREF19 , machine translation BIBREF20 , BIBREF21 
Question: Does the model evaluated on NLG datasets or dialog datasets?
Answer: NLG datasets
NLG datasets
Paper: Dynamic Compositional Neural Networks over Tree Structure
Learning the distributed representation for long spans of text from its constituents has been a key step for various natural language processing (NLP) tasks, such as text classification BIBREF0 , BIBREF1 , semantic matching BIBREF2 , BIBREF3 , and machine translation BIBREF4 . Existing deep learning approaches take a compositional function with different forms to compose word vectors recursively until obtaining a sentential representation. Typically, these compositional functions involve recurrent neural networks BIBREF5 , BIBREF6 , convolutional neural networks BIBREF7 , BIBREF8 , and tree-structured neural networks BIBREF9 , BIBREF10 .
Question: What tasks do they experiment with?
Paper: Dynamic Compositional Neural Networks over Tree Structure
In this section, we briefly describe the tree-structured neural networks.
Question: What is the meta knowledge specifically?
Paper: PitchNet: Unsupervised Singing Voice Conversion with Pitch Adversarial Network
Singing is an important way of human expression and the techniques of singing synthesis have broad applications in different prospects including virtual human, movie dubbing and so on. Traditional singing synthesize systems are based on concatenative BIBREF1 or HMM BIBREF2 based approaches. With the success of deep learning in Text-to-Speech, some neural singing synthesis methods have also been proposed recently. For example, BIBREF3 introduces a singing synthesis method using an architecture similar to WaveNet BIBREF4. It adopts lyrics and notes as input and generates vocoder features autoregressively for final singing voice synthesis.
Question: Are there elements, other than pitch, that can potentially result in out of key converted singing?
Paper: PitchNet: Unsupervised Singing Voice Conversion with Pitch Adversarial Network
Our method follows the autoencoder architecture in BIBREF0 except that there is an additional pitch regression network to separate pitch information out of the latent space. The architecture of PitchNet is illustrated in Fig. FIGREF1. It consists of five parts, an encoder, a decoder, a Look Up Table (LUT) of speaker embedding vectors, a singer classification network, and a pitch regression network.
Question: How is the quality of singing voice measured?
Answer: Automatic: Normalized cross correlation (NCC)
Manual: Mean Opinion Score (MOS)
Paper: Pyramidal Recurrent Unit for Language Modeling
Long short term memory (LSTM) units BIBREF1 are popular for many sequence modeling tasks and are used extensively in language modeling. A key to their success is their articulated gating structure, which allows for more control over the information passed along the recurrence. However, despite the sophistication of the gating mechanisms employed in LSTMs and similar recurrent units, the input and context vectors are treated with simple linear transformations prior to gating. Non-linear transformations such as convolutions BIBREF2 have been used, but these have not achieved the performance of well regularized LSTMs for language modeling BIBREF3 .
Question: what data did they use?
Paper: Pyramidal Recurrent Unit for Language Modeling
Multiple methods, including a variety of gating structures and transformations, have been proposed to improve the performance of recurrent neural networks (RNNs). We first describe these approaches and then provide an overview of recent work in language modeling.
Question: what previous RNN models do they compare with?
Answer: Variational LSTM, CharCNN, Pointer Sentinel-LSTM, RHN, NAS Cell, SRU, QRNN, RAN, 4-layer skip-connection LSTM, AWD-LSTM, Quantized LSTM
Paper: Translation Artifacts in Cross-lingual Transfer Learning
While most NLP resources are English-specific, there have been several recent efforts to build multilingual benchmarks. One possibility is to collect and annotate data in multiple languages separately BIBREF0, but most existing datasets have been created through translation BIBREF1, BIBREF2. This approach has two desirable properties: it relies on existing professional translation services rather than requiring expertise in multiple languages, and it results in parallel evaluation sets that offer a meaningful measure of the cross-lingual transfer gap of different models. The resulting multilingual datasets are generally used for evaluation only, relying on existing English datasets for training.
Question: What are examples of these artificats?
Paper: Translation Artifacts in Cross-lingual Transfer Learning
Current cross-lingual models work by pre-training multilingual representations using some form of language modeling, which are then fine-tuned on the relevant task and transferred to different languages. Some authors leverage parallel data to that end BIBREF5, BIBREF6, but training a model akin to BERT BIBREF7 on the combination of monolingual corpora in multiple languages is also effective BIBREF8. Closely related to our work, BIBREF4 showed that replacing segments of the training data with their translation during fine-tuning is helpful. However, they attribute this behavior to a data augmentation effect, which we believe should be reconsidered given the new evidence we provide.
Question: What are the languages they use in their experiment?
Answer: English
French
Spanish
German
Greek
Bulgarian
Russian
Turkish
Arabic
Vietnamese
Thai
Chinese
Hindi
Swahili
Urdu
Finnish
Paper: Translation Artifacts in Cross-lingual Transfer Learning
Most benchmarks covering a wide set of languages have been created through translation, as it is the case of XNLI BIBREF1 for NLI, PAWS-X BIBREF9 for adversarial paraphrase identification, and XQuAD BIBREF2 and MLQA BIBREF10 for Question Answering (QA). A notable exception is TyDi QA BIBREF0, a contemporaneous QA dataset that was separately annotated in 11 languages. Other cross-lingual datasets leverage existing multilingual resources, as it is the case of MLDoc BIBREF11 for document classification and Wikiann BIBREF12 for named entity recognition. Concurrent to our work, BIBREF13 combine some of these datasets into a single multilingual benchmark, and evaluate some well-known methods on it.
Question: Does the professional translation or the machine translation introduce the artifacts?
Paper: Translation Artifacts in Cross-lingual Transfer Learning
Several studies have shown that NLI datasets like SNLI BIBREF14 and MultiNLI BIBREF15 contain spurious patterns that can be exploited to obtain strong results without making real inferential decisions. For instance, BIBREF16 and BIBREF17 showed that a hypothesis-only baseline performs better than chance due to cues on their lexical choice and sentence length. Similarly, BIBREF18 showed that NLI models tend to predict entailment for sentence pairs with a high lexical overlap. Several authors have worked on adversarial datasets to diagnose these issues and provide a more challenging benchmark BIBREF19, BIBREF20, BIBREF21. Besides NLI, other tasks like QA have also been found to be susceptible to annotation artifacts BIBREF22, BIBREF23. While previous work has focused on the monolingual scenario, we show that translation can interfere with these artifacts in multilingual settings.
Question: Do they recommend translating the premise and hypothesis together?
Paper: Translation Artifacts in Cross-lingual Transfer Learning
Translated texts are known to have unique features like simplification, explicitation, normalization and interference, which are refer to as translationese BIBREF24. This phenomenon has been reported to have a notable impact in machine translation evaluation BIBREF25, BIBREF26. For instance, back-translation brings large BLEU gains for reversed test sets (i.e. when translationese is on the source side and original text is used as reference), but its effect diminishes in the natural direction BIBREF27. While connected, the phenomenon we analyze is different in that it arises from translation inconsistencies due to the lack of context, and affects cross-lingual transfer learning rather than machine translation.
Question: Is the improvement over state-of-the-art statistically significant?
Paper: Translation Artifacts in Cross-lingual Transfer Learning
Our goal is to analyze the effect of both human and machine translation in cross-lingual models. For that purpose, the core idea of our work is to (i) use machine translation to either translate the training set into other languages, or generate English paraphrases of it through back-translation, and (ii) evaluate the resulting systems on original, human translated and machine translated test sets in comparison with systems trained on original data. We next describe the models used in our experiments (§SECREF6), the specific training variants explored (§SECREF8), and the evaluation procedure followed (§SECREF10).
Question: What are examples of these artifacts?
Paper: Translation Artifacts in Cross-lingual Transfer Learning
We experiment with two models that are representative of the state-of-the-art in monolingual and cross-lingual pre-training: (i) Roberta BIBREF28, which is an improved version of BERT that uses masked language modeling to pre-train an English Transformer model, and (ii) XLM-R BIBREF8, which is a multilingual extension of the former pre-trained on 100 languages. In both cases, we use the large models released by the authors under the fairseq repository. As discussed next, we explore different variants of the training set to fine-tune each model on different tasks. At test time, we try both machine translating the test set into English (Translate-Test) and, in the case of XLM-R, using the actual test set in the target language (Zero-Shot).
Question: What languages do they use in their experiments?
Paper: Predicting Annotation Difficulty to Improve Task Routing and Model Performance for Biomedical Information Extraction
Assembling training corpora of annotated natural language examples in specialized domains such as biomedicine poses considerable challenges. Experts with the requisite domain knowledge to perform high-quality annotation tend to be expensive, while lay annotators may not have the necessary knowledge to provide high-quality annotations. A practical approach for collecting a sufficiently large corpus would be to use crowdsourcing platforms like Amazon Mechanical Turk (MTurk). However, crowd workers in general are likely to provide noisy annotations BIBREF0 , BIBREF1 , BIBREF2 , an issue exacerbated by the technical nature of specialized content. Some of this noise may reflect worker quality and can be modeled BIBREF0 , BIBREF1 , BIBREF3 , BIBREF4 , but for some instances lay people may simply lack the domain knowledge to provide useful annotation.
Question: How much higher quality is the resulting annotated data?
Paper: Predicting Annotation Difficulty to Improve Task Routing and Model Performance for Biomedical Information Extraction
Crowdsourcing annotation is now a well-studied problem BIBREF7 , BIBREF0 , BIBREF1 , BIBREF2 . Due to the noise inherent in such annotations, there have also been considerable efforts to develop aggregation models that minimize noise BIBREF0 , BIBREF1 , BIBREF3 , BIBREF4 .
Question: How do they match annotators to instances?
Answer: Annotations from experts are used if they have already been collected.
Paper: Predicting Annotation Difficulty to Improve Task Routing and Model Performance for Biomedical Information Extraction
Our specific application concerns annotating abstracts of articles that describe the conduct and results of randomized controlled trials (RCTs). Experimentation in this domain has become easy with the recent release of the EBM-NLP BIBREF5 corpus, which includes a reasonably large training dataset annotated via crowdsourcing, and a modest test set labeled by individuals with advanced medical training. More specifically, the training set comprises 4,741 medical article abstracts with crowdsourced annotations indicating snippets (sequences) that describe the Participants (p), Interventions (i), and Outcome (o) elements of the respective RCT, and the test set is composed of 191 abstracts with p, i, o sequence annotations from three medical experts.
Question: How much data is needed to train the task-specific encoder?
Answer: 57,505 sentences
57,505 sentences