z-uo/qasper-squad huggingface dataset:
Paper: Cross-lingual Pre-training Based Transfer for Zero-shot Neural Machine Translation
Although Neural Machine Translation (NMT) has dominated recent research on translation tasks BIBREF0, BIBREF1, BIBREF2, NMT heavily relies on large-scale parallel data, resulting in poor performance on low-resource or zero-resource language pairs BIBREF3. Translation between these low-resource languages (e.g., Arabic$\rightarrow $Spanish) is usually accomplished with pivoting through a rich-resource language (such as English), i.e., Arabic (source) sentence is translated to English (pivot) first which is later translated to Spanish (target) BIBREF4, BIBREF5. However, the pivot-based method requires doubled decoding time and suffers from the propagation of translation errors.
Question: which multilingual approaches do they compare with?
Paper: Cross-lingual Pre-training Based Transfer for Zero-shot Neural Machine Translation
In recent years, zero-shot translation in NMT has attracted widespread attention in academic research. Existing methods are mainly divided into four categories: pivot-based method, transfer learning, multilingual NMT, and unsupervised NMT.
Question: what are the pivot-based baselines?
Paper: Cross-lingual Pre-training Based Transfer for Zero-shot Neural Machine Translation
In this section, we will present a cross-lingual pre-training based transfer approach. This method is designed for a common zero-shot scenario where there are a lot of source$\leftrightarrow $pivot and pivot$\leftrightarrow $target bilingual data but no source$\leftrightarrow $target parallel data, and the whole training process can be summarized as follows step by step:
Question: which datasets did they experiment with?
Paper: Cross-lingual Pre-training Based Transfer for Zero-shot Neural Machine Translation
Two existing cross-lingual pre-training methods, Masked Language Modeling (MLM) and Translation Language Modeling (TLM), have shown their effectiveness on XNLI cross-lingual classification task BIBREF11, BIBREF28, but these methods have not been well studied on cross-lingual generation tasks in zero-shot condition. We attempt to take advantage of the cross-lingual ability of the two methods for zero-shot translation.
Question: what language pairs are explored?
Answer: De-En, En-Fr, Fr-En, En-Es, Ro-En, En-De, Ar-En, En-Ru
Paper: pioNER: Datasets and Baselines for Armenian Named Entity Recognition
Named entity recognition is an important task of natural language processing, featuring in many popular text processing toolkits. This area of natural language processing has been actively studied in the latest decades and the advent of deep learning reinvigorated the research on more effective and accurate models. However, most of existing approaches require large annotated corpora. To the best of our knowledge, no such work has been done for the Armenian language, and in this work we address several problems, including the creation of a corpus for training machine learning models, the development of gold-standard test corpus and evaluation of the effectiveness of established approaches for the Armenian language.
Question: what ner models were evaluated?
Paper: pioNER: Datasets and Baselines for Armenian Named Entity Recognition
We used Sysoev and Andrianov's modification of the Nothman et al. approach to automatically generate data for training a named entity recognizer. This approach uses links between Wikipedia articles to generate sequences of named-entity annotated tokens.
Question: what is the source of the news sentences?
Paper: pioNER: Datasets and Baselines for Armenian Named Entity Recognition
The main steps of the dataset extraction system are described in Figure FIGREF3 .
Question: did they use a crowdsourcing platform for manual annotations?
Paper: Identifying Dogmatism in Social Media: Signals and Models
“I'm supposed to trust the opinion of a MS minion? The people that produced Windows ME, Vista and 8? They don't even understand people, yet they think they can predict the behavior of new, self-guiding AI?” –anonymous
Question: what are the topics pulled from Reddit?
Answer: training data has posts from politics, business, science and other popular topics; the trained model is applied to millions of unannotated posts on all of Reddit
Paper: Identifying Dogmatism in Social Media: Signals and Models
Posts on Reddit capture debate and discussion across a diverse set of topics, making them a natural starting point for untangling domain-independent linguistic features of dogmatism.
Question: What predictive model do they build?
Paper: Adversarial Learning for Chinese NER from Crowd Annotations
There has been significant progress on Named Entity Recognition (NER) in recent years using models based on machine learning algorithms BIBREF0 , BIBREF1 , BIBREF2 . As with other Natural Language Processing (NLP) tasks, building NER systems typically requires a massive amount of labeled training data which are annotated by experts. In real applications, we often need to consider new types of entities in new domains where we do not have existing annotated data. For such new types of entities, however, it is very hard to find experts to annotate the data within short time limits and hiring experts is costly and non-scalable, both in terms of time and money.
Question: What accuracy does the proposed system achieve?
Answer: F1 scores of 85.99 on the DL-PS data,  75.15 on the EC-MT data and 71.53 on the EC-UQ data 
F1 of 85.99 on the DL-PS dataset (dialog domain); 75.15 on EC-MT and 71.53 on EC-UQ (e-commerce domain)
Paper: Adversarial Learning for Chinese NER from Crowd Annotations
Our work is related to three lines of research: Sequence labeling, Adversarial training, and Crowdsourcing.
Question: What crowdsourcing platform is used?
Answer: They did not use any platform, instead they hired undergraduate students to do the annotation.
Paper: Addressing word-order Divergence in Multilingual Neural Machine Translation for extremely Low Resource Languages
Deep Learning approaches have achieved impressive results on various NLP tasks BIBREF0 , BIBREF1 , BIBREF2 and have become the de facto approach for any NLP task. However, these deep learning techniques have found to be less effective for low-resource languages when the available training data is very less BIBREF3 . Recently, several approaches like Multi-task learning BIBREF4 , multilingual learning BIBREF5 , semi-supervised learning BIBREF2 , BIBREF6 and transfer learning BIBREF7 , BIBREF3 have been explored by the deep learning community to overcome data sparsity in low-resource languages. Transfer learning trains a model for a parent task and fine-tunes the learned parent model weights (features) for a related child task BIBREF7 , BIBREF8 . This effectively reduces the requirement on training data for the child task as the model would have learned relevant features from the parent task data thereby, improving the performance on the child task.
Question: How do they match words before reordering them?
Paper: Addressing word-order Divergence in Multilingual Neural Machine Translation for extremely Low Resource Languages
 BIBREF3 explored transfer learning for NMT on low-resource languages. They studied the influence of language divergence between languages chosen for training the parent and child model, and showed that choosing similar languages for training the parent and child model leads to better improvements from transfer learning. A limitation of BIBREF3 approach is that they ignore the lexical similarity between languages and also the source language embeddings are randomly initialized. BIBREF10 , BIBREF11 , BIBREF12 take advantage of lexical similarity between languages in their work. BIBREF10 proposed to use Byte-Pair Encoding (BPE) to represent the sentences in both the parent and the child language to overcome the above limitation. They show using BPE benefits transfer learning especially when the involved languages are closely-related agglutinative languages. Similarly, BIBREF11 utilize lexical similarity between the source and assisting languages by training a character-level NMT system. BIBREF12 address lexical divergence by using bilingual embeddings and mixture of universal token embeddings. One of the languages' vocabulary, usually English vocabulary is considered as universal tokens and every word in the other languages is represented as a mixture of universal tokens. They show results on extremely low-resource languages.
Question: On how many language pairs do they show that preordering assisting language sentences helps translation quality?
Answer: 5
Paper: Addressing word-order Divergence in Multilingual Neural Machine Translation for extremely Low Resource Languages
To the best of our knowledge, no work has addressed word order divergence in transfer learning for multilingual NMT. However, some work exists for other NLP tasks that could potentially address word order. For Named Entity Recognition (NER), BIBREF14 use a self-attention layer after the Bi-LSTM layer to address word-order divergence for Named Entity Recognition (NER) task. The approach does not show any significant improvements over multiple languages. A possible reason is that the divergence has to be addressed before/during construction of the contextual embeddings in the Bi-LSTM layer, and the subsequent self-attention layer does not address word-order divergence. BIBREF15 use adversarial training for cross-lingual question-question similarity ranking in community question answering. The adversarial training tries to force the encoder representations of similar sentences from different input languages to have similar representations.
Question: Which dataset(s) do they experiment with?
Paper: A Corpus for Automatic Readability Assessment and Text Simplification of German
Simplified language is a variety of standard language characterized by reduced lexical and syntactic complexity, the addition of explanations for difficult concepts, and clearly structured layout. Among the target groups of simplified language commonly mentioned are persons with cognitive impairment or learning disabilities, prelingually deaf persons, functionally illiterate persons, and foreign language learners BIBREF0.
Question: Which information about text structure is included in the corpus?
Answer: paragraph, lines, textspan element (paragraph segmentation, line segmentation, Information on physical page segmentation(for PDF only))
Paper: A Corpus for Automatic Readability Assessment and Text Simplification of German
A number of corpora for use in automatic readability assessment and automatic text simplification exist. The most well-known example is the Parallel Wikipedia Simplification Corpus (PWKP) compiled from parallel articles of the English Wikipedia and Simple English Wikipedia BIBREF13 and consisting of around 108,000 sentence pairs. The corpus profile is shown in Table TABREF2. While the corpus represents the largest dataset involving simplified language to date, its application has been criticized for various reasons BIBREF15, BIBREF14, BIBREF16; among these, the fact that Simple English Wikipedia articles are not necessarily direct translations of articles from the English Wikipedia stands out. hwang-et-al-2015 provided an updated version of the corpus that includes a total of 280,000 full and partial matches between the two Wikipedia versions. Another frequently used data collection for English is the Newsela Corpus BIBREF14 consisting of 1,130 news articles, each simplified into four school grade levels by professional editors. Table TABREF3 shows the profile of the Newsela Corpus. The table obviates that the difference in vocabulary size between the English and the simplified English side of the PWKP Corpus amounts to only 18%, while the corresponding number for the English side and the level representing the highest amount of simplification in the Newsela Corpus (Simple-4) is 50.8%. Vocabulary size as an indicator of lexical richness is generally taken to correlate positively with complexity BIBREF17.
Question: Which information about typography is included in the corpus?
Paper: Improved Neural Relation Detection for Knowledge Base Question Answering
Knowledge Base Question Answering (KBQA) systems answer questions by obtaining information from KB tuples BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 . For an input question, these systems typically generate a KB query, which can be executed to retrieve the answers from a KB. Figure 1 illustrates the process used to parse two sample questions in a KBQA system: (a) a single-relation question, which can be answered with a single $<$ head-entity, relation, tail-entity $>$ KB tuple BIBREF6 , BIBREF7 , BIBREF2 ; and (b) a more complex case, where some constraints need to be handled for multiple entities in the question. The KBQA system in the figure performs two key tasks: (1) entity linking, which links $n$ -grams in questions to KB entities, and (2) relation detection, which identifies the KB relation(s) a question refers to.
Question: On which benchmarks they achieve the state of the art?
Paper: Improved Neural Relation Detection for Knowledge Base Question Answering
Previous research BIBREF4 , BIBREF20 formulates KB relation detection as a sequence matching problem. However, while the questions are natural word sequences, how to represent relations as sequences remains a challenging problem. Here we give an overview of two types of relation sequence representations commonly used in previous work.
Question: What they use in their propsoed framework?
Paper: Improved Neural Relation Detection for Knowledge Base Question Answering
This section describes our hierarchical sequence matching with residual learning approach for relation detection. In order to match the question to different aspects of a relation (with different abstraction levels), we deal with three problems as follows on learning question/relation representations.
Question: What does KBQA abbreviate for
Paper: Improved Neural Relation Detection for Knowledge Base Question Answering
We provide our model with both types of relation representation: word-level and relation-level. Therefore, the input relation becomes $\mathbf {r}=\lbrace r^{word}_1,\cdots ,r^{word}_{M_1}\rbrace  \cup \lbrace r^{rel}_1,\cdots ,r^{rel}_{M_2}\rbrace $ , where the first $M_1$ tokens are words (e.g. {episode, written}), and the last $M_2$ tokens are relation names, e.g., {episode_written} or {starring_roles, series} (when the target is a chain like in Figure 1 (b)). We transform each token above to its word embedding then use two BiLSTMs (with shared parameters) to get their hidden representations $[\mathbf {B}^{word}_{1:M_1}:\mathbf {B}^{rel}_{1:M_2}]$ (each row vector $\mathbf {\beta }_i$ is the concatenation between forward/backward representations at $i$ ). We initialize the relation sequence LSTMs with the final state representations of the word sequence, as a back-off for unseen relations. We apply one max-pooling on these two sets of vectors and get the final relation representation $\mathbf {h}^r$ .
Question: What is te core component for KBQA?
Paper: How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings
The application of deep learning methods to NLP is made possible by representing words as vectors in a low-dimensional continuous space. Traditionally, these word embeddings were static: each word had a single vector, regardless of context BIBREF0, BIBREF1. This posed several problems, most notably that all senses of a polysemous word had to share the same representation. More recent work, namely deep neural language models such as ELMo BIBREF2 and BERT BIBREF3, have successfully created contextualized word representations, word vectors that are sensitive to the context in which they appear. Replacing static embeddings with contextualized representations has yielded significant improvements on a diverse array of NLP tasks, ranging from question-answering to coreference resolution.
Question: What experiments are proposed to test that upper layers produce context-specific embeddings?
Answer: They measure self-similarity, intra-sentence similarity and maximum explainable variance of the embeddings in the upper layers.
They plot the average cosine similarity between uniformly random words increases exponentially from layers 8 through 12.  
They plot the average self-similarity of uniformly randomly sampled words in each layer of BERT, ELMo, and GPT-2 and shown that the higher layer produces more context-specific embeddings.
They plot that word representations in a sentence become more context-specific in upper layers, they drift away from one another.
Paper: How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings
Skip-gram with negative sampling (SGNS) BIBREF0 and GloVe BIBREF1 are among the best known models for generating static word embeddings. Though they learn embeddings iteratively in practice, it has been proven that in theory, they both implicitly factorize a word-context matrix containing a co-occurrence statistic BIBREF7, BIBREF8. Because they create a single representation for each word, a notable problem with static word embeddings is that all senses of a polysemous word must share a single vector.
Question: How do they calculate a static embedding for each word?
Answer: They use the first principal component of a word's contextualized representation in a given layer as its static embedding.
Paper: Sensitive Data Detection and Classification in Spanish Clinical Text: Experiments with BERT
During the first two decades of the 21st century, the sharing and processing of vast amounts of data has become pervasive. This expansion of data sharing and processing capabilities is both a blessing and a curse. Data helps build better information systems for the digital era and enables further research for advanced data management that benefits the society in general. But the use of this very data containing sensitive information conflicts with private data protection, both from an ethical and a legal perspective.
Question: What is the performance of BERT on the task?
Answer: F1 scores are:
HUBES-PHI: Detection(0.965), Classification relaxed (0.95), Classification strict (0.937)
Medoccan: Detection(0.972), Classification (0.967)
Paper: Sensitive Data Detection and Classification in Spanish Clinical Text: Experiments with BERT
The state of the art in the field of Natural Language Processing (NLP) has reached an important milestone in the last couple of years thanks to deep-learning architectures, increasing in several points the performance of new models for almost any text processing task.
Question: What are the other algorithms tested?
Paper: Sensitive Data Detection and Classification in Spanish Clinical Text: Experiments with BERT
The aim of this paper is to evaluate BERT's multilingual model and compare it to other established machine-learning algorithms in a specific task: sensitive data detection and classification in Spanish clinical free text. This section describes the data involved in the experiments and the systems evaluated. Finally, we introduce the experimental setup.
Question: Does BERT reach the best performance among all the algorithms compared?
Paper: Sensitive Data Detection and Classification in Spanish Clinical Text: Experiments with BERT
Two datasets are exploited in this article. Both datasets consist of plain text containing clinical narrative written in Spanish, and their respective manual annotations of sensitive information in BRAT BIBREF13 standoff format. In order to feed the data to the different algorithms presented in Section SECREF7, these datasets were transformed to comply with the commonly used BIO sequence representation scheme BIBREF14.
Question: What are the clinical datasets used in the paper?
Paper: Massively Multilingual Neural Grapheme-to-Phoneme Conversion
Accurate grapheme-to-phoneme conversion (g2p) is important for any application that depends on the sometimes inconsistent relationship between spoken and written language. Most prominently, this includes text-to-speech and automatic speech recognition. Most work on g2p has focused on a few languages for which extensive pronunciation data is available BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 . Most languages lack these resources. However, a low resource language's writing system is likely to be similar to the writing systems of languages that do have sufficient pronunciation data. Therefore g2p may be possible for low resource languages if this high resource data can be properly utilized.
Question: how is model compactness measured?
Answer: Using file size on disk
Paper: Massively Multilingual Neural Grapheme-to-Phoneme Conversion
Our approach is similar in goal to deri2016grapheme's model for adapting high resource g2p models for low resource languages. They trained weighted finite state transducer (wFST) models on a variety of high resource languages, then transferred those models to low resource languages, using a language distance metric to choose which high resource models to use and a phoneme distance metric to map the high resource language's phonemes to the low resource language's phoneme inventory. These distance metrics are computed based on data from Phoible BIBREF4 and URIEL BIBREF5 .
Question: what was the baseline?
Paper: Massively Multilingual Neural Grapheme-to-Phoneme Conversion
In recent years, neural networks have emerged as a common way to use data from several languages in a single system. Google's zero-shot neural machine translation system BIBREF7 shares an encoder and decoder across all language pairs. In order to facilitate this multi-way translation, they prepend an artificial token to the beginning of each source sentence at both training and translation time. The token identifies what language the sentence should be translated to. This approach has three benefits: it is far more efficient than building a separate model for each language pair; it allows for translation between languages that share no parallel data; and it improves results on low-resource languages by allowing them to implicitly share parameters with high-resource languages. Our g2p system is inspired by this approach, although it differs in that there is only one target “language”, IPA, and the artificial tokens identify the language of the source instead of the language of the target.
Question: what evaluation metrics were used?
Paper: Massively Multilingual Neural Grapheme-to-Phoneme Conversion
g2p is the problem of converting the orthographic representation of a word into a phonemic representation. A phoneme is an abstract unit of sound which may have different realizations in different contexts. For example, the English phoneme has two phonetic realizations (or allophones):
Question: what datasets did they use?
Paper: Abstractive Summarization for Low Resource Data using Domain Transfer and Data Synthesis
Recently, with the emergence of neural seq2seq models, abstractive summarization methods have seen great performance strides BIBREF0, BIBREF1, BIBREF2. However, complex neural summarization models with thousands of parameters usually require a large amount of training data. In fact, much of the neural summarization work has been trained and tested in news domains where numerous large datasets exist. For example, the CNN/DailyMail (CNN/DM) BIBREF3, BIBREF4 and New York Times (NYT) datasets are in the magnitude of 300k and 700k documents, respectively. In contrast, in other domains such as student reflections, summarization datasets are only in the magnitude of tens or hundreds of documents (e.g., BIBREF5). We hypothesize that training complex neural abstractive summarization models in such domains will not yield good performing models, and we will indeed later show that this is the case for student reflections.
Question: What is the interannotator agreement for the human evaluation?