z-uo/qasper-squad huggingface dataset:
Paper: Arabic Diacritic Recovery Using a Feature-Rich biLSTM Model
Automatic diacritics restoration has been investigated for many different language such as European languages (e.g. Romanian BIBREF8, BIBREF9, French BIBREF10, and Croatian BIBREF11), African languages (e.g. Yorba BIBREF12), Southeast Asian languages (e.g. Vietnamese BIBREF13), Semitic language (e.g. Arabic and Hebrew BIBREF14), and many others BIBREF15. For many languages, diacritic (or accent restoration) is limited to a handful of letters. However, for Semitic languages, diacritic recovery extends to most letters. Many general approaches have been explored for this problem including linguistically motivated rule-based approaches, machine learning approaches, such as Hidden Markov Models (HMM) BIBREF14 and Conditional Random Fields (CRF) BIBREF16, and lately deep learning approaches such as Arabic BIBREF17, BIBREF18, BIBREF19, Slovak BIBREF20, and Yorba BIBREF12. Aside from rule-based approaches BIBREF21, different methods were used to recover diacritics in Arabic text. Using a hidden Markov model (HMM) BIBREF14, BIBREF22 with an input character sequence, the model attempts to find the best state sequence given previous observations. BIBREF14 reported a 14% word error rate (WER) while BIBREF22 achieved a 4.1% diacritic error rate (DER) on the Quran (CA). BIBREF23 combined both morphological, acoustic, and contextual features to build a diacritizer trained on FBIS and LDC CallHome ECA collections. They reported a 9% (DER) without CE, and 28% DER with CE. BIBREF24 employed a cascade of a finite state transducers. The cascade stacked a word language model (LM), a charachter LM, and a morphological model. The model achieved an accuracy of 7.33% WER without CE and and 23.61% WER with CE. BIBREF25 employed a maximum entropy model for sequence classification. The system was trained on the LDC’s Arabic Treebank (ATB) and evaluated on a 600 articles from An-Nahar Newspaper (340K words) and achieved 5.5% DER and 18% WER on words without CE.
Question: what are the previous state of the art?
Paper: Arabic Diacritic Recovery Using a Feature-Rich biLSTM Model
For MSA, we acquired the diacritized corpus that was used to train the RDI BIBREF7 diacritizer and the Farasa diacritizer BIBREF31. The corpus contains 9.7M tokens with approximately 194K unique surface forms (excluding numbers and punctuation marks). The corpus covers multiple genres such as politics and sports and is a mix of MSA and CA. This corpus is considerably larger than the Arabic Treebank BIBREF35 and is more consistent in its diacritization. For testing, we used the freely available WikiNews test set BIBREF31, which is composed of 70 MSA WikiNews articles (18,300 tokens) and evenly covers a variety of genres including politics, economics, health, science and technology, sports, arts and culture.
Question: what surface-level features are used?
Paper: Arabic Diacritic Recovery Using a Feature-Rich biLSTM Model
Arabic words are typically derived from a limited set of roots by fitting them into so-called stem-templates (producing stems) and may accept a variety of prefixes and suffixes such as prepositions, determiners, and pronouns (producing words). Word stems specify the lexical selection and are typically unaffected by the attached affixes. We used 4 feature types, namely:
Question: what linguistics features are used?
Answer: POS, gender/number and stem POS
Paper: MCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledge
Ambiguity and implicitness are inherent properties of natural language that cause challenges for computational models of language understanding. In everyday communication, people assume a shared common ground which forms a basis for efficiently resolving ambiguities and for inferring implicit information. Thus, recoverable information is often left unmentioned or underspecified. Such information may include encyclopedic and commonsense knowledge. This work focuses on commonsense knowledge about everyday activities, so-called scripts.
Question: what dataset statistics are provided?
Answer: More than 2,100 texts were paired with 15 questions each, resulting in a total number of approx. 32,000 annotated questions. 13% of the questions are not answerable.  Out of the answerable questions, 10,160 could be answered from the text directly (text-based) and 3,914 questions required the use of commonsense knowledge (script-based).  The final dataset comprises 13,939 questions, 3,827 of which require commonsense knowledge (i.e. 27.4%).
Distribution of category labels, number of answerable-not answerable questions, number of text-based and script-based questions, average text, question, and answer length, number of questions per text
Paper: MCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledge
Machine comprehension datasets consist of three main components: texts, questions and answers. In this section, we describe our data collection for these 3 components. We first describe a series of pilot studies that we conducted in order to collect commonsense inference questions (Section SECREF4 ). In Section SECREF5 , we discuss the resulting data collection of questions, texts and answers via crowdsourcing on Amazon Mechanical Turk (henceforth MTurk). Section SECREF17 gives information about some necessary postprocessing steps and the dataset validation. Lastly, Section SECREF19 gives statistics about the final dataset.
Question: what is the size of their dataset?
Paper: MCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledge
As a starting point for our pilots, we made use of texts from the InScript corpus BIBREF10 , which provides stories centered around everyday situations (see Section SECREF7 ). We conducted three different pilot studies to determine the best way of collecting questions that require inference over commonsense knowledge:
Question: what crowdsourcing platform was used?
Paper: MCScript: A Novel Dataset for Assessing Machine Comprehension Using Script Knowledge
As mentioned in the previous section, we decided to base the question collection on script scenarios rather than specific texts. As a starting point for our data collection, we use scenarios from three script data collections BIBREF3 , BIBREF11 , BIBREF12 . Together, these resources contain more than 200 scenarios. To make sure that scenarios have different complexity and content, we selected 80 of them and came up with 20 new scenarios. Together with the 10 scenarios from InScript, we end up with a total of 110 scenarios.
Question: how was the data collected?
Answer: The data was collected using 3 components: describe a series of pilot studies that were conducted to collect commonsense inference questions, then discuss the resulting data collection of questions, texts and answers via crowdsourcing on Amazon Mechanical Turk and gives information about some necessary postprocessing steps and the dataset validation.
Paper: Neural Architectures for Fine-Grained Propaganda Detection in News
In the age of information dissemination without quality control, it has enabled malicious users to spread misinformation via social media and aim individual users with propaganda campaigns to achieve political and financial gains as well as advance a specific agenda. Often disinformation is complied in the two major forms: fake news and propaganda, where they differ in the sense that the propaganda is possibly built upon true information (e.g., biased, loaded language, repetition, etc.).
Question: What is best performing model among author's submissions, what performance it had?
Answer: For SLC task, the "ltuorp" team  has the best performing  model (0.6323/0.6028/0.6649 for F1/P/R  respectively) and for FLC task the "newspeak" team  has the best performing  model (0.2488/0.2863/0.2201 for F1/P/R respectively).
Paper: Neural Architectures for Fine-Grained Propaganda Detection in News
Some of the propaganda techniques BIBREF3 involve word and phrases that express strong emotional implications, exaggeration, minimization, doubt, national feeling, labeling , stereotyping, etc. This inspires us in extracting different features (Table TABREF1) including the complexity of text, sentiment, emotion, lexical (POS, NER, etc.), layout, etc. To further investigate, we use topical features (e.g., document-topic proportion) BIBREF4, BIBREF5, BIBREF6 at sentence and document levels in order to determine irrelevant themes, if introduced to the issue being discussed (e.g., Red Herring).
Question: What extracted features were most influencial on performance?
Answer: Linguistic
Paper: Neural Architectures for Fine-Grained Propaganda Detection in News
Figure FIGREF2 (left) describes the three components of our system for SLC task: features, classifiers and ensemble. The arrows from features-to-classifier indicate that we investigate linguistic, layout and topical features in the two binary classifiers: LogisticRegression and CNN. For CNN, we follow the architecture of DBLP:conf/emnlp/Kim14 for sentence-level classification, initializing the word vectors by FastText or BERT. We concatenate features in the last hidden layer before classification.
Question: Did ensemble schemes help in boosting peformance, by how much?
Answer: The best ensemble topped the best single model by 0.029 in F1 score on dev (external).
They increased F1 Score by 0.029 in Sentence Level Classification, and by 0.044 in Fragment-Level classification
Paper: Neural Architectures for Fine-Grained Propaganda Detection in News
Figure FIGREF2 (right) describes our system for FLC task, where we design sequence taggers BIBREF9, BIBREF10 in three modes: (1) LSTM-CRF BIBREF11 with word embeddings ($w\_e$) and character embeddings $c\_e$, token-level features ($t\_f$) such as polarity, POS, NER, etc. (2) LSTM-CRF+Multi-grain that jointly performs FLC and SLC with FastTextWordEmb and BERTSentEmb, respectively. Here, we add binary sentence classification loss to sequence tagging weighted by a factor of $\alpha $. (3) LSTM-CRF+Multi-task that performs propagandistic span/fragment detection (PFD) and FLC (fragment detection + 19-way classification).
Question: Which basic neural architecture perform best by itself?
Answer: BERT
Paper: Neural Architectures for Fine-Grained Propaganda Detection in News
Data: While the SLC task is binary, the FLC consists of 18 propaganda techniques BIBREF3. We split (80-20%) the annotated corpus into 5-folds and 3-folds for SLC and FLC tasks, respectively. The development set of each the folds is represented by dev (internal); however, the un-annotated corpus used in leaderboard comparisons by dev (external). We remove empty and single token sentences after tokenization. Experimental Setup: We use PyTorch framework for the pre-trained BERT model (Bert-base-cased), fine-tuned for SLC task. In the multi-granularity loss, we set $\alpha = 0.1$ for sentence classification based on dev (internal, fold1) scores. We use BIO tagging scheme of NER in FLC task. For CNN, we follow DBLP:conf/emnlp/Kim14 with filter-sizes of [2, 3, 4, 5, 6], 128 filters and 16 batch-size. We compute binary-F1and macro-F1 BIBREF12 in SLC and FLC, respectively on dev (internal).
Question: What participating systems had better results than ones authors submitted?
Answer: For SLC task : Ituorp, ProperGander and YMJA  teams had better results.
For FLC task: newspeak and Antiganda teams had better results.
Paper: Neural Architectures for Fine-Grained Propaganda Detection in News
Table TABREF10 shows the scores on dev (internal and external) for SLC task. Observe that the pre-trained embeddings (FastText or BERT) outperform TF-IDF vector representation. In row r2, we apply logistic regression classifier with BERTSentEmb that leads to improved scores over FastTextSentEmb. Subsequently, we augment the sentence vector with additional features that improves F1 on dev (external), however not dev (internal). Next, we initialize CNN by FastTextWordEmb or BERTWordEmb and augment the last hidden layer (before classification) with BERTSentEmb and feature vectors, leading to gains in F1 for both the dev sets. Further, we fine-tune BERT and apply different thresholds in relaxing the decision boundary, where $\tau \ge 0.35$ is found optimal.
Question: What is specific to multi-granularity and multi-tasking neural arhiteture design?
Answer: An output layer for each task
Multi-tasking is addressed by neural sequence tagger based on LSTM-CRF and linguistic features, while multi-granularity is addressed by ensemble of LSTM-CRF and BERT.
Paper: When to reply? Context Sensitive Models to Predict Instructor Interventions in MOOC Forums
Massive Open Online Courses (MOOCs) have strived to bridge the social gap in higher education by bringing quality education from reputed universities to students at large. Such massive scaling through online classrooms, however, disrupt co-located, synchronous two-way communication between the students and the instructor.
Question: Do they report results only on English data?
Paper: When to reply? Context Sensitive Models to Predict Instructor Interventions in MOOC Forums
A thread INLINEFORM0 consists of a series of posts INLINEFORM1 through INLINEFORM2 where INLINEFORM3 is an instructor's post when INLINEFORM4 is intervened, if applicable. INLINEFORM5 is considered intervened if an instructor had posted at least once. The problem of predicting instructor intervention is cast as a binary classification problem. Intervened threads are predicted as 1 given while non-intervened threads are predicted as 0 given posts INLINEFORM6 through INLINEFORM7 .
Question: What aspects of discussion are relevant to instructor intervention, according to the attention mechanism?
Paper: When to reply? Context Sensitive Models to Predict Instructor Interventions in MOOC Forums
Context has been used and modelled in various ways for different problems in discussion forums. In a work on a closely related problem of forum thread retrieval BIBREF2 models context using inter-post discourse e.g., Question-Answer. BIBREF3 models the structural dependencies and relationships between forum posts using a conditional random field in their problem to infer the reply structure. Unlike BIBREF2 , BIBREF3 can be used to model any structural dependency and is, therefore, more general. In this paper, we seek to infer general dependencies between a reply and its previous context whereas BIBREF3 inference is limited to pairs of posts. More recently BIBREF4 proposed a context based model which factorises attention over threads of different lengths. Differently, we do not model length but the context before a post. However, our attention models cater to threads of all lengths.
Question: What was the previous state of the art for this task?
Paper: When to reply? Context Sensitive Models to Predict Instructor Interventions in MOOC Forums
The problem of predicting instructor intervention in MOOCs was proposed by BIBREF0 . Later BIBREF7 evaluated baseline models by BIBREF0 over a larger corpus and found the results to vary widely across MOOCs. Since then subsequent works have used similar diverse evaluations on the same prediction problem BIBREF1 , BIBREF8 . BIBREF1 proposed models with discourse features to enable better prediction over unseen MOOCs. BIBREF8 recently showed interventions on Coursera forums to be biased by the position at which a thread appears to an instructor viewing the forum interface and proposed methods for debiased prediction.
Question: What type of latent context is used to predict instructor intervention?
Paper: Distilling Translations with Visual Awareness
We build and test our MMT models on the Multi30K dataset BIBREF21 . Each image in Multi30K contains one English (EN) description taken from Flickr30K BIBREF22 and human translations into German (DE), French (FR) and Czech BIBREF23 , BIBREF24 , BIBREF25 . The dataset contains 29,000 instances for training, 1,014 for development, and 1,000 for test. We only experiment with German and French, which are languages for which we have in-house expertise for the type of analysis we present. In addition to the official Multi30K test set (test 2016), we also use the test set from the latest WMT evaluation competition, test 2018 BIBREF25 .
Question: Do they report results only on English dataset?
Paper: Distilling Translations with Visual Awareness
In addition to using the Multi30K dataset as is (standard setup), we probe the ability of our models to address the three linguistic phenomena where additional context has been proved important (Section ): ambiguities, gender-neutral words and noisy input. In a controlled experiment where we aim to remove the influence of frequency biases, we degrade the source sentences by masking words through three strategies to replace words by a placeholder: random source words, ambiguous source words and gender unmarked source words. The procedure is applied to the train, validation and test sets. For the resulting dataset generated for each setting, we compare models having access to text-only context versus additional text and multimodal contexts. We seek to get insights into the contribution of each type of context to address each type of degradation.
Question: What dataset does this approach achieve state of the art results on?
Paper: From English To Foreign Languages: Transferring Pre-trained Language Models
Pre-trained models BIBREF0, BIBREF1 have received much of attention recently thanks to their impressive results in many down stream NLP tasks. Additionally, multilingual pre-trained models enable many NLP applications for other languages via zero-short cross-lingual transfer. Zero-shot cross-lingual transfer has shown promising results for rapidly building applications for low resource languages. BIBREF2 show the potential of multilingual-BERT BIBREF0 in zero-shot transfer for a large number of languages from different language families on five NLP tasks, namely, natural language inference, document classification, named entity recognition, part-of-speech tagging, and dependency parsing.
Question: How much training data from the non-English language is used by the system?
Answer: No data. Pretrained model is used.
Paper: From English To Foreign Languages: Transferring Pre-trained Language Models
We first provide some background of pre-trained language models. Let $_e$ be English word-embeddings and $\Psi ()$ be the Transformer BIBREF10 encoder with parameters $$. Let $_{w_i}$ denote the embedding of word $w_i$ (i.e., $_{w_i} = _e[w_1]$). We omit positional embeddings and bias for clarity. A pre-trained LM typically performs the following computations: (i) transform a sequence of input tokens to contextualized representations $[_{w_1},\dots ,_{w_n}] = \Psi (_{w_1}, \dots , _{w_n}; )$, and (ii) predict an output word $y_i$ at $i^{\text{th}}$ position $p(y_i | _{w_i}) \propto \exp (_{w_i}^\top _{y_i})$.
Question: Is the system tested on low-resource languages?
Paper: From English To Foreign Languages: Transferring Pre-trained Language Models
Our approach to learn the initial foreign word embeddings $_f \in ^{|V_f| \times d}$ is based on the idea of mapping the trained English word embeddings $_e \in ^{|V_e| \times d}$ to $_f$ such that if a foreign word and an English word are similar in meaning then their embeddings are similar. Borrowing the idea of universal lexical sharing from BIBREF11, we represent each foreign word embedding $_f[i] \in ^d$ as a linear combination of English word embeddings $_e[j] \in ^d$
Question: What languages are the model transferred to?
Paper: From English To Foreign Languages: Transferring Pre-trained Language Models
Given an English-foreign parallel corpus, we can estimate word translation probability $p(e\,|\,f)$ for any (English-foreign) pair $(e, f)$ using popular word-alignment BIBREF12 toolkits such as fast-align BIBREF13. We then assign:
Question: How is the model transferred to other languages?
Answer: Build a bilingual language model,   learn the target language specific parameters starting from a pretrained English LM , fine-tune both English and target model to obtain the bilingual LM.
Paper: From English To Foreign Languages: Transferring Pre-trained Language Models
For low resource languages, parallel data may not be available. In this case, we rely only on monolingual data (e.g., Wikipedias). We estimate word translation probabilities from word embeddings of the two languages. Word vectors of these languages can be learned using fastText BIBREF14 and then are aligned into a shared space with English BIBREF15, BIBREF16. Unlike learning contextualized representations, learning word vectors is fast and computationally cheap. Given the aligned vectors $\bar{}_f$ of foreign and $\bar{}_e$ of English, we calculate the word translation matrix $\in ^{|V_f|\times |V_e|}$ as
Question: What metrics are used for evaluation?
Paper: From English To Foreign Languages: Transferring Pre-trained Language Models
After initializing foreign word-embeddings, we replace English word-embeddings in the English pre-trained LM with foreign word-embeddings to obtain the foreign LM. We then fine-tune only foreign word-embeddings on monolingual data. The training objective is the same as the training objective of the English pre-trained LM (i.e., masked LM for BERT). Since the trained encoder $\Psi ()$ is good at capturing association, the purpose of this step is to further optimize target embeddings such that the target LM can utilized the trained encoder for association task. For example, if the words Albert Camus presented in a French input sequence, the self-attention in the encoder more likely attends to words absurde and existentialisme once their embeddings are tuned.
Question: What datasets are used for evaluation?
Paper: Embedding Geographic Locations for Modelling the Natural Environment using Flickr Tags and Structured Data
Users of photo-sharing websites such as Flickr often provide short textual descriptions in the form of tags to help others find the images. With the availability of GPS systems in current electronic devices such as smartphones, latitude and longitude coordinates are nowadays commonly made available as well. The tags associated with such georeferenced photos often describe the location where these photos were taken, and Flickr can thus be regarded as a source of environmental information. The use of Flickr for modelling urban environments has already received considerable attention. For instance, various approaches have been proposed for modelling urban regions BIBREF0 , and for identifying points-of-interest BIBREF1 and itineraries BIBREF2 , BIBREF3 . However, the usefulness of Flickr for characterizing the natural environment, which is the focus of this paper, is less well-understood.
Question: what are the existing approaches?
Paper: Embedding Geographic Locations for Modelling the Natural Environment using Flickr Tags and Structured Data
The use of low-dimensional vector space embeddings for representing objects has already proven effective in a large number of applications, including natural language processing (NLP), image processing, and pattern recognition. In the context of NLP, the most prominent example is that of word embeddings, which represent word meaning using vectors of typically around 300 dimensions. A large number of different methods for learning such word embeddings have already been proposed, including Skip-gram and the Continuous Bag-of-Words (CBOW) model BIBREF8 , GloVe BIBREF9 , and fastText BIBREF14 . They have been applied effectively in many downstream NLP tasks such as sentiment analysis BIBREF15 , part of speech tagging BIBREF16 , BIBREF17 , and text classification BIBREF18 , BIBREF19 . The model we consider in this paper builds on GloVe, which was designed to capture linear regularities of word-word co-occurrence. In GloVe, there are two word vectors INLINEFORM0 and INLINEFORM1 for each word in the vocabulary, which are learned by minimizing the following objective: DISPLAYFORM0 
Question: what dataset is used in this paper?