z-uo/qasper-squad huggingface dataset:
Paper: A Simple Approach to Multilingual Polarity Classification in Twitter
Sentiment analysis is a crucial task in opinion mining field where the goal is to extract opinions, emotions, or attitudes to different entities (person, objects, news, among others). Clearly, this task is of interest for all languages; however, there exists a significant gap between English state-of-the-art methods and other languages. It is expected that some researchers decided to test the straightforward approach which consists in, first, translating the messages to English, and, then, use a high performing English sentiment classifier (for instance, see BIBREF0 and BIBREF1 ) instead of creating a sentiment classifier optimized for a given language. However, the advantages of a properly tuned sentiment classifier have been studied for different languages (for instance, see BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 ).
Question: How many texts/datapoints are in the SemEval, TASS and SENTIPOLC datasets?
Answer: Total number of annotated data:
Semeval'15: 10712
Semeval'16: 28632
Tass'15: 69000
Sentipol'14: 6428
Paper: A Simple Approach to Multilingual Polarity Classification in Twitter
We propose a method for multilingual polarity classification that can serve as a baseline as well as a framework to build more complex sentiment analysis systems due to its simplicity and availability as an open source software. As we mentioned, this baseline algorithm for multilingual Sentiment Analysis (B4MSA) was designed with the purpose of being multilingual and easy to implement. B4MSA is not a naïve baseline which is experimentally proved by evaluating it on several international competitions.
Question: In which languages did the approach outperform the reported results?
Answer: Arabic, German, Portuguese, Russian, Swedish
Paper: A Simple Approach to Multilingual Polarity Classification in Twitter
We defined cross-language features as a set of features that could be applied in most similar languages, not only related language families such as Germanic languages (English, German, etc.), Romance languages (Spanish, Italian, etc.), among others; but also similar surface features such as punctuation, diacritics, symbol duplication, case sensitivity, etc. Later, the combination of these features will be explored to find the best configuration for a given classifier.
Question: What eight language are reported on?
Paper: A Simple Approach to Multilingual Polarity Classification in Twitter
The following features are language dependent because they use specific information from the language concerned. Usually, the use of stopwords, stemming and negations are traditionally used in Sentiment Analysis. The users of this approach could add other features such as part of speech, affective lexicons, etc. to improve the performance BIBREF13 .
Question: What are the components of the multilingual framework?
Paper: Markov Chain Monte-Carlo Phylogenetic Inference Construction in Computational Historical Linguistics
Bayesian inference of phylogeny has great impact on evolutionary biology. It is believed that all the species are related through a history of a common descent BIBREF0, that is to say, the reason we have various wildlife, including human beings, is because of evolution. We can show the process of evolution and solve the problems, like what is the phylogeny of life, by showing a phylogenetic tree (see Figure FIGREF1).
Question: Is the proposed method compared to previous methods?
Paper: Markov Chain Monte-Carlo Phylogenetic Inference Construction in Computational Historical Linguistics
A great number of algorithms and mechanisms to antomatic cognate detection which could be applied in historical linguistics have been used and tested if they are working by many linguists and computer scientists BIBREF2, BIBREF4, BIBREF5, BIBREF6, BIBREF7. In detail, many of these works are very similar to each other, which consist of two main stages. For the first stage, they first extract the words with same meaning from the wordlists of different languages, either same or different language families, and compare them and use the distance calculation matrix to compute how similar they are. Regarding the second stage, a flat cluster algorithm or a network partitioning algorithm is used to partition all words into cognate sets, and also take the information in the matrix of word pairs as basis BIBREF4, BIBREF5. However, the methods in which those researchers use to compare the word pairs are totally different in that people could use different methods to pre-process their language datasets, or even use different algorithms to finish the comparison and clustering task. For example, intuitively, people will start the automated word comparison by computing the distance between the words, such as word embedding in NLP, GloVe BIBREF8, which computes the semantic similarities between the two words. In computational historical linguistics, phonetic segments are used to calculate how close the two words are instead, because the semantics of a single word is not changing as easy as phonetics is. The problem is since the program involves the data pre-processing, then the whole dataset would be traverse twice and the computation would be a big problem when the dataset is about to be over 100 languages. Consequently, people began to think about a faster method to help.
Question: What metrics are used to evaluate results?
Paper: Phonetic Temporal Neural Model for Language Identification
Language identification (LID) lends itself to a wide range of applications, such as mixed-lingual (code-switching) speech recognition. Humans use many cues to discriminate languages, and better accuracy can be achieved with the use of more cues. Various LID approaches have been developed, based on different types of cues.
Question: Which is the baseline model?
Answer: The three baseline models are the i-vector model, a standard RNN LID system and a multi-task RNN LID system.
Paper: Phonetic Temporal Neural Model for Language Identification
There are more than 5000 languages in the world, and each language has distinct properties at different levels, from acoustic to semantics BIBREF0 , BIBREF1 , BIBREF2 . A number of studies have investigated how humans use these properties as cues to distinguish between languages BIBREF3 . For example, Muthusamy BIBREF4 found that familiarity with a language is an important factor affecting LID accuracy, and that longer speech samples are easier to identify. Moreover, people can easily tell what cues they use for identification, including phonemic inventory, word usage, and prosody. More thorough investigations were conducted by others by modifying speech samples to promote one or several factors. For example, Mori et al. BIBREF5 found that people are able to identify Japanese and English fairly reliably even when phone information is reduced. They argued that other non-linguistic cues such as intensity and pitch were used to decide the language. Navratil BIBREF6 evaluated the importance of various types of knowledge, including lexical, phonotactic and prosodic, by asking humans to identify five languages, Chinese, English, French, German and Japanese. Subjects were presented with unaltered speech samples, samples with randomly altered syllables, and samples with the vocal-tract information removed to leave only the F0 and amplitude. Navratil found that the speech samples with random syllables are more difficult to identify compared to the original samples (73.9% vs 96%), and removing vocal-tract information leads to significant performance reduction (73.9% vs 49.4%). This means that with this 5-language LID task, the lexical and phonotactic information is important for human decision making.
Question: How big is the Babel database?
Paper: Phonetic Temporal Neural Model for Language Identification
Based on the different types of cues, multiple LID approaches have been proposed. Early work generally focused on feature-level cues. Feature-based methods use strong statistical models built on raw acoustic features to make the LID decision. For instance, Cimarusti used LPC features BIBREF7 , and Foil et al. BIBREF8 investigated formant features. Dynamic features that involve temporal information were also demonstrated to be effective BIBREF9 . The statistical models used include Gaussian mixture models (GMMs) BIBREF10 , BIBREF11 , hidden Markov models (HMMs) BIBREF12 , BIBREF13 , neural networks (NNs) BIBREF14 , BIBREF15 , and support vector machines (SVMs) BIBREF16 . More recently, a low-rank GMM model known as the i-vector model was proposed and achieved significant success BIBREF17 , BIBREF18 . This model constrains the mean vectors of the GMM components in a low-dimensional space to improve the statistical strength for model training, and uses a task-oriented discriminative model (e.g., linear discriminative analysis, LDA) to improve the decision quality at run-time, leading to improved LID performance. Due to the short-time property of the features, most feature-based methods model the distributional characters rather than the temporal characters of speech signals.
Question: What is the main contribution of the paper? 
Answer: Proposing an improved RNN model, the phonetic temporal neural LID approach, based on phonetic features that results in better performance
Paper: On Evaluating the Generalization of LSTM Models in Formal Languages
Recurrent Neural Networks (RNNs) are powerful machine learning models that can capture and exploit sequential data. They have become standard in important natural language processing tasks such as machine translation BIBREF0 , BIBREF1 and speech recognition BIBREF2 . Despite the ubiquity of various RNN architectures in natural language processing, there still lies an unanswered fundamental question: What classes of languages can, empirically or theoretically, be learned by neural networks? This question has drawn much attention in the study of formal languages, with previous results on both the theoretical BIBREF3 , BIBREF4 and empirical capabilities of RNNs, showing that different RNN architectures can learn certain regular BIBREF5 , BIBREF6 , context-free BIBREF7 , BIBREF8 , and context-sensitive languages BIBREF9 .
Question: What training settings did they try?
Paper: On Evaluating the Generalization of LSTM Models in Formal Languages
It has been shown that RNNs with a finite number of states can process regular languages by acting like a finite-state automaton using different units in their hidden layers BIBREF5 , BIBREF6 . RNNs, however, are not limited to recognizing only regular languages. BIBREF3 and BIBREF4 showed that first-order RNNs (with rational state weights and infinite numeric precision) can simulate a pushdown automaton with two-stacks, thereby demonstrating that RNNs are Turing-complete. In theory, RNNs with infinite numeric precision are capable of expressing recursively enumerable languages. Yet, in practice, modern machine architectures do not contain computational structures that support infinite numeric precision. Thus, the computational power of RNNs with finite precision may not necessarily be the same as that of RNNs with infinite precision.
Question: How do they get the formal languages?
Answer: These are well-known formal languages some of which was used in the literature to evaluate the learning capabilities of RNNs.
Paper: On Evaluating the Generalization of LSTM Models in Formal Languages
Following the traditional approach adopted by BIBREF7 , BIBREF12 , BIBREF9 and many other studies, we train our neural network as follows. At each time step, we present one input character to our model and then ask it to predict the set of next possible characters, based on the current character and the prior hidden states. Given a vocabulary $\mathcal {V}^{(i)}$ of size $d$ , we use a one-hot representation to encode the input values; therefore, all the input vectors are $d$ -dimensional binary vectors. The output values are $(d+1)$ -dimensional though, since they may further contain the termination symbol $\dashv $ , in addition to the symbols in $\mathcal {V}^{(i)}$ . The output values are not always one-hot encoded, because there can be multiple possibilities for the next character in the sequence, therefore we instead use a $k$ -hot representation to encode the output values. Our objective is to minimize the mean-squared error (MSE) of the sequence predictions. During testing, we use an output threshold criterion of $0.5$ for the sigmoid output layer to indicate which characters were predicted by the model. We then turn this prediction task into a classification task by accepting a sample if our model predicts all of its output values correctly and rejecting it otherwise.
Question: Are the unobserved samples from the same distribution as the training data?
Paper: Adapt or Get Left Behind: Domain Adaptation through BERT Language Model Finetuning for Aspect-Target Sentiment Classification
Sentiment Analysis (SA) is an active field of research in Natural Language Processing and deals with opinions in text. A typical application of classical SA in an industrial setting would be to classify a document like a product review into positve, negative or neutral sentiment polarity.
Question: By how much does their model outperform the baseline in the cross-domain evaluation?
Paper: Adapt or Get Left Behind: Domain Adaptation through BERT Language Model Finetuning for Aspect-Target Sentiment Classification
We separate our discussion of related work into two areas: First, neural methods applied to ATSC that have improved performance solely by model architecture improvements. Secondly, methods that additionally aim to transfer knowledge from semantically related tasks or domains.
Question: What are the performance results?
Paper: Bi-Directional Lattice Recurrent Neural Networks for Confidence Estimation
 Recent years have seen an increased usage of spoken language technology in applications ranging from speech transcription BIBREF0 to personal assistants BIBREF1 . The quality of these applications heavily depends on the accuracy of the underlying automatic speech recognition (ASR) system yielding 1-best hypotheses and how well ASR errors are mitigated. The standard approach to ASR error mitigation is confidence scores BIBREF2 , BIBREF3 . A low confidence can give a signal to downstream applications about the high uncertainty of the ASR in its prediction and measures can be taken to mitigate the risk of making a wrong decision. However, confidence scores can also be used in upstream applications such as speaker adaptation BIBREF4 and semi-supervised training BIBREF5 , BIBREF6 to reflect uncertainty among multiple possible alternative hypotheses. Downstream applications, such as machine translation and information retrieval, could similarly benefit from using multiple hypotheses.
Question: What is a confusion network or lattice?
Answer: graph-like structures where arcs connect nodes representing multiple hypothesized words, thus allowing multiple incoming arcs unlike 1-best sequences
Paper: Representation Learning for Discovering Phonemic Tone Contours
Tonal languages use pitch to distinguish different words, for example, yi in Mandarin may mean `one', `to move', `already', or `art', depending on the pitch contour. Of over 6000 languages in the world, it is estimated that as many as 60-70% are tonal BIBREF0, BIBREF1. A few of these are national languages (e.g., Mandarin Chinese, Vietnamese, and Thai), but many tonal languages have a small number of speakers and are scarcely documented. There is a limited availability of trained linguists to perform language documentation before these languages become extinct, hence the need for better tools to assist linguists in these tasks.
Question: What dataset is used for training?
Paper: Representation Learning for Discovering Phonemic Tone Contours
Mandarin Chinese (1.1 billion speakers) and Cantonese (74 million speakers) are two tonal languages in the Sinitic family BIBREF0. Mandarin has four lexical tones: high (55), rising (25), low-dipping (214), and falling (51). The third tone sometimes undergoes sandhi, addressed in section SECREF3. We exclude a fifth, neutral tone, which can only occur in word-final positions and has no fixed pitch.
Question: How close do clusters match to ground truth tone categories?
Answer: NMI between cluster assignments and ground truth tones for all sylables is:
Mandarin: 0.641
Cantonese: 0.464
Paper: Robust Multilingual Named Entity Recognition with Shallow Semi-Supervised Features
A named entity can be mentioned using a great variety of surface forms (Barack Obama, President Obama, Mr. Obama, B. Obama, etc.) and the same surface form can refer to a variety of named entities. For example, according to the English Wikipedia, the form `Europe' can ambiguously be used to refer to 18 different entities, including the continent, the European Union, various Greek mythological entities, a rock band, some music albums, a magazine, a short story, etc. Furthermore, it is possible to refer to a named entity by means of anaphoric pronouns and co-referent expressions such as `he', `her', `their', `I', `the 35 year old', etc. Therefore, in order to provide an adequate and comprehensive account of named entities in text it is necessary to recognize the mention of a named entity and to classify it by a pre-defined type (e.g, person, location, organization). Named Entity Recognition and Classification (NERC) is usually a required step to perform Named Entity Disambiguation (NED), namely to link `Europe' to the right Wikipedia article, and to resolve every form of mentioning or co-referring to the same entity.
Question: what are the evaluation metrics?
Answer: Precision, Recall, F1
Paper: Robust Multilingual Named Entity Recognition with Shallow Semi-Supervised Features
The main contributions of this paper are the following: First, we show how to easily develop robust NERC systems across datasets and languages with minimal human intervention, even for languages with declension and/or complex morphology. Second, we empirically show how to effectively use various types of simple word representation features thereby providing a clear methodology for choosing and combining them. Third, we demonstrate that our system still obtains very competitive results even when the supervised data is reduced by half (even less in some cases), alleviating the dependency on costly hand annotated data. These three main contributions are based on:
Question: which datasets were used in evaluation?
Answer: CoNLL 2003, GermEval 2014, CoNLL 2002, Egunkaria, MUC7, Wikigold, MEANTIME, SONAR-1, Ancora 2.0
Paper: Robust Multilingual Named Entity Recognition with Shallow Semi-Supervised Features
The Named Entity Recognition and Classification (NERC) task was first defined for the Sixth Message Understanding Conference (MUC 6) BIBREF39 . The MUC 6 tasks focused on Information Extraction (IE) from unstructured text and NERC was deemed to be an important IE sub-task with the aim of recognizing and classifying nominal mentions of persons, organizations and locations, and also numeric expressions of dates, money, percentage and time. In the following years, research on NERC increased as it was considered to be a crucial source of information for other Natural Language Processing tasks such as Question Answering (QA) and Textual Entailment (RTE) BIBREF39 . Furthermore, while MUC 6 was solely devoted to English as target language, the CoNLL shared tasks (2002 and 2003) boosted research on language independent NERC for 3 additional target languages: Dutch, German and Spanish BIBREF40 , BIBREF41 .
Question: what are the baselines?
Answer: Perceptron model using the local features.
Paper: Irony Detection in a Multilingual Context
Figurative language makes use of figures of speech to convey non-literal meaning BIBREF0, BIBREF1. It encompasses a variety of phenomena, including metaphor, humor, and irony. We focus here on irony and uses it as an umbrella term that covers satire, parody and sarcasm.
Question: What multilingual word representations are used?
Paper: Irony Detection in a Multilingual Context
Arabic dataset (Ar=$11,225$ tweets). Our starting point was the corpus built by BIBREF13 that we extended to different political issues and events related to the Middle East and Maghreb that hold during the years 2011 to 2018. Tweets were collected using a set of predefined keywords (which targeted specific political figures or events) and containing or not Arabic ironic hashtags (سخرية>#, مسخرة>#, تهكم>#, استهزاء>#) . The collection process resulted in a set of $6,809$ ironic tweets ($I$) vs. $15,509$ non ironic ($NI$) written using standard (formal) and different Arabic language varieties: Egypt, Gulf, Levantine, and Maghrebi dialects.
Question: Do the authors identify any cultural differences in irony use?
Paper: Irony Detection in a Multilingual Context
It is important to note that our aim is not to outperform state-of-the-art models in monolingual ID but to investigate which of the monolingual architectures (neural or feature-based) can achieve comparable results with existing systems. The result can show which kind of features works better in the monolingual settings and can be employed to detect irony in a multilingual setting. In addition, it can show us to what extend ID is language dependent by comparing their results to multilingual results. Two models have been built, as explained below. Prior to learning, basic preprocessing steps were performed for each language (e.g., removing foreign characters, ironic hashtags, mentions, and URLs).
Question: What neural architectures are used?
Paper: Irony Detection in a Multilingual Context
We use the previous CNN architecture with bilingual embedding and the RF model with surface features (e.g., use of personal pronoun, presence of interjections, emoticon or specific punctuation) to verify which pair of the three languages: (a) has similar ironic pragmatic devices, and (b) uses similar text-based pattern in the narrative of the ironic tweets. As continuous word embedding spaces exhibit similar structures across (even distant) languages BIBREF35, we use a multilingual word representation which aims to learn a linear mapping from a source to a target embedding space. Many methods have been proposed to learn this mapping such as parallel data supervision and bilingual dictionaries BIBREF35 or unsupervised methods relying on monolingual corpora BIBREF36, BIBREF37, BIBREF38. For our experiments, we use Conneau et al 's approach as it showed superior results with respect to the literature BIBREF36. We perform several experiments by training on one language ($lang_1$) and testing on another one ($lang_2$) (henceforth $lang_1\rightarrow lang_2$). We get 6 configurations, plus two others to evaluate how irony devices are expressed cross-culturally, i.e. in European vs. non European languages. In each experiment, we took 20% from the training to validate the model before the testing process. Table TABREF11 presents the results.
Question: What text-based features are used?