z-uo/qasper-squad huggingface dataset:
Paper: QA4IE: A Question Answering based Framework for Information Extraction
To overcome the above weaknesses of existing IE systems, we propose a novel IE framework named QA4IE to perform document level general IE with the help of state-of-the-art approaches in Question Answering (QA) and Machine Reading Comprehension (MRC) area.
Question: Was this benchmark automatically created from an existing dataset?
Paper: A Resource for Studying Chatino Verbal Morphology
The recent years have seen unprecedented forward steps for Natural Language Processing (NLP) over almost every NLP subtask, relying on the advent of large data collections that can be leveraged to train deep neural networks. However, this progress has solely been observed in languages with significant data resources, while low-resource languages are left behind.
Question: How does morphological analysis differ from morphological inflection?
Paper: A Resource for Studying Chatino Verbal Morphology
Chatino is a group of languages spoken in Oaxaca, Mexico. Together with the Zapotec language group, the Chatino languages form the Zapotecan branch of the Otomanguean language family. There are three main Chatino languages: Zenzontepec Chatino (ZEN, ISO 639-2 code czn), Tataltepec Chatino (TAT, cta), and Eastern Chatino (ISO 639-2 ctp, cya, ctz, and cly) (E.Cruz 2011 and Campbell 2011). San Juan Quiahije Chatino (SJQ), the language of the focus of this study, belongs to Eastern Chatino, and is used by about 3000 speakers.
Question: What was the criterion used for selecting the lemmata?
Paper: A Resource for Studying Chatino Verbal Morphology
Eastern Chatino languages , including SJQ Chatino, are intensively tonal BIBREF2, BIBREF3. Tones mark both lexical and grammatical distinctions in Eastern Chatino languages.
Question: What are the architectures used for the three tasks?
Paper: A Resource for Studying Chatino Verbal Morphology
SJQ Chatino verb inflection distinguishes four aspect/mood categories: completive (`I did'), progressive (`I am doing'), habitual (`I habitually do') and potential (`I might do'). In each of these categories, verbs inflect for three persons (first, second, third) and two numbers (singular, plural) and distinguish inclusive and exclusive categories of the first person plural (`we including you' vs `we excluding you'). Verbs can be classified into dozens of different conjugation classes. Each conjugation class involves its own tone pattern; each tone pattern is based on a series of three person/number (PN) triplets. A PN triplet [X, Y, Z] consists of three tones: tone X is employed in the third person singular as well as in all plural forms; tone Y is employed in the second person singular, and tone Z, in the third person singular. Thus, a verb's membership in a particular conjugation class entails the assignment of one tone triplet to completive forms, another to progressive forms, and a third to habitual and potential forms. The paradigm of the verb lyu1 `fall' in Table illustrates: the conjugation class to which this verb belongs entails the assignment of the triplet [1, 42, 20] to the completive, [1, 42, 32] to the progressive, and [20, 42, 32] to the habitual and potential. Verbs in other conjugation classes exhibit other triplet series.
Question: Which language family does Chatino belong to?
Paper: A Resource for Studying Chatino Verbal Morphology
We provide a hand-curated collection of complete inflection tables for 198 lemmata. The morphological tags follow the guidelines of the UniMorph schema BIBREF6, BIBREF7, in order to allow for the potential of cross-lingual transfer learning, and they are tagged with respect to:
Question: What system is used as baseline?
Paper: A Resource for Studying Chatino Verbal Morphology
Inflectional realization defines the inflected forms of a lexeme/lemma. As a computational task, often referred to as simply “morphological inflection," inflectional realization is framed as a mapping from the pairing of a lemma with a set of morphological tags to the corresponding word form. For example, the inflectional realization of SJQ Chatino verb forms entails a mapping of the pairing of the lemma lyu1 `fall' with the tag-set 1;SG;PROG to the word form nlyon32.
Question: How was annotation done?
Paper: A Resource for Studying Chatino Verbal Morphology
Morphological analysis is the task of creating a morphosyntactic description for a given word. It can be framed in a context-agnostic manner (as in our case) or within a given context, as for instance for the SIGMORPHON 2019 second shared task BIBREF11. We trained an encoder-decoder model that receives the form as character-level input, encodes it with a BiLSTM encoder, and then an attention enhanced decoder BIBREF14 outputs the corresponding sequence of morphological tags, implemented in DyNet. The baseline results are shown in Table . The exact-match accuracy of 67% is lower than the average accuracy that context-aware systems can achieve, and it highlights the challenge that the complexity of the tonal system of SJQ Chatino can pose.
Question: How was the data collected?
Paper: N-GrAM: New Groningen Author-profiling Model
With the rise of social media, more and more people acquire some kind of on-line presence or persona, mostly made up of images and text. This means that these people can be considered authors, and thus that we can profile them as such. Profiling authors, that is, inferring personal characteristics from text, can reveal many things, such as their age, gender, personality traits, location, even though writers might not consciously choose to put indicators of those characteristics in the text. The uses for this are obvious, for cases like targeted advertising and other use cases, such as security, but it is also interesting from a linguistic standpoint.
Question: How do their results compare against other competitors in the PAN 2017 shared task on Author Profiling?
Answer: They achieved best result in the PAN 2017 shared task with accuracy for Variety prediction task 0.0013 more than the 2nd best baseline, accuracy for Gender prediction task 0.0029 more than 2nd best baseline and accuracy for Joint prediction task 0.0101 more than the 2nd best baseline
Paper: N-GrAM: New Groningen Author-profiling Model
After an extensive grid-search we submitted as our final run, a simple SVM system (using the scikit-learn LinearSVM implementation) that uses character 3- to 5-grams and word 1- to 2-grams with tf-idf weighting with sublinear term frequency scaling, where instead of the standard term frequency the following is used:
Question: On which task does do model do worst?
Answer: Gender prediction task
Paper: N-GrAM: New Groningen Author-profiling Model
The training dataset provided consist of 11400 sets of tweets, each set representing a single author. The target labels are evenly distributed across variety and gender. The labels for the gender classification task are `male' and `female'. Table TABREF4 shows the labels for the language variation task and also shows the data distribution across languages.
Question: On which task does do model do best?
Answer: Variety prediction task
Paper: Multi-modal Sentiment Analysis using Super Characters Method on Low-power CNN Accelerator Device
The need to classify sentiment based on the multi-modal input arises in many different problems in customer related marketing fields. Super Characters BIBREF0 is a two-step method for sentiment analysis. It first converts text into images; then feeds the images into CNN models to classify the sentiment. Sentiment classification performance on large text contents from customer online comments shows that the Super Character method is superior to other existing methods. The Super Characters method also shows that the pretrained models on a larger dataset help improve accuracy by finetuning the CNN model on a smaller dataset. Compared with from-scratch trained Super Characters model, the finetuned one improves the accuracy from 95.7% to 97.8% on the well-known Chinese dataset of Fudan Corpus. Squared English Word (SEW) BIBREF1 is an extension of the Super Characters method into Latin Languages. With the wide availability of low-power CNN accelerator chips BIBREF2 BIBREF3, Super Characters method has the great potential to be deployed in large scale by saving power and fast inference speed. In addition, it is easy to deploy as well. The recent work also extend its applications to chatbot BIBREF4, image captioning BIBREF5, and also tabular data machine learning BIBREF6.
Question: Is their implementation on CNN-DSA compared to GPU implementation in terms of power consumption, accuracy and speed?
Paper: Multi-modal Sentiment Analysis using Super Characters Method on Low-power CNN Accelerator Device
For multi-modal sentiment analysis, we can simply split the image into two parts. One for the text input, and the other for the tabular data. Such that both can be embedded into the Super Characters image. The CNN accelerator chip comes together with a Model Development Kit (MDK) for CNN model training, which feeds the two-dimensional Super Characters images into MDK and then obtain the fixed point model. Then, using the Software Development Kit (SDK) to load the model into the chip and send command to the CNN accelerator chip, such as to read an image, or to forward pass the image through the network to get the inference result. The advantage of using the CNN accelerator is low-power, it consumes only 300mw for an input of size 3x224x224 RGB image at the speed of 140fps. Compared with other models using GPU or FPGA, this solution implement the heavy-lifting DNN computations in the CNN accelerator chip, and the host computer is only responsible for memory read/write to generate the designed Super Character image. This has shown good result on system implementations for NLP applications BIBREF9.
Question: Does this implementation on CNN-DSA lead to diminishing of performance?
Paper: Multi-modal Sentiment Analysis using Super Characters Method on Low-power CNN Accelerator Device
The training data set has 12,860 samples with 16 columns. The first ten columns are attributes, including sentenceid, author, nchar, created_utc, score, subreddit, label, full_text, wordcount, and id. And the other six columns are labels for each of the tasks of Emotion_disclosure, Information_disclosure, Support, Emmotion_support, Information_support, and General_support. Each task is a binary classification problem based on the ten attributes. So there will be 60 models to be trained for a 10-fold validation. The test data set has 5000 samples with only the ten columns of attributes. The system run will give labels on these test samples based on the 10-fold training.
Question: How is Super Character method modified to handle tabular data also?
Paper: Nefnir: A high accuracy lemmatizer for Icelandic
In text mining and Natural Language Processing (NLP), a lemmatizer is a tool used to determine the basic form of a word (lemma). Lemmatization differs from stemming in the way this base form is determined. While stemmers chop off word endings to reach the common stem of words, lemmatizers take into account the morphology of the words in order to produce the common morphological base form, i.e., the form of the word found in a dictionary. This type of text normalization is an important step in pre-processing morphologically complex languages, like Icelandic, before conducting various tasks, such as machine translation, text mining and information retrieval.
Question: How are the substitution rules built?
Paper: Nefnir: A high accuracy lemmatizer for Icelandic
The most basic approach to lemmatization is a simple look-up in a lexicon. This method has the obvious drawback that words that are not in the lexicon cannot be processed. To solve this, word transformation rules have been used to analyze the surface form of the word (the token) in order to produce the base form. These rules can either be hand-crafted or learned automatically using machine learning. When hand-crafting the rules that are used to determine the lemmas, a thorough knowledge of the morphological features of the language is needed. This is a time-consuming task, further complicated in Icelandic by the extensive inflectional system BIBREF1 . An example of a hand-crafted lemmatizer is the morphological analyzer that is part of the Czech Dependency Treebank BIBREF3 .
Question: Which dataset do they use?
Paper: Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation
Since machine learning algorithms learn to model patterns present in training datasets, what they learn is affected by data quality. Analysis has found that model predictions directly reflect the biases found in training datasets, such as image classifiers learning to associate ethnicity with specific activities BIBREF1. Recent work in natural language processing has found similar biases, such as in word embeddings BIBREF2, BIBREF3, BIBREF4, object classification BIBREF5, natural language inference BIBREF6, and coreference resolution BIBREF7. Less work has focused on the biases present in dialogue utterances BIBREF8, BIBREF9, despite bias being clearly present in human interactions, and the rapid development of dialogue agents for real-world use-cases, such as interactive assistants. In this work we aim to address this by focusing on mitigating gender bias.
Question: What baseline is used to compare the experimental results against?
Paper: Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation
Recent work in dialogue incorporates personas, or personality descriptions that ground speaker's chat, such as I love fishing BIBREF12, BIBREF13, BIBREF14, BIBREF15, BIBREF16. Personas have been shown to increase engagingness and improve consistency. However, they can be a starting point for bias BIBREF17, BIBREF18, BIBREF9, as bias in the personas propagates to subsequent conversations.
Question: How does counterfactual data augmentation aim to tackle bias?
Answer: The training dataset is augmented by swapping all gendered words by their other gender counterparts
Paper: Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation
Analyzing the personas in LIGHT qualitatively, we find many examples of bias. For example, the character girl contains the line I regularly clean and cook dinner. Further examples are given in Table TABREF1.
Question: In the targeted data collection approach, what type of data is targetted?
Answer: Gendered characters in the dataset
Paper: Efficient Vector Representation for Documents through Corruption
Text understanding starts with the challenge of finding machine-understandable representation that captures the semantics of texts. Bag-of-words (BoW) and its N-gram extensions are arguably the most commonly used document representations. Despite its simplicity, BoW works surprisingly well for many tasks BIBREF0 . However, by treating words and phrases as unique and discrete symbols, BoW often fails to capture the similarity between words or phrases and also suffers from sparsity and high dimensionality.
Question: Which language models do they compare against?
Paper: Efficient Vector Representation for Documents through Corruption
Text representation learning has been extensively studied. Popular representations range from the simplest BoW and its term-frequency based variants BIBREF9 , language model based methods BIBREF10 , BIBREF11 , BIBREF12 , topic models BIBREF13 , BIBREF3 , Denoising Autoencoders and its variants BIBREF14 , BIBREF15 , and distributed vector representations BIBREF8 , BIBREF2 , BIBREF16 . Another prominent line of work includes learning task-specific document representation with deep neural networks, such as CNN BIBREF17 or LSTM based approaches BIBREF18 , BIBREF19 .
Question: Is their approach similar to making an averaged weighted sum of word vectors, where weights reflect word frequencies?
Paper: Efficient Vector Representation for Documents through Corruption
Several works BIBREF6 , BIBREF5 showcased that syntactic and semantic regularities of phrases and sentences are reasonably well preserved by adding or subtracting word embeddings learned through Word2Vec. It prompts us to explore the option of simply representing a document as an average of word embeddings. Figure FIGREF9 illustrates the new model architecture.
Question: How do they determine which words are informative?
Answer: Informative are those that will not be suppressed by regularization performed.
Paper: Microsoft Research Asia's Systems for WMT19
We participated in the WMT19 shared news translation task in 11 translation directions. We achieved first place for 8 directions: German$\leftrightarrow $English, German$\leftrightarrow $French, Chinese$\leftrightarrow $English, English$\rightarrow $Lithuanian, English$\rightarrow $Finnish, and Russian$\rightarrow $English, and three other directions were placed second (ranked by teams), which included Lithuanian$\rightarrow $English, Finnish$\rightarrow $English, and English$\rightarrow $Kazakh.
Question: What is their best performance on the largest language direction dataset?
Paper: Microsoft Research Asia's Systems for WMT19
The core idea of dual learning is to leverage the duality between the primal task (mapping from domain $\mathcal {X}$ to domain $\mathcal {Y}$) and dual task (mapping from domain $\mathcal {Y}$ to $\mathcal {X}$ ) to boost the performances of both tasks. MADL BIBREF0 extends the dual learning BIBREF1, BIBREF2 framework by introducing multiple primal and dual models. It was integrated into our submitted systems for German$\leftrightarrow $English and German$\leftrightarrow $French translations.
Question: How does soft contextual data augmentation work?
Paper: Microsoft Research Asia's Systems for WMT19
Pre-training and fine-tuning have achieved great success in language understanding. MASS BIBREF3, a pre-training method designed for language generation, adopts the encoder-decoder framework to reconstruct a sentence fragment given the remaining part of the sentence: its encoder takes a sentence with randomly masked fragment (several consecutive tokens) as input, and its decoder tries to predict this masked fragment. It was integrated into our submitted systems for Chinese$\rightarrow $English and English$\rightarrow $Lithuanian translations.
Question: How does muli-agent dual learning work?
Paper: Microsoft Research Asia's Systems for WMT19
As well known, the evolution of neural network architecture plays a key role in advancing neural machine translation. Neural architecture optimization (NAO), our newly proposed method BIBREF4, leverages the power of a gradient-based method to conduct optimization and guide the creation of better neural architecture in a continuous and more compact space given the historically observed architectures and their performances. It was applied in English$\leftrightarrow $Finnish translations in our submitted systems.
Question: Which language directions are machine translation systems of WMT evaluated on?
Paper: Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
Exploiting scale in both training data and model size has been central to the success of deep learning. When datasets are sufficiently large, increasing the capacity (number of parameters) of neural networks can give much better prediction accuracy. This has been shown in domains such as text BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , images BIBREF4 , BIBREF5 , and audio BIBREF6 , BIBREF7 . For typical deep learning models, where the entire model is activated for every example, this leads to a roughly quadratic blow-up in training costs, as both the model size and the number of training examples increase. Unfortunately, the advances in computing power and distributed computation fall short of meeting such demand.
Question: Approximately how much computational cost is saved by using this model?
Paper: Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
Our approach to conditional computation is to introduce a new type of general purpose neural network component: a Sparsely-Gated Mixture-of-Experts Layer (MoE). The MoE consists of a number of experts, each a simple feed-forward neural network, and a trainable gating network which selects a sparse combination of the experts to process each input (see Figure FIGREF8 ). All parts of the network are trained jointly by back-propagation.
Question: What improvement does the MOE model make over the SOTA on machine translation?
Paper: Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
Since its introduction more than two decades ago BIBREF16 , BIBREF17 , the mixture-of-experts approach has been the subject of much research. Different types of expert architectures hae been proposed such as SVMs BIBREF18 , Gaussian Processes BIBREF19 , BIBREF20 , BIBREF21 , Dirichlet Processes BIBREF22 , and deep networks. Other work has focused on different expert configurations such as a hierarchical structure BIBREF23 , infinite numbers of experts BIBREF24 , and adding experts sequentially BIBREF25 . BIBREF26 suggest an ensemble model in the format of mixture of experts for machine translation. The gating network is trained on a pre-trained ensemble NMT model.
Question: What improvement does the MOE model make over the SOTA on language modelling?
Answer: Perpexity is improved from 34.7 to 28.0.
Paper: Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
The Mixture-of-Experts (MoE) layer consists of a set of INLINEFORM0 “expert networks" INLINEFORM1 , and a “gating network" INLINEFORM2 whose output is a sparse INLINEFORM3 -dimensional vector. Figure FIGREF8 shows an overview of the MoE module. The experts are themselves neural networks, each with their own parameters. Although in principle we only require that the experts accept the same sized inputs and produce the same-sized outputs, in our initial investigations in this paper, we restrict ourselves to the case where the models are feed-forward networks with identical architectures, but with separate parameters.
Question: How is the correct number of experts to use decided?
Paper: Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
A simple choice of non-sparse gating function BIBREF17 is to multiply the input by a trainable weight matrix INLINEFORM0 and then apply the INLINEFORM1 function. DISPLAYFORM0 
Question: What equations are used for the trainable gating network?
Paper: Evaluation of basic modules for isolated spelling error correction in Polish texts
Spelling error correction is a fundamental NLP task. Most language processing applications benefit greatly from being provided clean texts for their best performance. Human users of computers also often expect competent help in making spelling of their texts correct.
Question: What is the difference in performance between the interpretable system (e.g. vectors and cosine distance) and LSTM with ELMo system?
Answer: Accuracy of best interpretible system was 0.3945 while accuracy of LSTM-ELMo net was 0.6818.
Paper: Evaluation of basic modules for isolated spelling error correction in Polish texts
Published work on language correction for Polish dates back at least to 1970s, when simplest Levenshtein distance solutions were used for cleaning mainframe inputs BIBREF5 , BIBREF6 . Spelling correction tests described in literature have tended to focus on one approach applied to a specific corpus. Limited examples include works on spellchecking mammography reports and tweets BIBREF7 , BIBREF4 . These works emphasized the importance of tailoring correction systems to specific problems of corpora they are applied to. For example, mammography reports suffer from poor typing, which in this case is a repetitive work done in relative hurry. Tweets, on the other hand, tend to contain emoticons and neologisms that can trick solutions based on rules and dictionaries, such as LanguageTool. The latter is, by itself, fairly well suited for Polish texts, since a number of extensions to the structure of this application was inspired by problems with morphology of Polish language BIBREF3 .
Question: What solutions are proposed for error detection and context awareness?