z-uo/qasper-squad huggingface dataset:
Paper: Autocompletion interfaces make crowd workers slower, but their use promotes response diversity
Crowdsourcing applications vary from basic, self-contained tasks such as image recognition or labeling BIBREF0 all the way to open-ended and creative endeavors such as collaborative writing, creative question proposal, or more general ideation BIBREF1 . Yet scaling the crowd to very large sets of creative tasks may require prohibitive numbers of workers. Scalability is one of the key challenges in crowdsourcing: how to best apply the valuable but limited resources provided by crowd workers and how to help workers be as efficient as possible.
Question: What was the task given to workers?
Paper: Autocompletion interfaces make crowd workers slower, but their use promotes response diversity
An important goal of crowdsourcing research is achieving efficient scalability of the crowd to very large sets of tasks. Efficiency in crowdsourcing manifests both in receiving more effective information per worker and in making individual workers faster and/or more accurate. The former problem is a significant area of interest BIBREF5 , BIBREF6 , BIBREF7 while less work has been put towards the latter.
Question: How was lexical diversity measured?
Answer: By computing number of unique responses and number of responses divided by the number of unique responses to that question for each of the questions
Paper: Autocompletion interfaces make crowd workers slower, but their use promotes response diversity
Here we describe the task we studied and its input data, worker recruitment, the design of our experimental treatment and control, the “instrumentation” we used to measure the speeds of workers as they performed our task, and our procedures to post-process and rate the worker responses to our task prior to subsequent analysis.
Question: How many responses did they obtain?
Paper: Autocompletion interfaces make crowd workers slower, but their use promotes response diversity
We recruited 176 AMT workers to participate in our conceptualization task. Of these workers, 90 were randomly assigned to the Control group and 86 to the AUI group. These workers completed 1001 tasks: 496 tasks in the control and 505 in the AUI. All responses were gathered within a single 24-hour period during April, 2017.
Question: What crowdsourcing platform was used?
Paper: Regularizing Output Distribution of Abstractive Chinese Social Media Text Summarization for Improved Semantic Consistency
Abstractive test summarization is an important text generation task. With the applying of the sequence-to-sequence model and the publication of large-scale datasets, the quality of the automatic generated summarization has been greatly improved BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 . However, the semantic consistency of the automatically generated summaries is still far from satisfactory.
Question: Are results reported only for English data?
Paper: Regularizing Output Distribution of Abstractive Chinese Social Media Text Summarization for Improved Semantic Consistency
Base on the fact that the spurious correspondence is not stable and its realization in the model is prone to change, we propose to alleviate the issue heuristically by regularization. We use the cross-entropy with an annealed output distribution as the regularization term in the loss so that the little fluctuation in the distribution will be depressed and more robust and stable correspondence will be learned. By correspondence, we mean the relation between (a) the current output, and (b) the source content and the partially generated output. Furthermore, we propose to use an additional output layer to generate the annealed output distribution. Due to the same fact, the two output layers will differ more in the words that superficially co-occur, so that the output distribution can be better regularized.
Question: Which existing models does this approach outperform?
Answer: RNN-context, SRB, CopyNet, RNN-distract, DRGD
Paper: Regularizing Output Distribution of Abstractive Chinese Social Media Text Summarization for Improved Semantic Consistency
Typically, in the training of the sequence-to-sequence model, only the one-hot hard target is used in the cross-entropy based loss function. For an example in the training set, the loss of an output vector is DISPLAYFORM0 
Question: What human evaluation method is proposed?
Answer: comparing the summary with the text instead of the reference and labeling the candidate bad if it is incorrect or irrelevant
Paper: A Discrete CVAE for Response Generation on Short-Text Conversation
Open-domain response generation BIBREF0, BIBREF1 for single-round short text conversation BIBREF2, aims at generating a meaningful and interesting response given a query from human users. Neural generation models are of growing interest in this topic due to their potential to leverage massive conversational datasets on the web. These generation models such as encoder-decoder models BIBREF3, BIBREF2, BIBREF4, directly build a mapping from the input query to its output response, which treats all query-response pairs uniformly and optimizes the maximum likelihood estimation (MLE). However, when the models converge, they tend to output bland and generic responses BIBREF5, BIBREF6, BIBREF7.
Question: How is human evaluation performed, what were the criteria?
Paper: A Discrete CVAE for Response Generation on Short-Text Conversation
In this section, we briefly review recent advancement in encoder-decoder models and CVAE-based models for response generation.
Question: What automatic metrics are used?
Paper: A Discrete CVAE for Response Generation on Short-Text Conversation
Encoder-decoder models for short-text conversation BIBREF3, BIBREF2 maximize the likelihood of responses given queries. During testing, a decoder sequentially generates a response using search strategies such as beam search. However, these models frequently generate bland and generic responses.
Question: What other kinds of generation models are used in experiments?
Paper: A Discrete CVAE for Response Generation on Short-Text Conversation
A few works indicate that it is worth trying to apply the CVAE to dialogue generation which is originally used in image generation BIBREF16, BIBREF17 and optimized with the variational lower bound of the conditional log-likelihood. For task-oriented dialogues, BIBREF22 wen2017latent use the latent variable to model intentions in the framework of neural variational inference. For chit-chat multi-round conversations, BIBREF23 serban2017hierarchical model the generative process with multiple levels of variability based on a hierarchical sequence-to-sequence model with a continuous high-dimensional latent variable. BIBREF14 zhao2017learning make use of the CVAE and the latent variable is used to capture discourse-level variations. BIBREF24 gu2018dialogwae propose to induce the latent variables by transforming context-dependent Gaussian noise. BIBREF15 shen2017conditional present a conditional variational framework for generating specific responses based on specific attributes. Yet, it is observed in other tasks such as image captioning BIBREF25 and question generation BIBREF26 that the CVAE-based generation models suffer from the low output diversity problem, i.e. multiple sampled variables point to the same generated sequences. In this work, we utilize a discrete latent variable with an interpretable meaning to alleviate this low output diversity problem on short-text conversation.
Question: How does discrete latent variable has an explicit semantic meaning to improve the CVAE on short-text conversation?
Paper: Short-Text Classification Using Unsupervised Keyword Expansion
The web has provided researchers with vast amounts of unlabeled text data, and enabled the development of increasingly sophisticated language models which can achieve state of the art performance despite having no task specific training BIBREF0, BIBREF1, BIBREF2. It is desirable to adapt these models for bespoke tasks such as short text classification.
Question: What news dataset was used?
Paper: Short-Text Classification Using Unsupervised Keyword Expansion
Document expansion methods have typically focused on creating new features with the help of custom models. Word co-occurrence models BIBREF4, topic modeling BIBREF5, latent concept expansion BIBREF6, and word embedding clustering BIBREF7, are all examples of document expansion methods that must first be trained using either the original dataset or an external dataset from within the same domain. The expansion models may therefore only be used when there is a sufficiently large training set.
Question: How do they determine similarity between predicted word and topics?
Paper: Short-Text Classification Using Unsupervised Keyword Expansion
The News Category Dataset BIBREF11 is a collection of headlines published by HuffPost BIBREF12 between 2012 and 2018, and was obtained online from Kaggle BIBREF13. The full dataset contains 200k news headlines with category labels, publication dates, and short text descriptions. For this analysis, a sample of roughly 33k headlines spanning 23 categories was used. Further analysis can be found in table SECREF12 in the appendix.
Question: What is the language model pre-trained on?
Paper: Language Identification on Massive Datasets of Short Message using an Attention Mechanism CNN
Language Identification (LID) is the Natural Language Processing (NLP) task of automatically recognizing the language that a document is written in. While this task was called "solved" by some authors over a decade ago, it has seen a resurgence in recent years thanks to the rise in popularity of social media BIBREF0, BIBREF1, and the corresponding daily creation of millions of new messages in dozens of different languages including rare ones that are not often included in language identification systems. Moreover, these messages are typically very short (Twitter messages were until recently limited to 140 characters) and very noisy (including an abundance of spelling mistakes, non-word tokens like URLs, emoticons, or hashtags, as well as foreign-language words in messages of another language), whereas LID was solved using long and clean documents. Indeed, several studies have shown that LID systems trained to a high accuracy on traditional documents suffer significant drops in accuracy when applied to short social-media texts BIBREF2, BIBREF3.
Question: What languages are represented in the dataset?
Answer: EN, JA, ES, AR, PT, KO, TH, FR, TR, RU, IT, DE, PL, NL, EL, SV, FA, VI, FI, CS, UK, HI, DA, HU, NO, RO, SR, LV, BG, UR, TA, MR, BN, IN, KN, ET, SL, GU, CY, ZH, CKB, IS, LT, ML, SI, IW, NE, KM, MY, TL, KA, BO
Paper: Language Identification on Massive Datasets of Short Message using an Attention Mechanism CNN
In this section, we will consider recent advances on the specific challenge of language identification in short text messages. Readers interested in a general overview of the area of LID, including older work and other challenges in the area, are encouraged to read the thorough survey of BIBREF0.
Question: Which existing language ID systems are tested?
Paper: Language Identification on Massive Datasets of Short Message using an Attention Mechanism CNN
One of the first, if not the first, systems for LID specialized for short text messages is the graph-based method of BIBREF5. Their graph is composed of vertices, or character n-grams (n = 3) observed in messages in all languages, and of edges, or connections between successive n-grams weighted by the observed frequency of that connection in each language. Identifying the language of a new message is then done by identifying the most probable path in the graph that generates that message. Their method achieves an accuracy of 0.975 on their own Twitter corpus.
Question: How was the one year worth of data collected?
Paper: Controlling Utterance Length in NMT-based Word Segmentation with Attention
All over the world, languages are disappearing at an unprecedented rate, fostering the need for specific tools aimed to aid field linguists to collect, transcribe, analyze, and annotate endangered language data (e.g. BIBREF0, BIBREF1). A remarkable effort in this direction has improved the data collection procedures and tools BIBREF2, BIBREF3, enabling to collect corpora for an increasing number of endangered languages (e.g. BIBREF4).
Question: Which language family does Mboshi belong to?
Paper: Controlling Utterance Length in NMT-based Word Segmentation with Attention
In this section, we briefly review the main concepts of recurrent architectures for machine translation introduced in BIBREF18, BIBREF19, BIBREF20. In our setting, the source and target sentences are always observed and we are mostly interested in the attention mechanism that is used to induce word segmentation.
Question: Does the paper report any alignment-only baseline?
Paper: Controlling Utterance Length in NMT-based Word Segmentation with Attention
Sequence-to-sequence models transform a variable-length source sequence into a variable-length target output sequence. In our context, the source sequence is a sequence of words $w_1, \ldots , w_J$ and the target sequence is an unsegmented sequence of phonemes or characters $\omega _1, \ldots , \omega _I$. In the RNN encoder-decoder architecture, an encoder consisting of a RNN reads a sequence of word embeddings $e(w_1),\dots ,e(w_J)$ representing the source and produces a dense representation $c$ of this sentence in a low-dimensional vector space. Vector $c$ is then fed to an RNN decoder producing the output translation $\omega _1,\dots ,\omega _I$ sequentially.
Question: What is the dataset used in the paper?
Paper: Controlling Utterance Length in NMT-based Word Segmentation with Attention
Encoding a variable-length source sentence in a fixed-length vector can lead to poor translation results with long sentences BIBREF19. To address this problem, BIBREF20 introduces an attention mechanism which provides a flexible source context to better inform the decoder's decisions. This means that the fixed context vector $c$ in Equations (DISPLAY_FORM5) and (DISPLAY_FORM6) is replaced with a position-dependent context $c_i$, defined as:
Question: How is the word segmentation task evaluated?
Paper: Global Greedy Dependency Parsing
Dependency parsing predicts the existence and type of linguistic dependency relations between words (as shown in Figure FIGREF1), which is a critical step in accomplishing deep natural language processing. Dependency parsing has been well developed BIBREF0, BIBREF1, and it generally relies on two types of parsing models: transition-based models and graph-based models. The former BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF4 traditionally apply local and greedy transition-based algorithms, while the latter BIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10, BIBREF11, BIBREF12 apply globally optimized graph-based algorithms.
Question: What are performance compared to former models?
Paper: Global Greedy Dependency Parsing
The global greedy parser will build its dependency trees in a stepwise manner without backtracking, which takes a general greedy decoding algorithm as in easy-first parsers.
Question: How faster is training and decoding compared to former models?
Answer: Proposed vs best baseline:
Decoding: 8541 vs 8532 tokens/sec
Training: 8h vs 8h
Paper: Tagged Back-Translation
Neural Machine Translation (NMT) has made considerable progress in recent years BIBREF0 , BIBREF1 , BIBREF2 . Traditional NMT has relied solely on parallel sentence pairs for training data, which can be an expensive and scarce resource. This motivates the use of monolingual data, usually more abundant BIBREF3 . Approaches using monolingual data for machine translation include language model fusion for both phrase-based BIBREF4 , BIBREF5 and neural MT BIBREF6 , BIBREF7 , back-translation BIBREF8 , BIBREF9 , unsupervised machine translation BIBREF10 , BIBREF11 , dual learning BIBREF12 , BIBREF13 , BIBREF14 , and multi-task learning BIBREF15 .
Question: What datasets was the method evaluated on?
Paper: From Speech-to-Speech Translation to Automatic Dubbing
Automatic dubbing can be regarded as an extension of the speech-to-speech translation (STST) task BIBREF0, which is generally seen as the combination of three sub-tasks: (i) transcribing speech to text in a source language (ASR), (ii) translating text from a source to a target language (MT) and (iii) generating speech from text in a target language (TTS). Independently from the implementation approach BIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5, BIBREF6, the main goal of STST is producing an output that reflects the linguistic content of the original sentence. On the other hand, automatic dubbing aims to replace all speech contained in a video document with speech in a different language, so that the result sounds and looks as natural as the original. Hence, in addition to conveying the same content of the original utterance, dubbing should also match the original timbre, emotion, duration, prosody, background noise, and reverberation.
Question: Is the model evaluated against a baseline?
Paper: From Speech-to-Speech Translation to Automatic Dubbing
With some approximation, we consider here automatic dubbing of the audio track of a video as the task of STST, i.e. ASR + MT + TTS, with the additional requirement that the output must be temporally, prosodically and acoustically close to the original audio. We investigate an architecture (see Figure 1) that enhances the STST pipeline with (i) enhanced MT able to generate translations of variable lengths, (ii) a prosodic alignment module that temporally aligns the MT output with the speech segments in the original audio, (iii) enhanced TTS to accurately control the duration of each produce utterance, and, finally, (iv) audio rendering that adds to the TTS output background noise and reverberation extracted from the original audio. In the following, we describe each component in detail, with the exception of ASR, for which we use BIBREF16 an of-the-shelf online service. ˜
Question: How many people are employed for the subjective evaluation?
Paper: Learning Rare Word Representations using Semantic Bridging
The prominent model for representing semantics of words is the distributional vector space model BIBREF2 and the prevalent approach for constructing these models is the distributional one which assumes that semantics of a word can be predicted from its context, hence placing words with similar contexts in close proximity to each other in an imaginary high-dimensional vector space. Distributional techniques, either in their conventional form which compute co-occurrence matrices BIBREF2 , BIBREF3 and learn high-dimensional vectors for words, or the recent neural-based paradigm which directly learns latent low-dimensional vectors, usually referred to as embeddings BIBREF4 , rely on a multitude of occurrences for each individual word to enable accurate representations. As a result of this statistical nature, words that are infrequent or unseen during training, such as domain-specific words, will not have reliable embeddings. This is the case even if massive corpora are used for training, such as the 100B-word Google News dataset BIBREF5 .
Question: What other embedding models are tested?
Paper: Learning Rare Word Representations using Semantic Bridging
We take an existing semantic space INLINEFORM0 and enrich it with rare and unseen words on the basis of the knowledge encoded for them in an external knowledge base (KB) INLINEFORM1 . The procedure has two main steps: we first embed INLINEFORM2 to transform it from a graph representation into a vector space representation (§ SECREF2 ), and then map this space to INLINEFORM3 (§ SECREF7 ). Our methodology is illustrated in Figure 1.
Question: How is performance measured?
Paper: Learning Rare Word Representations using Semantic Bridging
Our coverage enhancement starts by transforming the knowledge base INLINEFORM0 into a vector space representation that is comparable to that of the corpus-based space INLINEFORM1 . To this end, we use two techniques for learning low-dimensional feature spaces from knowledge graphs: DeepWalk and node2vec. DeepWalk uses a stream of short random walks in order to extract local information for a node from the graph. By treating these walks as short sentences and phrases in a special language, the approach learns latent representations for each node. Similarly, node2vec learns a mapping of nodes to continuous vectors that maximizes the likelihood of preserving network neighborhoods of nodes. Thanks to a flexible objective that is not tied to a particular sampling strategy, node2vec reports improvements over DeepWalk on multiple classification and link prediction datasets. For both these systems we used the default parameters and set the dimensionality of output representation to 100. Also, note than nodes in the semantic graph of WordNet represent synsets. Hence, a polysemous word would correspond to multiple nodes. In our experiments, we use the MaxSim assumption of BIBREF11 in order to map words to synsets.
Question: How are rare words defined?
Paper: Localization of Fake News Detection via Multitask Transfer Learning
There is a growing interest in research revolving around automated fake news detection and fact checking as its need increases due to the dangerous speed fake news spreads on social media BIBREF0. With as much as 68% of adults in the United States regularly consuming news on social media, being able to distinguish fake from non-fake is a pressing need.
Question: What other datasets are used?
Paper: Localization of Fake News Detection via Multitask Transfer Learning
We provide a baseline model as a comparison point, using a few-shot learning-based technique to benchmark transfer learning against methods designed with low resource settings in mind. After which, we show three TL techniques that we studied and adapted to the task of fake news detection.
Question: What is the size of the dataset?
Paper: Localization of Fake News Detection via Multitask Transfer Learning
We use a siamese neural network, shown to perform state-of-the-art few-shot learning BIBREF11, as our baseline model.
Question: What is the source of the dataset?
Answer: Online sites tagged as fake news site by Verafiles and NUJP and news website in the Philippines, including Pilipino Star Ngayon, Abante, and Bandera
Paper: Localization of Fake News Detection via Multitask Transfer Learning
ULMFiT BIBREF5 was introduced as a TL method for Natural Language Processing (NLP) that works akin to ImageNet BIBREF13 pretraining in Computer Vision.
Question: What were the baselines?
Answer: Siamese neural network consisting of an embedding layer, a LSTM layer and a feed-forward layer with ReLU activations
Paper: Spatial Concept Acquisition for a Mobile Robot that Integrates Self-Localization and Unsupervised Word Discovery from Spoken Sentences
Autonomous robots, such as service robots, operating in the human living environment with humans have to be able to perform various tasks and language communication. To this end, robots are required to acquire novel concepts and vocabulary on the basis of the information obtained from their sensors, e.g., laser sensors, microphones, and cameras, and recognize a variety of objects, places, and situations in an ambient environment. Above all, we consider it important for the robot to learn the names that humans associate with places in the environment and the spatial areas corresponding to these names; i.e., the robot has to be able to understand words related to places. Therefore, it is important to deal with considerable uncertainty, such as the robot's movement errors, sensor noise, and speech recognition errors.
Question: How do they show that acquiring names of places helps self-localization?