z-uo/qasper-squad huggingface dataset:
Paper: A Sketch-Based System for Semantic Parsing
The dataset MSParS is published by NLPCC 2019 evaluation task. The whole dataset consists of 81,826 samples annotated by native English speakers. 80% of them are used as training set. 10% of them are used as validation set while the rest is used as test set. 3000 hard samples are selected from the test set. Metric for this dataset is the exactly matching accuracy on both full test set and hard test subset. Each sample is composed of the question, the logical form, the parameters(entity/value/type) and question type as the Table TABREF3 demonstrates.
Question: What is the difference between the full test set and the hard test set?
Paper: Progressive Joint Modeling in Unsupervised Single-channel Overlapped Speech Recognition
The cocktail party problem BIBREF0 , BIBREF1 , referring to multi-talker overlapped speech recognition, is critical to enable automatic speech recognition (ASR) scenarios such as automatic meeting transcription, automatic captioning for audio/video recordings, and multi-party human-machine interactions, where overlapping speech is commonly observed and all streams need to be transcribed. The problem is still one of the hardest problems in ASR, despite encouraging progresses BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 .
Question: How is the discriminative training formulation different from the standard ones?
Paper: Progressive Joint Modeling in Unsupervised Single-channel Overlapped Speech Recognition
Unsupervised single-channel overlapped speech recognition refers to the speech recognition problem when multiple unseen talkers speak at the same time and only a single channel of overlapped speech is available. Different from supervised mode, there's not any prior knowledge of speakers in the evaluation stage.
Question: How are the two datasets artificially overlapped?
Paper: NIHRIO at SemEval-2018 Task 3: A Simple and Accurate Neural Network Model for Irony Detection in Twitter
Mining Twitter data has increasingly been attracting much research attention in many NLP applications such as in sentiment analysis BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 and stock market prediction BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 . Recently, Davidov2010 and Reyes2013 have shown that Twitter data includes a high volume of “ironic” tweets. For example, a user can use positive words in a Twitter message to her intended negative meaning (e.g., “It is awesome to go to bed at 3 am #not”). This especially results in a research challenge to assign correct sentiment labels for ironic tweets BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 , BIBREF15 .
Question: What baseline system is used?
Paper: NIHRIO at SemEval-2018 Task 3: A Simple and Accurate Neural Network Model for Irony Detection in Twitter
The dataset consists of 4,618 tweets (2,222 ironic + 2,396 non-ironic) that are manually labelled by three students. Some pre-processing steps were applied to the dataset, such as the emoji icons in a tweet are replaced by a describing text using the Python emoji package. Additionally, all the ironic hashtags, such as #not, #sarcasm, #irony, in the dataset have been removed. This makes difficult to correctly predict the label of a tweet. For example, “@coreybking thanks for the spoiler!!!! #not” is an ironic tweet but without #not, it probably is a non-ironic tweet. The dataset is split into the training and test sets as detailed in Table TABREF5 .
Question: What type of lexical, syntactic, semantic and polarity features are used?
Paper: What comes next? Extractive summarization by next-sentence prediction
Writing a summary is a different task compared to producing a longer article. As a consequence, it is likely that the topic and discourse moves made in summaries differ from those in regular articles. In this work, we present a powerful extractive summarization system which exploits rich summary-internal structure to perform content selection, redundancy reduction, and even predict the target summary length, all in one joint model.
Question: How does nextsum work?
Paper: Translation of Patent Sentences with a Large Vocabulary of Technical Terms Using Neural Machine Translation
Neural machine translation (NMT), a new approach to solving machine translation, has achieved promising results BIBREF0 , BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 . An NMT system builds a simple large neural network that reads the entire input source sentence and generates an output translation. The entire neural network is jointly trained to maximize the conditional probability of a correct translation of a source sentence with a bilingual corpus. Although NMT offers many advantages over traditional phrase-based approaches, such as a small memory footprint and simple decoder implementation, conventional NMT is limited when it comes to larger vocabularies. This is because the training complexity and decoding complexity proportionally increase with the number of target words. Words that are out of vocabulary are represented by a single unknown token in translations, as illustrated in Figure 1 . The problem becomes more serious when translating patent documents, which contain several newly introduced technical terms.
Question: Can the approach be generalized to other technical domains as well? 
Answer: There is no reason to think that this approach wouldn't also be successful for other technical domains. Technical terms are replaced with tokens, therefore so as long as there is a corresponding process for identifying and replacing technical terms in the new domain this approach could be viable.
Paper: Forex trading and Twitter: Spam, bots, and reputation manipulation
Currency trading (Forex) is the largest world market in terms of volume. We analyze trading and tweeting about the EUR-USD currency pair over a period of three years. First, a large number of tweets were manually labeled, and a Twitter stance classification model is constructed. The model then classifies all the tweets by the trading stance signal: buy, hold, or sell (EUR vs. USD). The Twitter stance is compared to the actual currency rates by applying the event study methodology, well-known in financial economics. It turns out that there are large differences in Twitter stance distribution and potential trading returns between the four groups of Twitter users: trading robots, spammers, trading companies, and individual traders. Additionally, we observe attempts of reputation manipulation by post festum removal of tweets with poor predictions, and deleting/reposting of identical tweets to increase the visibility without tainting one's Twitter timeline. 
Question: How many tweets were manually labelled?
Paper: Unsupervised Text Summarization via Mixed Model Back-Translation
Machine summarization systems have made significant progress in recent years, especially in the domain of news text. This has been made possible among other things by the popularization of the neural sequence-to-sequence (seq2seq) paradigm BIBREF0, BIBREF1, BIBREF2, the development of methods which combine the strengths of extractive and abstractive approaches to summarization BIBREF3, BIBREF4, and the availability of large training datasets for the task, such as Gigaword or the CNN-Daily Mail corpus which comprise of over 3.8M shorter and 300K longer articles and aligned summaries respectively. Unfortunately, the lack of datasets of similar scale for other text genres remains a limiting factor when attempting to take full advantage of these modeling advances using supervised training algorithms.
Question: What dataset they use for evaluation?
Answer: The same 2K set from Gigaword used in BIBREF7
Paper: Putting Self-Supervised Token Embedding on the Tables
Today most of business-related information is transmitted in an electronic form, such as emails. Therefore, converting these messages into an easily analyzable representation could open numerous business opportunities, as a lot of them are not used fully because of the difficulty to build bespoke parsing methods. In particular, a great number of these transmissions are semi-structured text, which doesn’t necessarily follows the classic english grammar. As seen in Fig. 1 , they can be under the form of tables containing diverse elements, words and numbers, afterwards referred to as tokens.
Question: What is the source of the tables?
Paper: English verb regularization in books and tweets
Human language reflects cultural, political, and social evolution. Words are the atoms of language. Their meanings and usage patterns reveal insight into the dynamical process by which society changes. Indeed, the increasing frequency with which electronic text is used as a means of communicating, e.g., through email, text messaging, and social media, offers us the opportunity to quantify previously unobserved mechanisms of linguistic development.
Question: Which regions of the United States do they consider?
Answer: all regions except those that are colored black
Paper: English verb regularization in books and tweets
To be consistent with prior work, we chose the verb list for our project to match that of Michel et al. BIBREF1 . When comparing BE with AE, we use the subset of verbs that form the irregular past tense with the suffix -t. When calculating frequencies or token counts for the `past tense' we use both the preterite and past participle of the verb. See #1 for a complete tabulation of all verb forms.
Question: Why did they only consider six years of published books?
Paper: HuggingFace's Transformers: State-of-the-art Natural Language Processing
In the past 18 months, advances on many Natural Language Processing (NLP) tasks have been dominated by deep learning models and, more specifically, the use of Transfer Learning methods BIBREF0 in which a deep neural network language model is pretrained on a web-scale unlabelled text dataset with a general-purpose training objective before being fine-tuned on various downstream tasks. Following noticeable improvements using Long Short-Term Memory (LSTM) architectures BIBREF1, BIBREF2, a series of works combining Transfer Learning methods with large-scale Transformer architectures BIBREF3 has repeatedly advanced the state-of-the-art on NLP tasks ranging from text classification BIBREF4, language understanding BIBREF5, BIBREF6, BIBREF7, machine translation BIBREF8, and zero-short language generation BIBREF9 up to co-reference resolution BIBREF10 and commonsense inference BIBREF11.
Question: What state-of-the-art general-purpose pretrained models are made available under the unified API?
Paper: A framework for streamlined statistical prediction using topic models
For the past 20 years, topic models have been used as a means of dimension reduction on text data, in order to ascertain underlying themes, or `topics', from documents. These probabilistic models have frequently been applied to machine learning problems, such as web spam filtering BIBREF0 , database sorting BIBREF1 and trend detection BIBREF2 .
Question: How is performance measured?
Answer: they use ROC curves and cross-validation
Paper: LeafNATS: An Open-Source Toolkit and Live Demo System for Neural Abstractive Text Summarization
Being one of the prominent natural language generation tasks, neural abstractive text summarization (NATS) has gained a lot of popularity BIBREF0 , BIBREF1 , BIBREF2 . Different from extractive text summarization BIBREF3 , BIBREF4 , BIBREF5 , NATS relies on modern deep learning models, particularly sequence-to-sequence (Seq2Seq) models, to generate words from a vocabulary based on the representations/features of source documents BIBREF0 , BIBREF6 , so that it has the ability to generate high-quality summaries that are verbally innovative and can also easily incorporate external knowledge BIBREF1 . Many NATS models have achieved better performance in terms of the commonly used evaluation measures (such as ROUGE BIBREF7 score) compared to extractive text summarization approaches BIBREF2 , BIBREF8 , BIBREF9 .
Question: What models are included in the toolkit?
Paper: Language-Based Image Editing with Recurrent Attentive Models
 In this work, we aim to develop an automatic Language-Based Image Editing (LBIE) system. Given a source image, which can be a sketch, a grayscale image or a natural image, the system will automatically generate a target image by editing the source image following natural language instructions provided by users. Such a system has a wide range of applications from Computer-Aided Design (CAD) to Virtual Reality (VR). As illustrated in Figure 1 , a fashion designer presents a sketch of a pair of new shoes (i.e., the source image) to a customer, who can provide modifications on the style and color in verbal description, which can then be taken by the LBIE system to change the original design. The final output (i.e., the target image) is the revised and enriched design that meets the customer’s requirement. Figure 2 showcases the use of LBIE for VR. While most VR systems still use button-controlled or touchscreen interface, LBIE provides a natural user interface for future VR systems, where users can easily modify the virtual environment via natural language instructions.
Question: Is there any human evaluation involved in evaluating this famework?
Paper: MULTEXT-East
The MULTEXT-East project, (Multilingual Text Tools and Corpora for Central and Eastern European Languages) ran from ’95 to ’97 and developed standardised language resources for six Central and Eastern European languages, as well as for English, the “hub” language of the project BIBREF0. The project was a spin-off of the MULTEXT project BIBREF1, which pursued similar goals for six Western European languages. The main results of the project were morphosyntactic specifications, defining the tagsets for lexical and corpus annotations in a common format, lexical resources and annotated multilingual corpora. In addition to delivering resources, a focus of the project was also the adoption and promotion of encoding standardization. On the one hand, the morphosyntactic annotations and lexicons were developed in the formalism used in MULTEXT, itself based on the specifications of the Expert Advisory Group on Language Engineering Standards, EAGLES BIBREF2. On the other, the corpus resources were encoded in SGML, using CES, the Corpus Encoding Standard BIBREF3, a derivative of the Text Encoding Initiative Guidelines, version P3, BIBREF4.
Question: How big is multilingual dataset?
Paper: BERT Goes to Law School: Quantifying the Competitive Advantage of Access to Large Legal Corpora in Contract Understanding
Businesses rely on contracts to capture critical obligations with other parties, such as: scope of work, amounts owed, and cancellation policies. Various efforts have gone into automatically extracting and classifying these terms. These efforts have usually been modeled as: classification, entity and relation extraction tasks. In this paper we focus on classification, but in our application we have found that our findings apply equally and sometimes, more profoundly, on other tasks.
Question: How big is dataset used for fine-tuning BERT?
Paper: Proposal Towards a Personalized Knowledge-powered Self-play Based Ensemble Dialog System
Prompt: What is your team’s vision for your Socialbot? How do you want your customers to feel at the end of an interaction with your socialbot? How would your team measure success in competition?
Question: How big are datasets for 2019 Amazon Alexa competition?
Paper: Proposal Towards a Personalized Knowledge-powered Self-play Based Ensemble Dialog System
Prompt: Please share a sample interaction/conversation you expect your Socialbot to achieve by the end of the Competition.
Question: What is novel in author's approach?
Answer: They use self-play learning , optimize the model for specific metrics, train separate models per user, use model  and response classification predictors, and filter the dataset to obtain higher quality training data.
Paper: Dynamic Prosody Generation for Speech Synthesis using Linguistics-Driven Acoustic Embedding Selection
Corresponding author email: tshubhi@amazon.com. Paper submitted to IEEE ICASSP 2020
Question: What dataset is used for train/test of this method?
Answer: Training datasets: TTS System dataset and embedding selection dataset. Evaluation datasets: Common Prosody Errors dataset and LFR dataset.
Paper: DCN+: Mixed Objective and Deep Residual Coattention for Question Answering
Existing state-of-the-art question answering models are trained to produce exact answer spans for a question and a document. In this setting, a ground truth answer used to supervise the model is defined as a start and an end position within the document. Existing training approaches optimize using cross entropy loss over the two positions. However, this suffers from a fundamental disconnect between the optimization, which is tied to the position of a particular ground truth answer span, and the evaluation, which is based on the textual content of the answer. This disconnect is especially harmful in cases where answers that are textually similar to, but distinct in positions from, the ground truth are penalized in the same fashion as answers that are textually dissimilar. For example, suppose we are given the sentence “Some believe that the Golden State Warriors team of 2017 is one of the greatest teams in NBA history”, the question “which team is considered to be one of the greatest teams in NBA history”, and a ground truth answer of “the Golden State Warriors team of 2017”. The span “Warriors” is also a correct answer, but from the perspective of traditional cross entropy based training it is no better than the span “history”.
Question: How much is the gap between using the proposed objective and using only cross-entropy objective?
Answer: The mixed objective improves EM by 2.5% and F1 by 2.2%
Paper: Bootstrapping Generators from Noisy Data
A core step in statistical data-to-text generation concerns learning correspondences between structured data representations (e.g., facts in a database) and paired texts BIBREF0 , BIBREF1 , BIBREF2 . These correspondences describe how data representations are expressed in natural language (content realisation) but also indicate which subset of the data is verbalised in the text (content selection).
Question: What is the multi-instance learning?
Paper: Competency Questions and SPARQL-OWL Queries Dataset and Analysis
Within the field of ontology engineering, Competency Questions (CQs) BIBREF0 are natural language questions outlining the scope of knowledge represented by an ontology. They represent functional requirements in the sense that the developed ontology or an ontology-based information system should be able to answer them; hence contain all the relevant knowledge. For example, a CQ may be What are the implementations of C4.5 algorithm?, indicating that the ontology needs to contain classes, such as Algorithm and C4.5 as subclass of Algorithm, and something about implementations such that the answer to the CQ will be non-empty.
Question: How many domains of ontologies do they gather data from?
Answer: 5 domains: software, stuff, african wildlife, healthcare, datatypes
Paper: Question Answering via Integer Programming over Semi-Structured Knowledge
Answering questions posed in natural language is a fundamental AI task, with a large number of impressive QA systems built over the years. Today's Internet search engines, for instance, can successfully retrieve factoid style answers to many natural language queries by efficiently searching the Web. Information Retrieval (IR) systems work under the assumption that answers to many questions of interest are often explicitly stated somewhere BIBREF0 , and all one needs, in principle, is access to a sufficiently large corpus. Similarly, statistical correlation based methods, such as those using Pointwise Mutual Information or PMI BIBREF1 , work under the assumption that many questions can be answered by looking for words that tend to co-occur with the question words in a large corpus.
Question: How is the semi-structured knowledge base created?
Paper: Comparing morphological complexity of Spanish, Otomi and Nahuatl
Morphology deals with the internal structure of words BIBREF0 , BIBREF1 . Languages of the world have different word production processes. Morphological richness vary from language to language, depending on their linguistic typology. In natural language processing (NLP), taking into account the morphological complexity inherent to each language could be important for improving or adapting the existing methods, since the amount of semantic and grammatical information encoded at the word level, may vary significantly from language to language.
Question: what is the practical application for this paper?
Answer: Improve existing NLP methods. Improve linguistic analysis. Measure impact of word normalization tools.
Paper: Unsupervised, Knowledge-Free, and Interpretable Word Sense Disambiguation
The notion of word sense is central to computational lexical semantics. Word senses can be either encoded manually in lexical resources or induced automatically from text. The former knowledge-based sense representations, such as those found in the BabelNet lexical semantic network BIBREF0 , are easily interpretable by humans due to the presence of definitions, usage examples, taxonomic relations, related words, and images. The cost of such interpretability is that every element mentioned above is encoded manually in one of the underlying resources, such as Wikipedia. Unsupervised knowledge-free approaches, e.g. BIBREF1 , BIBREF2 , require no manual labor, but the resulting sense representations lack the above-mentioned features enabling interpretability. For instance, systems based on sense embeddings are based on dense uninterpretable vectors. Therefore, the meaning of a sense can be interpreted only on the basis of a list of related senses.
Question: Do they use a neural model for their task?
Paper: Summary Level Training of Sentence Rewriting for Abstractive Summarization
The task of automatic text summarization aims to compress a textual document to a shorter highlight while keeping salient information of the original text. In general, there are two ways to do text summarization: Extractive and Abstractive BIBREF0. Extractive approaches generate summaries by selecting salient sentences or phrases from a source text, while abstractive approaches involve a process of paraphrasing or generating sentences to write a summary.
Question: What's the method used here?
Answer: Two neural networks: an extractor based on an encoder (BERT) and a decoder (LSTM Pointer Network BIBREF22) and an abstractor identical to the one proposed in BIBREF8.