z-uo/qasper-squad huggingface dataset:
Paper: Recurrent Neural Network Encoder with Attention for Community Question Answering
Community question answering (cQA) is a paradigm that provides forums for users to ask or answer questions on any topic with barely any restrictions. In the past decade, these websites have attracted a great number of users, and have accumulated a large collection of question-comment threads generated by these users. However, the low restriction results in a high variation in answer quality, which makes it time-consuming to search for useful information from the existing content. It would therefore be valuable to automate the procedure of ranking related questions and comments for users with a new question, or when looking for solutions from comments of an existing question.
Question: What supplemental tasks are used for multitask learning?
Answer: Multitask learning is used for the task of predicting relevance of a comment on a different question to a given question, where the supplemental tasks are predicting relevance between the questions, and between the comment and the corresponding question
Paper: Recurrent Neural Network Encoder with Attention for Community Question Answering
Earlier work of community question answering relied heavily on feature engineering, linguistic tools, and external resource. BIBREF3 and BIBREF4 utilized rich non-textual features such as answer's profile. BIBREF5 syntactically analyzed the question and extracted name entity features. BIBREF6 demonstrated a textual entailment system can enhance cQA task by casting question answering to logical entailment.
Question: Is the improvement actually coming from using an RNN?
Paper: Recurrent Neural Network Encoder with Attention for Community Question Answering
In this section, we first discuss long short-term memory (LSTM) units and an associated attention mechanism. Next, we explain how we can encode a pair of sentences into a dense vector for predicting relationships using an LSTM with an attention mechanism. Finally, we apply these models to predict question-question similarity, question-comment similarity, and question-external comment similarity.
Question: How much performance gap between their approach and the strong handcrafted method?
Answer: 0.007 MAP on Task A, 0.032 MAP on Task B, 0.055 MAP on Task C
Paper: Recurrent Neural Network Encoder with Attention for Community Question Answering
LSTMs have shown great success in many different fields. An LSTM unit contains a memory cell with self-connections, as well as three multiplicative gates to control information flow. Given input vector $x_t$ , previous hidden outputs $h_{t-1}$ , and previous cell state $c_{t-1}$ , LSTM units operate as follows: 
Question: What is a strong feature-based method?
Paper: Recurrent Neural Network Encoder with Attention for Community Question Answering
A traditional RNN encoder-decoder approach BIBREF11 first encodes an arbitrary length input sequence into a fixed-length dense vector that can be used as input to subsequent classification models, or to initialize the hidden state of a secondary decoder. However, the requirement to compress all necessary information into a single fixed length vector can be problematic. A neural attention model BIBREF12 BIBREF13 has been recently proposed to alleviate this issue by enabling the network to attend to past outputs when decoding. Thus, the encoder no longer needs to represent an entire sequence with one vector; instead, it encodes information into a sequence of vectors, and adaptively chooses a subset of the vectors when decoding.
Question: Did they experimnet in other languages?
Paper: Attentional Encoder Network for Targeted Sentiment Classification
Targeted sentiment classification is a fine-grained sentiment analysis task, which aims at determining the sentiment polarities (e.g., negative, neutral, or positive) of a sentence over “opinion targets” that explicitly appear in the sentence. For example, given a sentence “I hated their service, but their food was great”, the sentiment polarities for the target “service” and “food” are negative and positive respectively. A target is usually an entity or an entity aspect.
Question: Do they use multi-attention heads?
Paper: Attentional Encoder Network for Targeted Sentiment Classification
The research approach of the targeted sentiment classification task including traditional machine learning methods and neural networks methods.
Question: How big is their model?
Answer: Proposed model has 1.16 million parameters and 11.04 MB.
Paper: Attentional Encoder Network for Targeted Sentiment Classification
Given a context sequence INLINEFORM0 and a target sequence INLINEFORM1 , where INLINEFORM2 is a sub-sequence of INLINEFORM3 . The goal of this model is to predict the sentiment polarity of the sentence INLINEFORM4 over the target INLINEFORM5 .
Question: How is their model different from BERT?
Paper: ThisIsCompetition at SemEval-2019 Task 9: BERT is unstable for out-of-domain samples
Opinion mining BIBREF0 is a huge field that covers many NLP tasks ranging from sentiment analysis BIBREF1 , aspect extraction BIBREF2 , and opinion summarization BIBREF3 , among others. Despite the vast literature on opinion mining, the task on suggestion mining has given little attention. Suggestion mining BIBREF4 is the task of collecting and categorizing suggestions about a certain product. This is important because while opinions indirectly give hints on how to improve a product (e.g. analyzing reviews), suggestions are direct improvement requests (e.g. tips, advice, recommendations) from people who have used the product.
Question: What datasets were used?
Paper: ThisIsCompetition at SemEval-2019 Task 9: BERT is unstable for out-of-domain samples
We present our model JESSI, which stands for Joint Encoders for Stable Suggestion Inference, shown in Figure FIGREF4 . Given a sentence INLINEFORM0 , JESSI returns a binary suggestion label INLINEFORM1 . JESSI consists of four important components: (1) A BERT-based encoder that leverages general knowledge acquired from a large pre-trained language model, (2) A CNN-based encoder that learns task-specific sentence representations, (3) an MLP classifier that predicts the label given the joint encodings, and (4) a domain adversarial training module that prevents the model to distinguish between the two domains.
Question: How did they do compared to other teams?
Paper: DENS: A Dataset for Multi-class Emotion Analysis
Humans experience a variety of complex emotions in daily life. These emotions are heavily reflected in our language, in both spoken and written forms.
Question: Which tested technique was the worst performer?
Paper: DENS: A Dataset for Multi-class Emotion Analysis
Using the categorical basic emotion model BIBREF3, BIBREF4, BIBREF5 studied creating lexicons from tweets for use in emotion analysis. Recently, BIBREF1, BIBREF6 and BIBREF2 proposed shared-tasks for multi-class emotion analysis based on tweets.
Question: How many emotions do they look at?
Answer: 9
Paper: DENS: A Dataset for Multi-class Emotion Analysis
In this section, we describe the process used to collect and annotate the dataset.
Question: What are the baseline benchmarks?
Paper: DENS: A Dataset for Multi-class Emotion Analysis
The dataset is annotated based on a modified Plutchik’s wheel of emotions.
Question: What is the size of this dataset?
Paper: DENS: A Dataset for Multi-class Emotion Analysis
We selected both classic and modern narratives in English for this dataset. The modern narratives were sampled based on popularity from Wattpad. We parsed selected narratives into passages, where a passage is considered to be eligible for annotation if it contained between 40 and 200 tokens.
Question: How many annotators were there?
Paper: Multitask Learning with CTC and Segmental CRF for Speech Recognition
State-of-the-art speech recognition accuracy has significantly improved over the past few years since the application of deep neural networks BIBREF0 , BIBREF1 . Recently, it has been shown that with the application of both neural network acoustic model and language model, an automatic speech recognizer can approach human-level accuracy on the Switchboard conversational speech recognition benchmark using around 2,000 hours of transcribed data BIBREF2 . While progress is mainly driven by well engineered neural network architectures and a large amount of training data, the hidden Markov model (HMM) that has been the backbone for speech recognition for decades is still playing a central role. Though tremendously successful for the problem of speech recognition, the HMM-based pipeline factorizes the whole system into several components, and building these components separately may be less computationally efficient when developing a large-scale system from thousands to hundred of thousands of examples BIBREF3 .
Question: Can SCRF be used to pretrain the model?
Paper: Filling Gender&Number Gaps in Neural Machine Translation with Black-box Context Injection
A common way for marking information about gender, number, and case in language is morphology, or the structure of a given word in the language. However, different languages mark such information in different ways – for example, in some languages gender may be marked on the head word of a syntactic dependency relation, while in other languages it is marked on the dependent, on both, or on none of them BIBREF0 . This morphological diversity creates a challenge for machine translation, as there are ambiguous cases where more than one correct translation exists for the same source sentence. For example, while the English sentence “I love language” is ambiguous with respect to the gender of the speaker, Hebrew marks verbs for the gender of their subject and does not allow gender-neutral translation. This allows two possible Hebrew translations – one in a masculine and the other in a feminine form. As a consequence, a sentence-level translator (either human or machine) must commit to the gender of the speaker, adding information that is not present in the source. Without additional context, this choice must be done arbitrarily by relying on language conventions, world knowledge or statistical (stereotypical) knowledge.
Question: What conclusions are drawn from the syntactic analysis?
Paper: Filling Gender&Number Gaps in Neural Machine Translation with Black-box Context Injection
Different languages use different morphological features marking different properties on different elements. For example, English marks for number, case, aspect, tense, person, and degree of comparison. However, English does not mark gender on nouns and verbs. Even when a certain property is marked, languages differ in the form and location of the marking BIBREF0 . For example, marking can occur on the head of a syntactic dependency construction, on its argument, on both (requiring agreement), or on none of them. Translation systems must generate correct target-language morphology as part of the translation process. This requires knowledge of both the source-side and target-side morphology. Current state-of-the-art translation systems do capture many aspects of natural language, including morphology, when a relevant context is available BIBREF2 , BIBREF3 , but resort to “guessing” based on the training-data statistics when it is not. Complications arise when different languages convey different kinds of information in their morphological systems. In such cases, a translation system may be required to remove information available in the source sentence, or to add information not available in it, where the latter can be especially tricky.
Question: What type of syntactic analysis is performed?
Paper: Filling Gender&Number Gaps in Neural Machine Translation with Black-box Context Injection
Our goal is to supply an NMT system with knowledge regarding the speaker and interlocutor of first-person sentences, in order to produce the desired target-side morphology when the information is not available in the source sentence. The approach we take in the current work is that of black-box injection, in which we attempt to inject knowledge to the input in order to influence the output of a trained NMT system, without having access to its internals or its training procedure as proposed by vanmassenhove-hardmeier-way:2018:EMNLP.
Question: How is it demonstrated that the correct gender and number information is injected using this system?
Paper: Filling Gender&Number Gaps in Neural Machine Translation with Black-box Context Injection
To demonstrate our method in a black-box setting, we focus our experiments on Google's machine translation system (GMT), accessed through its Cloud API. To test the method on real-world sentences, we consider a monologue from the stand-up comedy show “Sarah Silverman: A Speck of Dust”. The monologue consists of 1,244 English sentences, all by a female speaker conveyed to a plural, gender-neutral audience. Our parallel corpora consists of the 1,244 English sentences from the transcript, and their corresponding Hebrew translations based on the Hebrew subtitles. We translate the monologue one sentence at a time through the Google Cloud API. Eyeballing the results suggest that most of the translations use the incorrect, but default, masculine and singular forms for the speaker and the audience, respectively. We expect that by adding the relevant condition of “female speaking to an audience” we will get better translations, affecting both the gender of the speaker and the number of the audience.
Question: Which neural machine translation system is used?
Paper: Filling Gender&Number Gaps in Neural Machine Translation with Black-box Context Injection
We compare the different conditions by comparing BLEU BIBREF5 with respect to the reference Hebrew translations. We use the multi-bleu.perl script from the Moses toolkit BIBREF6 . Table shows BLEU scores for the different prefixes. The numbers match our expectations: Generally, providing an incorrect speaker and/or audience information decreases the BLEU scores, while providing the correct information substantially improves it - we see an increase of up to 2.3 BLEU over the baseline. We note the BLEU score improves in all cases, even when given the wrong gender of either the speaker or the audience. We hypothesise this improvement stems from the addition of the word “said” which hints the model to generate a more “spoken” language which matches the tested scenario. Providing correct information for both speaker and audience usually helps more than providing correct information to either one of them individually. The one outlier is providing “She” for the speaker and “her” for the audience. While this is not the correct scenario, we hypothesise it gives an improvement in BLEU as it further reinforces the female gender in the sentence.
Question: What are the components of the black-box context injection system?
Paper: Exploring End-to-End Techniques for Low-Resource Speech Recognition
Although development of the first speech recognition systems began half a century ago, there has been a significant increase of the accuracy of ASR systems and number of their applications for the recent ten years, even for low-resource languages BIBREF0 , BIBREF1 .
Question: What normalization techniques are mentioned?
Paper: Exploring End-to-End Techniques for Low-Resource Speech Recognition
Development of CTC-based systems originates from the paper BIBREF3 where CTC loss was introduced. This loss is a total probability of labels sequence given observation sequence, which takes into account all possible alignments induced by a given words sequence.
Question: What features do they experiment with?
Paper: Exploring End-to-End Techniques for Low-Resource Speech Recognition
For all experiments we used conversational speech from IARPA Babel Turkish Language Pack (LDC2016S10). This corpus contains about 80 hours of transcribed speech for training and 10 hours for development. The dataset is rather small compared to widely used benchmarks for conversational speech: English Switchboard corpus (300 hours, LDC97S62) and Fisher dataset (2000 hours, LDC2004S13 and LDC2005S13).
Question: Which architecture is their best model?
Paper: Exploring End-to-End Techniques for Low-Resource Speech Recognition
We tried to explore the behavior of different neural network architectures in case when rather small data is available. We used multi-layer bidirectional LSTM networks, tried fully-convolutional architecture similar to Wav2Letter BIBREF8 and explored DeepSpeech-like architecture developed by Salesforce (DS-SF) BIBREF14 .
Question: What kind of spontaneous speech is used?
Paper: Tag-based Multi-Span Extraction in Reading Comprehension
The task of reading comprehension, where systems must understand a single passage of text well enough to answer arbitrary questions about it, has seen significant progress in the last few years. With models reaching human performance on the popular SQuAD dataset BIBREF0, and with much of the most popular reading comprehension datasets having been solved BIBREF1, BIBREF2, a new dataset, DROP BIBREF3, was recently published.
Question: What approach did previous models use for multi-span questions?
Answer: Only MTMSM specifically tried to tackle the multi-span questions. Their approach consisted of two parts: first train a dedicated categorical variable to predict the number of spans to extract and the second was to generalize the single-span head method of extracting a span
Paper: Tag-based Multi-Span Extraction in Reading Comprehension
Numerically-aware QANet (NAQANet) BIBREF3 was the model released with DROP. It uses QANET BIBREF5, at the time the best-performing published model on SQuAD 1.1 BIBREF0 (without data augmentation or pretraining), as the encoder. On top of QANET, NAQANet adds four different output layers, which we refer to as "heads". Each of these heads is designed to tackle a specific question type from DROP, where these types where identified by DROP's authors post-creation of the dataset. These four heads are (1) Passage span head, designed for producing answers that consist of a single span from the passage. This head deals with the type of questions already introduced in SQuAD. (2) Question span head, for answers that consist of a single span from the question. (3) Arithmetic head, for answers that require adding or subtracting numbers from the passage. (4) Count head, for answers that require counting and sorting entities from the text. In addition, to determine which head should be used to predict an answer, a 4-way categorical variable, as per the number of heads, is trained. We denote this categorical variable as the "head predictor".
Question: How they use sequence tagging to answer multi-span questions?
Paper: Tag-based Multi-Span Extraction in Reading Comprehension
Problem statement. Given a pair $(x^P,x^Q)$ of a passage and a question respectively, both comprised of tokens from a vocabulary $V$, we wish to predict an answer $y$. The answer could be either a collection of spans from the input, or a number, supposedly arrived to by performing arithmetic reasoning on the input. We want to estimate $p(y;x^P,x^Q)$.
Question: What is difference in peformance between proposed model and state-of-the art on other question types?
Answer: For single-span questions, the proposed LARGE-SQUAD improve performance of the MTMSNlarge baseline for 2.1 EM and 1.55 F1.
For number type question,  MTMSNlarge baseline  have improvement over LARGE-SQUAD for 3,11  EM and  2,98 F1. 
For date question,  LARGE-SQUAD have improvements in 2,02 EM but MTMSNlarge have improvement of 4,39 F1.
Paper: Tag-based Multi-Span Extraction in Reading Comprehension
Assume there are $K$ answer heads in the model and their weights denoted by $\theta $. For each pair $(x^P,x^Q)$ we assume a latent categorical random variable $z\in \left\lbrace 1,\ldots \,K\right\rbrace $ such that the probability of an answer $y$ is
Question: What is the performance of proposed model on entire DROP dataset?
Answer: The proposed model achieves  EM 77,63 and F1 80,73  on the test and EM  76,95 and  F1 80,25 on the dev
Paper: Tag-based Multi-Span Extraction in Reading Comprehension
Before going over the answer heads, two additional components should be introduced - the summary vectors, and the head predictor.
Question: What is the previous model that attempted to tackle multi-span questions as a part of its design?
Paper: Transfer Learning Between Related Tasks Using Expected Label Proportions
Data annotation is a key bottleneck in many data driven algorithms. Specifically, deep learning models, which became a prominent tool in many data driven tasks in recent years, require large datasets to work well. However, many tasks require manual annotations which are relatively hard to obtain at scale. An attractive alternative is lightly supervised learning BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , in which the objective function is supplemented by a set of domain-specific soft-constraints over the model's predictions on unlabeled data. For example, in label regularization BIBREF0 the model is trained to fit the true label proportions of an unlabeled dataset. Label regularization is special case of expectation regularization (XR) BIBREF0 , in which the model is trained to fit the conditional probabilities of labels given features.
Question: How much more data does the model trained using XR loss have access to, compared to the fully supervised model?
Paper: Transfer Learning Between Related Tasks Using Expected Label Proportions
An effective way to supplement small annotated datasets is to use lightly supervised learning, in which the objective function is supplemented by a set of domain-specific soft-constraints over the model's predictions on unlabeled data. Previous work in lightly-supervised learning focused on training classifiers by using prior knowledge of label proportions BIBREF2 , BIBREF3 , BIBREF10 , BIBREF0 , BIBREF11 , BIBREF12 , BIBREF7 , BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 , BIBREF8 or prior knowledge of features label associations BIBREF1 , BIBREF17 , BIBREF18 , BIBREF19 , BIBREF20 . In the context of NLP, BIBREF17 suggested to use distributional similarities of words to train sequence models for part-of-speech tagging and a classified ads information extraction task. BIBREF19 used background lexical information in terms of word-class associations to train a sentiment classifier. BIBREF21 , BIBREF22 suggested to exploit the bilingual correlations between a resource rich language and a resource poor language to train a classifier for the resource poor language in a lightly supervised manner.
Question: Does the system trained only using XR loss outperform the fully supervised neural system?
Paper: Transfer Learning Between Related Tasks Using Expected Label Proportions
Expectation Regularization (XR) BIBREF0 is a lightly supervised learning method, in which the model is trained to fit the conditional probabilities of labels given features. In the context of NLP, XR was used by BIBREF20 to train twitter-user attribute prediction using hundreds of noisy distributional expectations based on census demographics. Here, we suggest using XR to train a target task (aspect-level sentiment) based on the output of a related source-task classifier (sentence-level sentiment).
Question: How accurate is the aspect based sentiment classifier trained only using the XR loss?
Answer: BiLSTM-XR-Dev Estimation accuracy is 83.31 for SemEval-15 and 87.68 for SemEval-16.
BiLSTM-XR accuracy is 83.31 for SemEval-15 and 88.12 for SemEval-16.