z-uo/qasper-squad huggingface dataset:
Paper: Multiplicative Models for Recurrent Language Modeling
One of the principal challenges in computational linguistics is to account for the word order of the document or utterance being processed BIBREF0 . Of course, the numbers of possible phrases grows exponentially with respect to a given phrase length, requiring an approximate approach to summarizing its content. rnn are such an approach, and they are used in various tasks in nlp, such as machine translation BIBREF1 , abstractive summarization BIBREF2 and question answering BIBREF3 . However, rnn, as approximations, suffer from numerical troubles that have been identified, such as that of recovering from past errors when generating phrases. We take interest in a model that mitigates this problem, mrnn, and how it has been and can be combined for new models. To evaluate these models, we use the task of recurrent language modeling, which consists in predicting the next token (character or word) in a document. This paper is organized as follows: rnn and mrnn are introduced respectively in Sections SECREF2 and SECREF3 . Section SECREF4 presents new and existing multiplicative models. Section SECREF5 describes the datasets and experiments performed, as well as results obtained. Sections SECREF6 discusses and concludes our findings.
Question: Do they compare results against state-of-the-art language models?
Paper: Multiplicative Models for Recurrent Language Modeling
rnn are powerful tools of sequence modeling that can preserve the order of words or characters in a document. A document is therefore a sequence of words, INLINEFORM0 . Given the exponential growth of possible histories with respect to the sequence length, the probability of observing a given sequence needs to be approximated. rnn will make this approximation using the product rule, INLINEFORM1 
Question: Do they integrate the second-order term in the mLSTM?
Paper: Multiplicative Models for Recurrent Language Modeling
Most recurrent neural network architectures, including lstm and gru share the following building block: DISPLAYFORM0 
Question: Which dataset do they train their models on?
Paper: HULK: An Energy Efficiency Benchmark Platform for Responsible Natural Language Processing
Environmental concerns of machine learning research has been rising as the carbon emission of certain tasks like neural architecture search reached an exceptional “ocean boiling” level BIBREF7. Increased carbon emission has been one of the key factors to aggravate global warming . Research and development process like parameter search further increase the environment impact. When using cloud-based machines, the environment impact is strongly correlated with budget.
Question: How much does it minimally cost to fine-tune some model according to benchmarking framework?
Answer: $1,728
Paper: HULK: An Energy Efficiency Benchmark Platform for Responsible Natural Language Processing
For pretraining phase, the benchmark is designed to favor energy efficient models in terms of time and cost that each model takes to reach certain multi-task performance pretrained from scratch. For example, we keep track of the time and cost of a BERT model pretrained from scratch. After every thousand of pretraining steps, we clone the model for fine-tuning and see if the final performance can reach our cut-off level. When the level is reached, time and cost for pretraining is used for comparison. Models faster or cheaper to pretrain are recommended.
Question: What models are included in baseline benchmarking results?
Answer: BERT, XLNET RoBERTa, ALBERT, DistilBERT
Paper: Evaluation and Improvement of Chatbot Text Classification Data Quality Using Plausible Negative Examples
Smart conversational agents are increasingly used across business domains BIBREF0 . We focus on recruitment chatbots that connect recruiters and job-seekers. The recruiter teams we work with are motivated by reasons of scale and accessibility to build and maintain chatbots that provide answers to frequently asked questions (FAQs) based on ML/NLP datasets. Our enterprise clients may have up to INLINEFORM0 employees, and commensurate hiring rate. We have found that almost INLINEFORM1 of end-user (job-seeker) traffic occurs outside of working hours BIBREF1 , which is consistent with the anecdotal reports of our clients that using the chatbot helped reduce email and ticket inquiries of common FAQs. The usefulness of these question-answering conversational UIs depends on building and maintaining the ML/NLP components used in the overall flow (see Fig. FIGREF4 ).
Question: did they compare with other evaluation metrics?
Paper: Evaluation and Improvement of Chatbot Text Classification Data Quality Using Plausible Negative Examples
Chatbots, or “text messaging-based conversational agents”, have received particular attention in 2010s BIBREF0 . Many modern text-based chatbots use relatively simple NLP tools BIBREF7 , or avoid ML/NLP altogether BIBREF2 , relying on conversation flow design and non-NLP inputs like buttons and quick-replies. Conversational natural-language interfaces for question-answering have an extensive history, which distinguishes open-domain and closed-domain systems BIBREF8 . ML-based chatbots rely on curated data to provide examples for classes (commonly, “intents”), and must balance being widely-accessible to many end-users, but typically specialized in the domain and application goal BIBREF9 . In practice, design and development of a chatbot might assume a domain more focused, or different, than real use reveals.
Question: which datasets were used in validation?
Paper: Learning to Paraphrase for Question Answering
Enabling computers to automatically answer questions posed in natural language on any domain or topic has been the focus of much research in recent years. Question answering (QA) is challenging due to the many different ways natural language expresses the same information need. As a result, small variations in semantically equivalent questions, may yield different answers. For example, a hypothetical QA system must recognize that the questions “who created microsoft” and “who started microsoft” have the same meaning and that they both convey the founder relation in order to retrieve the correct answer from a knowledge base.
Question: It looks like learning to paraphrase questions, a neural scoring model and a answer selection model cannot be trained end-to-end. How are they trained?
Answer: using multiple pivot sentences
Paper: Evaluating Multimodal Representations on Visual Semantic Textual Similarity
Language understanding is a task proving difficult to automatize, because, among other factors, much of the information that is needed for the correct interpretation of an utterance is not explicit in text BIBREF0. This contrasts with how natural is language understanding for humans, who can cope easily with information absent in text, using common sense and background knowledge like, for instance, typical spatial relations between objects. From another perspective, it is well-known that the visual modality provides complementary information to that in the text. In fact, recent advances in deep learning research have led the field of computer vision and natural language processing to significant progress in tasks that involve visual and textual understanding. Tasks that include visual and textual content include Image Captioning BIBREF1, Visual Question Answering BIBREF2, and Visual Machine Translation BIBREF3, among others.
Question: What multimodal representations are used in the experiments?
Paper: Evaluating Multimodal Representations on Visual Semantic Textual Similarity
The task of Visual Semantic Textual Similarity stems from previous work on textual inference tasks. In textual entailment, given a textual premise and a textual hypothesis, systems need to decide whether the first entails the second, they are in contradiction, or none of the previous BIBREF4. Popular datasets include the Stanford Natural Language Inference dataset BIBREF5. As an alternative to entailment, STS datasets comprise pairs of sentences which have been annotated with similarity scores. STS systems are usually evaluated on the STS benchmark dataset BIBREF6. In this paper we present an extension of STS, so we present the task in more detail in the next section.
Question: How much better is inference that has addition of image representation compared to text-only representations?
Paper: Evaluating Multimodal Representations on Visual Semantic Textual Similarity
STS assesses the degree to which two sentences are semantically equivalent to each other. The annotators measure the similarity among sentences, with higher scores for more similar sentences. The annotations of similarity were guided by the scale in Table TABREF4, ranging from 0 for no meaning overlap to 5 for meaning equivalence. Intermediate values reflect interpretable levels of partial overlap in meaning.
Question: How they compute similarity between the representations?
Paper: Evaluating Multimodal Representations on Visual Semantic Textual Similarity
The data collection of sentence-image pairs comprised several steps, including the selection of pairs to be annotated, the annotation methodology, and a final filtering stage.
Question: How big is vSTS training data?
Paper: Exploring Multilingual Syntactic Sentence Representations
Recent success in language modelling and representation learning have largely focused on learning the semantic structures of language BIBREF0. Syntactic information, such as part-of-speech (POS) sequences, is an essential part of language and can be important for tasks such as authorship identification, writing-style analysis, translation, etc. Methods that learn syntactic representations have received relatively less attention, with focus mostly on evaluating the semantic information contained in representations produced by language models.
Question: Which evaluation metrics do they use for language modelling?
Paper: Exploring Multilingual Syntactic Sentence Representations
Training semantic embeddings based on multilingual data was studied by MUSE BIBREF1 and LASER BIBREF2 at the word and sentence levels respectively. Multi-task training for disentangling semantic and syntactic information was studied in BIBREF6. This work also used a nearest neighbour method to evaluate the syntactic properties of models, though their focus was on disentanglement rather than embedding quality.
Question: Do they do quantitative quality analysis of learned embeddings?
Paper: Exploring Multilingual Syntactic Sentence Representations
We iterated upon the model architecture proposed in LASER BIBREF2. The model consists of a two-layer Bi-directional LSTM (BiLSTM) encoder and a single-layer LSTM decoder. The encoder is language agnostic as no language context is provided as input. In contrast to LASER, we use the concatenation of last hidden and cell states of the encoder to initialize the decoder through a linear projection.
Question: Do they evaluate on downstream tasks?
Paper: Exploring Multilingual Syntactic Sentence Representations
Training was performed using an aligned parallel corpus. Given a source-target aligned sentence pair (as in machine translation), we:
Question: Which corpus do they use?
Paper: Conclusion-Supplement Answer Generation for Non-Factoid Questions
Question Answering (QA) modules play particularly important roles in recent dialog-based Natural Language Understanding (NLU) systems, such as Apple's Siri and Amazon's Echo. Users chat with AI systems in natural language to get the answers they are seeking. QA systems can deal with two types of question: factoid and non-factoid ones. The former sort asks, for instance, for the name of a thing or person such as “What/Who is $X$?”. The latter sort includes more diverse questions that cannot be answered by a short fact. For instance, users may ask for advice on how to make a long-distance relationship work well or for opinions on public issues. Significant progress has been made in answering factoid questions BIBREF0, BIBREF1; however, answering non-factoid questions remains a challenge for QA modules.
Question: How much more accurate is the model than the baseline?
Answer: For the Oshiete-goo dataset, the NAGM model's ROUGE-L score is higher than the highest performing conventional model, Trans, by 0.021, and its BLEU-4 score is higher than the highest performing model CLSTM by 0.037.  For the nfL6 dataset, the NAGM model's ROUGE-L score is higher than the highest performing conventional model, CLSTM, by 0.028, and its BLEU-4 score is higher than the highest performing model CLSTM by 0.040. Human evaluation of the NAGM's generated outputs for the Oshiete-goo dataset had 47% ratings of (1), the highest rating, while CLSTM only received 21% ratings of (1). For the nfL6 dataset, the comparison of (1)'s was NAGM's 50% to CLSTM's 30%.
Paper: Syntax-Enhanced Self-Attention-Based Semantic Role Labeling
The task of semantic role labeling (SRL) is to recognize arguments for a given predicate in one sentence and assign labels to them, including “who” did “what” to “whom”, “when”, “where”, etc. Figure FIGREF1 is an example sentence with both semantic roles and syntactic dependencies. Since the nature of semantic roles is more abstract than the syntactic dependencies, SRL has a wide range of applications in different areas, e.g., text classification BIBREF0, text summarization BIBREF1, BIBREF2, recognizing textual entailment BIBREF3, BIBREF4, information extraction BIBREF5, question answering BIBREF6, BIBREF7, and so on.
Question: How big is improvement over the old  state-of-the-art performance on CoNLL-2009 dataset?
Paper: Syntax-Enhanced Self-Attention-Based Semantic Role Labeling
Traditional semantic role labeling task BIBREF17 presumes that the syntactic structure of the sentence is given, either being a constituent tree or a dependency tree, like in the CoNLL shared tasks BIBREF18, BIBREF19, BIBREF20. Recent neural-network-based approaches can be roughly categorized into two classes: 1) making use of the syntactic information BIBREF21, BIBREF22, BIBREF23, BIBREF24, and 2) pure end-to-end learning from tokens to semantic labels, e.g., BIBREF25, BIBREF26.
Question: What is new state-of-the-art performance on CoNLL-2009 dataset?
Answer: In closed setting 84.22 F1 and in open 87.35 F1.
Paper: Syntax-Enhanced Self-Attention-Based Semantic Role Labeling
In this section, we first introduce the basic architecture of our self-attention-based SRL model, and then present two different ways to encode the syntactic dependency information. Afterwards, we compare three approaches to incorporate the syntax into the base model, concatenation to the input embedding, LISA, and our proposed relation-aware method.
Question: How big is CoNLL-2009 dataset?
Paper: Syntax-Enhanced Self-Attention-Based Semantic Role Labeling
Our basic model is a multi-head self-attention-based model, which is effective in SRL task as previous work proves BIBREF35. The model consists of three layers: the input layer, the encoder layer and the prediction layer as shown in Figure FIGREF5.
Question: What different approaches of encoding syntactic information authors present?
Paper: Syntax-Enhanced Self-Attention-Based Semantic Role Labeling
The input layer contains three types of embeddings: token embedding, predicate embedding, and positional embedding.
Question: What are two strong baseline methods authors refer to?
Answer: Marcheggiani and Titov (2017) and Cai et al. (2018)
Paper: Multi-modal Dense Video Captioning
The substantial amount of freely available video material has brought up the need for automatic methods to summarize and compactly represent the essential content. One approach would be to produce a short video skim containing the most important video segments as proposed in the video summarization task BIBREF0. Alternatively, the video content could be described using natural language sentences. Such an approach can lead to a very compact and intuitive representation and is typically referred to as video captioning in the literature BIBREF1. However, producing a single description for an entire video might be impractical for long unconstrained footage. Instead, dense video captioning BIBREF2 aims, first, at temporally localizing events and, then, at producing natural language description for each of them. Fig. FIGREF1 illustrates dense video captions for an example video sequence.
Question: How many category tags are considered?
Answer: 14 categories
Paper: Multi-modal Dense Video Captioning
Early works in video captioning applied rule-based models BIBREF13, BIBREF14, BIBREF15, where the idea was to identify a set of video objects and use them to fill predefined templates to generate a sentence. Later, the need for sentence templates was omitted by casting the captioning problem as a machine translation task BIBREF16. Following the success of neural models in translation systems BIBREF17, similar methods became widely popular in video captioning BIBREF18, BIBREF19, BIBREF20, BIBREF1, BIBREF21, BIBREF22, BIBREF23, BIBREF24, BIBREF25. The rationale behind this approach is to train two Recurrent Neural Networks (RNNs) in an encoder-decoder fashion. Specifically, an encoder inputs a set of video features, accumulates its hidden state, which is passed to a decoder for producing a caption.
Question: What domain does the dataset fall into?
Answer: YouTube videos
Paper: Multi-modal Dense Video Captioning
Inspired by the idea of the dense image captioning task BIBREF37, Krishna BIBREF2 introduced a problem of dense video captioning and released a new dataset called ActivityNet Captions which leveraged the research in the field BIBREF4, BIBREF5, BIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF38, BIBREF10. In particular, BIBREF5 adopted the idea of the context-awareness BIBREF2 and generalized the temporal event proposal module to utilize both past and future contexts as well as an attentive fusion to differentiate captions from highly overlapping events. Meanwhile, the concept of Single Shot Detector (SSD) BIBREF39 was also used to generate event proposals and reward maximization for better captioning in BIBREF6.
Question: What ASR system do they use?
Paper: Multi-modal Dense Video Captioning
A few attempts has been made to include additional cues like audio and speech BIBREF38, BIBREF42, BIBREF43 for dense video captioning task. Rahman BIBREF38 utilized the idea of cycle-consistency BIBREF40 to build a model with visual and audio inputs. However, due to weak supervision, the system did not reach high performance. Hessel BIBREF42 and Shi BIBREF43 employ a transformer architecture BIBREF3 to encode both video frames and speech segments to generate captions for instructional (cooking) videos. Yet, the high results on a dataset which is restricted to instructional video appear to be not evidential as the speech and the captions are already very close to each other in such videos BIBREF41.
Question: What is the state of the art?
Paper: Deep Learning for Automatic Tracking of Tongue Surface in Real-time Ultrasound Videos, Landmarks instead of Contours
Ultrasound technology is a widespread technology in speech research for studying tongue movement and speech articulation BIBREF0 due to its attractive characteristics, such as imaging at a reasonably rapid frame rate, which empowers researchers to envision subtle and swift gestures of the tongue in real-time. Besides, ultrasound technology is portable, relatively affordable, and clinically safe and non-invasive BIBREF1. The mid-sagittal view is regularly adapted in ultrasound data as it displays relative backness, height, and the slope of various areas of the tongue. Quantitative analysis of tongue motion needs the tongue contour to be extracted, tracked, and visualized.
Question: How big are datasets used in experiments?
Paper: Deep Learning for Automatic Tracking of Tongue Surface in Real-time Ultrasound Videos, Landmarks instead of Contours
Similar to facial landmark detection methods BIBREF15, we considered the problem of tongue contour extraction as a simple landmark detection and tracking. For this reason, we first developed a customized annotator software that can extract equally separated and randomized markers from segmented tongue contours in different databases. Meanwhile, the same software could fit B-spline curves on the extracted markers to revert the process for evaluation purposes.
Question: What previously annotated databases are available?
Paper: From quantum foundations via natural language meaning to a theory of everything
No physicists! ...the symbol INLINEFORM0 above does not stand for the operation that turns two Hilbert spaces into the smallest Hilbert space in which the two given ones bilinearly embed. No category-theoreticians! ...neither does it stand for the composition operation that turns any pair of objects (and morphisms) in a monoidal category into another object, and that is subject to a horrendous bunch of conditions that guaranty coherence with the remainder of the structure. Instead, this is what it means: INLINEFORM1 
Question: Do they address abstract meanings and concepts separately?
Paper: From quantum foundations via natural language meaning to a theory of everything
So, how does one go about formalising the concept of togetherness? While we don't want an explicit description of the foo involved, we do need some kind of means for identifying foo. Therefore, we simple give each foo a name, say INLINEFORM0 . Then, INLINEFORM1 represents the togetherness of INLINEFORM2 and INLINEFORM3 . We also don't want an explicit description of INLINEFORM4 , so how can we say anything about INLINEFORM5 without explicitly describing INLINEFORM6 , INLINEFORM7 and INLINEFORM8 ?
Question: Do they argue that all words can be derived from other (elementary) words?
Paper: From quantum foundations via natural language meaning to a theory of everything
But we can still do a lot better. What a resource theory fails to capture (on purpose in fact) is the actual process that converts one resource into another one. So let's fix that problem, and explicitly account for processes.
Question: Do they break down word meanings into elementary particles as in the standard model of quantum theory?
Paper: VATEX Captioning Challenge 2019: Multi-modal Information Fusion and Multi-stage Training Strategy for Video Captioning
Video captioning has drawn more attention and shown promising results recently. To translate content-rich video into human language is a extremely complex task, which should not only extract abundant multi-modal information from video but also cross the semantic gap to generate accurate and fluent language. Thanks to the recent developments of useful deep learning frameworks, such as LSTM BIBREF1 networks, as well as of machine translation techniques such as BIBREF2, the dominant approach in video captioning is currently based on sequence learning using an encoder-decoder framework.
Question: How big is the dataset used?
Paper: On the Relationship between Self-Attention and Convolutional Layers
Recent advances in Natural Language Processing (NLP) are largely attributed to the rise of the transformer BIBREF1. Pre-trained to solve an unsupervised task on large corpora of text, transformer-based architectures, such as GPT-2 BIBREF2, BERT BIBREF3 and Transformer-XL BIBREF4, seem to possess the capacity to learn the underlying structure of text and, as a consequence, to learn representations that generalize across tasks. The key difference between transformers and previous methods, such as recurrent neural networks BIBREF5 and convolutional neural networks (CNN), is that the former can simultaneously attend to every word of their input sequence. This is made possible thanks to the attention mechanism—originally introduced in Neural Machine Translation to better handle long-range dependencies BIBREF6. With self-attention in particular, the similarity of two words in a sequence is captured by an attention score measuring the distance of their representations. The representation of each word is then updated based on those words whose attention score is highest.
Question: How they prove that multi-head self-attention is at least as powerful as convolution layer?