z-uo/qasper-squad huggingface dataset:
Paper: Can neural networks understand monotonicity reasoning?
Natural language inference (NLI), also known as recognizing textual entailment (RTE), has been proposed as a benchmark task for natural language understanding. Given a premise $P$ and a hypothesis $H$ , the task is to determine whether the premise semantically entails the hypothesis BIBREF0 . A number of recent works attempt to test and analyze what type of inferences an NLI model may be performing, focusing on various types of lexical inferences BIBREF1 , BIBREF2 , BIBREF3 and logical inferences BIBREF4 , BIBREF5 .
Question: Do they release MED?
Paper: Can neural networks understand monotonicity reasoning?
As an example of a monotonicity inference, consider the example with the determiner every in ( "Monotonicity" ); here the premise $P$ entails the hypothesis $H$ .
Question: What NLI models do they analyze?
Paper: Can neural networks understand monotonicity reasoning?
To create monotonicity inference problems, we should satisfy three requirements: (a) detect the monotonicity operators and their arguments; (b) based on the syntactic structure, induce the polarity of the argument positions; and (c) replace the phrase in the argument position with a more general or specific phrase in natural and various ways (e.g., by using lexical knowledge or logical connectives). For (a) and (b), we first conduct polarity computation on a syntactic structure for each sentence, and then select premises involving upward/downward expressions.
Question: How do they define upward and downward reasoning?
Answer: Upward reasoning is defined as going from one specific concept to a more general one. Downward reasoning is defined as the opposite, going from a general concept to one that is more specific.
Paper: Can neural networks understand monotonicity reasoning?
We also collect monotonicity inference problems from previous manually curated datasets and linguistics publications. The motivation is that previous linguistics publications related to monotonicity reasoning are expected to contain well-designed inference problems, which might be challenging problems for NLI models.
Question: What is monotonicity reasoning?
Paper: Enriching Existing Conversational Emotion Datasets with Dialogue Acts using Neural Annotators.
With the growing demand for human-computer/robot interaction systems, detecting the emotional state of the user can heavily benefit a conversational agent to respond at an appropriate emotional level. Emotion recognition in conversations has proven important for potential applications such as response recommendation or generation, emotion-based text-to-speech, personalisation, etc. Human emotional states can be expressed verbally and non-verbally BIBREF0, BIBREF1, however, while building an interactive dialogue system, the interface needs dialogue acts. A typical dialogue system consists of a language understanding module which requires to determine the meaning of and intention in the human input utterances BIBREF2, BIBREF3. Also, in discourse or conversational analysis, dialogue acts are the main linguistic features to consider BIBREF4. A dialogue act provides an intention and performative function in an utterance of the dialogue. For example, it can infer a user's intention by distinguishing Question, Answer, Request, Agree/Reject, etc. and performative functions such as Acknowledgement, Conversational-opening or -closing, Thanking, etc. The dialogue act information together with emotional states can be very useful for a spoken dialogue system to produce natural interaction BIBREF5.
Question: What other relations were found in the datasets?
Paper: Enriching Existing Conversational Emotion Datasets with Dialogue Acts using Neural Annotators.
There are two emotion taxonomies: (1) discrete emotion categories (DEC) and (2) fined-grained dimensional basis of emotion states (DBE). The DECs are Joy, Sadness, Fear, Surprise, Disgust, Anger and Neutral; identified by Ekman et al. ekman1987universalemos. The DBE of the emotion is usually elicited from two or three dimensions BIBREF1, BIBREF11, BIBREF12. A two-dimensional model is commonly used with Valence and Arousal (also called activation), and in the three-dimensional model, the third dimension is Dominance. IEMOCAP is annotated with all DECs and two additional emotion classes, Frustration and Excited. IEMOCAP is also annotated with three DBE, that includes Valance, Arousal and Dominance BIBREF6. MELD BIBREF8, which is an evolved version of the Emotionlines dataset developed by BIBREF13, is annotated with exactly 7 DECs and sentiments (positive, negative and neutral).
Question: How does the ensemble annotator extract the final label?
Paper: Enriching Existing Conversational Emotion Datasets with Dialogue Acts using Neural Annotators.
There have been many taxonomies for dialogue acts: speech acts BIBREF14 refer to the utterance, not only to present information but to the action at is performed. Speech acts were later modified into five classes (Assertive, Directive, Commissive, Expressive, Declarative) BIBREF15. There are many such standard taxonomies and schemes to annotate conversational data, and most of them follow the discourse compositionality. These schemes have proven their importance for discourse or conversational analysis BIBREF16. During the increased development of dialogue systems and discourse analysis, the standard taxonomy was introduced in recent decades, called Dialogue Act Markup in Several Layers (DAMSL) tag set. According to DAMSL, each DA has a forward-looking function (such as Statement, Info-request, Thanking) and a backwards-looking function (such as Accept, Reject, Answer) BIBREF17.
Question: How were dialogue act labels defined?
Paper: Enriching Existing Conversational Emotion Datasets with Dialogue Acts using Neural Annotators.
We adopted the neural architectures based on Bothe et al. bothe2018discourse where two variants are: non-context model (classifying at utterance level) and context model (recognizing the dialogue act of the current utterance given a few preceding utterances). From conversational analysis using dialogue acts in Bothe et al. bothe2018interspeech, we learned that the preceding two utterances contribute significantly to recognizing the dialogue act of the current utterance. Hence, we adapt this setting for the context model and create a pool of annotators using recurrent neural networks (RNNs). RNNs can model the contextual information in the sequence of words of an utterance and in the sequence of utterances of a dialogue. Each word in an utterance is represented with a word embedding vector of dimension 1024. We use the word embedding vectors from pre-trained ELMo (Embeddings from Language Models) embeddings BIBREF22. We have a pool of five neural annotators as shown in Figure FIGREF6. Our online tool called Discourse-Wizard is available to practice automated dialogue act labeling. In this tool we use the same neural architectures but model-trained embeddings (while, in this work we use pre-trained ELMo embeddings, as they are better performant but computationally and size-wise expensive to be hosted in the online tool). The annotators are:
Question: How many models were used?
Paper: Synchronising audio and ultrasound by learning cross-modal embeddings
Ultrasound tongue imaging (UTI) is a non-invasive way of observing the vocal tract during speech production BIBREF0 . Instrumental speech therapy relies on capturing ultrasound videos of the patient's tongue simultaneously with their speech audio in order to provide a diagnosis, design treatments, and measure therapy progress BIBREF1 . The two modalities must be correctly synchronised, with a minimum shift of INLINEFORM0 45ms if the audio leads and INLINEFORM1 125ms if the audio lags, based on synchronisation standards for broadcast audiovisual signals BIBREF2 . Errors beyond this range can render the data unusable – indeed, synchronisation errors do occur, resulting in significant wasted effort if not corrected. No mechanism currently exists to automatically correct these errors, and although manual synchronisation is possible in the presence of certain audiovisual cues such as stop consonants BIBREF3 , it is time consuming and tedious.
Question: Do they compare their neural network against any other model?
Paper: Synchronising audio and ultrasound by learning cross-modal embeddings
Ultrasound and audio are recorded using separate components, and hardware synchronisation is achieved by translating information from the visual signal into audio at recording time. Specifically, for every ultrasound frame recorded, the ultrasound beam-forming unit releases a pulse signal, which is translated by an external hardware synchroniser into an audio pulse signal and captured by the sound card BIBREF6 , BIBREF7 . Synchronisation is achieved by aligning the ultrasound frames with the audio pulse signal, which is already time-aligned with the speech audio BIBREF8 .
Question: Do they annotate their own dataset or use an existing one?
Answer: Use an existing one
Paper: Synchronising audio and ultrasound by learning cross-modal embeddings
Speech audio is generated by articulatory movement and is therefore fundamentally correlated with other manifestations of this movement, such as lip or tongue videos BIBREF10 . An alternative to the hardware approach is to exploit this correlation to find the offset. Previous approaches have investigated the effects of using different representations and feature extraction techniques on finding dimensions of high correlation BIBREF11 , BIBREF12 , BIBREF13 . More recently, neural networks, which learn features directly from input, have been employed for the task. SyncNet BIBREF4 uses a two-stream neural network and self-supervision to learn cross-modal embeddings, which are then used to synchronise audio with lip videos. It achieves near perfect accuracy ( INLINEFORM0 99 INLINEFORM1 ) using manual evaluation where lip-sync error is not detectable to a human. It has since been extended to use different sample creation methods for self-supervision BIBREF5 , BIBREF14 and different training objectives BIBREF14 . We adopt the original approach BIBREF4 , as it is both simpler and significantly less expensive to train than the more recent variants.
Question: Does their neural network predict a single offset in a recording?
Paper: Synchronising audio and ultrasound by learning cross-modal embeddings
Videos of lip movement can be obtained from various sources including TV, films, and YouTube, and are often cropped to include only the lips BIBREF4 . UTI data, on the other hand, is recorded in clinics by trained therapists BIBREF15 . An ultrasound probe placed under the chin of the patient captures the midsaggital view of their oral cavity as they speak. UTI data consists of sequences of 2D matrices of raw ultrasound reflection data, which can be interpreted as greyscale images BIBREF15 . There are several challenges specifically associated with UTI data compared with lip videos, which can potentially lower the performance of models relative to results reported on lip video data. These include:
Question: What kind of neural network architecture do they use?
Answer: CNN
Paper: Basic tasks of sentiment analysis
School of Computer Science and Engineering, Nanyang Technological University, Singapore
Question: How are aspects identified in aspect extraction?
Paper: Generalisation in Named Entity Recognition: A Quantitative Analysis
Named entity recognition and classification (NERC, short NER), the task of recognising and assigning a class to mentions of proper names (named entities, NEs) in text, has attracted many years of research BIBREF0 , BIBREF1 , analyses BIBREF2 , starting from the first MUC challenge in 1995 BIBREF3 . Recognising entities is key to many applications, including text summarisation BIBREF4 , search BIBREF5 , the semantic web BIBREF6 , topic modelling BIBREF7 , and machine translation BIBREF8 , BIBREF9 .
Question: What web and user-generated NER datasets are used for the analysis?
Answer: MUC, CoNLL, ACE, OntoNotes, MSM, Ritter, UMBC
Paper: wav2vec: Unsupervised Pre-training for Speech Recognition
Current state of the art models for speech recognition require large amounts of transcribed audio data to attain good performance BIBREF1 . Recently, pre-training of neural networks has emerged as an effective technique for settings where labeled data is scarce. The key idea is to learn general representations in a setup where substantial amounts of labeled or unlabeled data is available and to leverage the learned representations to improve performance on a downstream task for which the amount of data is limited. This is particularly interesting for tasks where substantial effort is required to obtain labeled data, such as speech recognition.
Question: Which unlabeled data do they pretrain with?
Answer: 1000 hours of WSJ audio data
Paper: wav2vec: Unsupervised Pre-training for Speech Recognition
Given an audio signal as input, we optimize our model (§ SECREF3 ) to predict future samples from a given signal context. A common problem with these approaches is the requirement to accurately model the data distribution INLINEFORM0 , which is challenging. We avoid this problem by first encoding raw speech samples INLINEFORM1 into a feature representation INLINEFORM2 at a lower temporal frequency and then implicitly model a density function INLINEFORM3 similar to BIBREF15 .
Question: How many convolutional layers does their model have?
Answer: wav2vec has 12 convolutional layers
Paper: wav2vec: Unsupervised Pre-training for Speech Recognition
Our model takes raw audio signal as input and then applies two networks. The encoder network embeds the audio signal in latent space and the context network combines multiple time-steps of the encoder to obtain contextualized representations (Figure FIGREF2 ). Both networks are then used to compute the objective function (§ SECREF4 ).
Question: Do they explore how much traning data is needed for which magnitude of improvement for WER?
Paper: Cross-lingual, Character-Level Neural Morphological Tagging
State-of-the-art morphological taggers require thousands of annotated sentences to train. For the majority of the world's languages, however, sufficient, large-scale annotation is not available and obtaining it would often be infeasible. Accordingly, an important road forward in low-resource NLP is the development of methods that allow for the training of high-quality tools from smaller amounts of data. In this work, we focus on transfer learning—we train a recurrent neural tagger for a low-resource language jointly with a tagger for a related high-resource language. Forcing the models to share character-level features among the languages allows large gains in accuracy when tagging the low-resource languages, while maintaining (or even improving) accuracy on the high-resource language.
Question: How are character representations from various languages joint?
Paper: Cross-lingual, Character-Level Neural Morphological Tagging
Many languages in the world exhibit rich inflectional morphology: the form of individual words mutates to reflect the syntactic function. For example, the Spanish verb soñar will appear as sueño in the first person present singular, but soñáis in the second person present plural, depending on the bundle of syntaco-semantic attributes associated with the given form (in a sentential context). For concreteness, we list a more complete table of Spanish verbal inflections in tab:paradigm. [author=Ryan,color=purple!40,size=,fancyline,caption=,]Notation in table is different. Note that some languages, e.g. the Northeastern Caucasian language Archi, display a veritable cornucopia of potential forms with the size of the verbal paradigm exceeding 10,000 BIBREF10 .
Question: On which dataset is the experiment conducted?
Paper: Neural Cross-Lingual Relation Extraction Based on Bilingual Word Embedding Mapping
Relation extraction (RE) is an important information extraction task that seeks to detect and classify semantic relationships between entities like persons, organizations, geo-political entities, locations, and events. It provides useful information for many NLP applications such as knowledge base construction, text mining and question answering. For example, the entity Washington, D.C. and the entity United States have a CapitalOf relationship, and extraction of such relationships can help answer questions like “What is the capital city of the United States?"
Question: Do they train their own RE model?
Paper: Neural Cross-Lingual Relation Extraction Based on Bilingual Word Embedding Mapping
We summarize the main steps of our neural cross-lingual RE model transfer approach as follows.
Question: How big are the datasets?
Answer: In-house dataset consists of  3716 documents 
ACE05 dataset consists of  1635 documents
Paper: Neural Cross-Lingual Relation Extraction Based on Bilingual Word Embedding Mapping
In recent years, vector representations of words, known as word embeddings, become ubiquitous for many NLP applications BIBREF12, BIBREF13, BIBREF14.
Question: What languages do they experiment on?
Paper: Neural Cross-Lingual Relation Extraction Based on Bilingual Word Embedding Mapping
To build monolingual word embeddings for the source and target languages, we use a variant of the Continuous Bag-of-Words (CBOW) word2vec model BIBREF13.
Question: What datasets are used?
Paper: Visual Natural Language Query Auto-Completion for Estimating Instance Probabilities
This work focuses on the problem of finding objects in an image based on natural language descriptions. Existing solutions take into account both the image and the query BIBREF0, BIBREF1, BIBREF2. In our problem formulation, rather than having the entire text, we are given only a prefix of the text which requires completing the text based on a language model and the image, and finding a relevant object in the image. We decompose the problem into three components: (i) completing the query from text prefix and an image; (ii) estimating probabilities of objects based on the completed text, and (iii) segmenting and classifying all instances in the image. We combine, extend, and modify state of the art components: (i) we extend a FactorCell LSTM BIBREF3, BIBREF4 which conditionally completes text to complete a query from both a text prefix and an image; (ii) we fine tune a BERT embedding to compute instance probabilities from a complete sentence, and (iii) we use Mask-RCNN BIBREF5 for instance segmentation.
Question: How better does auto-completion perform when using both language and vision than only language?
Paper: Visual Natural Language Query Auto-Completion for Estimating Instance Probabilities
Figure FIGREF2 shows the architecture of our approach. First, we extract image features with a pre-trained CNN. We incorporate the image features into a modified FactorCell LSTM language model along with the user query prefix to complete the query. The completed query is then fed into a fine-tuned BERT embedding to estimate instance probabilities, which in turn are used for instance selection.
Question: How big is data provided by this research?
Paper: Visual Natural Language Query Auto-Completion for Estimating Instance Probabilities
We utilize the FactorCell (FC) adaptation of an LSTM with coupled input and forget gates BIBREF4 to autocomplete queries. The FactorCell is an LSTM with a context-dependent weight matrix $\mathbf {W^{\prime }} = \mathbf {W} + \mathbf {A}$ in place of $\mathbf {W}$. Given a character embedding $w_t \in \mathbb {R}^e$, a previous hidden state $h_{t-1} \in \mathbb {R}^h$ , the adaptation matrix, $\mathbf {A}$, is formed by taking the product of the context, c, with two basis tensors $\mathbf {Z_L} \in \mathbb {R}^{m\times (e+h)\times r}$ and $\mathbf {Z_R} \in \mathbb {R}^{r\times h \times m}$.
Question: How they complete a user query prefix conditioned upon an image?
Paper: Translating Navigation Instructions in Natural Language to a High-Level Plan for Behavioral Robot Navigation
Enabling robots to follow navigation instructions in natural language can facilitate human-robot interaction across a variety of applications. For instance, within the service robotics domain, robots can follow navigation instructions to help with mobile manipulation BIBREF0 and delivery tasks BIBREF1 .
Question: Did the collection process use a WoZ method?
Paper: Translating Navigation Instructions in Natural Language to a High-Level Plan for Behavioral Robot Navigation
This section reviews relevant prior work on following navigation instructions. Readers interested in an in-depth review of methods to interpret spatial natural language for robotics are encouraged to refer to BIBREF11 .
Question: By how much did their model outperform the baseline?
Paper: Translating Navigation Instructions in Natural Language to a High-Level Plan for Behavioral Robot Navigation
Our goal is to translate navigation instructions in text form into a sequence of behaviors that a robot can execute to reach a desired destination from a known start location. We frame this problem under a behavioral approach to indoor autonomous navigation BIBREF5 and assume that prior knowledge about the environment is available for the translation task. This prior knowledge is a topological map, in the form of a behavioral navigation graph (Fig. FIGREF4 (b)). The nodes of the graph correspond to semantically-meaningful locations for the navigation task, and its directed edges are visuo-motor behaviors that a robot can use to move between nodes. This formulation takes advantage of the rich semantic structure behind man-made environments, resulting in a compact route representation for robot navigation.
Question: What baselines did they compare their model with?
Answer: the baseline where path generation uses a standard sequence-to-sequence model augmented with attention mechanism and path verification uses depth-first search
Paper: Translating Navigation Instructions in Natural Language to a High-Level Plan for Behavioral Robot Navigation
We view the behavioral graph INLINEFORM0 as a knowledge base that encodes a set of navigational rules as triplets INLINEFORM1 , where INLINEFORM2 and INLINEFORM3 are adjacent nodes in the graph, and the edge INLINEFORM4 is an executable behavior to navigate from INLINEFORM5 to INLINEFORM6 . In general, each behaviors includes a list of relevant navigational attributes INLINEFORM7 that the robot might encounter when moving between nodes.
Question: What was the performance of their model?
Answer: For test-repeated set, EM score of 61.17, F1 of 93.54, ED of 0.75 and GM of 61.36. For test-new set, EM score of 41.71, F1 of 91.02, ED of 1.22 and GM of 41.81
Paper: Translating Navigation Instructions in Natural Language to a High-Level Plan for Behavioral Robot Navigation
We leverage recent advances in deep learning to translate natural language instructions to a sequence of navigation behaviors in an end-to-end fashion. Our proposed model builds on the sequence-to-sequence translation model of BIBREF23 , which computes a soft-alignment between a source sequence (natural language instructions in our case) and the corresponding target sequence (navigation behaviors).
Question: What evaluation metrics are used?
Answer: exact match, f1 score, edit distance and goal match
Paper: Translating Navigation Instructions in Natural Language to a High-Level Plan for Behavioral Robot Navigation
We created a new dataset for the problem of following navigation instructions under the behavioral navigation framework of BIBREF5 . This dataset was created using Amazon Mechanical Turk and 100 maps of simulated indoor environments, each with 6 to 65 rooms. To the best of our knowledge, this is the first benchmark for comparing translation models in the context of behavioral robot navigation.
Question: Did the authors use a crowdsourcing platform?