z-uo/qasper-squad huggingface dataset:
Paper: Char-RNN and Active Learning for Hashtag Segmentation
To our knowledge there is no available dataset for hashtag segmentation in Russian, so we faced the need to create our own dataset. Our approach to the dataset creation was twofold: the training data was created from social network texts by selecting frequent $n$-grams and generating hashtags following some hashtag patterns. The test dataset consists of real hashtags collected from vk.com (a Russian social network) and were segmented manually.
Question: Which languages are used in the paper?
Paper: Generating Narrative Text in a Switching Dynamical System
In this section, we give a brief overview of Switching Dynamical systems and how they can be used to capture both a scaffold of the narrative as well as the narrative dynamics. We then describe in detail the components of our model and its relation to existing models.
Question: What metrics are used for evaluation?
Paper: Generating Narrative Text in a Switching Dynamical System
The specifics of the narrative (characters, setting, etc.), will differ between stories, but as BIBREF0 notes, the way they transition to the next point in the narrative (what we refer to as “narrative dynamics") is often shared. Let's say that, as done often, we represent the `narrative specifics' at time step $i$ with a latent vector $Z_i$. A natural way to explicitly model how this state evolves over time that fits with the above observation is as a Linear Dynamical System:
Question: What baselines are used?
Paper: Learning Explicit and Implicit Structures for Targeted Sentiment Analysis
Accepted as a long paper in EMNLP 2019 (Conference on Empirical Methods in Natural Language Processing).
Question: Which model is used to capture the implicit structure?
Paper: Learning Explicit and Implicit Structures for Targeted Sentiment Analysis
Our objective is to design a model to extract targets as well as their associated targeted sentiments for a given sentence in a joint manner. As we mentioned before, we believe that both explicit and implicit structures are crucial for building a successful model for TSA. Specifically, we first present an approach to learn flexible explicit structures based on latent CRF, and next present an approach to efficiently learn the rich implicit structures for exponentially many possible combinations of targets.
Question: How is the robustness of the model evaluated?
Paper: Learning Explicit and Implicit Structures for Targeted Sentiment Analysis
Motivated by BIBREF11, we design an approach based on latent CRF to model flexible sentiment spans to capture better explicit structures in the output space. To do so, we firstly integrate target and targeted sentiment information into a label sequence by using 3 types of tags in our EI model: $\mathbf {B}_p$, $\mathbf {A}_p$, and $\mathbf {E}_{\epsilon ,p}$, where $p \in \lbrace +, -, 0\rbrace $ indicates the sentiment polarity and $\epsilon \in \lbrace \textit {B,M,E,S}\rbrace $ denotes the BMES tagging scheme. We explain the meaning of each type of tags as follows.
Question: How is the effectiveness of the model evaluated?
Paper: End-to-End Information Extraction without Token-Level Supervision
Humans spend countless hours extracting structured machine readable information from unstructured information in a multitude of domains. Promising to automate this, information extraction (IE) is one of the most sought-after industrial applications of natural language processing. However, despite substantial research efforts, in practice, many applications still rely on manual effort to extract the relevant information.
Question: Do they assume sentence-level supervision?
Paper: Recurrently Controlled Recurrent Networks
Recurrent neural networks (RNNs) live at the heart of many sequence modeling problems. In particular, the incorporation of gated additive recurrent connections is extremely powerful, leading to the pervasive adoption of models such as Gated Recurrent Units (GRU) BIBREF0 or Long Short-Term Memory (LSTM) BIBREF1 across many NLP applications BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 . In these models, the key idea is that the gating functions control information flow and compositionality over time, deciding how much information to read/write across time steps. This not only serves as a protection against vanishing/exploding gradients but also enables greater relative ease in modeling long-range dependencies.
Question: By how much do they outperform BiLSTMs in Sentiment Analysis?
Answer: Proposed RCRN outperforms ablative baselines BiLSTM by +2.9% and 3L-BiLSTM by +1.1% on average across 16 datasets.
Paper: Recurrently Controlled Recurrent Networks
RNN variants such as LSTMs and GRUs are ubiquitous and indispensible building blocks in many NLP applications such as question answering BIBREF12 , BIBREF9 , machine translation BIBREF2 , entailment classification BIBREF13 and sentiment analysis BIBREF14 , BIBREF15 . In recent years, many RNN variants have been proposed, ranging from multi-scale models BIBREF16 , BIBREF17 , BIBREF18 to tree-structured encoders BIBREF19 , BIBREF20 . Models that are targetted at improving the internals of the RNN cell have also been proposed BIBREF21 , BIBREF22 . Given the importance of sequence encoding in NLP, the design of effective RNN units for this purpose remains an active area of research.
Question: Does their model have more parameters than other models?
Paper: End-to-End Multi-View Networks for Text Classification
State-of-the-art deep neural networks leverage task-specific architectures to develop hierarchical representations of their input, with each layer building a refined abstraction of the layer that came before it BIBREF0 . For text classification, one can think of this as a single reader building up an increasingly refined understanding of the content. In a departure from this philosophy, we propose a divide-and-conquer approach, where a team of readers each focus on different aspects of the text, and then combine their representations to make a joint decision.
Question: what state of the accuracy did they obtain?
Answer: 51.5
Paper: End-to-End Multi-View Networks for Text Classification
The MVN architecture is depicted in Figure FIGREF1 . First, individual selection vectors INLINEFORM0 are created, each formed by a distinct softmax weighted sum over the word vectors of the input text. Next, these selections are sequentially transformed into views INLINEFORM1 , with each view influencing the views that come after it. Finally, all views are concatenated and fed into a two-layer perceptron for classification.
Question: what models did they compare to?
Answer: High-order CNN, Tree-LSTM, DRNN, DCNN, CNN-MC, NBoW and SVM
Paper: End-to-End Multi-View Networks for Text Classification
Each selection INLINEFORM0 is constructed by focusing on a different subset of words from the original text, as determined by a softmax weighted sum BIBREF8 . Given a piece of text with INLINEFORM1 words, we represent it as a bag-of-words feature matrix INLINEFORM2 INLINEFORM3 . Each row of the matrix corresponds to one word, which is represented by a INLINEFORM4 -dimensional vector, as provided by a learned word embedding table. The selection INLINEFORM5 for the INLINEFORM6 view is the softmax weighted sum of features: DISPLAYFORM0 
Question: which benchmark tasks did they experiment on?
Answer:  They used Stanford Sentiment Treebank benchmark for sentiment classification task and   AG English news corpus for the text classification task.
Paper: On the Robustness of Projection Neural Networks For Efficient Text Representation: An Empirical Study
At the core of Natural Language Processing (NLP) neural models are pre-trained word embeddings like Word2Vec BIBREF0, GloVe BIBREF1 and ELMo BIBREF2. They help initialize the neural models, lead to faster convergence and have improved performance for numerous application such as Question Answering BIBREF3, Summarization BIBREF4, Sentiment Analysis BIBREF5. While word embeddings are powerful in unlimited constraints such as computation power and compute resources, it becomes challenging to deploy them to on-device due to their huge size.
Question: Are recurrent neural networks trained on perturbed data?
Paper: On the Robustness of Projection Neural Networks For Efficient Text Representation: An Empirical Study
The Projection function, $\mathbb {P}$ (Figure FIGREF1), BIBREF9 used in SGNN models BIBREF6 extracts token (or character) n-gram & skip-gram features from a raw input text, $\textbf {x}$ and dynamically generates a binary projection representation, $\mathbb {P}(\mathbf {x}) \in [0,1]^{T.d}$ after a Locality-Sensitive Hashing (LSH) based transformation, $\mathbb {L}$ as in
Question: How does their perturbation algorihm work?
Paper: Experiments in Cuneiform Language Identification
As discussed in a recent survey BIBREF0 , discriminating between similar languages, national language varieties, and dialects is an important challenge faced by state-of-the-art language identification systems. The topic has attracted more and more attention from the CL/NLP community in recent years with publications on similar languages of the Iberian peninsula BIBREF1 , and varieties and dialects of several languages such as Greek BIBREF2 and Romanian BIBREF3 to name a few.
Question: Which language is divided into six dialects in the task mentioned in the paper?
Paper: Experiments in Cuneiform Language Identification
Since its first edition in 2014, shared tasks on similar language and dialect identification have been organized together with the VarDial workshop co-located with international conferences such as COLING, EACL, and NAACL. The first and most well-attended of these competitions was the Discrminating between Similar Languages (DSL) shared task which has been organized between 2014 and 2017 BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 . The DSL provided the first benchmark for evaluation of language identification systems developed for similar languages and language varieties using the DSL Corpus Collection (DSLCC) BIBREF13 , a multilingual benchmarked dataset compiled for this purpose. In 2017 and 2018, VarDial featured evaluation campaigns with multiple shared tasks not only on language and dialect identification but also on other NLP tasks related to language and dialect variation (e.g. morphosyntactic tagging, and cross-lingual dependency parsing). With the exception of the DSL, the language and dialect identification competitions organized at VarDial focused on groups of dialects from the same language such as Arabic (ADI shared task) and German (GDI shared task).
Question: What is one of the first writing systems in the world?
Paper: Brundlefly at SemEval-2016 Task 12: Recurrent Neural Networks vs. Joint Inference for Clinical Temporal Information Extraction
This work discusses two information extraction systems for identifying temporal information in clinical text, submitted to SemEval-2016 Task 12 : Clinical TempEval BIBREF0 . We participated in tasks from both phases: (1) identifying text spans of time and event mentions; and (2) predicting relations between clinical events and document creation time.
Question: How do they obtain distant supervision rules for predicting relations?
Paper: Brundlefly at SemEval-2016 Task 12: Recurrent Neural Networks vs. Joint Inference for Clinical Temporal Information Extraction
Phase 1 of the challenge required parsing clinical documents to identify Timex3 and Event temporal entity mentions in text. Timex3 entities are expressions of time, ranging from concrete dates to phrases describing intervals like “the last few months." Event entities are broadly defined as anything relevant to a patient's clinical timeline, e.g., diagnoses, illnesses, procedures. Entity mentions are tagged using a document collection of clinic and pathology notes from the Mayo Clinic called the THYME (Temporal History of Your Medical Events) corpus BIBREF2 .
Question: Which structured prediction approach do they adopt for temporal entity extraction?
Paper: Improving Information Retrieval Results for Persian Documents using FarsNet
In Information Retrieval (IR), the searched query has always been an integral part. When a user enters a query in the information retrieval system the keywords they use might be different from the ones used in the documents or they might be expressing it in a different form. Considering this situation, the information retrieval systems should be intelligent and provide the requested information to the user. According to Spink (2001), each user in the web uses 2.4 words in their query; having said that, the probability of the input query being close to those of the documents is extremely low [22]. The latest algorithms implement query indexing techniques and covers only the user's history of search. This simply brings the problem of keywords mismatch; the queries entered by user don't match with the ones in the documents, this problem is called the lexical problem. The lexical problem originates from synonymy. Synonymy is the state that two or more words have the same meaning. Thus, expanding the query by enriching each word with their synonyms will enhance the IR results.
Question: Which evaluation metric has been measured?
Paper: Improving Information Retrieval Results for Persian Documents using FarsNet
One of the first researchers who used the method for indexing was Maron (1960) [11]. Aforementioned paper described a meticulous and novel method to retrieve information from the books in the library. This paper is also one of the pioneers of the relevance and using probabilistic indexing. Relevance feedback is the process to involve user in the retrieved documents. It was mentioned in Rocchio (1971) [15], Ide (1971) [8], and Salton (1971) [19]. In the Relevance feedback the user's opinion for the retrieved documents is asked, then by the help of the user's feedbacks the relevance and irrelevance of the documents is decided. In the later researches, relevance feedback has been used in combination with other methods. For instance, Rahimi (2014) [14] used relevance feedback and Latent Semantic Analysis (LSA) to increase user's satisfaction. Other researches regarding the usage of relevance feedback are Salton (1997) [18], Rui (1997) [16], and Rui (1998) [17].
Question: What is the WordNet counterpart for Persian?
Paper: Excitation-based Voice Quality Analysis and Modification
Since early times of computer-based speech synthesis research, voice quality (the perceived timbre of speech) analysis/modification has attracted interest of researchers BIBREF0. The topic of voice quality analysis finds application in various areas of speech processing such as high-quality parametric speech synthesis, expressive/emotional speech synthesis, speaker identification, emotion recognition, prosody analysis, speech therapy. Due to availability of reviews such as BIBREF1 and space limitations, a review of voice quality analysis methods will not be presented here.
Question: What large corpus is used for experiments?
Paper: TLT-school: a Corpus of Non Native Children Speech
We have acquired large sets of both written and spoken data during the implementation of campaigns aimed at assessing the proficiency, at school, of Italian pupils learning both German and English. Part of the acquired data has been included in a corpus, named "Trentino Language Testing" in schools (TLT-school), that will be described in the following.
Question: Are any of the utterances ungrammatical?
Paper: TLT-school: a Corpus of Non Native Children Speech
In Trentino, an autonomous region in northern Italy, there is a series of evaluation campaigns underway for testing L2 linguistic competence of Italian students taking proficiency tests in both English and German. A set of three evaluation campaigns is underway, two having been completed in 2016 and 2018, and a final one scheduled in 2020. Note that the “TLT-school” corpus refers to only the 2018 campaign, that was split in two parts: 2017 try-out data set (involving about 500 pupils) and the actual 2018 data (about 2500 pupils). Each of the three campaigns (i.e. 2016, 2018 and 2020) involves about 3000 students ranging from 9 to 16 years, belonging to four different school grade levels and three proficiency levels (A1, A2, B1). The schools involved in the evaluations are located in most part of the Trentino region, not only in its main towns; Table highlights some information about the pupils that took part to the campaigns. Several tests, aimed at assessing the language learning skills of the students, were carried out by means of multiple-choice questions, which can be evaluated automatically. However, a detailed linguistic evaluation cannot be performed without allowing the students to express themselves in both written sentences and spoken utterances, which typically require the intervention of human experts to be scored.
Question: How is the proficiency score calculated?
Answer: They used 6 indicators for proficiency (same for written and spoken) each marked by bad, medium or good by one expert.
Paper: TLT-school: a Corpus of Non Native Children Speech
The speaking part of the proficiency tests in 2017/2018 consists of 47 question prompts provided in written form: 24 in English and 23 in German, divided according to CEFR levels. Apart from A1 level, which differs in the number of questions (11 for English; 10 for German), both English and German A2 and B1 levels have respectively 6 and 7 questions each. As for A1 level, the first four introductory questions are the same (How old are you?, Where do you live?, What are your hobbies?, Wie alt bist du?, Wo wohnst du?, Was sind deine Hobbys?) or slightly different (What's your favourite pet?, Welche Tiere magst du?) in both languages, whereas the second part of the test puts the test-takers in the role of a customer in a pizzeria (English) or in a bar (German).
Question: What proficiency indicators are used to the score the utterances?
Answer: 6 indicators:
- lexical richness
- pronunciation and fluency
- syntactical correctness
- fulfillment of delivery
- coherence and cohesion
- communicative, descriptive, narrative skills
Paper: TLT-school: a Corpus of Non Native Children Speech
Table reports some statistics extracted from the written data collected so far. In this table, the number of pupils taking part in the English and German evaluation is reported, along with the number of sentences and tokens, identified as character sequences bounded by spaces.
Question: What accuracy is achieved by the speech recognition system?
Answer: Accuracy not available: WER results are reported 42.6 German, 35.9 English
Paper: TLT-school: a Corpus of Non Native Children Speech
Table reports some statistics extracted from the acquired spoken data. Speech was recorded in classrooms, whose equipment depended on each school. In general, around 20 students took the test together, at the same time and in the same classrooms, so it is quite common that speech of mates or teachers often overlaps with the speech of the student speaking in her/his microphone. Also, the type of microphone depends on the equipment of the school. On average, the audio signal quality is nearly good, while the main problem is caused by a high percentage of extraneous speech. This is due to the fact that organisers decided to use a fixed duration - which depends on the question - for recording spoken utterances, so that all the recordings for a given question have the same length. However, while it is rare that a speaker has not enough time to answer, it is quite common that, especially after the end of the utterance, some other speech (e.g. comments, jokes with mates, indications from the teachers, etc.) is captured. In addition, background noise is often present due to several sources (doors, steps, keyboard typing, background voices, street noises if the windows are open, etc). Finally, it has to be pointed out that many answers are whispered and difficult to understand.
Question: How is the speech recognition system evaluated?
Answer: Speech recognition system is evaluated using WER metric.
Paper: TLT-school: a Corpus of Non Native Children Speech
In order to create both an adaptation and an evaluation set for ASR, we manually transcribed part of the 2017 data sets. We defined an initial set of guidelines for the annotation, which were used by 5 researchers to manually transcribe about 20 minutes of audio data. This experience led to a discussion, from which a second set of guidelines originated, aiming at reaching a reasonable trade-off between transcription accuracy and speed. As a consequence, we decided to apply the following transcription rules:
Question: How many of the utterances are transcribed?
Answer: Total number of transcribed utterances including Train and Test for both Eng and Ger language is 5562 (2188 cleaned)
Paper: TLT-school: a Corpus of Non Native Children Speech
From the above description it appears that the corpus can be effectively used in many research directions.
Question: How many utterances are in the corpus?
Answer: Total number of utterances available is: 70607 (37344 ENG + 33263 GER)
Paper: Efficient Summarization with Read-Again and Copy Mechanism
Encoder-decoder models have been widely used in sequence to sequence tasks such as machine translation ( BIBREF0 , BIBREF1 ). They consist of an encoder which represents the whole input sequence with a single feature vector. The decoder then takes this representation and generates the desired output sequence. The most successful models are LSTM and GRU as they are much easier to train than vanilla RNNs.
Question: By how much does their model outperform both the state-of-the-art systems?
Answer: w.r.t Rouge-1 their model outperforms by 0.98% and w.r.t Rouge-L their model outperforms by 0.45%
Paper: Efficient Summarization with Read-Again and Copy Mechanism
In the past few years, there has been a lot of work on extractive summarization, where a summary is created by composing words or sentences from the source text. Notable examples are BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 and BIBREF14 . As a consequence of their extractive nature the summary is restricted to words (sentences) in the source text.
Question: What is the state-of-the art?
Answer: neural attention model with a convolutional encoder with an RNN decoder and RNN encoder-decoder
Paper: Exploiting Task-Oriented Resources to Learn Word Embeddings for Clinical Abbreviation Expansion
Abbreviations and acronyms appear frequently in the medical domain. Based on a popular online knowledge base, among the 3,096,346 stored abbreviations, 197,787 records are medical abbreviations, ranked first among all ten domains. An abbreviation can have over 100 possible explanations even within the medical domain. Medical record documentation, the authors of which are mainly physicians, other health professionals, and domain experts, is usually written under the pressure of time and high workload, requiring notation to be frequently compressed with shorthand jargon and acronyms. This is even more evident within intensive care medicine, where it is crucial that information is expressed in the most efficient manner possible to provide time-sensitive care to critically ill patients, but can result in code-like messages with poor readability. For example, given a sentence written by a physician with specialty training in critical care medicine, “STAT TTE c/w RVS. AKI - no CTA. .. etc”, it is difficult for non-experts to understand all abbreviations without specific context and/or knowledge. But when a doctor reads this, he/she would know that although “STAT” is widely used as the abbreviation of “statistic”, “statistics” and “statistical” in most domains, in hospital emergency rooms, it is often used to represent “immediately”. Within the arena of medical research, abbreviation expansion using a natural language processing system to automatically analyze clinical notes may enable knowledge discovery (e.g., relations between diseases) and has potential to improve communication and quality of care.
Question: How do they identify abbreviations?