z-uo/qasper-squad huggingface dataset:
Paper: A Deep Learning Architecture for De-identification of Patient Notes: Implementation and Evaluation
 Electronic Health Records (EHR) have become ubiquitous in recent years in the United States, owing much to the The Health Information Technology for Economic and Clinical Health (HITECH) Act of 2009. BIBREF0 Their ubiquity have given researchers a treasure trove of new data, especially in the realm of unstructured textual data. However, this new data source comes with usage restrictions in order to preserve the privacy of individual patients as mandated by the Health Insurance Portability and Accountability Act (HIPAA). HIPAA demands any researcher using this sensitive data to first strip the medical records of any protected health information (PHI), a process known as de-identification.
Question: What performance is achieved?
Paper: A Deep Learning Architecture for De-identification of Patient Notes: Implementation and Evaluation
The task of automatic de-identification has been heavily studied recently, in part due to two main challenges organized by i2b2 in 2006 and in 2014. The task of de-identification can be classified as a named entity recognition (NER) problem which has been extensively studied in machine learning literature. Automated de-identification systems can be roughly broken down into four main categories:
Question: Do they use BERT?
Paper: A Deep Learning Architecture for De-identification of Patient Notes: Implementation and Evaluation
Rule-based systems make heavy use of pattern matching such as dictionaries (or gazetteers), regular expressions and other patterns. BIBREF2 Systems such as the ones described in BIBREF5 , BIBREF6 do not require the use any labeled data. Hence, they are considered as unsupervised learning systems. Advantages of such systems include their ease of use, ease of adding new patterns and easy interpretability. However, these methods suffer from a lack of robustness with regards to the input. For example, different casings of the same word could be misinterpreted as an unknown word. Furthermore, typographical errors are almost always present in most documents and rule-based systems often cannot correctly handle these types of inaccuracies present in the data. Critically, these systems cannot handle context which could render a medical text unreadable. For example, a diagnosis of “Lou Gehring disease” could be misidentified by such a system as a PHI of type Name. The system might replace the tokens “Lou” and “Gehring” with randomized names rendering the text meaningless if enough of these tokens were replaced.
Question: What is their baseline?
Paper: A Deep Learning Architecture for De-identification of Patient Notes: Implementation and Evaluation
The drawbacks of such rule-based systems led researchers to adopt a machine learning approach. A comprehensive review of such systems can be found in BIBREF7 , BIBREF8 . In machine learning systems, given a sequence of input vectors INLINEFORM0 , a machine learning algorithm outputs label predictions INLINEFORM1 . Since the task of de-identification is a classification task, traditional classification algorithms such as support vector machines, conditional random fields (CRFs) and decision trees BIBREF9 have been used for building de-identification systems.
Question: Which two datasets is the system tested on?
Paper: Cross Lingual Cross Corpus Speech Emotion Recognition
Speech conveys human emotions most naturally. In recent years there has been an increased research interest in speech emotion recognition domain. The first step in a typical SER system is extracting linguistic and acoustic features from speech signal. Some para-linguistic studies find Low-Level Descriptor (LLD) features of the speech signal to be most relevant to studying emotions in speech. These features include frequency related parameters like pitch and jitter, energy parameters like shimmer and loudness, spectral parameters like alpha ratio and other parameters that convey cepstral and dynamic information. Feature extraction is followed with a classification task to predict the emotions of the speaker.
Question: Which four languages do they experiment with?
Answer: German, English, Italian, Chinese
Paper: Location-Relative Attention Mechanisms For Robust Long-Form Speech Synthesis
Sequence-to-sequence models that use an attention mechanism to align the input and output sequences BIBREF0, BIBREF1 are currently the predominant paradigm in end-to-end TTS. Approaches based on the seminal Tacotron system BIBREF2 have demonstrated naturalness that rivals that of human speech for certain domains BIBREF3. Despite these successes, there are sometimes complaints of a lack of robustness in the alignment procedure that leads to missing or repeating words, incomplete synthesis, or an inability to generalize to longer utterances BIBREF4, BIBREF5, BIBREF6.
Question: Does DCA or GMM-based attention perform better in experiments?
Answer: About the same performance
Paper: Location-Relative Attention Mechanisms For Robust Long-Form Speech Synthesis
The system that we use in this paper is based on the original Tacotron system BIBREF2 with architectural modifications from the baseline model detailed in the appendix of BIBREF11. We use the CBHG encoder from BIBREF2 to produce a sequence of encoder outputs, $\lbrace j\rbrace _{j=1}^L$, from a length-$L$ input sequence of target phonemes, $\lbrace \mathbf {x}_j\rbrace _{j=1}^L$. Then an attention RNN, (DISPLAY_FORM2), produces a sequence of states, $\lbrace \mathbf {s}_i\rbrace _{i=1}^T$, that the attention mechanism uses to compute $\mathbf {\alpha }_i$, the alignment at decoder step $i$. Additional arguments to the attention function in () depend on the specific attention mechanism (e.g., whether it is content-based, location-based, or both). The context vector, $\mathbf {c}_i$, that is fed to the decoder RNN is computed using the alignment, $\mathbf {\alpha }_i$, to produce a weighted average of encoder states. The decoder is fed both the context vector and the current attention RNN state, and an output function produces the decoder output, $\mathbf {y}_i$, from the decoder RNN state, $\mathbf {d}_i$.
Question: How they compare varioius mechanisms in terms of naturalness?
Paper: Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack
The detection of offensive language has become an important topic as the online community has grown, as so too have the number of bad actors BIBREF2. Such behavior includes, but is not limited to, trolling in public discussion forums BIBREF3 and via social media BIBREF4, BIBREF5, employing hate speech that expresses prejudice against a particular group, or offensive language specifically targeting an individual. Such actions can be motivated to cause harm from which the bad actor derives enjoyment, despite negative consequences to others BIBREF6. As such, some bad actors go to great lengths to both avoid detection and to achieve their goals BIBREF7. In that context, any attempt to automatically detect this behavior can be expected to be adversarially attacked by looking for weaknesses in the detection system, which currently can easily be exploited as shown in BIBREF8, BIBREF9. A further example, relevant to the natural langauge processing community, is the exploitation of weaknesses in machine learning models that generate text, to force them to emit offensive language. Adversarial attacks on the Tay chatbot led to the developers shutting down the system BIBREF1.
Question: What evaluation metric is used?
Answer: F1 and Weighted-F1
Paper: Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack
The task of detecting offensive language has been studied across a variety of content classes. Perhaps the most commonly studied class is hate speech, but work has also covered bullying, aggression, and toxic comments BIBREF13.
Question: What datasets are used?
Paper: ViGGO: A Video Game Corpus for Data-To-Text Generation in Open-Domain Conversation
The recent adoption of deep learning methods in natural language generation (NLG) for dialogue systems resulted in an explosion of neural data-to-text generation models, which depend on large training data. These are typically trained on one of the few parallel corpora publicly available, in particular the E2E BIBREF0 and the WebNLG BIBREF1 datasets. Crowdsourcing large NLG datasets tends to be a costly and time-consuming process, making it impractical outside of task-oriented dialogue systems. At the same time, current neural NLG models struggle to replicate the high language diversity of the training sentences present in these large datasets, and instead they learn to produce the same generic type of sentences as with considerably less training data BIBREF2, BIBREF3, BIBREF4.
Question: Is the origin of the dialogues in corpus some video game and what game is that?
Paper: ViGGO: A Video Game Corpus for Data-To-Text Generation in Open-Domain Conversation
ViGGO features more than 100 different video game titles, whose attributes were harvested using free API access to two of the largest online video game databases: IGDB and GiantBomb. Using these attributes, we generated a set of 2,300 structured MRs. The human reference utterances for the generated MRs were then crowdsourced using vetted workers on the Amazon Mechanical Turk (MTurk) platform BIBREF18, resulting in 6,900 MR-utterance pairs altogether. With the goal of creating a clean, high-quality dataset, we strived to obtain reference utterances with correct mentions of all slots in the corresponding MR through post-processing.
Question: Is any data-to-text generation model trained on this new corpus, what are the results?
Answer: Yes, Transformer based seq2seq is evaluated with average BLEU 0.519, METEOR 0.388, ROUGE 0.631 CIDEr 2.531 and SER 2.55%.
Paper: ViGGO: A Video Game Corpus for Data-To-Text Generation in Open-Domain Conversation
The MRs in the ViGGO dataset range from 1 to 8 slot-value pairs, and the slots come from a set of 14 different video game attributes. Table TABREF6 details how these slots may be distributed across the 9 different DAs. The inform DA, represented by 3,000 samples, is the most prevalent one, as the average number of slots it contains is significantly higher than that of all the other DAs. Figure FIGREF7 visualizes the MR length distribution across the entire dataset.
Question: How the authors made sure that corpus is clean despite being crowdsourced?
Paper: A human-editable Sign Language representation for software editing---and a writing system?
The main motivation here is to equip Sign Language (SL) with software and foster implementation as available tools for SL are paradoxically limited in such digital times. For example, translation assisting software would help respond to the high demand for accessible content and information. But equivalent text-to-text software relies on source and target written forms to work, whereas similar SL support seems impossible without major revision of the typical user interface.
Question: Do they build a generative probabilistic language model for sign language?
Paper: Contextual LSTM (CLSTM) models for Large scale NLP tasks
Documents have sequential structure at different hierarchical levels of abstraction: a document is typically composed of a sequence of sections that have a sequence of paragraphs, a paragraph is essentially a sequence of sentences, each sentence has sequences of phrases that are comprised of a sequence of words, etc. Capturing this hierarchical sequential structure in a language model (LM) BIBREF0 can potentially give the model more predictive accuracy, as we have seen in previous work BIBREF1 , BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 .
Question: Does CLSTM have any benefits over BERT?
Paper: Ensemble-Based Deep Reinforcement Learning for Chatbots
Humans in general find it relatively easy to have chat-like conversations that are both coherent and engaging at the same time. While not all human chat is engaging, it is arguably coherent BIBREF0 , and it can cover large vocabularies across a wide range of conversational topics. In addition, each contribution by a partner conversant may exhibit multiple sentences, such as greeting+question or acknowledgement+statement+question. The topics raised in a conversation may go back and forth without losing coherence. All of these phenomena represent big challenges for current data-driven chatbots.
Question: How do they obtain human generated policies?
Paper: Ensemble-Based Deep Reinforcement Learning for Chatbots
A reinforcement learning agent induces its behaviour from interacting with an environment through trial and error, where situations (representations of sentences in a dialogue history) are mapped to actions (follow-up sentences) by maximising a long-term reward signal. Such an agent is typically characterised by: (i) a finite set of states INLINEFORM0 that describe all possible situations in the environment; (ii) a finite set of actions INLINEFORM1 to change in the environment from one situation to another; (iii) a state transition function INLINEFORM2 that specifies the next state INLINEFORM3 for having taken action INLINEFORM4 in the current state INLINEFORM5 ; (iv) a reward function INLINEFORM6 that specifies a numerical value given to the agent for taking action INLINEFORM7 in state INLINEFORM8 and transitioning to state INLINEFORM9 ; and (v) a policy INLINEFORM10 that defines a mapping from states to actions BIBREF1 , BIBREF29 . The goal of a reinforcement learning agent is to find an optimal policy by maximising its cumulative discounted reward defined as DISPLAYFORM0 
Question: How many agents do they ensemble over?
Paper: Impact of Coreference Resolution on Slot Filling
Coreference resolution systems group noun phrases (mentions) that refer to the same entity into the same chain. Mentions can be full names (e.g., John Miller), pronouns (e.g., he), demonstratives (e.g., this), comparatives (e.g., the first) or descriptions of the entity (e.g. the 40-year-old) BIBREF0 . Although coreference resolution has been a research focus for several years, systems are still far away from being perfect. Nevertheless, there are many tasks in natural language processing (NLP) which would benefit from coreference information, such as information extraction, question answering or summarization BIBREF1 . In BIBREF2 , for example, we showed that coreference information can also be incorporated into word embedding training. In general, coreference resolution systems can be used as a pre-processing step or as a part of a pipeline of different modules.
Question: What is the task of slot filling?
Paper: Fully Automated Fact Checking Using External Sources
Recent years have seen the proliferation of deceptive information online. With the increasing necessity to validate the information from the Internet, automatic fact checking has emerged as an important research topic. It is at the core of multiple applications, e.g., discovery of fake news, rumor detection in social media, information verification in question answering systems, detection of information manipulation agents, and assistive technologies for investigative journalism. At the same time, it touches many aspects, such as credibility of users and sources, information veracity, information verification, and linguistic aspects of deceptive language.
Question: Do they report results only on English data?
Paper: Fully Automated Fact Checking Using External Sources
Given a claim, our system searches for support information on the Web in order to verify whether the claim is likely to be true. The three steps in this process are (i) external support retrieval, (ii) text representation, and (iii) veracity prediction.
Question: Does this system improve on the SOTA?
Paper: Fully Automated Fact Checking Using External Sources
This step consists of generating a query out of the claim and querying a search engine (here, we experiment with Google and Bing) in order to retrieve supporting documents. Rather than querying the search engine with the full claim (as on average, a claim is two sentences long), we generate a shorter query following the lessons highlighted in BIBREF0 .
Question: How are the potentially relevant text fragments identified?
Answer:  Generate a query out of the claim and querying a search engine, rank the words by means of TF-IDF, use IBM's AlchemyAPI to identify named entities, generate queries of 5–10 tokens, which execute against a search engine, and collect the snippets and the URLs in the results, skipping any result that points to a domain that is considered unreliable.
Paper: Fully Automated Fact Checking Using External Sources
Next, we build the representation of a claim and the corresponding snippets and Web pages. First, we calculate three similarities (a) between the claim and a snippet, or (b) between the claim and a Web page: (i) cosine with tf-idf, (ii) cosine over embeddings, and (iii) containment BIBREF1 . We calculate the embedding of a text as the average of the embeddings of its words; for this, we use pre-trained embeddings from GloVe BIBREF2 . Moreover, as a Web page can be long, we first split it into a set of rolling sentence triplets, then we calculate the similarities between the claim and each triplet, and we take the highest scoring triplet. Finally, as we have up to ten hits from the search engine, we take the maximum and also the average of the three similarities over the snippets and over the Web pages.
Question: What algorithm and embedding dimensions are used to build the task-specific embeddings?
Paper: Fully Automated Fact Checking Using External Sources
Next, we build classifiers: neural network (NN), support vector machines (SVM), and a combination thereof (SVM+NN).
Question: What data is used to build the task-specific embeddings?
Paper: A Probabilistic Generative Grammar for Semantic Parsing
Accurate and efficient semantic parsing is a long-standing goal in natural language processing. There are countless applications for methods that provide deep semantic analyses of sentences. Leveraging semantic information in text may provide improved algorithms for many problems in NLP, such as named entity recognition BIBREF0 , BIBREF1 , BIBREF2 , word sense disambiguation BIBREF3 , BIBREF4 , semantic role labeling BIBREF5 , co-reference resolution BIBREF6 , BIBREF7 , etc. A sufficiently expressive semantic parser may directly provide the solutions to many of these problems. Lower-level language processing tasks, such as those mentioned, may even benefit by incorporating semantic information, especially if the task can be solved jointly during semantic parsing.
Question: Do they evaluate the syntactic parses?
Paper: A Probabilistic Generative Grammar for Semantic Parsing
Our model is an extension of context-free grammars (CFGs) BIBREF18 that couples syntax and semantics. To generate a sentence in our framework, the semantic statement is first drawn from a prior. A grammar then recursively constructs a syntax tree top-down, randomly selecting production rules from distributions that depend on the semantic statement. We present a particular incarnation of a grammar in this framework, where hierarchical Dirichlet processes (HDPs) BIBREF19 are used to select production rules randomly. The application of HDPs in our setting is novel, requiring a new inference technique.
Question: What knowledge bases do they use?
Paper: Harnessing the richness of the linguistic signal in predicting pragmatic inferences
An important property of human communication is that listeners can infer information beyond the literal meaning of an utterance. One well-studied type of inference is scalar inference BIBREF0, BIBREF1, whereby a listener who hears an utterance with a scalar item like some infers the negation of a stronger alternative with all:
Question: Which dataset do they use?
Paper: Enhancing Pre-trained Chinese Character Representation with Word-aligned Attention
Pre-trained language Models (PLM) such as ELMo BIBREF0, BERT BIBREF1, ERNIE BIBREF2 and XLNet BIBREF3 have been proven to capture rich language information from text and then benefit many NLP applications by simple fine-tuning, including sentiment classification, natural language inference, named entity recognition and so on.
Question: What pre-trained models did they compare to?
Paper: Enhancing Pre-trained Chinese Character Representation with Word-aligned Attention
The primary goal of this work is to inject the word segmentation knowledge into character-level Chinese PLM and enhance original models. Given the strong performance of recent deep transformers trained on language modeling, we adopt BERT and its updated variants (ERNIE, BERT-wwm) as the basic encoder for our work, and the outputs $\mathbf {H}$ from the last layer of encoder are treated as the enriched contextual representations.
Question: How does the fusion method work?
Paper: Enhancing Pre-trained Chinese Character Representation with Word-aligned Attention
Although the character-level PLM can well capture language knowledge from text, it neglects the semantic information expressed in the word level. Therefore we apply a word-aligned layer on top of the encoder to integrate the word boundary information into representation of character with the attention aggregation mechanism.
Question: What dataset did they use?
Answer: weibo-100k, Ontonotes, LCQMC and XNLI
Paper: Enhancing Pre-trained Chinese Character Representation with Word-aligned Attention
As mentioned in Section SECREF1, our proposed word-aligned attention relies on the segmentation results of CWS tool $\pi $. Unfortunately, a segmenter is usually unreliable due to the risk of ambiguous and non-formal input, especially on out-of-domain data, which may lead to error propagation and an unsatisfactory model performance. In practice, The ambiguous distinction between morphemes and compound words leads to the cognitive divergence of words concepts, thus different $\pi $ may provide diverse $\pi (S)$ with various granularities. To reduce the impact of segmentation error and effectively mine the common knowledge of different segmenters, it’s natural to enhance the word-aligned attention layer with multi-source segmentation input. Formally, assume that there are $M$ popular CWS tools employed, we can obtain $M$ different representations $\overline{\textbf {H}}^1, ..., \overline{\textbf {H}}^M $ by Eq. DISPLAY_FORM11. Then we propose to fuse these semantically different representations as follows:
Question: What benchmarks did they experiment on?
Paper: Towards Task-Oriented Dialogue in Mixed Domains
In this work, we investigate the problem of task-oriented dialogue in mixed-domain settings. Our work is related to two lines of research in Spoken Dialogue System (SDS), namely task-oriented dialogue system and multi-domain dialogue system. We briefly review the recent literature related to these topics as follows.
Question: What were the evaluation metrics used?
Paper: Towards Task-Oriented Dialogue in Mixed Domains
In this section, we present briefly two methods that we use in our experiments which have been mentioned in the previous section. The first method is the Sequicity framework and the second one is the state-of-the-art multi-domain dialogue state tracking approach.
Question: What is the size of the dataset?
Answer: 3029
Paper: Towards Task-Oriented Dialogue in Mixed Domains
Figure FIGREF1 shows the architecture of the Sequicity framework as described in BIBREF8. In essence, in each turn, the Sequicity model first takes a bspan ($B_1$) and a response ($R_1$) which are determined in the previous step, and the current human question ($U_2$) to generate the current bspan. This bspan is then used together with a knowledge base to generate the corresponding machine answer ($R_2$), as shown in the right part of Figure FIGREF1.
Question: What multi-domain dataset is used?
Paper: Towards Task-Oriented Dialogue in Mixed Domains
Figure FIGREF8 shows the architecture of the multi-domain belief tracking with knowledge sharing as described in BIBREF9. This is the state-of-the-art belief tracker for multi-domain dialogue.
Question: Which domains did they explored?