z-uo/qasper-squad huggingface dataset:
Paper: Exploiting Task-Oriented Resources to Learn Word Embeddings for Clinical Abbreviation Expansion
The task of abbreviation disambiguation in biomedical documents has been studied by various researchers using supervised machine learning algorithms BIBREF2 , BIBREF3 , BIBREF4 , BIBREF5 , BIBREF6 . However, the performance of these supervised methods mainly depends on a large amount of labeled data which is extremely difficult to obtain for our task since intensive care medicine texts are very rare resources in clinical domain due to the high cost of de-identification and annotation. Tengstrand et al. tengstrand2014eacl proposed a distributional semantics-based approach for abbreviation expansion in Swedish but they focused only on expanding single words and cannot handle multi-word phrases. In contrast, we use word embeddings combined with task-oriented resources and knowledge, which can handle multiword expressions.
Question: What kind of model do they build to expand abbreviations?
Paper: Exploiting Task-Oriented Resources to Learn Word Embeddings for Clinical Abbreviation Expansion
The overview of our approach is shown in Figure FIGREF6 . Within ICU notes (e.g., text example in top-left box in Figure 2), we first identify all abbreviations using regular expressions and then try to find all possible expansions of these abbreviations from domain-specific knowledge base as candidates. We train word embeddings using the clinical notes data with task-oriented resources such as Wikipedia articles of candidates and medical scientific papers and compute the semantic similarity between an abbreviation and its candidate expansions based on their embeddings (vector representations of words).
Question: Do they use any knowledge base to expand abbreviations?
Paper: Exploiting Task-Oriented Resources to Learn Word Embeddings for Clinical Abbreviation Expansion
Given an abbreviation as input, we expect the correct expansion to be the most semantically similar to the abbreviation, which requires the abbreviation and the expansion share similar contexts. For this reason, we exploit rich task-oriented resources such as the Wikipedia articles of all the possible candidates, research papers and books written by the intensive care medicine fellows. Together with our clinical notes data which functions as a corpus, we train word embeddings since the expansions of abbreviations in the clinical notes are likely to appear in these resources and also share the similar contexts to the abbreviation's contexts.
Question: In their used dataset, do they study how many abbreviations are ambiguous?
Paper: Exploiting Task-Oriented Resources to Learn Word Embeddings for Clinical Abbreviation Expansion
In most cases, an abbreviation's expansion is a multi-word phrase. Therefore, we need to obtain the phrase's embedding so that we can compute its semantic similarity to the abbreviation.
Question: Which dataset do they use to build their model?
Paper: CM-Net: A Novel Collaborative Memory Network for Spoken Language Understanding
Spoken Language Understanding (SLU) is a core component in dialogue systems. It typically aims to identify the intent and semantic constituents for a given utterance, which are referred as intent detection and slot filling, respectively. Past years have witnessed rapid developments in diverse deep learning models BIBREF0, BIBREF1 for SLU. To take full advantage of supervised signals of slots and intents, and share knowledge between them, most of existing works apply joint models that mainly based on CNNs BIBREF2, BIBREF3, RNNs BIBREF4, BIBREF5, and asynchronous bi-model BIBREF6. Generally, these joint models encode words convolutionally or sequentially, and then aggregate hidden states into a utterance-level representation for the intent prediction, without interactions between representations of slots and intents.
Question: What is the domain of their collected corpus?
Paper: CM-Net: A Novel Collaborative Memory Network for Spoken Language Understanding
In principle, the slot filling is treated as a sequence labeling task, and the intent detection is a classification problem. Formally, given an utterance $X = \lbrace x_1, x_2, \cdots , x_N \rbrace $ with $N$ words and its corresponding slot tags $Y^{slot} = \lbrace y_1, y_2, \cdots , y_N \rbrace $, the slot filling task aims to learn a parameterized mapping function $f_{\theta } : X \rightarrow Y $ from input words to slot tags. For the intent detection, it is designed to predict the intent label $\hat{y}^{int}$ for the entire utterance $X$ from the predefined label set $S^{int}$.
Question: What was the performance on the self-collected corpus?
Answer: F1 scores of 86.16 on slot filling and 94.56 on intent detection
Paper: CM-Net: A Novel Collaborative Memory Network for Spoken Language Understanding
In this section, we start with a brief overview of our CM-Net and then proceed to introduce each module. As shown in Figure FIGREF16, the input utterance is firstly encoded with the Embedding Layer, and then is transformed by multiple CM-blocks with the assistance of slot and intent memories, and finally make predictions in the Inference Layer.
Question: What is the size of their dataset?
Answer: 10,001 utterances
Paper: CM-Net: A Novel Collaborative Memory Network for Spoken Language Understanding
The pre-trained word embeddings has been indicated as a de-facto standard of neural network architectures for various NLP tasks. We adapt the cased, 300d Glove BIBREF17 to initialize word embeddings, and keep them frozen.
Question: What is the source of the CAIS dataset?
Paper: CM-Net: A Novel Collaborative Memory Network for Spoken Language Understanding
It has been demonstrated that character level information (e.g. capitalization and prefix) BIBREF18 is crucial for sequence labeling. We use one layer of CNN followed by max pooling to generate character-aware word embeddings.
Question: What were the baselines models?
Paper: "What is Relevant in a Text Document?": An Interpretable Machine Learning Approach
A number of real-world problems related to text data have been studied under the framework of natural language processing (NLP). Example of such problems include topic categorization, sentiment analysis, machine translation, structured information extraction, or automatic summarization. Due to the overwhelming amount of text data available on the Internet from various sources such as user-generated content or digitized books, methods to automatically and intelligently process large collections of text documents are in high demand. For several text applications, machine learning (ML) models based on global word statistics like TFIDF BIBREF0 , BIBREF1 or linear classifiers are known to perform remarkably well, e.g. for unsupervised keyword extraction BIBREF2 or document classification BIBREF3 . However more recently, neural network models based on vector space representations of words (like BIBREF4 ) have shown to be of great benefit to a large number of tasks. The trend was initiated by the seminal work of BIBREF5 and BIBREF6 , who introduced word-based neural networks to perform various NLP tasks such as language modeling, chunking, named entity recognition, and semantic role labeling. A number of recent works (e.g. BIBREF6 , BIBREF7 ) also refined the basic neural network architecture by incorporating useful structures such as convolution, pooling, and parse tree hierarchies, leading to further improvements in model predictions. Overall, these ML models have permitted to assign automatically and accurately concepts to entire documents or to sub-document levels like phrases; the assigned information can then be mined on a large scale.
Question: Are the document vectors that the authors introduce evaluated in any way other than the new way the authors propose?
Paper: "What is Relevant in a Text Document?": An Interpretable Machine Learning Approach
Explanation of individual classification decisions in terms of input variables has been studied for a variety of machine learning classifiers such as additive classifiers BIBREF18 , kernel-based classifiers BIBREF19 or hierarchical networks BIBREF11 . Model-agnostic methods for explanations relying on random sampling have also been proposed BIBREF20 , BIBREF21 , BIBREF22 . Despite their generality, the latter however incur an additional computational cost due to the need to process the whole sample to provide a single explanation. Other methods are more specific to deep convolutional neural networks used in computer vision: the authors of BIBREF8 proposed a network propagation technique based on deconvolutions to reconstruct input image patterns that are linked to a particular feature map activation or prediction. The work of BIBREF9 aimed at revealing salient structures within images related to a specific class by computing the corresponding prediction score derivative with respect to the input image. The latter method reveals the sensitivity of the classifier decision to some local variation of the input image, and is related to sensitivity analysis BIBREF23 , BIBREF24 . In contrast, the LRP method of BIBREF12 corresponds to a full decomposition of the classifier output for the current input image. It is based on a layer-wise conservation principle and reveals parts of the input space that either support or speak against a specific classification decision. Note that the LRP framework can be applied to various models such as kernel support vector machines and deep neural networks BIBREF12 , BIBREF17 . We refer the reader to BIBREF14 for a comparison of the three explanation methods, and to BIBREF13 for a view of particular instances of LRP as a “deep Taylor decomposition” of the decision function.
Question: According to the authors, why does the CNN model exhibit a higher level of explainability?
Paper: "What is Relevant in a Text Document?": An Interpretable Machine Learning Approach
In this section we describe our method for identifying words in a text document, that are relevant with respect to a given category of a classification problem. For this, we assume that we are given a vector-based word representation and a neural network that has already been trained to map accurately documents to their actual category. Our method can be divided in four steps: (1) Compute an input representation of a text document based on word vectors. (2) Forward-propagate the input representation through the convolutional neural network until the output is reached. (3) Backward-propagate the output through the network using the layer-wise relevance propagation (LRP) method, until the input is reached. (4) Pool the relevance scores associated to each input variable of the network onto the words to which they belong. As a result of this four-step procedure, a decomposition of the prediction score for a category onto the words of the documents is obtained. Decomposed terms are called relevance scores. These relevance scores can be viewed as highlighted text or can be used to form a list of top-words in the document. The whole procedure is also described visually in Figure 1 . While we detail in this section the LRP method for a specific network architecture and with predefined choices of layers, the method can in principle be extended to any architecture composed of similar or larger number of layers.
Question: Does the LRP method work in settings that contextualize the words with respect to one another?
Paper: Fortia-FBK at SemEval-2017 Task 5: Bullish or Bearish? Inferring Sentiment towards Brands from Financial News Headlines
Real time information is key for decision making in highly technical domains such as finance. The explosive growth of financial technology industry (Fintech) continued in 2016, partially due to the current interest in the market for Artificial Intelligence-based technologies.
Question: How do they incorporate lexicon into the neural network?
Paper: Fortia-FBK at SemEval-2017 Task 5: Bullish or Bearish? Inferring Sentiment towards Brands from Financial News Headlines
While image and sound come with a natural high dimensional embedding, the issue of which is the best representation is still an open research problem in the context of natural language and text. It is beyond the scope of this paper to do a thorough overview of word representations, for this we refer the interest reader to the excellent review provided by BIBREF1 . Here, we will just introduce the main representations that are related to the proposed method.
Question: What is the source of their lexicon?
Paper: Fortia-FBK at SemEval-2017 Task 5: Bullish or Bearish? Inferring Sentiment towards Brands from Financial News Headlines
The data consists of a set of financial news headlines, crawled from several online outlets such as Yahoo Finance, where each sentence contains one or more company names/brands.
Question: What was their performance?
Paper: Fortia-FBK at SemEval-2017 Task 5: Bullish or Bearish? Inferring Sentiment towards Brands from Financial News Headlines
In Figure FIGREF5 , we can see the overall architecture of our model.
Question: How long is the dataset used for training?
Paper: Fortia-FBK at SemEval-2017 Task 5: Bullish or Bearish? Inferring Sentiment towards Brands from Financial News Headlines
Minimal preprocessing was adopted in our approach: we replaced the target company's name with a fixed word <company> and numbers with <number>. The sentences were then tokenized using spaces as separator and keeping punctuation symbols as separate tokens.
Question: What embeddings do they use?
Paper: BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
Understanding what facts can be inferred to be true or false from text is an essential part of natural language understanding. In many cases, these inferences can go well beyond what is immediately stated in the text. For example, a simple sentence like “Hanna Huyskova won the gold medal for Belarus in freestyle skiing." implies that (1) Belarus is a country, (2) Hanna Huyskova is an athlete, (3) Belarus won at least one Olympic event, (4) the USA did not win the freestyle skiing event, and so on. Work completed while interning at Google. Also affiliated with Columbia University, work done at Google.
Question: did they use other pretrained language models besides bert?
Paper: BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
Yes/No questions make up a subset of the reading comprehension datasets CoQA BIBREF3 , QuAC BIBREF4 , and HotPotQA BIBREF5 , and are present in the ShARC BIBREF10 dataset. These datasets were built to challenge models to understand conversational QA (for CoQA, ShARC and QuAC) or multi-step reasoning (for HotPotQA), which complicates our goal of using yes/no questions to test inferential abilities. Of the four, QuAC is the only one where the question authors were not allowed to view the text being used to answer their questions, making it the best candidate to contain naturally occurring questions. However, QuAC still heavily prompts users, including limiting their questions to be about pre-selected Wikipedia articles, and is highly class imbalanced with 80% “yes" answers.
Question: how was the dataset built?
Answer: Questions are gathered from anonymized, aggregated queries to the Google search engine. Queries that are likely to be yes/no questions are heuristically identified: we found selecting queries where the first word is in a manually constructed set of indicator words and are of sufficient length, to be effective. Questions are only kept if a Wikipedia page is returned as one of the first five results, in which case the question and Wikipedia page are given to a human annotator for further processing. Annotators label question/article pairs in a three-step process. First, they decide if the question is good, meaning it is comprehensible, unambiguous, and requesting factual information. This judgment is made before the annotator sees the Wikipedia page. Next, for good questions, annotators find a passage within the document that contains enough information to answer the question. Annotators can mark questions as “not answerable" if the Wikipedia article does not contain the requested information. Finally, annotators mark whether the question's answer is “yes" or “no"
Paper: BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
An example in our dataset consists of a question, a paragraph from a Wikipedia article, the title of the article, and an answer, which is either “yes" or “no". We include the article title since it can potentially help resolve ambiguities (e.g., coreferent phrases) in the passage, although none of the models presented in this paper make use of them.
Question: what is the size of BoolQ dataset?
Paper: Database of Parliamentary Speeches in Ireland, 1919-2013
Almost all political decisions and political opinions are, in one way or another, expressed in written or spoken texts. Great leaders in history become famous for their ability to motivate the masses with their speeches; parties publish policy programmes before elections in order to provide information about their policy objectives; parliamentary decisions are discussed and deliberated on the floor in order to exchange opinions; members of the executive in most political systems are legally obliged to provide written or verbal answers to questions from legislators; and citizens express their opinions about political events on internet blogs or in public online chats. Political texts and speeches are everywhere that people express their political opinions and preferences.
Question: what processing was done on the speeches before being parsed?
Answer: Remove numbers and interjections
Paper: A Lightweight Front-end Tool for Interactive Entity Population
Entity extraction is one of the most major NLP components. Most NLP tools (e.g., NLTK, Stanford CoreNLP, etc.), including commercial services (e.g., Google Cloud API, Alchemy API, etc.), provide entity extraction functions to recognize named entities (e.g., PERSON, LOCATION, ORGANIZATION, etc.) from texts. Some studies have defined fine-grained entity types and developed extraction methods BIBREF0 based on these types. However, these methods cannot comprehensively cover domain-specific entities. For instance, a real estate search engine needs housing equipment names to index these terms for providing fine-grained search conditions. There is a significant demand for constructing user-specific entity dictionaries, such as the case of cuisine and ingredient names for restaurant services. A straightforward solution is to prepare a set of these entity names as a domain-specific dictionary. Therefore, this paper focuses on the entity population task, which is a task of collecting entities that belong to an entity type required by a user.
Question: What programming language is the tool written in?
Paper: User Generated Data: Achilles' heel of BERT
In recent times, pre-trained contextual language models have led to significant improvement in the performance for many NLP tasks. Among the family of these models, the most popular one is BERT BIBREF0, which is also the focus of this work. The strength of the BERT model FIGREF2 stems from its transformerBIBREF1 based encoder architectureFIGREF1. While it is still not very clear as to why BERT along with its embedding works so well for downstream tasks when it is fine tuned, there has been some work in this direction that that gives some important cluesBIBREF2, BIBREF3.
Question: What is the performance change of the textual semantic similarity task when no error and maximum errors (noise) are present?
Answer: 10 Epochs: pearson-Spearman correlation  drops  60 points when error increase by 20%
50 Epochs: pearson-Spearman correlation  drops  55 points when error increase by 20%
Paper: User Generated Data: Achilles' heel of BERT
In recent years pre-trained language models ((e.g. ELMoBIBREF5, BERTBIBREF0) have made breakthroughs in several natural language tasks. These models are trained over large corpora that are not human annotated and are easily available. Chief among these models is BERTBIBREF0. The popularity of BERT stems from its ability to be fine-tuned for a variety of downstream NLP tasks such as text classification, regression, named-entity recognition, question answeringBIBREF0, machine translationBIBREF6 etc. BERT has been able to establish State-of-the-art (SOTA) results for many of these tasks. People have been able to show how one can leverage BERT to improve searchBIBREF7.
Question: Which sentiment analysis data set has a larger performance drop when a 10% error is introduced?
Answer: SST-2 dataset
Paper: User Generated Data: Achilles' heel of BERT
For our experiments, we use pre-trained BERT implementation as given by huggingface transformer library. We use the BERTBase uncased model. We work with three datasets namely - IMDB movie reviewsBIBREF11, Stanford Sentiment Treebank (SST-2) BIBREF12 and Semantic Textual Similarity (STS-B) BIBREF13.
Question: What kind is noise is present in typical industrial data?
Paper: User Generated Data: Achilles' heel of BERT
Let us discuss the results from the above mentioned experiments. We show the plots of accuracy vs noise for each of the tasks. For IMDB, we fine tune the model for the sentiment analysis task. We plot F1 score vs % of error, as shown in Figure FIGREF6. Figure FIGREF6imdba shows the performance after fine tuning for 10 epochs, while Figure FIGREF6imdbb shows the performance after fine tuning for 50 epochs.
Question: What is the reason behind the drop in performance using BERT for some popular task?
Paper: Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning
Pre-trained feature extractors, such as BERT BIBREF0 for natural language processing and VGG BIBREF1 for computer vision, have become effective methods for improving the performance of deep learning models. In the last year, models similar to BERT have become state-of-the-art in many NLP tasks, including natural language inference (NLI), named entity recognition (NER), sentiment analysis, etc. These models follow a pre-training paradigm: they are trained on a large amount of unlabeled text via a task that resembles language modeling BIBREF2, BIBREF3 and are then fine-tuned on a smaller amount of “downstream” data, which is labeled for a specific task. Pre-trained models usually achieve higher accuracy than any model trained on downstream data alone.
Question: How they observe that fine-tuning BERT on a specific task does not improve its prunability?
Paper: Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning
Neural network pruning involves examining a trained network and removing parts deemed to be unnecessary by some heuristic saliency criterion. One might remove weights, neurons, layers, channels, attention heads, etc. depending on which heuristic is used. Below, we describe three different lenses through which we might interpret pruning.
Question: How much is pre-training loss increased in Low/Medium/Hard level of pruning?
Answer: The increase is linearly from lowest on average 2.0 , medium around 3.5, and the largest is 6.0
Paper: Paying Attention to Attention: Highlighting Influential Samples in Sequential Analysis
The attention mechanism BIBREF1 in neural networks can be used to interpret and visualize model behavior by selecting the most pertinent pieces of information instead of all available information. For example, in BIBREF0 , a hierarchical attention network (Han) is created and tested on the classification of product and movie reviews. As a side effect of employing the attention mechanism, sentences (and words) that are considered important to the model can be highlighted, and color intensity corresponds to the level of importance (darker color indicates higher importance).
Question: How do they gather human reviews?