z-uo/qasper-squad huggingface dataset:
Paper: Learning to Create Sentence Semantic Relation Graphs for Multi-Document Summarization
We model the semantic relationship among sentences using a graph representation. In this graph, each vertex is a sentence $S_{i,j}$ ($j$'th sentence of document $D_i$) from the collection documents $C$ and an undirected edge between $S_{i_u,j_u}$ and $S_{i_v,j_v}$ indicates their degree of similarity. In order to compute the semantic similarity, we use the model of BIBREF6 trained on the English Wikipedia corpus. In this manner, we incorporate general knowledge (i.e. not domain-specific) that will complete the specialized sentence embeddings obtained during training (see Section SECREF5). We process sentences by their model and compute the cosine similarity between every sentence pair, resulting in a complete graph. However, having a complete graph alone does not allow the model to leverage the semantic structure across sentences significantly, as every sentence pair is connected, and likewise, a sparse graph does not contain enough information to exploit semantic similarities. Furthermore, all edges have a weight above zero, since it is very unlikely that two sentence embeddings are completely orthogonal. To overcome this problem, we introduce an edge-removal-method, where every edge below a certain threshold $t_{sim}^g$ is removed in order to emphasize high sentence similarity. Nonetheless, $t_{sim}^g$ should not be too large, as we otherwise found the model to be prone to overfitting. After removing edges below $t_{sim}^g$, our sentence semantic relation graph is used as the adjacency matrix $A$. The impact of $t_{sim}^g$ with different values is shown in Section SECREF26.
Question: How better are state-of-the-art results than this model?
Paper: A Deep Neural Architecture for Sentence-level Sentiment Classification in Twitter Social Networking
Twitter sentiment classification have intensively researched in recent years BIBREF0 BIBREF1 . Different approaches were developed for Twitter sentiment classification by using machine learning such as Support Vector Machine (SVM) with rule-based features BIBREF2 and the combination of SVMs and Naive Bayes (NB) BIBREF3 . In addition, hybrid approaches combining lexicon-based and machine learning methods also achieved high performance described in BIBREF4 . However, a problem of traditional machine learning is how to define a feature extractor for a specific domain in order to extract important features.
Question: What were their results on the three datasets?
Answer: accuracy of 86.63 on STS, 85.14 on Sanders and 80.9 on HCR
Paper: A Deep Neural Architecture for Sentence-level Sentiment Classification in Twitter Social Networking
Our proposed model consists of a deep learning classifier and a tweet processor. The deep learning classifier is a combination of DeepCNN and Bi-LSTM. The tweet processor standardizes tweets and then applies semantic rules on datasets. We construct a framework that treats the deep learning classifier and the tweet processor as two distinct components. We believe that standardizing data is an important step to achieve high accuracy. To formulate our problem in increasing the accuracy of the classifier, we illustrate our model in Figure. FIGREF4 as follows:
Question: What was the baseline?
Paper: A Deep Neural Architecture for Sentence-level Sentiment Classification in Twitter Social Networking
Stanford - Twitter Sentiment Corpus (STS Corpus): STS Corpus contains 1,600K training tweets collected by a crawler from BIBREF0 . BIBREF0 constructed a test set manually with 177 negative and 182 positive tweets. The Stanford test set is small. However, it has been widely used in different evaluation tasks BIBREF0 BIBREF5 BIBREF13 .
Question: Which datasets did they use?
Paper: A Deep Neural Architecture for Sentence-level Sentiment Classification in Twitter Social Networking
We firstly take unique properties of Twitter in order to reduce the feature space such as Username, Usage of links, None, URLs and Repeated Letters. We then process retweets, stop words, links, URLs, mentions, punctuation and accentuation. For emoticons, BIBREF0 revealed that the training process makes the use of emoticons as noisy labels and they stripped the emoticons out from their training dataset because BIBREF0 believed that if we consider the emoticons, there is a negative impact on the accuracies of classifiers. In addition, removing emoticons makes the classifiers learns from other features (e.g. unigrams and bi-grams) presented in tweets and the classifiers only use these non-emoticon features to predict the sentiment of tweets. However, there is a problem is that if the test set contains emoticons, they do not influence the classifiers because emoticon features do not contain in its training data. This is a limitation of BIBREF0 , because the emoticon features would be useful when classifying test data. Therefore, we keep emoticon features in the datasets because deep learning models can capture more information from emoticon features for increasing classification accuracy.
Question: Are results reported only on English datasets?
Paper: A Deep Neural Architecture for Sentence-level Sentiment Classification in Twitter Social Networking
In Twitter social networking, people express their opinions containing sub-sentences. These sub-sentences using specific PoS particles (Conjunction and Conjunctive adverbs), like "but, while, however, despite, however" have different polarities. However, the overall sentiment of tweets often focus on certain sub-sentences. For example:
Question: Which three Twitter sentiment classification datasets are used for experiments?
Paper: A Deep Neural Architecture for Sentence-level Sentiment Classification in Twitter Social Networking
To construct embedding inputs for our model, we use a fixed-sized word vocabulary INLINEFORM0 and a fixed-sized character vocabulary INLINEFORM1 . Given a word INLINEFORM2 is composed from characters INLINEFORM3 , the character-level embeddings are encoded by column vectors INLINEFORM4 in the embedding matrix INLINEFORM5 , where INLINEFORM6 is the size of the character vocabulary. For word-level embedding INLINEFORM7 , we use a pre-trained word-level embedding with dimension 200 or 300. A pre-trained word-level embedding can capture the syntactic and semantic information of words BIBREF17 . We build every word INLINEFORM8 into an embedding INLINEFORM9 which is constructed by two sub-vectors: the word-level embedding INLINEFORM10 and the character fixed-size feature vector INLINEFORM11 of INLINEFORM12 where INLINEFORM13 is the length of the filter of wide convolutions. We have INLINEFORM14 character fixed-size feature vectors corresponding to word-level embedding in a sentence.
Question: What semantic rules are proposed?
Answer: rules that compute polarity of words after POS tagging or parsing steps
Paper: Logic Attention Based Neighborhood Aggregation for Inductive Knowledge Graph Embedding
Knowledge graphs (KGs) such as Freebase BIBREF0 , DBpedia BIBREF1 , and YAGO BIBREF2 play a critical role in various NLP tasks, including question answering BIBREF3 , information retrieval BIBREF4 , and personalized recommendation BIBREF5 . A typical KG consists of numerous facts about a predefined set of entities. Each fact is in the form of a triplet INLINEFORM0 (or INLINEFORM1 for short), where INLINEFORM2 and INLINEFORM3 are two entities and INLINEFORM4 is a relation the fact describes. Due to the discrete and incomplete natures of KGs, various KG embedding models are proposed to facilitate KG completion tasks, e.g., link prediction and triplet classification. After vectorizing entities and relations in a low-dimensional space, those models predict missing facts by manipulating the involved entity and relation embeddings.
Question: Which knowledge graph completion tasks do they experiment with?
Paper: Logic Attention Based Neighborhood Aggregation for Inductive Knowledge Graph Embedding
In recent years, representation learning problems on KGs have received much attention due to the wide applications of the resultant entity and relation embeddings. Typical KG embedding models include TransE BIBREF11 , Distmult BIBREF12 , Complex BIBREF13 , Analogy BIBREF14 , to name a few. For more explorations, we refer readers to an extensive survey BIBREF15 . However, conventional approaches on KG embedding work in a transductive manner. They require that all entities should be seen during training. Such limitation hinders them from efficiently generalizing to emerging entities.
Question: Apart from using desired properties, do they evaluate their LAN approach in some other way?
Paper: Logic Attention Based Neighborhood Aggregation for Inductive Knowledge Graph Embedding
To relieve the issue of emerging entities, several inductive KG embedding models are proposed, including BIBREF16 xie2016representation, BIBREF6 shi2018open and BIBREF17 xie2016image which use description text or images as inputs. Although the resultant embeddings may be utilized for KG completion, it is not clear whether the embeddings are powerful enough to infer implicit or new facts beyond those expressed in the text/image. Moreover, when domain experts are recruited to introduce new entities via partial facts rather than text or images, those approaches may not help much.
Question: Do they evaluate existing methods in terms of desired properties?
Paper: Learning with Noisy Labels for Sentence-level Sentiment Classification
It is well known that sentiment annotation or labeling is subjective BIBREF0. Annotators often have many disagreements. This is especially so for crowd-workers who are not well trained. That is why one always feels that there are many errors in an annotated dataset. In this paper, we study whether it is possible to build accurate sentiment classifiers even with noisy-labeled training data. Sentiment classification aims to classify a piece of text according to the polarity of the sentiment expressed in the text, e.g., positive or negative BIBREF1, BIBREF0, BIBREF2. In this work, we focus on sentence-level sentiment classification (SSC) with labeling errors.
Question: How does the model differ from Generative Adversarial Networks?
Paper: Learning with Noisy Labels for Sentence-level Sentiment Classification
Our work is related to sentence sentiment classification (SSC). SSC has been studied extensively BIBREF18, BIBREF19, BIBREF20, BIBREF21, BIBREF22, BIBREF23, BIBREF24, BIBREF25, BIBREF26, BIBREF27, BIBREF28. None of them can handle noisy labels. Since many social media datasets are noisy, researchers have tried to build robust models BIBREF29, BIBREF30, BIBREF31. However, they treat noisy data as additional information and don't specifically handle noisy labels. A noise-aware classification model in BIBREF12 trains using data annotated with multiple labels. BIBREF32 exploited the connection of users and noisy labels of sentiments in social networks. Since the two works use multiple-labeled data or users' information (we only use single-labeled data, and we do not use any additional information), they have different settings than ours.
Question: What is the dataset used to train the model?
Paper: Learning with Noisy Labels for Sentence-level Sentiment Classification
Our model builds on CNN BIBREF25. The key idea is to train two CNNs alternately, one for addressing the input noisy labels and the other for predicting `clean' labels. The overall architecture of the proposed model is given in Figure FIGREF2. Before going further, we first introduce a proposition, a property, and an assumption below.
Question: What is the performance of the model?
Answer: Experiment 1: ACC around 0.5 with 50% noise rate in worst case - clearly higher than baselines for all noise rates
Experiment 2: ACC on real noisy datasets: 0.7 on Movie, 0.79 on Laptop, 0.86 on Restaurant (clearly higher than baselines in almost all cases)
Paper: Learning with Noisy Labels for Sentence-level Sentiment Classification
In this section, we evaluate the performance of the proposed NetAb model. we conduct two types of experiments. (1) We corrupt clean-labeled datasets to produce noisy-labeled datasets to show the impact of noises on sentiment classification accuracy. (2) We collect some real noisy data and use them to train models to evaluate the performance of NetAb.
Question: Is the model evaluated against a CNN baseline?
Paper: Keep Calm and Switch On! Preserving Sentiment and Fluency in Semantic Text Exchange
There has been significant research on style transfer, with the goal of changing the style of text while preserving its semantic content. The alternative where semantics are adjusted while keeping style intact, which we call semantic text exchange (STE), has not been investigated to the best of our knowledge. Consider the following example, where the replacement entity defines the new semantic context:
Question: Does the model proposed beat the baseline models for all the values of the masking parameter tested?
Paper: Keep Calm and Switch On! Preserving Sentiment and Fluency in Semantic Text Exchange
Word2Vec BIBREF3, BIBREF4 allows for analogy representation through vector arithmetic. We implement a baseline (W2V-STEM) using this technique. The Universal Sentence Encoder (USE) BIBREF5 encodes sentences and is trained on a variety of web sources and the Stanford Natural Language Inference corpus BIBREF6. Flair embeddings BIBREF7 are based on architectures such as BERT BIBREF8. We use USE for SMERTI as it is designed for transfer learning and shows higher performance on textual similarity tasks compared to other models BIBREF9.
Question: Has STES been previously used in the literature to evaluate similar tasks?
Paper: Keep Calm and Switch On! Preserving Sentiment and Fluency in Semantic Text Exchange
Text infilling is the task of filling in missing parts of sentences called masks. MaskGAN BIBREF10 is restricted to a single word per mask token, while SMERTI is capable of variable length infilling for more flexible output. BIBREF11 uses a transformer-based architecture. They fill in random masks, while SMERTI fills in masks guided by semantic similarity, resulting in more natural infilling and fulfillment of the STE task.
Question: What are the baseline models mentioned in the paper?
Paper: CN-CELEB: a challenging Chinese speaker recognition dataset
Speaker recognition including identification and verification, aims to recognize claimed identities of speakers. After decades of research, performance of speaker recognition systems has been vastly improved, and the technique has been deployed to a wide range of practical applications. Nevertheless, the present speaker recognition approaches are still far from reliable in unconstrained conditions where uncertainties within the speech recordings could be arbitrary. These uncertainties might be caused by multiple factors, including free text, multiple channels, environmental noises, speaking styles, and physiological status. These uncertainties make the speaker recognition task highly challenging BIBREF0, BIBREF1.
Question: What was the performance of both approaches on their dataset?
Answer: ERR of 19.05 with i-vectors and 15.52 with x-vectors
Paper: CN-CELEB: a challenging Chinese speaker recognition dataset
The original purpose of the CN-Celeb dataset is to investigate the true difficulties of speaker recognition techniques in unconstrained conditions, and provide a resource for researchers to build prototype systems and evaluate the performance. Ideally, it can be used as a standalone data source, and can be also used with other datasets together, in particular VoxCeleb which is free and large. For this reason, CN-Celeb tries to be distinguished from but also complementary to VoxCeleb from the beginning of the design. This leads to three features that we have discussed in the previous section: Chinese focused, complex genres, and quality guarantee by human check.
Question: What kind of settings do the utterances come from?
Paper: CN-CELEB: a challenging Chinese speaker recognition dataset
Table TABREF13 summarizes the main difference between CN-Celeb and VoxCeleb. Compared to VoxCeleb, CN-Celeb is a more complex dataset and more challenging for speaker recognition research. More details of these challenges are as follows.
Question: What genres are covered?
Answer: genre, entertainment, interview, singing, play, movie, vlog, live broadcast, speech, drama, recitation and advertisement
Paper: CN-CELEB: a challenging Chinese speaker recognition dataset
CN-Celeb was collected following a two-stage strategy: firstly we used an automated pipeline to extract potential segments of the Person of Interest (POI), and then applied a human check to remove incorrect segments. This process is much faster than purely human-based segmentation, and reduces errors caused by a purely automated process.
Question: Do they experiment with cross-genre setups?
Paper: CN-CELEB: a challenging Chinese speaker recognition dataset
In this section, we present a series of experiments on speaker recognition using VoxCeleb and CN-Celeb, to compare the complexity of the two datasets.
Question: Which of the two speech recognition models works better overall on CN-Celeb?
Answer: x-vector
Paper: CN-CELEB: a challenging Chinese speaker recognition dataset
VoxCeleb: The entire dataset involves two parts: VoxCeleb1 and VoxCeleb2. We used SITW BIBREF21, a subset of VoxCeleb1 as the evaluation set. The rest of VoxCeleb1 was merged with VoxCeleb2 to form the training set (simply denoted by VoxCeleb). The training set involves $1,236,567$ utterances from $7,185$ speakers, and the evaluation set involves $6,445$ utterances from 299 speakers (precisely, this is the Eval. Core set within SITW).
Question: By how much is performance on CN-Celeb inferior to performance on VoxCeleb?
Answer: For i-vector system, performances are 11.75% inferior to voxceleb. For x-vector system, performances are 10.74% inferior to voxceleb
Paper: Conditional BERT Contextual Augmentation
Deep neural network-based models are easy to overfit and result in losing their generalization due to limited size of training data. In order to address the issue, data augmentation methods are often applied to generate more training samples. Recent years have witnessed great success in applying data augmentation in the field of speech area BIBREF0 , BIBREF1 and computer vision BIBREF2 , BIBREF3 , BIBREF4 . Data augmentation in these areas can be easily performed by transformations like resizing, mirroring, random cropping, and color shifting. However, applying these universal transformations to texts is largely randomized and uncontrollable, which makes it impossible to ensure the semantic invariance and label correctness. For example, given a movie review “The actors is good", by mirroring we get “doog si srotca ehT", or by random cropping we get “actors is", both of which are meaningless.
Question: On what datasets is the new model evaluated on?
Paper: Conditional BERT Contextual Augmentation
Language model pre-training has attracted wide attention and fine-tuning on pre-trained language model has shown to be effective for improving many downstream natural language processing tasks. Dai BIBREF7 pre-trained unlabeled data to improve Sequence Learning with recurrent networks. Howard BIBREF8 proposed a general transfer learning method, Universal Language Model Fine-tuning (ULMFiT), with the key techniques for fine-tuning a language model. Radford BIBREF9 proposed that by generative pre-training of a language model on a diverse corpus of unlabeled text, large gains on a diverse range of tasks could be realized. Radford BIBREF9 achieved large improvements on many sentence-level tasks from the GLUE benchmark BIBREF10 . BERT BIBREF11 obtained new state-of-the-art results on a broad range of diverse tasks. BERT pre-trained deep bidirectional representations which jointly conditioned on both left and right context in all layers, following by discriminative fine-tuning on each specific task. Unlike previous works fine-tuning pre-trained language model to perform discriminative tasks, we aim to apply pre-trained BERT on generative tasks by perform the masked language model(MLM) task. To generate sentences that are compatible with given labels, we retrofit BERT to conditional BERT, by introducing a conditional masked language model task and fine-tuning BERT on the task.
Question: How do the authors measure performance?
Answer: Accuracy across six datasets
Paper: Conditional BERT Contextual Augmentation
Text data augmentation has been extensively studied in natural language processing. Sample-based methods includes downsampling from the majority classes and oversampling from the minority class, both of which perform weakly in practice. Generation-based methods employ deep generative models such as GANs BIBREF12 or VAEs BIBREF13 , BIBREF14 , trying to generate sentences from a continuous space with desired attributes of sentiment and tense. However, sentences generated in these methods are very hard to guarantee the quality both in label compatibility and sentence readability. In some specific areas BIBREF15 , BIBREF16 , BIBREF17 . word replacement augmentation was applied. Wang BIBREF18 proposed the use of neighboring words in continuous representations to create new instances for every word in a tweet to augment the training dataset. Zhang BIBREF19 extracted all replaceable words from the given text and randomly choose $r$ of them to be replaced, then substituted the replaceable words with synonyms from WordNet BIBREF5 . Kolomiyets BIBREF20 replaced only the headwords under a task-specific assumption that temporal trigger words usually occur as headwords. Kolomiyets BIBREF20 selected substitute words with top- $K$ scores given by the Latent Words LM BIBREF21 , which is a LM based on fixed length contexts. Fadaee BIBREF22 focused on the rare word problem in machine translation, replacing words in a source sentence with only rare words. A word in the translated sentence is also replaced using a word alignment method and a rightward LM. The work most similar to our research is Kobayashi BIBREF6 . Kobayashi used a fill-in-the-blank context for data augmentation by replacing every words in the sentence with language model. In order to prevent the generated words from reversing the information related to the labels of the sentences, Kobayashi BIBREF6 introduced a conditional constraint to control the replacement of words. Unlike previous works, we adopt a deep bidirectional language model to apply replacement, and the attention mechanism within our model allows a more structured memory for handling long-term dependencies in text, which resulting in more general and robust improvement on various downstream tasks.
Question: Does the new objective perform better than the original objective bert is trained on?
Paper: Conditional BERT Contextual Augmentation
In general, the language model(LM) models the probability of generating natural language sentences or documents. Given a sequence $\textbf {\textit {S}}$ of N tokens, $<t_1,t_2,...,t_N>$ , a forward language model allows us to predict the probability of the sequence as: 
Question: Are other pretrained language models also evaluated for contextual augmentation?
Paper: Conditional BERT Contextual Augmentation
As shown in Fig 1 , our conditional BERT shares the same model architecture with the original BERT. The differences are the input representation and training procedure.
Question: Do the authors report performance of conditional bert on tasks without data augmentation?