pwc-archive/papers-with-abstracts huggingface dataset:
Bi-Matching Mechanism to Combat the Long Tail of Word Sense Disambiguation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Few-Shot Learning, Word Sense Disambiguation

The long tail phenomenon of word sense distribution in linguistics causes the Word Sense Disambiguation (WSD) task to face a serious polarization of word sense distribution, that is, Most Frequent Senses (MFSs) with huge sample sizes and Long Tail Senses (LTSs) with small sample sizes. The single matching mechanism model that does not distinguish between the two senses will cause LTSs to be ignored because LTSs are in a weak position. The few-shot learning method that mainly focuses on LTSs is not conducive to grasping the advantage of easy identification of MFSs. This paper proposes a bi-matching mechanism to serve the WSD model to deal with two kinds of senses in a targeted manner, namely definition matching and collocation feature matching. The experiment is carried out under the evaluation framework of English all-words WSD and is better than the baseline models. Moreover, state-of-the-art performance is achieved through data enhancement.
Quantum-inspired Representation for Long-tail Senses of Word Sense Disambiguation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Data Augmentation, Word Sense Disambiguation

Data imbalance, also known as the long-tailed distribution of data, is an important challenge for data-driven models. Due to the long tail phenomenon of word sense distribution in linguistics, it is difficult to learn accurate representations for Long-Tail Senses (LTSs) in Word Sense Disambiguation (WSD) tasks. Without data augmentation, exploring representation methods that do not rely on large sample sizes is an important means to combat the long tail. In this paper, inspired by the superposition state in quantum mechanics, a representation method in Hilbert space is proposed to reduce the dependence on large sample sizes. We theoretically prove the correctness of the method, verify its effectiveness on the WSD task, and achieve state-of-the-art performance.
Deep-to-bottom Weights Decay: A Systemic Knowledge Review Learning Technique for Transformer Layers in Knowledge Distillation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Knowledge Distillation

There are millions of parameters and huge computational power consumption behind the outstanding performance of pre-trained language models in natural language processing tasks. Knowledge distillation is considered as a compression strategy to address this problem. However, previous works (i) distill partial transformer layers of the teacher model, which ignore the importance of bottom base information, or (ii) neglect the difficulty differences of knowledge from deep to shallow, which corresponds to different level information of teacher model. We introduce a deep-to-bottom weights decay review mechanism to knowledge distillation, which fuses teacher-side information taking each layer’s difficulty level into consideration. To validate our claims, we distill a 12-layer BERT into a 6-layer model and evaluate it on the GLUE dataset. Experimental results show that our review approach is able to outperform other existing techniques.
OrderSum: Reading Order-Aware Unsupervised Opinion Summarization
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Opinion Summarization, Unsupervised Opinion Summarization

Opinion summarization aims to create a concise summary reflecting subjective information conveyed by multiple user reviews about the same product. To avoid the high expense of curating golden summaries for training, many unsupervised methods have been recently developed. Most state-of-the-art methods utilize the extracted segments following their salience ranking as pseudo labels to train a summary generator. However, the extracted salient segments can be verbose and their reading order has been long overlooked. In this paper, we propose a reading order-aware framework, OrderSum, aiming to generate concise and logical summaries. Specifically, we first formulate the segment ordering problem in pseudo labels as path-choosing and solve it using reinforcement learning. Moreover, to generate a more concise summary, we propose to encourage the generative model to skip useless words based on the token link information derived from concise sentences, which can be collected easily from massive raw reviews by considering the ratio of sentiment/aspect words. Extensive experiments demonstrate that OrderSum benefits from the awareness of reading order and the conciseness modeling, thus being more effective than existing unsupervised methods and achieving the state-of-the-art performance.
Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Models
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Inductive Bias

Relations between words are governed by hierarchical structure rather than linear ordering. Sequence-to-sequence (seq2seq) models, despite their success in downstream NLP applications, often fail to generalize in a hierarchy-sensitive manner when performing syntactic transformations—for example, transforming declarative sentences into questions—instead generalizing linearly using positional surface heuristics. However, syntactic evaluations of seq2seq models have only observed models that were not pre-trained on natural language data before being trained to perform syntactic transformations, in spite of the fact that pre-training has been found to induce hierarchical linguistic generalizations in language models; in other words, the syntactic capabilities of seq2seq models may have been greatly understated. Here, we make use of the pre-trained seq2seq model T5 (and its multilingual variant mT5) and evaluate whether they generalize hierarchically on two syntactic transformations in two languages: question formation and passivization in English and German. We find that T5 and mT5 generalize hierarchically when performing syntactic transformations, whereas non-pre-trained baseline models do not. This result presents additional evidence for the learnability of hierarchical syntactic information from non-annotated natural language text while also demonstrating that seq2seq models are capable of syntactic generalization.
Is Attention Explanation? An Introduction to the Debate
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

The performance of deep learning models in NLP and other fields of machine learning has led to a rise in their popularity, and so the need for explanations of these models becomes paramount. Attention has been seen as a solution to increase performance, while providing some explanations. However, a debate has started to cast doubt on the explanatory power of attention in neural networks. Although the debate has created a vast literature thanks to contributions from various areas, the lack of communication is becoming more and more tangible. In this paper, we provide a clear overview of the insights on the debate by critically confronting works from these different areas. This holistic vision can be of great interest for future works in all the communities concerned by this debate. We sum up the main challenges spotted in these areas, and we conclude by discussing the most promising future avenues on attention as an explanation.
A Bit Bayesian Facilitates Efficient Training in Token Classification
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Classification, token-classification, Token Classification

Token classification is a fundamental subject matter in computational linguistics. Token classification models, like other modern deep neural network models, are usually trained on the entire training set in each epoch, while research has found all of the training data may not be needed in late epochs of training. Inspired by human pedagogy, we propose a teacher-aware structure to accelerate the training of token classification models. After each epoch of training, the teacher samples data that it is uncertain to and data it predicts differently from the student, which are passed into the structure for training in the next epoch. As a proof of concept, we use a Bayesian linear classifier as the teacher, and use two commonly used backbone models as the student. Experiments show that our method reduces the number of training iterations, speeding up the training without affecting the model's performance.
SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Abstractive Text Summarization, Mixture-of-Experts, Re-Ranking

Sequence-to-sequence neural networks have recently achieved great success in abstractive summarization, especially with the trend of fine-tuning large pre-trained language models on the downstream dataset. These models are typically decoded with beam search to generate a unique summary. However, the search space is very large, and due to exposure bias, such decoding is not optimal. In this paper, we show that it is possible to directly train a second-stage model performing re-ranking on a set of summary candidates. Our mixture-of-experts SummaReranker learns to select a better candidate and systematically improves the performance of the base model. With a base PEGASUS, we push ROUGE scores by 5.44% on CNN-DailyMail (47.16 ROUGE-1), 1.31% on XSum (48.12 ROUGE-1) and 9.34% on Reddit TIFU (29.83 ROUGE-1), reaching a new state-of-the-art.
Few-Shot Named Entity Recognition with Biaffine Span Representation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Few-shot NER, Named Entity Recognition, Named Entity Recognition (NER), Sentence

While Named Entity Recognition (NER) is a widely studied task, making inferences of entities with only a few labeled data (i.e., few-shot NER) has been challenging. Correspondingly, the N-way K-shot NER task is proposed to recognize entities in the given N categories with only K labeled samples for each category. Existing methods treat this task as a sequence labeling problem, while this paper regards it as an entity span classification problem and designs a Biaffine Span Representation (BSR) method to learn contextual span dependency representation to fit into the classification algorithm. The BSR applies a biaffine pooling module to establish the dependencies of each word on the whole sentence and to reduce the dimension of word features, thus, the span representation could gain contextual dependency information to help improve recognition accuracies. Experimental study on four standard NER datasets shows that our proposed BSR method outperforms pre-trained language models and existing N-way K-shot NER algorithms in two types of adaptations (i.e., Intra-Domain Cross-Type Adaptation and Cross-Domain Cross-Type Adaptation). Notably, F_1 value has increased by an average of 13.77% and 18.30% on the 5-way 1-shot task and the 5-way 5-shot task, respectively.
Detect Low-Resource Rumors in Microblog Posts via Adversarial Contrastive Learning
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Contrastive Learning

Massive false rumors emerging along with breaking news or trending topics severely hinder the truth. Exiting rumor detection approaches achieve promising performance on the yesterday's news, since there is enough corpus collected from the same domain for model training. However, they are poor at detecting rumors about unforeseen events such as COVID-19 due to the lack of training data and prior knowledge (i.e., low-resource rumors). In this paper, we propose an adversarial contrastive learning framework to detect low-resource rumors by adapting the features learned from well-resourced rumor data to that of the low-resourced. Our model explicitly overcomes the restriction of both domain and language usage via language alignment and contrastive training. Moreover, we develop an adversarial augmentation mechanism to further enhance the robustness of low-resource rumor representation. Extensive experiments conducted on two low-resource datasets collected from real-world microblog platforms demonstrate that our framework achieves much better performance than state-of-the-art methods and exhibits a superior capacity for detecting rumors at early stages.
CluSent – Combining Semantic Expansion and De-Noising for Dataset-Oriented Sentiment Analysis of Short Texts
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Sentiment Analysis

The lack of sufficient information, mainly in short texts, is a major challenge to building effective sentiment models. Short texts can be enriched with more complex semantic relationships that can better capture affective information, with a potential undesired side effect of noise introduced into the data. In this work, we propose a new strategy for customized dataset-oriented sentiment analysis -- CluSent -- that exploits a powerful, recently proposed concept for representing semantically related words -- CluWords. CluSent tackles the issues mentioned above of information shortage and noise by: (i) exploiting the semantic neighborhood of a given pre-trained word embedding to enrich document representation, and (ii) introducing dataset-oriented filtering and weighting mechanisms to cope with noise, which take advantage of the polarity and intensity information from lexicons. In our experimental evaluation, considering 19 datasets,  5 state-of-the-art baselines (including modern transformer architectures) and two metrics, CluSent was the best method in 30 out of 38 possibilities, with significant gains over the strongest baselines (over 14%).
Self-Supervised Contrastive Learning with Adversarial Perturbations for Robust Pretrained Language Models
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Adversarial Attack, Contrastive Learning

In this paper, we present an approach to improve the robustness of BERT language models against word substitution-based adversarial attacks by leveraging adversarial perturbations for self-supervised contrastive learning. We create an efficient word-level adversarial attack, and use it to finetune BERT on adversarial examples generated \textit{on the fly} during training.
In contrast with previous works, our method improves model robustness without using any labeled data. Experimental results show that our method improves robustness of BERT against four different word substitution-based adversarial attacks, and combining our method with adversarial training gives higher robustness than adversarial training alone.
As our method improves the robustness of BERT purely with unlabeled data, it opens up the possibility of using large text datasets to train robust language models.
Subword Information for Authorship Attribution: A Deep Learning Approach
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Authorship Attribution, Deep Learning

Authorship attribution is the process of unveiling the hidden identity of authors from a corpus of literary data. Many previous works on authorship attribution employed word-based models to capture an author's distinctive writing style. The vocabulary of the training corpus is heavily dependent on the pre-trained word vectors, which limits the performance of these models. Alternate methods using character-based models proposed to overcome the rare word problems arising from different linguistic features fail to capture the sequential relationship of words inherently present in the texts. The question we addressed in this paper is whether it is possible to tackle the ambiguity of hidden writing style (or words) as we introduce Gaussian noise while preserving the sequential context of the text to improve authorship-related tasks. In this work,  we propose two bidirectional long short-term memory (BLSTM) with a 2D convolutional neural network (CNN) over a two-dimensional pooling operation to capture sequential writing styles for distinguishing different authors. To determine the appropriate writing style representation, we used BLSTM to obtain the sequential relationship between characteristics using subword information and 2D CNN is adopted to understand the local syntactical position of the style from unlabelled input text. We extensively evaluate the model that leverages subword embedding and compare it against state-of-the-art methods for an extensive range of authors. Our methods improve 2.42\%, 0.96\% and 0.97\% on CCAT50, Blog50 and Twitter, respectively and produce comparable results on the remaining one.
Blackbird's language matrices (BLMs): a new benchmark to investigate disentangled generalisation in neural networks
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Current successes of machine learning architectures are based on computationally expensive algorithms and prohibitively large amounts of data. We need to develop tasks and data to train networks to reach more complex and more compositional skills. In this paper, we illustrate Blackbird's language matrices (BLMs), a novel grammatical dataset developed to test a linguistic variant of Raven's progressive matrices, an intelligence test usually based on visual stimuli. The dataset consists of roughly 48000 sentences, generatively constructed to support investigations of current models' linguistic mastery of grammatical rules and their ability to generalize them. We present the logic of the dataset, the method to automatically construct data on a large scale, and the architecture to learn them. Through error analysis and several experiments on variations of the dataset, we demonstrate that this language task and the data that instantiate it provide a new challenging testbed to understand generalization and abstraction.
Structure and Features Fusion with Evidential Graph Convolutional Neural Network for Node Classification
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Classification, Graph Representation Learning, Node Classification, Representation Learning

Recently, text-enhanced network representation learning has achieved great success by taking advantage of rich text information and network structure information. However, content-rich network representation learning and quantifying classiﬁcation uncertainty are challenging when it comes to integrating complex structural dependencies and rich content features at an evidence level. In this paper, we propose an evidential graph representation learning model (EGCN), which can not only fuse network structure and content information into a more complete and powerful representation for each node, but also assess the quality of graph node features to improve classification accuracy. To achieve better fusion, we integrate the node's features representation into structure-aware representation through a delivery operator. Besides, to overcome the difﬁculty of predicting node classification conﬁdence, we employ a novel module based on Dirichlet distribution theory of evidence and subject opinion learning to collect the evidence of the class probabilities. Experimental results on three real-world networks show that our model can improve both node classification accuracy and robustness as compared to all baselines.
Rumor Detection on Twitter Through SMOTE+RU Hybrid Top Features Extraction Model using Ensemble Method
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

In today's scenario, the prevalence of social media like Twitter or Weibo has increased a lot. Such platforms allow their users to discuss and share rumors due to their open nature. As the social media platforms impacts a large section of the community, the spread of rumors on Twitter may result in economic, political & geopolitical consequences for a state. Various rumor detection techniques has been reported that tracks the rumored posts on a social media platforms automatically. Existing rumor detection techniques implement either machine learning methods (like SVM or Random Forest classifier) or deep learning methods such as Convolutional Neural Networks etc. However, the traditional techniques fail to offer the high accuracy in rumor detection. In this paper, we propose a hybrid technique for rumor detection that is based on SMOTE+RU feature extraction model. The proposed technique initially use Pearson Correlation method to find and extract the highly correlated features from the data set. Then the synthetic minority oversampling technique (SMOTE) and random under sampling (RU) are used to balance the minority and majority class data. Further, the random forest method provides the important feature extraction. Then the proposed technique use ensemble method which is the hybrid of decision tree, k-nearest neighbor, ada-boost and neural network methods. The majority voting technique is used to get the best possible prediction from the ensemble model. Our proposed rumor detection technique offers increased level of accuracy as compared to the relevant existing techniques. The experimental evaluation exhibited more than 91 % results in terms of accuracy, precision, recall and F1-score metrics.
Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Contrastive Learning, Language Modeling, Language Modelling

Learning scientific document representations can be substantially improved through contrastive learning objectives, where the challenge lies in creating positive and negative training samples that encode the desired similarity semantics. Prior work relies on discrete citation relations to generate contrast samples. However, discrete citations enforce a hard cut-off to similarity. This is counter-intuitive to similarity-based learning, and ignores that scientific papers can be very similar despite lacking a direct citation - a core problem of finding related research. Instead, we use controlled nearest neighbor sampling over citation graph embeddings for contrastive learning. This control allows us to learn continuous similarity, to sample hard-to-learn negatives and positives, and also to avoid collisions between negative and positive samples by controlling the sampling margin between them. The resulting method SciNCL outperforms the state-of-the-art on the SciDocs benchmark. Furthermore, we demonstrate that it can train (or tune) models sample-efficiently, which improves compute efficiency, and that it can be combined with recent training-efficient methods. Perhaps surprisingly, even training a general-domain language model this way outperforms baselines pretrained in-domain.
BigFive: A Dataset of Coarse- and Fine-Grained Personality Characteristics
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Obtaining the personalities of users conveyed by their published short texts has a wide and important range of applications, from detecting abnormal behavior of online users to accurately customization recommendation. Advancement in this area can be improved using large-scale datasets with coarse- and fine-grained typologies, adaptable to multiple downstream tasks. Therefore, this paper introduces $BigFive$, a large, high quality dataset manually annotated by experts. $BigFive$ contains 13,478 Chinese phrases that belong to five categories (coarse-grained) and 30 categories (fine-grained). The reliability of five categories grouped by personality level and 30 categories grouped by dimension level is demonstrated via a detailed data analysis. In addition,  a strong baseline is build based on fine-tuning a BERT model. Our BERT-based model achieves an average F1-score of .33 (std=.24) in terms of 30 categories and an average F1-score of .66 (std=.05) in terms of five categories. The experimental results suggest that there is much room for improvement.