pwc-archive/papers-with-abstracts huggingface dataset:
GFDC: Graph Function Dependence for Logically Consistent Dialogue Response Beyond Persona Data
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Dialogue Generation, General Knowledge

The rigorous logical consistency between current and historical responses during dialogue is the basis of an excellent human speaker, and it is also essential for dialogue agents. Although maintaining consistent personas has made tremendous advancements, they are still limited to the scale of persona data. Then it is necessary to study further how to lead to generate consistent responses when there are beyond personas or even without personas. In this work, we show how the problems can be tackled by consistent dialogue generation turn into Graph Functional Dependencies of a consistent graph schema representation and utilized, called GFDC. Specifically, the model consists of an XLNet-based consistency module and a Transformer-based generation module. The consistency module is for learning the representation of the consistent graph schema. The generation module is for generating responses using consistent graph schema representations. In particular, to learn a consistent graph schema in the dialogue data, we construct the dialogue data as a graph. Then introduce the graph of general knowledge for constructing the consistent graph schema. Both automatic and human evaluations show that our model outperforms strong baselines in response quality and consistency under fewer persona data settings.
Improving Neural Topic Models by Contrastive Learning with BERT
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Contrastive Learning, Topic Models

We present a general plug-and-play contrastive learning framework that improves existing neural topic models (NTMs) by incorporating the knowledge distilled from pre-trained language models. Recent NTMs have been applied to many applications and shown promising improvement on text analysis. However, they mainly focus on word-occurrences and are often optimized by maximizing the likelihood-based objective, which could lead to suboptimal topic coherence and document representation. To overcome the above bottleneck, we introduce an additional contrastive loss that pushes the topical representation of a document learned by an NTM close to the semantic representation of the document obtained from pre-trained language models. In this way, the prior knowledge of the pre-trained language models can enrich the contextual information of the target corpus for NTMs. Comprehensive experiments show that the proposed framework achieve the state-of-the-art performance. Importantly, our framework is general approach to improve most of the existing NTMs.
Universal Phone Recognition for Language Agnostic Keyword Search
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Recently, significant advanced have been made in universal phone recognition. Certain of these methods allow researchers to recognize phones in thousands of languages. In this paper, we explore the usage of such universal phone recognition for phonetic keyword search (KWS). That is, we apply these methods to search for specific sequences of phones, corresponding to keywords, in a set of audio files. We find that truly universal phone recognition might not be viable for KWS, but phone recognition systems can be fine-tuned with small amounts of data (3-5 hours of recordings) to produce useful results.
AdapLeR: Speeding up Inference by Adaptive Length Reduction
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Pre-trained language models have shown stellar performance in various downstream tasks. But, this usually comes at the cost of high latency and computation, hindering their usage in resource-limited settings. In this work, we propose a novel approach for reducing the computational cost of BERT with minimal loss in downstream performance. Our model dynamically eliminates less contributing tokens through layers, resulting in shorter lengths and consequently lower computational cost. To determine the importance of each token representation, we train a Contribution Predictor for each layer using a gradient-based saliency method. Our experiments on several diverse classification tasks show speedups up to 17x during inference time. We also validate the quality of the selected tokens in our method using human annotations in the ERASER benchmark. In comparison to other widely used strategies for selecting important tokens, such as saliency and attention, our proposed method has significantly less false positive rate in generating rationales.
Detecting Rumor Veracity with Only Textual Information by Double-Channel Structure
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

We develop a double-channel classifier to detect the veracity of social media rumors, relying only on the most basic textual information. Our model first assigns each thread into a “certain” or “uncertain” category. Since authors with a proprietary source of information are likely to post threads with a certain textual tone, we apply lie detection algorithms to certain texts. In contrast, as uncertain threads are arbitrary, we examine whether the replies are in accordance with the threads instead of applying the lie detection algorithms. This approach yields a macro-F1 score of 0.4027, outperforming all the baseline models and the second-place winner of SemEval 2019 Task 7. Further, we show that dividing the sample into two subgroups significantly improves the classification accuracy, reinforcing our claim that applying appropriate classifiers is crucial in rumor veracity detection.
Repo4QA: Answering Complex Coding Questions via Dense Retrieval on GitHub Repositories
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Contrastive Learning, Information Retrieval, Retrieval

Open-source platforms such as Github and Stack Overflow both play important roles in our software ecosystem. It is crucial but time-consuming for programmers to raise their specific programming questions on coding forums such as Stack Overflow, which guides them to actual solutions on Github repositories. We show our interest in accelerating such a process and find that traditional Information Retrieval based methods fail to handle the long and complex questions in coding forums and thus cannot find the suitable coding repositories. In order to bridge the semantic gap between repositories and real-world coding questions effectively and efficiently, we introduce a specialized dataset named Repo4QA, which includes over 12,000 question-repository pairs constructed from Stack Overflow and Github. Furthermore, we propose QuReCL, a contrastive learning model based on CodeBERT, to jointly learn the representation of both questions and repositories. Experimental results demonstrate that our model can simultaneously capture the semantic features in both questions and repositories through jointly embedding, and outperforms existing state-of-art methods.
Fair NLP Models with Differentially Private Text Encoders
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Fairness

Encoded text representations often capture sensitive attributes about individuals (e.g., gender, race, or age), which can raise privacy concerns and contribute to making downstream models unfair to certain groups. In this work, we propose FEDERATE, an approach that combines ideas from differential privacy and adversarial learning to learn private text representations which also induces fairer models. We empirically evaluate the trade-off between the privacy of the representations and the fairness and accuracy of the downstream model on two challenging NLP tasks. Our results show that FEDERATE consistently improves upon previous methods.
Predicate-Argument Based Bi-Encoder for Paraphrase Identification
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Paraphrase Identification, Sentence

Paraphrase identification involves identifying whether a pair of sentences express the same or similar meanings. While cross-encoders have achieved high performances across several benchmarks, bi-encoders such as SBERT have been widely applied to sentence pair tasks. They exhibit substantially lower computation complexity and are better suited to symmetric tasks. In this work, we adopt a bi-encoder approach to the paraphrase identification task, and investigate the impact of explicitly incorporating predicate-argument information into SBERT through weighted aggregation. Experiments on six paraphrase identification datasets demonstrate that, with a minimal increase in parameters, the proposed model is able to outperform SBERT/SRoBERTa significantly. Further, ablation studies reveal that the predicate-argument based component plays a significant role in the performance gain.
Automatic Mining of Salient Events from Multiple Documents
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Reading Comprehension

This paper studies a new event knowledge extraction task, Event Chain Mining. Given multiple documents on a super event, it aims to mine a series of salient events in a temporal order. For example, the event chain of super event Mexico Earthquake in 2017 is {earthquake hit Mexico, destroy houses, kill people, block roads}. This task can help readers capture the gist of texts quickly, thereby improving reading efficiency and deepening text comprehension. To address this task, we regard an event as a cluster of different mentions of similar meanings. In this way, we can identify the different expressions of events, enrich their semantic knowledge and enhance order information among them. Taking events as the basic unit, we propose a novel and flexible unsupervised framework, EMiner. Specifically, we extract event mentions from texts and merge those of similar meanings into a cluster as an event. Then, essential events are selected and arranged into a chain in the order of their occurrences. We then develop a testbed for the proposed task, including a human-annotated benchmark and comprehensive evaluation metrics. Extensive experiments are conducted to verify the effectiveness of EMiner in terms of both automatic and human evaluations.
FrugalScore: Learning Cheaper, Lighter and Faster Evaluation Metrics for Automatic Text Generation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Text Generation, Translation

Fast and reliable evaluation metrics are key to R&D progress. While traditional natural language generation metrics are fast, they are not very reliable. Conversely, new metrics based on large pretrained language models are much more reliable, but require significant computational resources. In this paper, we propose FrugalScore, an approach to learn a fixed, low cost version of any expensive NLG metric, while retaining most of its original performance. Experiments with BERTScore and MoverScore on summarization and translation show that FrugalScore is on par with the original metrics (and sometimes better), while having several orders of magnitude less parameters and running several times faster. On average over all learned metrics, tasks, and variants, FrugalScore retains 96.8% of the performance, runs 24 times faster, and has 35 times less parameters than the original metrics. We make our trained metrics publicly available, to benefit the entire NLP community and in particular researchers and practitioners with limited resources.
Should a Bot be Sarcastic?\\Understanding User Preferences Towards Sarcasm Generation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Previous sarcasm generation research has focused on \emph{how} to generate text that people perceive as sarcastic to create more human-like interactions. In this paper, we argue that we should first turn our attention to the question of \emph{when} sarcasm should be generated, finding that humans consider sarcastic responses inappropriate to many input utterances. Next, we use a theory-driven framework for generating sarcastic responses, which allows us to control the linguistic devices included during generation. For each device, we investigate how much humans associate it with sarcasm, finding that pragmatic insincerity and emotional markers are devices crucial for making sarcasm recognisable.
On the Importance of Data Size in Probing Fine-tuned Models
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Diversity

Several studies have investigated the reasons behind the effectiveness of fine-tuning, usually through the lens of probing. However, these studies often neglect the role of the size of the dataset on which the model is fine-tuned. In this paper, we highlight the importance of this factor and its undeniable role in probing performance. We show that the extent of encoded linguistic knowledge depends on the number of fine-tuning samples, specifically the number of iterations for which the model is updated. The analysis also reveals that larger training data mainly affects higher layers, and that the extent of this change is a factor of the number of iterations in fine-tuning rather than the diversity of the training samples. Finally, we show through a set of experiments that fine-tuning introduces shallow and recoverable changes to model's representation.
Can Pre-trained Models Really Generate Single-Step Textual Entailment?
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Natural Language Inference, RTE, valid

We investigate the task of generating textual entailment (GTE). Different from prior works on recognizing textual entailment, also known as NLI, GTE requires the models with deeper reasoning capabilities - generating entailment from premises rather than making prediction on given premises and the entailment. We argue that existing adapted datasets are limited and inadequate to train and evaluate human-like reasoning in the GTE. In this paper, we propose a new large-scale benchmark, named \mydataset, targeted for learning and evaluating models' capabilities towards RTE. \mydataset consists of 15k instances with each containing a pair of premise statements and a human-annotated entailment. It is constructed by first retrieving instances from a knowledge base, and then augmenting each instance with several complementary instances by 7 manually crafted  transformations. We demonstrate that even extensively fine-tuned pre-trained
models perform poorly on \mydataset. The best generator models can only generate valid textual entailment 59.1\% of times. Further, to motivate future advances, we provide detailed analysis to show significant gaps between baselines and human performance.
Towards Improving Topic Models with the BERT-based Neural Topic Encoder
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Topic Models

Neural Topic Models (NTMs) have been popular for mining a set of topics from a collection of corpora. Recently, there is an emerging direction of combining NTMs with pre-trained language models such as BERT, which aims to use the contextual information to of BERT to help train better NTMs.
However, existing works in this direction either use the contextual information of pre-trained language models as the input of NTMs or align the outputs of the two kinds of models.
In this paper, we study how to build deeper interactions between NTMs and pre-trained language and propose a BERT-based neural topic encoder, which deeply integrates with the transformer layers of BERT. Our proposed encoder encodes both the BoW data and the sequence of words of a document, which can be complementary to each other for learning a better topic distribution for the document.
The proposed encoder is a better alternative to the ones used in existing NTMs.
Thanks to the in-depth integration with BERT, extensive experiments show that the proposed model achieves the state-of-art performances the comparisons with many advanced models.
Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token Embeddings
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: All, Language Modeling, Language Modelling, Machine Translation, Text Generation, Word Similarity

Recent studies have determined that the learned token embeddings of large-scale neural language models are degenerated to be anisotropic with a narrow-cone shape. This phenomenon, called the representation degeneration problem, facilitates an increase in the overall similarity between token embeddings that negatively affect the performance of the models. Although the existing methods that address the degeneration problem based on observations of the phenomenon triggered by the problem improves the performance of the text generation, the training dynamics of token embeddings behind the degeneration problem are still not explored. In this study, we analyze the training dynamics of the token embeddings focusing on rare token embedding. We demonstrate that the specific part of the gradient for rare token embeddings is the key cause of the degeneration problem for all tokens during training stage. Based on the analysis, we propose a novel method called, adaptive gradient gating(AGG). AGG addresses the degeneration problem by gating the specific part of the gradient for rare token embeddings. Experimental results from language modeling, word similarity, and machine translation tasks quantitatively and qualitatively verify the effectiveness of AGG.
Towards Automated Real-time Evaluation in Text-based Counseling
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Automated real-time evaluation of counselor-client interaction is important for ensuring quality counseling but the rules are difficult to articulate. Recent advancements in machine learning methods show the possibility of learning such rules automatically. However, these methods often demand large scale and high quality counseling data, which are difficult to collect. To address this issue, we build an online counseling platform, which allows professional psychotherapists to provide free counseling services to those are in need. In exchange, we collect the counseling transcripts. Within a year of its operation, we manage to get one of the largest set of (675) transcripts of counseling sessions. To further leverage the valuable data we have, we label our dataset using both coarse- and fine-grained labels and use a set of pretraining techniques. In the end, we are able to achieve practically useful accuracy in both labeling system.
TransSGAN: GAN based semi-superivsed learning for text classification with Transformer Encoder
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Data Augmentation, text-classification, Text Classification

Recent semi-supervised learning for text classification uses data augmentations. However, it is a time-consuming and tricky hyperparametric approach. To overcome this problem, we present GAN-based semi-supervised learning for text classification, TransSGAN, which has a simple architecture, fewer hyperparameters, and trains for less time than current SOTA models since it does not need data augmentation. By adding one transformer encoder block to Semi-Supervised GAN, we can get comparable performances with extremely few labeled data to previous SOTA models up to 1% difference and 25% better performance than baseline GAN based model. Furthermore, we provide an analysis of our model, what the generator makes, and what multi-head self-attention layer in the generator learns. Through this, we can validate that our generator makes qualified data.
WLASL-LEX: a Dataset for Recognising Phonological Properties in American Sign Language
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Translation

Signed Language Processing (SLP) concerns the automated processing of signed languages, the main means of communication of Deaf and hearing impaired individuals. SLP features many different tasks, ranging from sign recognition to translation and production of signed speech, but has been overlooked by the NLP community thus far.
In this paper, we bring to attention the task of modelling the phonology of sign languages. We leverage existing resources to construct a large-scale dataset of American Sign Language signs annotated with six different phonological properties. We then conduct an extensive empirical study to investigate whether data-driven end-to-end and feature-based approaches can be optimised to automatically recognise these properties. We find that, despite the inherent challenges of the task, graph-based neural networks that operate over skeleton features extracted from raw videos are able to succeed at the task to a varying degree. Most importantly, we show that this performance pertains even on signs unobserved during training.
Translating Embeddings in Document for Modeling Multi-relational Graphs
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Document-level Relation Extraction, Graph Embedding, Knowledge Graph Embedding, Relation Extraction

Document-level relation extraction (RE) aims at extracting heterogeneous relational graphs upon entities in document, which has been handled with graph neural networks (GNNs) and pre-trained language models (PLMs) effectively. 
However, a crucial problem is that most GNNs adopt a task-independent pseudo graph convoluation which cause the over-parameterization. 
Is this paper, we argue that excessive parameters are unnecessary, and further propose a novel light-weight model named \textbf{TransGCN}. Specifically, we obtain the representation of entities in a document through a PLM encoder and construct our transmission-based graph convolutional network (GCN) on them. Unlike the previous methods that require storing the parameters of GNNs, our transmission-based GCN is performed with message passing constrained with the transmitting scores, which can be calculated by knowledge graph embedding models. In this way, we reduce the number of parameters by about half compared to the SOTA model. 
We conduct experiments on DocRED, which is a large-scale human-annotated document RE dataset. The results show that we outperform the state-of-the-art model with only half amount of parameters.
Lexical Gender Made Simple: A Scalable Methodology for Gender Detection with Online Lexical Databases
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

The evaluation of gender bias in Natural Language Processing relies on the use of gendered expressions, such as pronouns and words with lexical gender. Up until this point, researchers have manually compiled lists that record lexical gender for individual words. However, manual compilation leads to static information if lists are not periodically updated and categorization requires value judgements by annotators and researchers. Moreover, words that are not covered by the list fall out of the range of analysis.
To address these issues, we devised a dictionary-based method to automatically detect lexical gender that can provide a dynamic, up-to-date analysis with high coverage. Our approach reaches 90 % accuracy in determining the lexical gender of words retrieved randomly from a Wikipedia sample, and when testing on a manually compiled list that the method aims to replace.