pwc-archive/papers-with-abstracts huggingface dataset:
Semantic Role Labeling as Dependency Parsing: Exploring Latent Tree Structures Inside Arguments
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Dependency Parsing, Semantic Role Labeling

Semantic role labeling (SRL) is a fundamental yet challenging task in the NLP community.
Recent works of SRL mainly fall into two lines: 1) BIO-based; 2) span-based.
Despite ubiquity, they share some intrinsic drawbacks of not explicitly considering internal argument structures, which may potentially hinder the model's expressiveness.
To remedy this, we propose to reduce SRL to a dependency parsing task and regard the flat argument spans as latent subtrees.
In particular, we equip our formulation with a novel span-constrained TreeCRF to make tree structures span-aware, and further extend it to the second-order case.
Experiments on CoNLL05 and CoNLL12 benchmarks reveal that the results of our methods outperform all previous works and achieve the state-of-the-art.
A Flexible Multi-Task Model for BERT Serving
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Knowledge Distillation, model

We present an efficient BERT-based multi-task (MT) framework that is particularly suitable for iterative and incremental development of the tasks. The proposed framework is based on the idea of partial fine-tuning, i.e. only fine-tune some top layers of BERT while keep the other layers frozen. For each task,  we train independently a single-task (ST) model using partial fine-tuning. Then we compress the task-specific layers in each ST model using knowledge distillation. Those compressed ST models are finally merged into one MT model so that the frozen layers of the former are shared across the tasks. We exemplify our approach on eight GLUE tasks, demonstrating that it is able to achieve 99.6\% of the performance of the full fine-tuning method, while reducing up to two thirds of its overhead.
Graph-based Fine-grained Multimodal Attention Mechanism for Sentiment Analysis
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: multimodal interaction, Multimodal Sentiment Analysis, Sentiment Analysis

Multimodal sentiment analysis is a popular research area in natural language processing. Mainstream multimodal learning models barely consider that the visual and acoustic behaviors often have a much higher temporal frequency than words. Therefore, these models lack the representation capability to accurately model multimodal interactions. In this paper, we propose an attachment called Graph-based Fine-grained Multimodal Attention Mechanism (GFMAM), which can utilize the multimodal information from different subspaces to achieve accurate multimodal interactions. Firstly, the attachment further splits the information of every modality into multiple subspaces.
Then, the fine-grained multimodal information from different subspaces is converted into multimodal interaction graphs dominant by the language modality. The multimodal interaction graph can capture significant interactions among multiple modalities at the subspace level.
Finally, the information of nonverbal modalities is additionally added to compensate for the loss of continuity caused by the splitting operation. Embedding GFMAM into BERT, we propose a new model called GFMAM-BERT that can directly accept nonverbal modalities in addition to language modality. We conducted experiments on both publicly available multimodal sentiment analysis datasets CMU-MOSI and CMU-MOSEI. The experiment results demonstrate that GFMAM-BERT exceeds the state-of-the-art models. Moreover, the proposed model outperforms humans on most metrics on the CMU-MOSI dataset.
Heterogeneous Language Model Optimization in Automatic Speech Recognition
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Language Modeling, Language Modelling, model, Model Optimization, speech-recognition, Speech Recognition

The rising data privacy risks make it difficult for automatic speech recognition (ASR) systems to acquire complete training data in practical application. Recently, the merge paradigm for acoustic model has been proposed to solve the issue. However, ASR still suffers from another salient issue on language model. Current efforts mainly focus on isomorphic neural network models, while language model optimization is characterized by merging and matching heterogeneous models including $n$-gram and neural network models. In this paper, we propose a novel Match-and-Merge paradigm to fill up the vacuum for the language model optimization. Based on different training datasets, we train multiple language model pairs. In order to merge them into a target pair with the best performance, we first propose a Genetic Match-and-Merge (GMM) method that can be specifically adopted to optimize heterogeneous models. To improve the algorithm efficiency, we further propose a Reinforced Match-and-Merge (RMM) method, which maintains superior recognition accuracy while reducing convergence time. Extensive experiments demonstrate the effectiveness and generalization of our proposed methods, which significantly establishes the new state-of-the-art.
FedParsing: a Semi-Supervised Federated Learning Model on Semantic Parsing
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Federated Learning, Semantic Parsing

Although many semantic parsing models have been proven to work effectively on "NL-to-SQL", the limitation of annotated datasets remains a great challenge. In many semi-supervised models, while they use unlabeled data to greatly improve the model accuracy, they fail to take data privacy of users into account. In this work, we focus on improving the performance of the semantic parsing model  and protecting the users’ data privacy without increasing the size of the labeled dataset. Our new model, which is named FedParsing, is a semi-supervised Federated Learning model. In order to solve the difficulty on convergence of traditional semi-supervised Federated Learning model, we incorporate the Mean Teacher algorithm and apply the Exponential Moving Average algorithm to update model parameters. Experiments on WikiSQL show that with extra unlabeled data, our model performs better than supervised training model and traditional semi-supervised Federated Learning model, which proves the effectiveness of FedParsing model.
Making Document-Level Information Extraction Right for the Right Reasons
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Sentence

Document-level information extraction is a flexible framework compatible with applications where information is not necessarily localized in a single sentence. For example, key features of a diagnosis in radiology a report may not be explicitly stated, but nevertheless can be inferred from the report's text. However, document-level neural models can easily learn spurious correlations from irrelevant information. This work studies how to ensure that these models make correct inferences from complex text and make those inferences in an auditable way: beyond just being right, are these models "right for the right reasons?" We experiment with post-hoc evidence extraction in a predict-select-verify framework using feature attribution techniques. While this basic approach can extract reasonable evidence, it can be regularized with small amounts of evidence supervision during training, which substantially improves the quality of extracted evidence. We evaluate on two domains: a small-scale labeled dataset of brain MRI reports and a large-scale modified version of DocRED (Yao et al., 2019) and show that models' plausibility can be improved with no loss in accuracy.
The impact of lexical and grammatical processing on generating code from natural language
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Code Translation, Decoder, Translation

Considering the seq2seq architecture of Yin and Neubig (2018) for natural language to code translation, we identify four key components of importance: grammatical constraints, lexical preprocessing, input representations, and copy mechanisms. To study the impact of these components, we use a state-of-the-art architecture that relies on BERT encoder and a grammar-based decoder for which a formalization is provided. The paper highlights the importance of the lexical substitution component in the current natural language to code systems.
How to be Helpful on Online Support Forums?
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Internet forums such as Reddit offer people a platform to ask for advice when they encounter various issues at work, school or in relationships. Telling helpful comments apart from unhelpful comments to these advice-seeking posts can help people and dialogue agents to become more helpful in offering advice. We propose a dataset that contains both helpful and unhelpful comments in response to such requests. We then relate helpfulness to the closely related construct of empathy. Finally, we are the first study to analyze the language features that are associated with helpful and unhelpful comments.
On a Benefit of Masked Language Model Pretraining: Robustness to Simplicity Bias
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Language Modeling, Language Modelling

Despite the success of pretrained masked language models (MLM), why MLM pretraining is useful is still a question not fully answered. In this work we theoretically and empirically that MLM pretraining makes models robust to lexicon-level spurious features, partly answering the question. Our explanation is that MLM pretraining may alleviate problems brought by simplicity bias (Shah et al., 2020), which refers to the phenomenon that a deep model tends to rely excessively on simple features. In NLP tasks, those simple features could be token-level features whose spurious association with the label can be learned easily. We show that MLM pretraining makes learning from the context easier. Thus, pretrained models are less likely to rely excessively on a single token. We also explore the theoretical explanations of MLM’s efficacy in causal settings. Compared with Wei et al. (2021), we achieve similar results with milder assumption. Finally, we close the gap between our theories and real-world practices by conducting experiments on real-world tasks.
Multimodal Sentiment Analysis with Common-sense Modulation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Common Sense Reasoning, Multimodal Sentiment Analysis, Sentiment Analysis

Our world is inherently multimodal and recent work highlights the importance of machine learning models leveraging multiple streams of information in making decisions. Multimodal sentiment analysis has been an active area of research that requires models to take advantage of the linguistic, acoustic, and visual signals available in an utterance. However, most current models do not take into account any social common-sense knowledge which is crucial in how we perceive sentiment in a conversation. To address that, in this paper, we aim to influence or modulate modality representations with common-sense knowledge obtained from a generative social common-sense knowledge base. We provide a novel way to modulate the linguistic, acoustic, and visual features corresponding to an utterance by scaling and shifting these representations. We use the knowledge base to obtain knowledge latent representations for an utterance corresponding to different states of the speaker such as the intent and the reaction, and we use it to shift and scale the three modalities. Our experiments on popular multimodal sentiment analysis benchmark datasets show that our proposed method is on par and often surpasses the current state-of-the art models.
Modular Domain Adaptation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Domain Adaptation, text-classification, Text Classification

Off-the-shelf models are widely used by computational social science researchers to measure properties of text, such as sentiment. However, without access to source data it is difficult to account for domain shift, which presents a threat to validity. Here, we treat domain adaptation as a modular process that involves separate model producers and model consumers, and show how they can independently cooperate to facilitate more accurate measurements of text. We introduce two lightweight techniques for this scenario, and demonstrate that they reliably increase out-of-domain accuracy on four multi-domain text classification datasets when used with linear and contextual embedding models. We conclude with recommendations for model producers and consumers, and release models and replication code to accompany this paper.
A New Dataset for Summarizing Radiology Reports
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Diagnostic

The radiology report summary is an important technology in smart healthcare. Compared with medical image processing and disease recognition which have been comprehensively studied, the research on radiology report summary is much limited, which is mainly due to the lack of a high-quality benchmark dataset. In this paper, we present a dataset called CRRsum for radiology report summary, where it is constructed from over 10K real radiology reports that contain diagnostic findings and diagnostic opinions. We perform extensive evaluation with the current state-of-the-art methods for radiology report summary on the proposed dataset. An extensive evaluation is performed with the current state-of-the-art methods for radiology report summary on our proposed dataset. Our experiments reveal the challenges of radiology report summary and provide many opportunities for research going forward. We also show that the CRRsum can be used in medical classification to facilitate the research in this task.
KinyaBERT: a Morphology-aware Kinyarwanda Language Model
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Language Modeling, Language Modelling, model, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER)

Pre-trained language models such as BERT have been successful at tackling many natural language processing tasks. However, the unsupervised sub-word tokenization methods commonly used in these models (e.g., byte-pair encoding - BPE) are sub-optimal at handling morphologically rich languages. Even given a morphological analyzer, naive sequencing of morphemes into a standard BERT architecture is inefficient at capturing morphological compositionality and expressing word-relative syntactic regularities. We address these challenges by proposing a simple two-tier BERT architecture that leverages a morphological analyzer and explicitly represents morphological compositionality. Despite the success of BERT, most of its evaluations have been conducted on high-resource languages, obscuring its applicability on low-resource languages. We evaluate our proposed method on the low-resource morphologically rich Kinyarwanda language, naming the proposed model architecture KinyaBERT. A robust set of experimental results reveal that KinyaBERT outperforms solid baselines by 2% F1 score on a named entity recognition task and by 4.3% average score of a machine-translated GLUE benchmark. KinyaBERT fine-tuning has better convergence and achieves more robust results on multiple tasks even in the presence of translation noise.
Efficient Long Sequence Encoding via Synchronization
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Pre-trained Transformer models have achieved successes in a wide range of NLP tasks, but are inefficient when dealing with long input sequences. Existing studies try to overcome this challenge via segmenting the long sequence followed by hierarchical encoding or post-hoc aggregation. We propose a synchronization mechanism for hierarchical encoding. Our approach first identifies anchor tokens across segments and groups them by their roles in the original input sequence. Then inside Transformer layer, anchor embeddings are synchronized within their group via a self-attention module. Our approach is a general framework with sufficient flexibility -- when adapted to a new task, it is easy to be enhanced with the task-specific anchor definitions. Experiments on two representative tasks with different types of long input texts, NarrativeQA summary setting and wild multi-hop reasoning from HotpotQA, demonstrate that our approach is able to improve the global information exchange among segments while maintaining efficiency.
Consecutive Task-oriented Dialog Policy Learning
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Continual Learning, Management, Q-Learning

A practical dialog policy agent must be flexible and can handle new scenarios easily. To achieve this, the agent should be able to expand its knowledge base. This must be done, however, without affecting the agent's performance. Nevertheless, existing dialog systems fail to do so. In practice, new knowledge expansion and old experience preservation concurrently are conflicting. This is a challenging task which occurs regularly in continual learning. We present a novel model which can conduct consecutive dialog policy learning for a series of tasks without catastrophic forgetting. We tackle the issue from three different aspects: (1) For effective old task preservation, we employ a continual Q-learning module which is based on replayed experience to retain the policy trained on historic tasks. (2) For efficient new task acquisition, we integrate an invariant risk minimization module to learn a stable policy predictor to avoid spurious corrections in the training data.  (3) For saving for storing the replayed experiences, we introduce a linear-decay replay buffer management. The effectiveness of the proposed model is evaluated theoretically and experimentally by both simulation and human.
Simple yet Powerful: An Overlooked Architecture for Nested Named Entity Recognition
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER, Nested Named Entity Recognition

Named Entity Recognition (NER) is an important task in Natural Language Processing that aims to identify text spans belonging to predefined categories. Traditional NER research ignores nested entities, which are entities contained in other entity mentions. Although several methods have been proposed to address this case, most of them rely on complex task-specific structures and ignore potentially useful baselines for the task. We argue that this creates an overly optimistic impression of their performance. This paper revisits the Multiple LSTM-CRF (MLC) model, a simple, overlooked, yet powerful approach based on training independent sequence labeling models for each entity type. Extensive experiments with three nested NER corpora show that, regardless of the simplicity of this model, its performance is better or at least as well as more sophisticated methods. Furthermore, we show that the MLC architecture achieves state-of-the-art results in the Chilean Waiting List corpus by including pre-trained language models. In addition, we propose new task-specific metrics that adequately measure the ability of models to detect nestings. The results show that standard NER metrics do not measure well the ability of a model to detect nested entities, while our task-specific metrics provide new evidence on how existing approaches handle the task.
Distantly Supervised Named Entity Recognition via Confidence-Based Multi-Class Positive and Unlabeled Learning
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER

In this paper, we study the named entity recognition (NER) problem under distant supervision. Due to the incompleteness of the external dictionaries and/or knowledge bases, such distantly annotated training data usually suffer from a high false negative rate. To this end, we formulate the Distantly Supervised NER (DS-NER) problem via Multi-class Positive and Unlabeled (MPU) learning and propose a theoretically and practically novel CONFidence-based MPU (Conf-MPU) approach. To handle the incomplete annotations, Conf-MPU consists of two steps. First, a confidence score is estimated for each token of being an entity token. Then, the proposed Conf-MPU risk estimation is applied to train a multi-class classifier for the NER task. Thorough experiments on two benchmark datasets labeled by various external knowledge demonstrate the superiority of the proposed Conf-MPU over existing DS-NER methods.
Contextual Fine-to-Coarse Distillation for Coarse-grained Response Selection in Open-Domain Conversations
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Retrieval

We study the problem of coarse-grained response selection in retrieval-based dialogue systems. The problem is equally important with fine-grained response selection, but is less explored in existing literature. In this paper, we propose a Contextual Fine-to-Coarse (CFC) distilled model for coarse-grained response selection in open-domain conversations. In our CFC model, dense representations of query, candidate contexts and responses is learned based on the multi-tower architecture using contextual matching, and richer knowledge learned from the one-tower architecture (fine-grained) is distilled into the multi-tower architecture (coarse-grained) to enhance the performance of the retriever. To evaluate the performance of the proposed model, we construct two new datasets based on the Reddit comments dump and Twitter corpus. Extensive experimental results on the two datasets show that the proposed method achieves huge improvement over all evaluation metrics compared with traditional baseline methods.
DialogVED: A Pre-trained Latent Variable Encoder-Decoder Model for Dialog Response Generation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Decoder, Diversity, Language Modeling, Language Modelling, Response Generation

Dialog response generation in open domain is an important research topic where the main challenge is to generate relevant and diverse responses. In this paper, we propose a new dialog pre-training framework called DialogVED, which introduces continuous latent variables into the enhanced encoder-decoder pre-training framework to increase the relevance and diversity of responses. With the help of a large dialog corpus (Reddit), we pre-train the model using the following 4 tasks, used in training language models (LMs) and Variational Autoencoders (VAEs) literature: 1) masked language model; 2) response generation; 3) bag-of-words prediction; and 4) KL divergence reduction. We also add additional parameters to model the turn structure in dialogs to improve the performance of the pre-trained model. We conduct experiments on PersonaChat, DailyDialog, and DSTC7-AVSD benchmarks for response generation. Experimental results show that our model achieves the new state-of-the-art results on all these datasets.