pwc-archive/papers-with-abstracts huggingface dataset:
Probing BERT’s priors with serial reproduction chains
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Language Modeling, Language Modelling, Masked Language Modeling

We can learn as much about language models from what they say as we learn from their performance on targeted benchmarks. Sampling is a promising bottom-up method for probing, but generating samples from successful models like BERT remains challenging. Taking inspiration from theories of iterated learning in cognitive science, we explore the use of serial reproduction chains to probe BERT's priors. Although the masked language modeling objective does not guarantee a consistent joint distribution, we observe that a unique and consistent estimator of the ground-truth joint distribution may be obtained by a GSN sampler, which randomly selects which word to mask and reconstruct on each step. We compare the lexical and syntactic statistics of sentences from the resulting prior distribution against those of the ground-truth corpus distribution and elicit a large empirical sample of naturalness judgments to investigate how, exactly, the model deviates from human speakers.  Our findings suggest the need to move beyond top-down evaluation methods toward bottom-up probing to capture the full richness of what has been learned about language.
Temporal Knowledge Graph Embedding based on Multivariate Gaussian Process
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Gaussian Processes, Graph Embedding, Knowledge Graph Embedding, Link Prediction

Recently, reasoning over Temporal Knowledge Graph (TKG), such as link prediction, has become an attractive research topic. 
Numerous Temporal Knowledge Graph Embedding (TKGE) methods have been proposed to map the entities and relations in TKG to the high-dimensional representations for further reasoning tasks. However, most existing TKGE methods \lh{which mainly based on deterministic vector embeddings, still} have two drawbacks. On the one hand, they mainly model temporal evolution of entities and relations by a deterministic function of time, which captures the global trends but fails at the surging local fluctuations. On the other hand, they mainly focus on the semantic meaning of embeddings, while losing the sight of temporal uncertainties of the embeddings. To tackle such limitations, in this paper, we propose a novel approach to mapping the entities and relations in TKG to multivariate Gaussian Processes (MGP). With the flexibility and capacity of MGP, the global trends as well as the local fluctuations can be simultaneously modeled. Moreover, the temporal uncertainties can be also captured with the kernel function and covariance matrix of MGP. Experimental results show the effectiveness of the proposed approach on two real-world benchmark datasets compared with  some state-of-the-art TKGE methods.
Sense Embeddings are also Biased -- Evaluating Social Biases in Static and Contextualised Sense Embeddings
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Word Embeddings

Sense embedding learning methods learn different embeddings for the different senses of an ambiguous word.
One sense of an ambiguous word might be socially biased while its other senses remain unbiased.
In comparison to the numerous prior work evaluating the social biases in pretrained word embeddings, the biases in sense embeddings have been relatively under studied.
In this paper, we create a benchmark dataset for evaluating the social biases in sense embeddings and propose novel sense-specific bias evaluation measures.
We conduct an extensive evaluation of multiple static and contextualised sense embeddings for various types of social biases using the proposed measures.
Our experimental results show that even in cases where no biases are found at word-level, there still exist worrying levels of social biases at sense-level, which are often ignored by the word-level bias evaluation measures.
More Informative Dialogue Generation via Multiple Knowledge Selection
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Dialogue Generation

Knowledge-grounded dialogue generation is a task of generating a fluent and informative response based on both dialogue context and a collection of external knowledge. There is a lot of noise in the knowledge pool, and appropriate knowledge selection plays an important role. Existing methods can only select one piece of knowledge to participate in the generation of the response, which inevitably loses some useful clues contained in the discarded candidates. In this work, we propose MSEL, a novel knowledge selector which could select multiple useful knowledge. MSEL takes the dialog context and knowledge pool as inputs and predicts a subset of knowledge pool in sequence-to-sequence manner. MSEL is easy to implement and can benefits from the generative pre-trained language models. Empirical results on the Wizard-of-Wikipedia dataset indicate that our model can significantly outperforms state-of-the-art approaches in both automatic and human evaluation.
Multilingual Document-Level Translation Enables Zero-Shot Transfer From Sentences to Documents
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Machine Translation, Sentence, Transfer Learning, Translation

Document-level neural machine translation (DocNMT) achieves coherent translations by incorporating cross-sentence context. However, for most language pairs there's a shortage of parallel documents, although parallel sentences are readily available. In this paper, we study whether and how contextual modeling in DocNMT is transferable via multilingual modeling. We focus on the scenario of zero-shot transfer from teacher languages with document level data to student languages with no documents but sentence level data, and for the first time treat document-level translation as a transfer learning problem. Using simple concatenation-based DocNMT, we explore the effect of 3 factors on the transfer: the number of teacher languages with document level data, the balance between document and sentence level data at training, and the data type of  document level data (genuine vs. back-translated). Our experiments on Europarl-7 and IWSLT-10 show the feasibility of multilingual transfer for DocNMT, particularly on document-specific metrics. We observe that more teacher languages and adequate data schedule both contribute to better transfer quality. Surprisingly, the transfer is less sensitive to the data type, where multilingual DocNMT delivers decent performance with either back-translated or genuine document pairs.
A Comparison of the Validity of Measurement Methods for the General English Proficiency through Dictation and Read-Aloud Performances
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

This paper compares three measurement methods for the general proficiency of learners of English as a second language (GEP). If students’ GEP can be measured on course materials frequently, for instance, at the beginning and end of a semester, English teachers can confirm students’ levels of learning achievement. So far, English teachers have two options for GEP measurement: calculating scores for read-aloud or those of dictation performance. This study expands an option to measure GEP using both dictation and read-aloud performances. When comparing the three types of measurement methods, the experimental results suggest that GEP should be measured by calculating dictation and read-aloud performances.
Fine-grained Category Discovery under Coarse-grained supervision with Hierarchical Weighted Self-contrastive Learning
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Contrastive Learning

In this paper, we propose a new task named Fine-grained Category Discovery under Coarse-grained supervision (FCDC). Without asking for any fine-grained knowledge, FCDC aims at discovering fine-grained categories with only coarse-grained labeled data, which can not only reduce significant labeling costs, but also adapt to novel fine-grained categories. It is also a challenging task since performing FCDC requires models to ensure fine-grained sample separability with only coarse-grained supervision and can easily make models overfit on the training set. Considering most current methods cannot transfer knowledge from coarse-grained level to fine-grained level, we propose a novel hierarchical weighted self-contrastive network to approach the FCDC task. Inspired by the hierarchy of pre-trained models (e.g. BERT), we combine supervised learning and contrastive learning to learn fine-grained knowledge from shallow to deep. Specifically, we use coarse-grained labels to train bottom layers of our model to learn surface knowledge, then we build a novel weighted self-contrastive module to train top layers of our model to learn more fine-grained knowledge. Extensive experiments on two public datasets show both effectiveness and efficiency of our model over state-of-the-art methods.
An Empirical Study on Explanations in Out-of-Domain Settings
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Recent work in Natural Language Processing has focused on developing approaches that extract faithful explanations, either via identifying the most important tokens in the input (i.e. post-hoc explanations) or by designing inherently faithful models that first select the most important tokens and then use them to predict the correct label (i.e. select-then-predict models). Currently, these approaches are largely evaluated on in-domain settings. Yet, little is known about how post-hoc explanations and inherently faithful models perform in out-of-domain settings. In this paper, we conduct an extensive empirical study that examines: (1) the out-of-domain faithfulness of post-hoc explanations, generated by five feature attribution methods; and (2) the out-of-domain performance of two inherently faithful models over six datasets. Contrary to our expectations, results show that in many cases out-of-domain post-hoc explanation faithfulness measured by sufficiency and comprehensiveness is higher compared to in-domain. We find this misleading and suggest using a random baseline as a yardstick for evaluating post-hoc explanation faithfulness. Our findings also show that select-then predict models demonstrate comparable predictive performance in out-of-domain settings to full-text trained models.
A Structure-Aware Argument Encoder for Literature Discourse Analysis
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Position, Representation Learning, Sentence

Existing research for argument representation learning mainly treats tokens in the sentence equally and ignores the implied structure information of argumentative context. In this paper, we propose to separate tokens into two groups, namely framing tokens and topic ones, to capture structural information of arguments. In addition, we consider high-level structure by incorporating paragraph-level position information. A novel structure-aware argument encoder is proposed for literature discourse analysis. Experimental results on both a self-constructed corpus and a public corpus show the effectiveness of our model.
Bottom Up Parsing via Sequence Labeling
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Discourse Parsing

We translate the sequence labeling framework, first introduced for top-down discourse parsing by Koto et al. (2021), to bottom-up discourse parsing. We introduce a novel parser that is not constrained by parsing direction (left-to-right or otherwise), and is conditioned on previous parsing decisions. We describe the unique training requirements of a (directionally) unconstrained parser and explore two different training procedures. Additionally, we introduce a novel dynamic oracle for unconstrained bottom-up parsing. Our proposed parser achieves state-of-the-art performance amongst bottom-up RST parsers.
DocEE: A Large-Scale and Fine-grained Benchmark for Document-level Event Extraction
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Document-level Event Extraction, Event Extraction, Sentence

Event extraction aims to identify an event and then extract the arguments participating in the event. Despite the great success in sentence-level event extraction, events are more naturally presented in the form of documents, with event arguments scattering in multiple sentences. However, a major barrier to promote document-level event extraction has been the lack of large-scale and practical training and evaluation datasets. In this paper, we present DocEE, a new document-level event extraction dataset including 20,000+ events, 100,000+ arguments. We highlight three features: large-scale manual annotations, fine-grained argument types and application-oriented settings. Experiments show that there is still a big gap between state-of-the-art models and human beings (43\% Vs 85\% in F1 score), indicating that DocEE is an open issue. We will publish DocEE upon acceptance.
VQN: Variable Quantization Noise for Neural Network Compression
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Neural Network Compression, Quantization

Quantization refers to a set of methods that compress a neural network by representing its parameters with fewer bits. However, applying quantization to a neural network after training often leads to severe performance regressions. Quantization Aware Training (QAT) addresses this problem by applying simulated training-time quantization for the model to learn robustness to inference-time quantization. One key drawback of this approach is that quantization functions induce biased gradient flow through the network during backpropagation, thus preventing the network from best-fitting to the learning task. Fan et al. addressed this issue by proposing Quant-Noise, in which simulated quantization is applied to a fixed proportion, called the quantization noise rate, of parameters during training. Our study, Variable Quantization Noise (VQN), builds upon their technique by exploring a variable quantization noise rate instead of a fixed one. We craft three candidate functions to vary noise rate during training and evaluate the variants with 3 datasets and 3 quantization schemes for each dataset. First, we report negative results on our hand-crafted candidate functions. Second, we observe somewhat positive results on a method, originally intended as an ablation study, of randomly varying the noise rate during training. This method outperforms Quant-Noise on two out of three quantization schemes for all three tested datasets. Moreover, on two of the datasets, this method at 4x compression matches or exceeds performance of even the uncompressed model. Future work should determine whether these unexpected results hold for more datasets and quantization schemes, as well as investigating other schemes for varying the noise rate during training.
Removing Out-of-Distribution Data Improves Adversarial Robustness
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Adversarial Robustness

Deep neural networks are vulnerable to adversarial examples crafted maliciously. Existing defense methods often improve the adversarial robustness of models by enlarging the training set with adversarial examples and fitting the models on the augmented training set. The introduced adversarial examples in the training stage may not be natural, which might even hurt the training distribution, resulting in lowter performance in the clean examples. Increasing the size of the training set also requires a long training time.We hypothesize that removing out-of-distribution samples from the training set could make the decision boundary of models smoother, leading to more robust models.We propose two methods to detect and remove out-of-distribution samples. Experimental results show that our methods can significantly boost the robustness of models without suffering any drop in clean accuracy.
A Structured Semantic Reinforcement method for Task-Oriented Dialogue
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Decision Making

Recently, many BERT based approachs have been proposed for task-oriented dialogue (TOD) task. Despite their impressive performance, the insufficient utilization of deep semantic information and long-distance context understanding makes it difficult for these methods to digest complex dialogue scenarios for they cannot obtain sufficient evidence from dialogue data to support dialogue decision-making.
In this work, we propose a novel structured semantics reinforcement (SSR) method to handle these issues.
SSR reorganized the end-to-end TOD structure, which mainly includes two key components:
  1. The dialogue symbolic memory, which cache the objects mentioned in the dialogue and the structure under the semantic relationship.
  2. semantic projection module,  understanding module, based on the previous structured results, determines the source of the slot extraction required for the current task.
And our approach achieves state-of-the-art results on dataset MultiWOZ 2.1, where we acquire a joint goal accuracy beyond 60\% and also gains a significant effect on dataset DSTC8.
Prototypical Verbalizer for Prompt-based Few-shot Tuning
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Contrastive Learning, Entity Typing, Few-Shot Learning, Topic Classification

Prompt-based tuning for pre-trained language models (PLMs) has shown its effectiveness in few-shot learning. Typically, prompt-based tuning wraps the input text into a cloze question. To make predictions, the model maps the output words to labels via a verbalizer, which is either manually designed or automatically built. However, manual verbalizers heavily depend on domain-specific prior knowledge and human efforts, while finding appropriate label words automatically still remains challenging.In this work, we propose the prototypical verbalizer (ProtoVerb) which is built directly from training data. Specifically, ProtoVerb learns prototype vectors as verbalizers by contrastive learning. In this way, the prototypes summarize training instances and are able to enclose rich class-level semantics. We conduct experiments on both topic classification and entity typing tasks, and the results demonstrate that ProtoVerb significantly outperforms current automatic verbalizers, especially when training data is extremely scarce. More surprisingly, ProtoVerb consistently boosts prompt-based tuning even on untuned PLMs, indicating an elegant non-tuning way to utilize PLMs.
Finding the Dominant Winning Ticket in Pre-Trained Language Models
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

The Lottery Ticket Hypothesis suggests that for any over-parameterized model, a small subnetwork exists to achieve competitive performance compared to the backbone architecture. In this paper, we study whether there is a winning lottery ticket for pre-trained language models, which allow the practitioners to fine-tune the parameters in the ticket but achieve good downstream performance. To achieve this, we regularize the fine-tuning process with L1 distance and explore the subnetwork structure (what we refer to as the "dominant winning ticket"). Empirically, we show that (a) the dominant winning ticket can achieve performance that is comparable with that of the full-parameter model, (b) the dominant winning ticket is transferable across different tasks, (c) and the dominant winning ticket has a natural structure within each parameter matrix. Strikingly, we find that a dominant winning ticket that takes up 0.05% of the parameters can already achieve satisfactory performance, indicating that the PLM is significantly reducible during fine-tuning.
CLGP: Multi-Feature Embedding based Cross-Attention for Chinese NER
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: NER

The previous works fused lexicon information while ignoring two important Chinese language characteristics: glyph and pinyin, which carry significant syntax and semantics information for sequence tagging tasks. This paper proposes CLGP, which utilizes three specific extractors to obtain the embeddings of the glyph, pinyin, and lexicon, and further uses a  network based on cross-attention to perform multi-feature embedding fusion. Specifically, we introduce the embedding scheme to preserve the lexicon matching results, and design two specific CNN architectures to extract glyph and pinyin embeddings. Moreover, we fuse the four embeddings by the cross-attention-based network to enhance the Chinese NER. The experimental results on four famous datasets show that CLGP achieves the SOTA performance.
One Agent To Rule Them All: Towards Multi-agent Conversational AI
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: All

The increasing volume of commercially available conversational agents (CAs) on the market has resulted in users being burdened with learning and adopting multiple agents to accomplish their tasks.
Though prior work has explored supporting a multitude of domains within the design of a single agent, the interaction experience suffers due to the large action space of desired capabilities.
To address these problems, we introduce a new task BBAI: Black-Box Agent Integration, focusing on combining the capabilities of multiple black-box CAs at scale.
We explore two techniques: question agent pairing and question response pairing aimed at resolving this task.
Leveraging these techniques, we design One For All (OFA), a scalable system that provides a unified interface to interact with multiple CAs.
Additionally, we introduce MARS: Multi-Agent Response Selection, a new encoder model for question response pairing that jointly encodes user question and agent response pairs.
We demonstrate that the OFA system is able to automatically and accurately integrate an ensemble of commercially available CAs spanning disparate domains.
Specifically, using the MARS encoder we achieve 88.5% accuracy on our BBAI task, outperforming strong baselines.
How to Translate Your Samples and Choose Your Shots? Analyzing Translate-train & Few-shot Cross-lingual Transfer
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Cross-Lingual Transfer, Machine Translation, Translation

Translate-train or few-shot cross-lingual transfer can be used to improve the zero-shot performance of multilingual pretrained language models. Few-shot utilizes high-quality low-quantity samples (often manually translated from the English corpus to the target language). Translate-train employs a machine translation of the English corpus, resulting in samples with lower quality that could be scaled to high quantity. Given the lower cost and higher availability of machine translation compared to manual professional translation, it is important to systematically compare few-shot and translate-train, understand when few-shot is beneficial, and whether choosing the shots to translate increases the few-shot gain. This work aims to fill this gap: we compare and quantify the performance gain of few-shot vs. translate-train using a varying number of samples for three tasks/datasets (XNLI, PAWS-X, XQuAD) spanning 17 languages. We show that scaling up the training data using machine translation gives a larger gain compared to using the small-scale (higher-quality) few-shot data. When few-shot is beneficial, we show that there are random sets of samples that perform better across languages and that the performance on English and on the machine-translation of the samples can both be used to choose the shots to manually translate for an increased few-shot gain.