pwc-archive/papers-with-abstracts huggingface dataset:
Towards a Fine-Grained Multi-Domain Neural Machine Translation Using Inter-Domain Relationships
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Domain Adaptation, Machine Translation, Translation

While research on the domain adaptation task in neural machine translation has become popular recently, there exists no agreement on what constituents a domain, and most previous studies only focus on coarse-grained domain adaptation and their methods cannot be generalized if the domain size is large. In this work, we argue the necessity to study a fine-grained domain adaptation problem. We build a new multilingual dataset from web sources that focus on fine-grained domains and inter-domain attributes and relationships. We also propose a simple but effective adaptation method to incorporate domain knowledge leveraging models in information networks.
AutoTriggER: Named Entity Recognition with Auxiliary Trigger Extraction
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Low Resource Named Entity Recognition, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER

Deep neural models for low-resource named entity recognition (NER) have shown impressive results by leveraging distant super-vision or other meta-level information (e.g. explanation). However, the costs of acquiring such additional information are generally prohibitive, especially in domains where existing resources (e.g. databases to be used for distant supervision) may not exist. In this paper, we present a novel two-stage framework (AutoTriggER) to improve NER performance by automatically generating and leveraging "entity triggers" which are essentially human-readable clues in the text that can help guide the model to make better decisions. Thus, the framework is able to both create and leverage auxiliary supervision by itself. Through experiments on three well-studied NER datasets, we show that our automatically extracted triggers are well-matched to human triggers, and AutoTriggER improves performance over a RoBERTa-CRFarchitecture by nearly 0.5 F1 points on average and much more in a low resource setting.
Question Answering Infused Pre-training of General-Purpose Contextualized Representations
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), Question Answering, Sentiment Analysis

We propose a pre-training objective based on question answering (QA) for learning general-purpose contextual representations,
motivated by the intuition that the representation of a phrase in a passage should encode all questions that the phrase can answer in context. To this end, we train a bi-encoder QA model, which independently encodes passages and questions, to match the predictions of a more accurate cross-encoder model on 80 million synthesized QA pairs. By encoding QA-relevant information, the bi-encoder's token-level representations are useful for non-QA downstream tasks without extensive (or in some cases, any) fine-tuning. We show large improvements over both RoBERTa-large and previous state-of-the-art results on zero-shot and few-shot paraphrase detection on four datasets, few-shot named entity recognition on two datasets, and zero-shot sentiment analysis on three datasets.
RE: A Study for Restorable Embeddings
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Language Modeling, Language Modelling, Passage Retrieval, Retrieval, Sentence

As the number of model parameters increased, large language models achieved linguistic fluency and exhibited high performance in various natural language tasks without gradient updates because the models could retain more knowledge.
However, the large model size makes difficult to apply the model to a task requiring domain knowledge not included in the training corpus, due to the fact that knowledge stored in model parameters is not controllable during generation and model parameter updates are costly.
To tackle the problem, we suggest separating the language model and knowledge, and divide the end-to-end language model into three parts: 1) encoding knowledge, 2) processing the encoded knowledge, and 3) restoring the processed knowledge embedding to natural language.
In this paper, we propose a model for learning restorable embeddings as a first step toward the study to separate the language model and knowledge.
The experimental results shows that the proposed model can restore most knowledge in 1-2 sentences by encoding knowledge in sentence-level embeddings and then restoring the embeddings back to the original sentence.
We also verify that the embeddings generated through our method significantly improves performance in the passage retrieval task.
How Distributed are Distributed Representations? An Observation on the Locality of Syntactic Information in Verb Agreement Tasks
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: feature selection, Sentence

This work addresses the question of the localization of the syntactic information encoding in the Transformers representations. We tackle this question from two perspectives, considering the object-past participle agreement in French, by identifying, first, in which part of the sentence and, second, in which part of the representation syntactic information is encoded. The results of our experiments, using probing, causal analysis and feature selection method, show that syntactic information is encoded locally in a way consistent with the French grammar.
Multi-head or Single-head? An Empirical Comparison for Transformer Training
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Multi-head attention plays a crucial role in the recent success of Transformer, which leads to consistent performance improvements over conventional attention in various applications. The popular belief is that its effectiveness stems from the ability to attend multiple positions jointly. In this paper, we first demonstrate that jointly attending multiple positions is not a unique feature of multi-head attention, as multi-layer single-head attention also attends multiple positions. Then, we suggest the main advantage of multi-head attention is the training stability, since it has fewer layers than the single-head attention when attending the same number of positions. Meanwhile, we show that, with recent advances in deep learning, we can successfully stabilize the training of the deep single-head Transformer. As the training difficulty is no longer a bottleneck, substantially deeper single-head Transformers achieve consistent performance improvements.
Can Language Models Be Specific? How?
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Language Modeling, Language Modelling, Specificity

A good speaker needs not only to be correct but also to be specific, and so are language models. In this paper, we propose to measure how specific the language of pre-trained language models (PLMs) is. To achieve this, we introduce a novel approach to build a benchmark for specificity testing by forming masked token prediction tasks with prompts. For instance, given ``J. K. Rowling was born in [MASK].'', we want to test whether a more specific answer will be better filled in by PLMs, e.g., Yate instead of England.
From our evaluations, we show that existing PLMs have only a slight preference for more specific answers, indicating that PLMs are weak in specificity. We identify underlying factors affecting the specificity and design two prompt-based methods to improve the specificity. Results show that the specificity of the models can be improved by the proposed methods without additional training. We believe this work can provide a new insight for language modeling and encourage the research community to further explore this important but understudied problem.
Control False Negative Instances In Contrastive Learning To ImproveLong-tailed Item Categorization
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Classification, Contrastive Learning, Representation Learning, Self-Learning, text-classification, Text Classification

Item categorization (IC) is an important core technology in e-commerce natural language processing (NLP). Given category labels' long-tailed distribution, IC performances on tail labels tend to be poor due to sporadic supervision. To address the long-tail issue in classification, an increasing number of methods have been proposed in the computer vision domain. In this paper, we adopted a new method, which consists of decoupling the entire classification task into (a) learning representations in a k-positive contrastive learning (KCL) way and (b) training a classifier on balanced data set, into IC tasks. Using SimCSE to be our self-learning backbone, we demonstrated that the proposed method works on the IC text classification task. In addition, we spotted a shortcoming in the KCL: false negative instances (FN) may harm the representation learning step. After eliminating FN instances, IC performance (measured by macro-F1) has been further improved.
Text-to-Table: A New Way of Information Extraction
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Language Modelling, Relation Extraction

We study a new problem setting of information extraction (IE), referred to as text-to-table. In text-to-table, given a text, one creates a table or several tables expressing the main content of the text, while the model is learned from text-table pair data. The problem setting differs from those of the existing methods for IE. First, the extraction can be carried out from long texts to large tables with complex structures. Second, the extraction is entirely data-driven, and there is no need to explicitly define the schemas.  As far as we know, there has been no previous work that studies the problem. In this work, we formalize text-to-table as a sequence-to-sequence (seq2seq) problem.  We first employ a seq2seq model fine-tuned from a pre-trained language model to perform the task.  We also develop a new method within the seq2seq approach, exploiting two additional techniques in table generation: table constraint and table relation embeddings. We consider text-to-table as an inverse problem of the well-studied table-to-text, and make use of four existing table-to-text datasets in our experiments on text-to-table. Experimental results show that the vanilla seq2seq model can outperform the baseline methods of using relation extraction and named entity extraction. The results also show that our method can further boost the performances of the vanilla seq2seq model. We further discuss the main challenges of the proposed task. The code and data will be made publicly available.
Exploring and Adapting Chinese GPT to Pinyin Input Method
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Text Generation

While GPT has become the de-facto method for text generation tasks, its application to pinyin input method remains unexplored.
In this work, we make the first exploration to leverage Chinese GPT for pinyin input method.
We find that a frozen GPT achieves state-of-the-art performance on perfect pinyin.
However, the performance drops dramatically when the input includes abbreviated pinyin.
A reason is that an abbreviated pinyin can be mapped to many perfect pinyin, which links to even larger amount of Chinese characters.
We mitigate this issue with two strategies, including enriching the context with pinyin and optimizing the training process to help distinguish homophones. 
To further facilitate the evaluation of pinyin input method, we create a dataset consisting of 270K instances from 15 domains.
Results show that our approach improves the performance on abbreviated pinyin across all domains.
Model analysis demonstrates that both strategies contribute to the performance boost.
Prompt-Guided Few-Shot Event Detection
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Event Detection, Event Extraction, Few-Shot Learning

Practical applications of event extraction systems have long been hindered by their need for heavy human annotation. In order to scale up to new domains and event types, models must learn to cope with limited supervision, as in few-shot learning settings. To this end, the major challenge is to let the model master the semantic of event types, without requiring abundant event mention annotations. In our study, we employ cloze prompts to elicit event-related knowledge from pre-trained language models and further use event definitions and keywords to pinpoint the trigger word. By formulating the event detection task as an "identify-then-localize" procedure, we minimize the number of type-specific parameters, enabling our model to quickly adapt to event detection tasks for new types. Experiments on three event detection benchmark datasets (ACE, FewEvent, MAVEN) show that our proposed method performs favorably under fully supervised settings and surpasses existing few-shot methods by 16% F1 on the FewEvent dataset and 23% on the MAVEN dataset when only 5 examples are provided for each event type.
GenRE: A Generative Model for Relation Extraction
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Knowledge Base Population, model, Question Answering, Relation, Relation Extraction, Sentence

Relation extraction (RE) is an important information extraction task which provides essential information to many NLP applications such as knowledge base population and question answering. In this paper, we present a novel generative model for relation extraction (which we call GenRE), where RE is modeled as a sequence-to-sequence generation task. We explore various encoding schemes for the source and target sequences, and design effective schemes that enable GenRE to achieve state-of-the-art performance on three benchmark RE datasets. In addition, we introduce negative sampling and decoding scaling techniques which provide a flexible tool to tune the precision and recall performance of our GenRE model. Our approach can be extended to extract all relation triples from a sentence in one pass. Although the one-pass approach incurs certain performance loss, it is much more computationally efficient.
The Power of Prompt Tuning for Low-Resource Semantic Parsing
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Semantic Parsing

Prompt tuning has recently emerged as an effective  method  for  adapting  pre-trained  language  models  to  a  number  of  language  understanding and generation tasks.   In this paper,  we investigate prompt tuning for semantic parsing—the task of mapping natural language  utterances  onto  formal  meaning  representations.    On  the  low-resource  splits  of Overnight and TOPv2, we find that a prompt tuned T5-xl significantly outperforms its fine-tuned counterpart, as well as strong GPT-3 and BART  baselines.    We  also  conduct  ablation studies  across  different  model  scales  and  target representations, finding that, with increasing model scale, prompt tuned T5 models improve at generating target representations that are far from the pre-training distribution.
Cross-Utterance Conditioned VAE for Non-Autoregressive Text-to-Speech
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Diversity, text-to-speech, Text to Speech

Modelling prosody variation is critical for synthesizing natural and expressive speech in end-to-end text-to-speech (TTS) systems. In this paper, a cross-utterance conditional VAE (CUC-VAE) is proposed to estimate a posterior probability distribution of the latent prosody features for each phoneme by conditioning on acoustic features, speaker information, and text features obtained from both past and future sentences. At inference time, instead of the standard Gaussian distribution used by VAE, CUC-VAE allows sampling from an utterance-specific prior distribution conditioned on cross-utterance information, which allows the prosody features generated by the TTS system to be related to the context and is more similar to how humans naturally produce prosody. The performance of CUC-VAE is evaluated via a qualitative listening test for naturalness, intelligibility and quantitative measurements, including word error rates and the standard deviation of prosody attributes. Experimental results on LJ-Speech and LibriTTS data show that the proposed CUC-VAE TTS system improves naturalness and prosody diversity with clear margins.
ATOGAN:Adaptive Training Objective Generative Adversarial Network for Cross-lingual Word Alignment in Non-Isomorphic Embedding Spaces
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Generative Adversarial Network, Translation, Word Alignment, Word Translation

Cross-lingual word alignment is a task for word translation from monolingual word embedding spaces of two languages. Recent works are mostly based on supervised approaches, which need specific bilingual seed dictionaries. The unsupervised adversarial approaches, which utilize the generative adversarial networks to map the whole monolingual space, do not need any aligned data. However these approaches pay no attention to the problem of mode collapse and gradient disappearance in generative adversarial networks(GAN). We proposed an adaptive training objective generative adversarial network(ATOGAN). We combined particle swarm optimization(PSO) with GAN to select the training objective in GAN's training, which alleviates the problem of mode collapse and gradient disappearance. Moreover, we improved the word alignment by bi-directional mapping and consistency loss. Experimental results demonstrate that our approach is better than several state-of-the-art approaches in distant language pairs(non-isomorphic embedding spaces).
On the Use of Entity Embeddings from Pre-Trained Language Models for Knowledge Graph Completion
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Entity Embeddings, Knowledge Graph Completion, Knowledge Graphs, Language Modeling, Language Modelling, Model Selection

Recent work has found that entity representations can be extracted from pre-trained language models to develop knowledge graph completion models that are more robust to the naturally occurring sparsity found in knowledge graphs. In this work, we explore how to best extract and incorporate those embeddings. We explore the suitability of the extracted embeddings for direct use in entity ranking and introduce both unsupervised and supervised processing methods that can lead to improved downstream performance. We then introduce supervised embedding extraction methods and demonstrate that we can extract more informative representations. We also examine the effect of language model selection and find that the choice of model can have a significant impact. We then synthesize our findings and develop a knowledge graph completion model that significantly outperforms recent neural models.
CST5: Data augmentation for Code-Switched Semantic Parsing
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Data Augmentation, Semantic Parsing

Extending semantic parsers to code-switched input has been a challenging problem, primarily due to lack of labeled data for supervision.  In this work, we introduce CST5, a new data augmentation technique that finetunes a T5 model using a small seed set ($\approx$100 utterances) to generate code-switched utterances from English utterances. We demonstrate the effectiveness of CST5 by comparing baseline models which are trained without data augmentation to models which are trained with augmented data for varying amount of training data. 
By using CST5, one can achieve the same semantic parsing performance by using up to 20x less labeled data. 
To aid further research,  we release over 10k human annotated Hindi-English (Hinglish) code-switched utterances along with 170K CST5 generated code-switched utterances from the TOPv2 dataset. The generated data is of good quality deemed over 98\% natural and 89\% semantically equivalent by native annotators.
Mixture-of-Graphs: Zero-shot Relational Learning for Knowledge Graph by Fusing Ontology and Textual Experts
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Graph Embedding, Knowledge Graph Completion, Knowledge Graph Embedding, Relation, Relational Reasoning, Zero-Shot Learning

Knowledge Graph Embedding (KGE) have been proposed and succeed utilized to knowledge Graph Completion (KGC). But dominant KGE models often fail in zero-shot relational learning because they cannot learn effective representations for unseen relations. Previous studies mainly separately utilize the textual description of relation and its neighbor relations to represent unseen relations. In fact, the semantics of a relation can be expressed by three kinds of graphs: factual graph, ontology graph and textual description graph, and they can complement and enhance each other. Therefore, to obtain more accurate representation of relation in zero-shot learning, we propose the mixture-of-graphs (MoG) experts to improve the effect of current KGE for unseen relations. We build multi-aspect associations between seen and unseen relations which will be used directly to guide previous KGE methods such as TransE and RotatE on zero-shot relational learning. The experiments on multiple public datasets verify the effectiveness of the proposed method, which improves the state-of-the-art zero-shot relational learning method by 12.84% in Hits@10 on average.
Relevance in Dialogue: Is Less More? An Empirical Comparison of Existing Metrics, and a Novel Simple Metric
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Sensitivity

In this work, we evaluate various existing dialogue relevance metrics, find strong dependencies on the dataset, often with poor correlation with human scores of relevance, and propose modifications to reduce data requirements and domain sensitivity while improving correlation. With these changes, our metric achieves state-of-the-art performance on the HUMOD dataset (Merdivan et al., 2020) while reducing measured sensitivity to dataset by 50%. We achieve this without fine-tuning, using only 3750 unannotated human dialogues and a single negative example. Despite these limitations, we demonstrate competitive performance on four datasets from different domains. Our code including our metric and experiments is open sourced.
Non-Linear Relational Information Probing in Word Embeddings
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Relation, Word Embeddings

Pre-trained word embeddings such as SkipGram and GloVe are known to contain a myriad of useful information about words. In this work, we use multilayer perceptrons (MLP) to probe the relational information contained in these word embeddings. Previous studies that use linear models on the analogy and relation induction tasks have shown that SkipGram generally outperforms GloVe, suggesting that SkipGram embeddings contain more relational information than GloVe embeddings. However, by using non-linear probe like MLP, our results instead suggest that GloVe embeddings contain more relational information than SkipGram embeddings, but a good amount of that is stored in a non-linear form and thus previous linear models failed to reveal that. Interpreting our relation probes using post-hoc analysis provides us with an explanation for this difference.