pwc-archive/papers-with-abstracts huggingface dataset:
Explicit Modeling the Context for Chinese NER
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Data Augmentation, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER

Named entity recognition (NER) is the foundation of many natural language processing tasks. Current NER models have achieved promising results. But as pointed by several studies, they fail with a high ratio on generalization tests such as invariance test because they heavily rely on name information. So, we propose a context module to explicitly model the contextual information, and a trainable balance factor is designed to incorporate the result of context module. To learn this factor, we propose several tailored data augmentation strategies to generate synthetic labels for it. These approaches help the model learn whether it should focus on the context. Our method achieves on average 1.2\% absolute improvement of F1 than BERT-CRF on three datasets. Moreover, our method performs on par with the best solutions who rely heavily on external features besides BERT. We also conduct invariance test to analyse the effect of the context information. The source code of our model and augmentation strategies will be available at anonymous.url.
An Unsupervised Multiple-Task and Multiple-Teacher Model for Cross-lingual Named Entity Recognition
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Cross-Lingual NER, Knowledge Distillation, Multi-Task Learning, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER, Transfer Learning

Cross-lingual named entity recognition task is one of the critical problem for evaluating the potential transfer learning techniques on low resource languages. Knowledge distillation using pre-trained multilingual language models between source and target languages have shown their superiority. However, existing cross-lingual distillation models merely consider the potential transferability between two identical single tasks across both domain. Other possible auxiliary tasks to improve the learning performance have not been fully investigated. In this study, based on the knowledge distillation framework and multi-task learning, we introduce the similarity metric model as an auxiliary task to improve the cross-lingual NER performance on target domain. Specifically, an entity recognizer and a similarity evaluator teachers are first trained in parallel from the source domain. Then, two tasks in the student model are supervised by the two teachers simultaneously. Empirical studies on the datasets across 7 different languages confirm the effectiveness of the proposed model.
A Multilingual Corpus for Socio-political Event Coreference Resolution
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: coreference-resolution, Coreference Resolution, Event Coreference Resolution

We propose a dataset for event coreference resolution, which is based on random samples drawn from multiple sources, languages, and countries. Early scholarship on event information collection has not quantified the contribution of event coreference resolution. We prepared and analyzed a representative multilingual corpus and measured the performance and contribution of the state-of-the-art event coreference resolution approaches. We found that almost half of the event mentions in documents co-occur with other event mentions and this makes it inevitable to obtain erroneous or partial event information. We showed that event coreference resolution could help improving this situation. Our contribution sheds light on a challenge that has been overlooked or hard to study to date. Future event information collection studies can be designed based on the results we present in this report.
Get the Point! Graph Enhanced Candidate Retrieval for Zero-shot Entity Linking
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Entity Linking, Entity Retrieval, Graph Neural Network, Retrieval, Sentence, Sentence Embedding, Sentence-Embedding, Sentence Embeddings

For the retrieval phase of the zero-shot entity linking task, BERT has been widely used to represent the mentions and entities with the sentence embeddings. However, the sentence embeddings obtained by BERT are dominated by the high-frequency words in the pre-training corpus, thus performing poorly especially when the mention/entity is a low-frequency word. To solve this issue, we propose a Graph enhanced Entity Retrieval (GER) framework, fusing word-level embedding with the sentence embedding from BERT.
Specifically, we construct a mention/entity centralized graph and design a Hierarchical Graph Neural Network (HGNN) to capture the word-level information. Experimental results on the ZESHEL dataset demonstrate that our proposal achieves a recall@64 of 84.72\%, a 2.66 points improvement compared to previous best results.
How do people talk about images? A study on open-domain conversation on images.
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Diversity

Open-domain conversation on images requires the model to consider the relation and balance between utterances and images in order to generate proper responses. This paper explore how human conduct conversation on images by investigating a well-constructed open-domain image conversation dataset, ImageChat. We examine the conversations on images from three perspectives: $\textit{image relevancy}$, $\textit{image information}$ and $\textit{utterance style}$. We show that objects in the image are indeed the most important element for conversations on image, which could be directly discussed or be a bait to other off-image conversations. Thus, being able to accurately detect objects in the image and knowing their attributes are essential to chat on image. Understanding the scenarios of the image, except extracting the image objects, is also a key factor to the conversation on images. Based on our analysis, we propose to enriching the image information with image caption and object tags, increasing the diversity and image-relevancy of generated responses. We believe that our analysis provides useful insights and directions that facilitate future research on open-domain conversation on images.
Contextual Representation Learning beyond Masked Language Modeling
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Language Modeling, Language Modelling, Masked Language Modeling, Representation Learning

Currently, masked language modeling (e.g., BERT) is the prime choice to learn contextualized representations. Due to the pervasiveness, it naturally raises an interesting question: how do masked language models (MLMs) learn contextual representations?  In this work, we analyze the learning dynamics of MLMs and find that it adopts sampled embeddings as anchors to estimate and inject contextual semantics to representations, which limits the efficiency and effectiveness of MLMs. To address these problems, we propose TACO, a simple yet effective representation learning approach to directly model global semantics. To be specific, TACO extracts and aligns contextual semantics hidden in contextualized representations to encourage models to attend global semantics when generating contextualized representations. Experiments on the GLUE benchmark show that TACO achieves up to 5x speedup and up to 1.2 points average improvement over MLM.
Multilingual pre-training with Language and Task Adaptation for Multilingual Text Style Transfer
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Style Transfer, Text Style Transfer

We exploit the pre-trained seq2seq model mBART for multilingual text style transfer. Using machine translated data as well as gold aligned English sentences yields state-of-the-art results in the three target languages we consider.  Besides, in view of the general scarcity of parallel data, we propose a modular approach for multilingual formality transfer, which consists of two training strategies that target adaptation to both language and task. Our approach achieves competitive performance without monolingual task-specific parallel data and can be applied to other style transfer tasks as well as to other languages.
Cross-lingual Word Embeddings in Hyperbolic Space
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Cross-Lingual Word Embeddings, Word Embeddings

Cross-lingual word embeddings can be applied to several natural language processing applications across multiple languages. Unlike prior works that use word embeddings based on the Euclidean space, this short paper presents a simple and effective cross-lingual Word2Vec model that adapts to the Poincaré  ball model of hyperbolic space to learn unsupervised cross-lingual word representations from a German-English parallel corpus. It has been shown that hyperbolic embeddings can capture and preserve hierarchical relationships.
We evaluate the model on both hypernymy and analogy tasks. The proposed model achieves comparable performance with the %standard 
vanilla Word2Vec model on the cross-lingual analogy task, the hypernymy task shows that the cross-lingual Poincaré Word2Vec model can capture latent hierarchical structure from free text across languages, which are absent from the Euclidean-based Word2Vec representations. Our results show that by preserving the latent hierarchical information, hyperbolic spaces can offer better representations for cross-lingual embeddings.
Input-specific Attention Subnetworks for Adversarial Detection
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Adversarial Attack

Self-attention heads are characteristic of Transformer models and have been well studied for interpretability and pruning.
In this work, we demonstrate an altogether different utility of attention heads, namely for adversarial detection. 
Specifically, we propose a method to construct input-specific attention subnetworks (IAS) from which we extract three features to discriminate between authentic and adversarial inputs. The resultant detector significantly improves (by over 7.5%) the state-of-the-art adversarial detection accuracy for the BERT encoder on 10 NLU datasets with 11 different adversarial attack types. We also demonstrate that our method (a) is more accurate for larger models which are likely to have more spurious correlations and thus vulnerable to adversarial attack, and (b) performs well even with modest training sets of adversarial examples.
MOROCCO: Model Resource Comparison Framework
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Computational Efficiency, model

A new generation of pre-trained transformer language models has established new state-of-the-art results on many tasks, even exceeding the human level in standard NLU benchmarks. Despite the rapid progress, the benchmark-based evaluation has generally relied on the downstream performance as a primary metric which limits the scope of model comparison in terms of their practical use. This paper presents MOdel ResOurCe COmparison (MOROCCO), a framework that allows to assess models with respect to their downstream quality combined with two computational efficiency metrics such as memory consumption and throughput during the inference stage. The framework allows for a flexible integration with popular leaderboards compatible with jiant environment that supports over 50 downstream tasks. We demonstrate the MOROCCO applicability by evaluating 10 transformer models on two multi-task GLUE-style benchmarks in English and Russian and provide the model analysis.
Calibration of Machine Reading Systems at Scale
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Claim Verification, Open-Domain Question Answering, Prediction, Question Answering, Reading Comprehension, Retrieval


In typical machine learning systems, an estimate of the probability of the prediction is used to assess the system's confidence in the prediction.
This confidence measure is usually uncalibrated; i.e.\ the system's confidence in the prediction does not match the true probability of the predicted output.
In this paper, we present an investigation into calibrating open setting machine reading systems
such as open-domain question answering and claim verification systems.
We show that calibrating such complex systems which contain discrete retrieval and deep reading components is challenging and current calibration techniques fail to scale to these settings. 
We propose simple extensions to existing calibration approaches that allows us to adapt them to these settings.
Our experimental results reveal that the approach works well, and can be useful to selectively predict answers when question answering systems are posed with unanswerable or out-of-the-training distribution questions.
HighIE: High-Order Inference for Entity Recognition, Relation Extraction, and Event Extraction
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Decoder, Event Extraction, Relation, Relation Extraction, Vocal Bursts Intensity Prediction

Most prior work on Information Extraction typically predicts labels of individual instances (e.g., event triggers, relations, entities) independently regardless of their interactions. We propose a novel framework, HighIE, that aims to integrate high-order cross-subtask and cross-instance dependencies in both learning and inference. High-order inference on label variables is an NP-hard problem. To address it, we propose a high-order decoder that is unfolded from an approximate inference algorithm. The experimental results show that our approach achieves consistent improvement compared with prior work.
A Meta-Learning Approach for Few-Shot (Dis)Agreement Identification in Online Discussions
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Meta-Learning, Opinion Mining, Transfer Learning

Online discussions are abundant with different opinions for a common topic, and identifying agreement and disagreement between online posts enables many opinion mining applications. Realizing the increasing needs to analyze opinions for emergent new topics (e.g., from "mask mandate" to "COVID vaccination") that however tend to lack annotations, we present the first meta-learning approach for few-shot (dis)agreement identification on a new topic with few labeled instances. We further design a lexicon based regularization loss and propose  domain-aware task augmentation for meta-training to enable the meta-learner to learn both domain-invariant cues and domain-specific expressions for (dis)agreement identification. Extensive experiments on two benchmark datasets and evaluation on three topic domains demonstrate the effectiveness of the meta-learning approach that consistently and noticeably outperforms the conventional transfer learning approach based on fine-tuning.
UNICON: Unsupervised Intent Discovery via Semantic-level Contrastive Learning
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Clustering, Contrastive Learning, Data Augmentation, Intent Discovery, Natural Language Understanding, Text Augmentation

Discovering new intents is crucial for expanding domains in dialogue systems or natural language understanding (NLU) systems. A typical approach is to leverage unsupervised and semi-supervised learning to train a neural encoder to produce representations of utterances that are adequate for clustering then perform clustering on the representations to detect unseen clusters of intents. Recently, instance-level contrastive learning has been proposed to improve representation quality for better clustering. However, the proposed method suffers from semantic distortion in text augmentation and even from representation inadequacy due to limitations of using representations of pre-trained language models, typically BERT. Neural encoders can be powerful representation learners, but the initial parameters of pre-trained language models do not reliably produce representations that are suitable for capturing semantic distances. To eliminate the necessity of data augmentation and reduce the negative impact of pre-trained language models as encoders, we propose UNICON, a novel contrastive learning method that utilizes auxiliary external representations to provide powerful guidance for the encoder. Neural encoders can be powerful representation learners, but the initial parameters of pre-trained language models do not reliably produce representations that are suitable for capturing semantic distances. To eliminate the necessity of data augmentation and reduce the negative impact of pre-trained language models as encoders, we propose UNICON, a novel contrastive learning method that utilizes auxiliary external representations to provide powerful guidance for the encoder.
Meta Learning for Code Summarization
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Code Summarization, Meta-Learning, Source Code Summarization

Source code summarization is the task of generating a high-level natural language description for a segment of programming language
code. Current neural models for the task differ in their architecture and the aspects of code they consider. In this paper, we show that three
SOTA models for code summarization work well on largely disjoint subsets of a large code-base. This complementarity motivates model
combination: We propose three meta-models that select the best candidate summary for a given code segment. The two neural models improve significantly over the performance of the best individual model, obtaining an improvement of 2.1 BLEU points on a dataset of
code segments where at least one of the individual models obtains a non-zero BLEU.
Context-Paraphrase Enhanced Commonsense Question Answering
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Machine Reading Comprehension, Question Answering, Reading Comprehension

Commonsense question answering (CQA) generally means that the machine use the mastered commonsense to answer questions without relevant background material, which is a challenging task in natural language processing. Many prior methods mainly retrieve question related evidences from the structured knowledge base as the background material of the question, while the extracted evidence is generally described through the entities and the relationship between the entities, making it difficult for the machine to understand the meaning of the evidence completely. In this paper, we integrate the paraphrase in WordNet and Wiktionary into the evidence extraction process and machine reading comprehension (MRC) model, and propose a context-paraphrase enhanced commonsense question answering method. Specifically, the context-paraphrase obtained by WordNet and Wiktionary is first incorporated into the construction process of the heterogeneous graph, and the question related triple is extracted based on the heterogeneous graph, the triple is converted to triple-text based on a relational template. Then, the triple-text is used as the context of the question to establish an association graph containing the relationship between the context entities and the paraphrases. We further integrate the association graph into the MRC model to better guide the model to answer. Experimental results on CommonsenseQA and OpenBookQA show that context-paraphrase is effective in improving the answer accuracy of the MRC model.
MetaPrompting: Learning to Learn Better Prompts
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Meta-Learning

Prompting method is regarded as one of the crucial progress for few-shot nature language processing. Recent research on prompting moves from discrete tokens based "hard prompts" to continuous "soft prompts", which employ learnable vectors as pseudo prompts and achieve better performance. Though showing promising prospects, these soft-prompting methods are observed to rely heavily on good initialization to take effect. Unfortunately, obtaining a perfect initialization for soft prompt requires understanding of inner language models working and elaborate design, which is no easy task and has to restart from scratch for each new task. To remedy this, we propose a generalized soft prompting method called MetaPrompting, which adopts the well-recognized model-agnostic meta-learning algorithm to automatically find better prompt initialization that facilitates fast adaptation to new prompting tasks. Experiments show MetaPrompting brings significant improvements on three different datasets (over 6.5 points improvements for 1-shot setting), and achieves new state-of-the-art performance.
Ask Me Anything in Your Native Language
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Cross-Lingual Question Answering, Question Answering, Retrieval

Cross-lingual question answering is a thriving field in the modern world, helping people to search information on the web more efficiently. One of the important scenarios is to give an answer even there is no answer in the language a person asks a question with. We present a novel approach to achieving a new state of the art in both retrieval and end-to-end tasks on the XOR TyDi dataset outperforming the previous results up to 10\% on several languages. We find that our approach can be generalized to more than 100 languages in zero-shot approach and outperform all previous models by 8\%.
Isomorphic Cross-lingual Embeddings for Low-Resource Languages
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Cross-Lingual Word Embeddings, Representation Learning, Word Embeddings

Recent research in cross-lingual representation learning has focused on offline mapping approaches due to their simplicity, computational efficacy, and ability to work with minimal parallel resources. However, they crucially depend on the assumption of embedding spaces being approximately isomorphic, which does not hold in practice, leading to poorer performance on low-resource and distant language pairs. In this paper, we introduce a framework to learn cross-lingual word embeddings, without assuming isometry, for low-resource pairs via joint exploitation of a related higher-resource language. Both the source and target monolingual embeddings are independently aligned to the related language, enabling the use of offline methods. We show that this approach successfully outperforms other methods on several low-resource language pairs in both bilingul lexicon induction as well as eigen value simialrity.