pwc-archive/papers-with-abstracts huggingface dataset:
Multi-stage Distillation Framework for Cross-Lingual Semantic Similarity Matching
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Contrastive Learning, Knowledge Distillation, Semantic Similarity, Semantic Textual Similarity, XLM-R

Previous studies have proved that cross-lingual knowledge distillation can significantly improve the performance of pre-trained models for cross-lingual similarity matching tasks. However, the student model needs to be large in this operation.  Otherwise, its performance will drop sharply, thus making it impractical to be deployed to memory-limited devices. To address this issue, we delve into cross-lingual knowledge distillation and propose a multi-stage distillation framework for constructing a small-size but high-performance cross-lingual model. In our framework, the contrastive learning and an assistant model are introduced to prevent performance from being compromised during the compression process. The experimental results demonstrate that our method can compress the size of XLM-R and MiniLM by more than 50%, while the performance is only reduced by about 1%. In addition, our framework is model-independent and applicable to all transformer-based models.
The Dark Side of the Language: Pre-trained Transformers in the DarkNet
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Natural Language Understanding

Pre-trained Transformers are challenging human performances in many natural language processing tasks. 
The gigantic datasets used for pre-training seem to be the key for their success on existing tasks. 
In this paper, we explore how a range of pre-trained natural language understanding models perform on truly novel and unexplored data, provided by classification tasks over a DarkNet corpus. Surprisingly, results show that syntactic and lexical neural networks largely outperform pre-trained Transformers. This seems to suggest that pre-trained Transformers have serious difficulties in adapting to radically novel texts.
Exploring Low-dimensional Intrinsic Task Subspace via Prompt Tuning
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Why can pre-trained language models (PLMs) learn universal representations and effectively adapt to broad NLP tasks differing a lot superficially? In this work, we empirically find evidence indicating that the adaptations of PLMs to various few-shot tasks can be reparameterized as optimizing only a few free parameters in a unified low-dimensional intrinsic task subspace, which may help us understand why PLMs could easily adapt to various NLP tasks with small-scale data. To find such a subspace and examine its universality, we propose an analysis pipeline called intrinsic prompt tuning (IPT). Specifically, we resort to the recent success of prompt tuning and decompose the soft prompts of multiple NLP tasks into the same low-dimensional nonlinear subspace, then we learn to adapt the PLM to unseen data or tasks by only tuning parameters in this subspace. In the experiments, we study diverse few-shot NLP tasks and surprisingly find that in a 5-dimensional subspace found with 100 tasks, by only tuning 5 free parameters, we can recover 87% and 65% of the full prompt tuning performance for 100 seen tasks (using different training data) and 20 unseen tasks, respectively, showing great generalization ability of the found intrinsic task subspace. Besides being an analysis tool, IPT could further bring practical benefits, such as improving the prompt tuning stability.
Improving Syntactic Parsing with Consistency Learning
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Dependency Parsing, Multi-Task Learning

In this paper, we propose using \emph{consistency learning} to improve constituency and dependency parsing performances on a multi-task setting. It utilizes a consistent constraint between the predictions. While multi-task learning implicitly learns shared representations for multiple sub-tasks, our method introduces an explicit consistency objective, which encourages shared representations that result in consistent predictions. Our intuition is that correct predictions are more likely consistent ones. To introduce consistent constraints, we propose a general method for introducing consistency objectives, as well as other prior knowledge, into existing neural models. This method only requires a boolean function that tells whether or not the multiple predictions are consistent, which does not need to be differentiable. We demonstrate the efficacy of our method by showing that it out-performs a state-of-the-art joint dependency and constituency parser on CTB.
CSL: A Large-scale Chinese Scientific Literature Dataset for Cross-task Evaluation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Scientific literature serves as a high-quality corpus, which could provide natural annotated data for many natural language processing (NLP) research. In this work, we introduce a Chinese Scientific Literature dataset – CSL, which contains the titles, abstracts, keywords and academic fields of 400,000 papers. The rich semantic information in these scientific literature creates extensive NLP tasks and provides a natural cross-task scenario. Based on this, we present a cross-task few-shot benchmark. To evaluate the cross-task transferability of the model, we design scenarios with different aspects and difficulties. Compared with previous cross-task benchmarks, these tasks are constructed from homogeneous corpus, allowing researchers to investigate the relationships between tasks, without being disturbed by heterogeneous data sources, annotation, and other factors. We analyze the behavior of existing text-to-text models on the proposed benchmark, and reveal the challenges for cross-task generalization, which provides a valuable reference for future research. Code and data are publicly available at https://github.com/CSL-Dataset/CSL_Dataset.
ELLE: Efficient Lifelong Pre-training for Emerging Data
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Lifelong learning

Current pre-trained language models (PLM) are typically trained with static data, ignoring that in real-world scenarios, streaming data of various sources may continuously grow. This requires PLMs to integrate the information from all the sources in a lifelong manner. Although this goal could be achieved by exhaustive pre-training on all the existing data, such a process is known to be computationally expensive. To this end, we propose ELLE, aiming at efficient lifelong pre-training for emerging data. Specifically, ELLE consists of (1) function preserved model expansion, which flexibly expands an existing PLM's width and depth to improve the efficiency of knowledge acquisition; and (2) pre-trained domain prompts, which disentangle the versatile knowledge learned during pre-training and stimulate the proper knowledge for downstream tasks. We experiment ELLE with streaming data from $5$ domains on BERT and GPT. The results show the superiority of ELLE over various lifelong learning baselines in both pre-training efficiency and downstream performances. All the data, model parameters and codes used will be available upon publication.
Listen to Both Sides and be Enlightened! -- Hierarchical Modality Fusion Network for Entity and Relation Extraction
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), Relation, Relation Extraction

Multimodal named entity recognition and relation extraction (MNER and MRE) is a fundamental and crucial branch in multimodal learning. 
However, existing approaches for MNER and MRE mainly suffer from 1)  error sensitivity when images contain irrelevant concepts not mentioned in texts; and 2) large modality gap between image and text features, especially hierarchical visual features. 
To deal with these issues, we propose a novel Hierarchical Modality fusion NeTwork (HMNeT) for visual-enhanced entity and relation extraction, aim to reduce the modality gap and achieve more effective and robust performance. Specifically, we innovatively leverage hierarchical pyramidal visual features to conduct multi-layer internal integration in Transformer. We further present a dynamic gated aggregation strategy to decide modality integration according to different images. Extensive  experiments on three benchmark datasets demonstrate the effectiveness of our method, and achieve  state-of-the-art performance.
BARCOR: Towards A Unified Framework for Conversational Recommendation
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Conversational Recommendation, Recommendation Systems, Response Generation

Recommendation systems focus on helping users find items of interest in the situations of information overload, where users' preferences are typically estimated by past observed behaviors. In contrast, conversational recommendation systems (CRS) aim to understand users' preferences via interactions in conversation flows. CRS is a complex problem that consists of two main tasks: (1) recommendation and (2) response generation. Previous work often tried to solve the problem in a modular manner, where recommenders and response generators are separate neural models. Such modular architectures often come with a complicated and unintuitive connection between the modules, leading to inefficient learning and other issues. In this work, we propose a unified framework based on BART for conversational recommendation, which tackles two tasks in a single model. Furthermore, we also design and collect a lightweight knowledge graph for CRS in the movie domain. The experimental results show that the proposed methods achieve the state-of-the-art.
Efficient Hyper-parameter Search for Knowledge Graph Embedding
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Graph Embedding, Knowledge Graph Embedding

While hyper-parameters (HPs) are important for knowledge graph (KG) embedding, existing methods fail to search them efficiently. To solve this problem, we first analyze the properties of different HPs and quantize the transferability from small subgraph to the large graph. Based on the analysis, we propose an efficient two-stage search algorithm, which efficiently explores HP configurations on small subgraph at the first stage and transfers the top configurations for fine-tuning on the large whole graph at the second stage. Experiments show that our method can consistently find better HPs than the baseline algorithms with the same time budget. We achieve 10.8% average relevant improvement for four embedding models on the large-scale KGs in open graph benchmark.
Knowledge Enhanced Embedding: Improve Model Generalization Through Knowledge Graphs
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Knowledge Graphs, Sentence

Pre-trained language models have achieved excellent results in NLP and NLI, and since the birth of Bert, various new types of Bert have emerged.They are able to grasp the ubiquitous linguistic representational information from large-scale corpora in different ways, but when reading texts, it is difficult for them to combine and use external knowledge to make inferences about other meanings that the text may contain, as people do.To this end, we propose a linguistic model (K2E-BERT) capable of simply incorporating external knowledge, which fuses information from the knowledge graph (triad) with the entity information in the original text.In order to better integrate external knowledge into the original text without letting it deviate from the original meaning of the sentence, we propose a method called EaKA (Entity and Knowledge Align), which can better distance and combine entities and knowledge so that the model can accept new external knowledge without losing the meaning of the original sentence; additionally, we can easily and beyond Bert without changing the internal structure of Bert, we can easily and go beyond the results of BERT, which shows that our approach is feasible.After our experiments, we found good results in several NLP tasks we selected, which indicated that K2E-BERT easily surpassed BERT in generalization ability, proving its effectiveness.
The Algorithmic Inflection and Morphological Variability of Russian
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

We present a set of deterministic algorithms for Russian inflection and automated text synthesis. These algorithms are implemented in a publicly available web-service www.passare.ru. This service provides functions for inflection of single words, word matching and synthesis of grammatically correct Russian text. The inflectional functions have been tested against the annotated corpus of Russian language OpenCorpora and used for estimating the morphological variability and complexity of different parts of speech in Russian.
MoEfication: Conditional Computation of Transformer Models for Efficient Inference
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Mixture-of-Experts

Transformer-based pre-trained language models achieve superior performance on most NLP tasks due to large parameter capacity, but also lead to huge computation cost. Fortunately, we observe that most inputs only activate a tiny ratio of neurons of large Transformer-based pre-trained models during inference. Hence, we propose to convert a model into its mixture-of-experts (MoE) version with the same parameters, namely MoEfication, which accelerates large-model inference by conditional computation based on the sparse activation phenomenon. Specifically, MoEfication consists of two phases: (1) splitting the parameters of feed-forward neural networks (FFNs) into multiple parts as experts, and (2) building expert routers to decide which experts will be used for each input. Experimental results show that MoEfication can save $80\%$ computation cost of FFNs while maintaining over $95\%$ original performance for different models, including models with different sizes (up to 3 billion parameters) and distilled models, on various downstream tasks. Moreover, we find that the MoEfied model achieves better performance than the MoE model pre-trained from scratch with the same model size. We will release all the code and models of this paper.
Pre-training and Fine-tuning Neural Topic Model: A Simple yet Effective Approach to Incorporating External Knowledge
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Topic Models, Word Embeddings

Recent years have witnessed growing interests in incorporating external knowledge such as pre-trained word embeddings (PWEs) or pre-trained language models (PLMs) into neural topic modeling. However, we found that employing PWEs and PLMs for topic modeling only achieved limited performance improvements but with huge computational overhead. In this paper, we propose a novel strategy to incorporate external knowledge into neural topic modeling where the neural topic model is pre-trained on a large corpus and then fine-tuned on the target dataset. Experiments have been conducted on three datasets and results show that the proposed approach significantly outperforms both current state-of-the-art neural topic models and some topic modeling approaches enhanced with PWEs or PLMs. Moreover, further study shows that the proposed approach greatly reduces the need for the huge size of training data.
EigenNoise: A Contrastive Prior to Warm-Start Representations
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

In this work, we present a naïve initialization scheme for word vectors based on a dense, independent co-occurrence model and provide preliminary results that suggest it is competitive, and warrants further investigation. Specifically, we demonstrate through information-theoretic minimum description length (MDL) probing that our model, EigenNoise, can approach the performance of empirically trained GloVe despite the lack of any pre-training data (in the case of EigenNoise). We present these preliminary results with interest to set the stage for further investigations into how this competitive initialization works without pre-training data, as well as to invite the exploration of more intelligent initialization schemes informed by the theory of harmonic linguistic structure. Our application of this theory likewise contributes a novel (and effective) interpretation of recent discoveries which have elucidated the underlying distributional information that linguistic representations capture from data and contrast distributions.
Distilling Causal Metaknowledge from Massive Knowledge Graph
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Knowledge Graphs

In recent years, the growing information overload facilitates the access to billions  of  relational  facts in the world, which are usually integrated in all manner of knowledge graphs. The metaknowledge, defined as the knowledge about knowledge, reveals the inner principle of arising these factual knowledge, and hence is of vital importance to be discovered for the understanding, exploiting and completion of knowledge. In this paper, we focus on capturing the causal component of metaknowledge, that is a metarule with causal semantic.
For the propose, we devise an efficient causal rule discovery algorithm called CaRules that distills the causal rules between two knowledge graph schemata abstracted from instances from massive knowledge graphs. Extensive experiments demonstrate that the quality and interpretability of the causation-based rules outperform the correlation-based rules, especially in the out-of-distribution tasks.
Class Embeddings for Improved Out-of-Scope Detection in Intent Classification
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: intent-classification, Intent Classification, Sentence, Sentence Embeddings

This work presents a deep investigation of {\emph class embeddings} for intent classification, which had only been timidly addressed before. We show that there are advantages in representing classes as dense vectors instead of one-hot-encoded ones, similar to the way sentence embeddings became the standard representation instead of one-hot-encoding of bags of words. Some of our results corroborate previous findings showing that class embeddings can be a better approach to deal with out-of-scope examples. But this paper goes further by arguing that there are gains by simply changing from the traditional one-hot-encoding with softmax to one-hot embeddings. We also show that there is a lot of room for the better use of class embeddings by exploring random embeddings. The results suggest that optimal class embeddings could exist, with significant improvement on classification metrics, particularly out-of-scope detection. Additionally, we observe that the dimension of the class embedding might depend on the objective function, and that can open up for research on new neural network architectures to make better use of class embeddings.
A Comprehensive and Large-Scale Dataset for Integrated Argument Mining Tasks
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Argument Mining, Articles, Claim-Evidence Pair Extraction (CEPE), Claim Extraction with Stance Classification (CESC), Stance Classification

Traditionally, a debate usually requires a manual preparation process, including reading plenty of articles, selecting the claims, identifying the stances of the claims, seeking the evidences for the claims, etc. As the AI debate attracts more attention these years, it is worth exploring the methods to automate the tedious process involved in the debating system. In this work, we introduce a comprehensive and large dataset, which can be applied to a series of argument mining tasks, including claim extraction, stance classification, evidence extraction, etc. Our dataset is collected from over 1k articles related to 123 topics. Near 70k sentences in the dataset are fully annotated based on their argument properties (e.g., claims, stances, evidences, etc.). We further propose two new integrated argument mining tasks associated with the debate preparation process: (1) claim extraction with stance classification (CESC) and (2) claim-evidence pair extraction (CEPE). We adopt a pipeline approach and an end-to-end method for each integrated task separately. Promising experimental results are reported to show the values and challenges of our proposed tasks, and motivate future research on argument mining.
Prompting as Multimodal Fusing
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Language Modeling, Language Modelling

Tsimpoukelli et al. (2021) devise Frozen, empowering a language model to solve multimodal tasks by pretraining a vision encoder whose outputs are prompts fed to the language model. The vision encoder has a dual objective: extracting image features and aligning image/text representation spaces. We propose to disentangle the objectives by using prompt vectors to align the spaces; this lets the vision encoder focus on extracting image features. We show that this disentangled approach is modular and parameter-efficient for processing tasks that involve two or more modalities.
Detecting Unassimilated Borrowings in Spanish: An Annotated Corpus and Approaches to Modeling
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Word Embeddings

This work presents a new resource for borrowing identification and analyzes the performance and errors of several models on this task.
We introduce a new annotated corpus of Spanish newswire rich in unassimilated lexical borrowings---words from one language that are introduced into another without orthographic adaptation---and use it to evaluate how several sequence labeling models (CRF, BiLSTM-CRF, and Transformer-based models) perform. The corpus contains 370,000 tokens and is larger, more borrowing-dense, OOV-rich, and topic-varied than previous corpora available for this task. Our results show that a BiLSTM-CRF model fed with either Transformer-based embeddings pretrained on codeswitched data or a combination of contextualized word embeddings (along with character embeddings and Spanish and English subword embeddings) outperforms results obtained by a multilingual BERT-based model.
Causal Transformers: Improving the Robustness on Spurious Correlations
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Machine Translation, Natural Language Inference, NMT, Translation

The fully-connected dependencies in self-attention over-fit spurious correlations and limit the generalization on out-of-distribution data. Pre-trained language models (PLMs) alleviate this problem benefitted from the appreciable counterexamples in large-scale pre-training corpora. However, there is no study to resolve this problem by improving the model structure. We enforced the causal independence	 mechanism in the self-attention network, which constrains attention mapping topologies (AMGs) as causal structures. To implement it, we defined a smooth loss on the Markov boundary constrained directed acyclic graph (DAG) with the Lagrange duality, and used it to optimize the AMGs towards causal structures. Further, this causal attention network was applied on Transformer (Causal Transformer). The empirical results on two spurious correlation challenging (SCC) datasets, neural machine translation (NMT) and natural language inference (NLI) tasks demonstrated that our Causal Transformer outperforms the state-of-the-art model and improves the out-of-distribution prediction.