G i BERT : Enhancing BERT with Linguistic Information using a Lightweight Gated Injection Method (2021)

Abstract: Large pre-trained language models such as BERT have been the driving force behind recent improvements across many NLP tasks. However, BERT is only trained to predict missing words – either through masking or next sentence prediction – and has no knowledge of lexical, syntactic or semantic information beyond what it picks up through unsupervised pre-training. We propose a novel method to explicitly inject linguistic information in the form of word embeddings into any layer of a pre-trained BERT. When injecting counter-fitted and dependency-based embeddings, the performance improvements on multiple semantic similarity datasets indicate that such information is beneficial and currently missing from the original model. Our qualitative analysis shows that counter-fitted embedding injection is particularly beneficial, with notable improvements on examples that require synonym resolution.

## introduction detecting the semantic similarity between a given text pair is at the core of many nlp tasks. it is a challenging problem due to the inherent variability of language and the limitations of surface form similarity. recent pre-trained language models such as elmo @xcite and bert @xcite have led to noticeable improvements in semantic similarity detection and subsequent work has explored how these architectures can be further improved. one line of work aims at model compression, making bert smaller and accessible while mostly preserving its performance @xcite @xcite @xcite . other studies seek to further improve model performance by enhancing bert with external information from knowledge bases @xcite or additional modalities @xcite . before the rise of contextualised models, transfer of pre-trained information between datasets and tasks in nlp was based on word embeddings. over many years, substantial effort was placed into the creation of such embeddings. while originally capturing mainly collocation patterns @xcite , subsequent work enriched these embeddings with additional information, such as dependencies @xcite , subword information @xcite and semantic lexicons @xcite . as a result, there exists a wealth of pre-trained embedding resources for many languages in a unified format which could provide complementary information for contemporary pre-trained contextual models. moreover, aligning contextual embeddings with static embeddings has shown to increase the performance of the former @xcite . we propose a new method for injecting pretrained linguistically-enriched embeddings into any layer of bert. the model maps any word embeddings into the same space as bert's hidden representations, then combines them using learned gating parameters. evaluation of this method on five semantic similarity tasks shows that injecting pre-trained dependency-based and counter-fitted embeddings can further enhance bert's performance. more specifically, we make the following contributions: ## results full model gibert with counter-fitted embeddings outperforms all other systems in both average f1 and average non-obvious f1 score (see f1 non-obvious f1 msrp quora semeval avg msrp quora semeval avg a b c a b c previous systems kelp ---.506 -----.199 --ecnu --.777 -----.707 ---bunji ----.197 -----.028 -bert .876 .902 .704 .473 .268 .645 .827 .860 .656 .243 .085 .534 sembert .876 .901 -.820 .874 -tbertdependency .882 .906 .780 .510 .242 .664 .827 .858 .728 .262 .090 .553 tbertcounter-fitted .879 .906 .756 .500 .215 .651 .824 .857 .699 .258 .064 .540 our implementation aibertdependency .863 .903 .738 .498 .282 .657 .792 .866 .681 .268 .090 .539 aibertcounter-fitted .877 .904 .724 .496 .263 .653 .835 .867 .662 .264 .076 .541 gibertdependency .883 .904 .768 .474 .238 .653 .849 .864 .704 .231 .087 .547 gibertcounter-fitted .884 .907 .780 .511 .256 .668 .858 .862 .719 .248 .090 .555 gibert has the fewest parameters of all bertenhancing models (see appendix f) and doesn't require any additional preprocessing tools (such as the neural srl tagger required by sembert), making it more efficient than sembert, tbert and aibert. the largest improvement of gibert over bert is observed with counter-fitted embeddings, especially on semeval a and b (the datasets with the highest proportion of examples involving synonym pairs, see our results indicate that semantic information is more important for the tasks at hand and syntactic information benefits from a late integration. gating mechanism catastrophic forgetting is a potential problem when introducing external information into a pre-trained model as the injected information could disturb or completely overwrite existing knowledge @xcite . in our proposed model, a gating mechanism is used to scale injected embeddings before adding them to the pre-trained internal bert representation (see section 3.1). to understand the importance of this mechanism, in counter-fitted embedding injection without gating improves on 4 out of 5 datasets, with further improvements when adding gating, outperforming the vanilla bert model across all datasets. in addition, gating makes model training more stable and reduces failed runs (where the model predicted only the majority class) on the particularly imbalanced semeval c dataset. this highlights the importance of the gating mechanism in our proposed method. ## conclusion in this paper, we introduced a new approach for injecting external information into bert. our proposed method adds linguistically enriched embeddings to bert's internal representation through a lightweight gating mechanism which requires fewer parameters than previous approaches. evaluating our injection method on multiple semantic similarity detection datasets, we demonstrated that injecting counter-fitted embeddings clearly improved performance over vanilla bert and on average outperformed all baselines on the task, while dependency embedding injection achieved slightly smaller gains. in comparison to the multihead attention injection mechanism, we found the gated method at least as effective, with comparable performance for dependency embeddings and improved results for counter-fitted embeddings. in ablation studies, we showed that the choice of injection location and the use of the proposed gating mechanism are crucial for our architecture. our qualitative analysis highlighted that counter-fitted injection was particularly helpful for instances involving synonym pairs. future work could explore combining multiple embedding sources or injecting other types of information. another direction is to investigate the usefulness of embedding injection for other tasks.