C o ELM : Construction-Enhanced Language Modeling (2024)

Abstract: Recent studies have shown that integrating constructional information can improve the performance of pre-trained language models (PLMs) in natural language understanding. However, exploration into leveraging constructional information to enhance generative language models for natural language generation has been limited. Additionally, probing studies indicate that PLMs primarily grasp the syntactic structure of constructions but struggle to capture their semantics. In this work, we encode constructions as inductive biases to explicitly embed constructional semantics and guide the generation process. We begin by presenting a construction grammar induction framework designed to automatically identify constructions from corpora. Subsequently, we propose the Construction-Enhanced Language Model (CoELM). It introduces a construction-guided language modeling approach that employs a dynamic sequence reassembly strategy during pre-training. Extensive experiments have demonstrated the superiority of CoELM across various benchmarks.

## introduction the constructions are linguistic structures that are viewed as a set of form-meaning pairs in construction grammar (cxg; @xcite . specifically, a construction is represented as a sequence of slot-constraints from a usage-based perspective @xcite , where the slots encompass different levels of abstraction (e.g., lexical, syntactic). based on these slots, constructions can be classified into three main types @xcite : (i) schematic constructions contain open slots solely and convey a general meaning. for example, "subject-verb-object1-object2" is a ditransitive construction @xcite that denotes the abstract † equal contribution and shared co-first authorship. * corresponding author. meaning of transfer. (ii) semi-idiomatic constructions are partially filled with lexical elements, for instance, "give-pronoun-a-break". (iii) idiomatic constructions are sequences composed of successive fixed words, such as "like a bat out of hell". recent studies @xcite have leveraged constructional information on encoder-only pre-trained language models (plms), such as bert and roberta @xcite . these studies demonstrate that incorporating constructional information can enhance the performance of language models in various natural language understanding (nlu) tasks. however, prior research has not explored the utilization of constructional information to augment generative language models for natural language generation (nlg). in particular, the prevalent approach in generative models features a decoder-only architecture @xcite , which contrasts with the encoder-only architecture in pre-training tasks. there is a significant gap that our work aims to address. while plms can access constructional information without explicit training @xcite , probing studies @xcite suggest that they primarily grasp the syntactic structures but fail to fully capture semantic aspects. in appendix b, we investigate the perception of language models for constructional semantics via probing experiments. larger models exhibit better awareness of constructional semantics for both semi-idiomatic and schematic constructions compared to smaller ones, which aligns with the findings by @xcite on plms' capacity to generalize hierarchical structures, albeit slowly. these observations indicate that constructions consist of hierarchically abstract slots, posing intrinsic challenges for plms as they involve noncompositional meanings not directly linked to individual words. consequently, this insight motivates us to adapt pre-training tasks to facilitate the swift acquisition of constructional information. in summary, our research focuses on developing effective pre-training strategies to efficiently learn constructional information, thereby enhancing language modeling capabilities essential for nlg. to this end, we propose a two-step approach. initially, we employ automated methods to identify constructions from corpora, creating a comprehensive inventory for pre-training. subsequently, these identified constructions are encoded as inductive biases during the pre-training phase, which allows model to explicitly capture constructional information and direct the generation process. in the first step, recognizing the labor-intensive nature of manual construction inventory creation, we prioritize automated methods for identifying constructions. these methods are advantageous as they can uncover a variety of patterns that linguists might overlook @xcite . in usage-based cxg, a construction is depicted as a sequence of slot-constraints. to quantify the relationship between individual words and generate these constraints, either frequency-based or associationbased models @xcite are applied. as suggested by @xcite , association-based models provide better generalizations for slot-constraints when compared to frequency-based counterparts. however, manual inspection of the inventory of constructions indicates that some constructions are relatively short or tend to contain generic labels (e.g., "preposition + his"; @xcite , lacking a clear mapping of form to a specific function or meaning. through empirical study, we explore the possible reasons. existing methods employ threshold-based strategies, such as frequency and association metrics @xcite , to assess the eligibility of adding new slots to a construction. a construction is truncated if the association strength between adjacent slots drops below the specified threshold. to tackle the issue, we introduce a construction grammar induction framework called cxglearner. within this framework, the association strength among slots is assessed through a plm-based association strength estimator (ase), which can consider more extended distances when assessing slot constraints. additionally, inspired by nucleus sampling @xcite , we establish a nucleus set that utilizes the ase's output distribution. this set guides the decision on whether to append slots to the sequence of candidate constructions, aiming to create complete constructions without resorting to rigid threshold-based truncation. for the second step, we propose a constructionguided language modeling approach with dynamic sequence reassembly strategy for pre-training. this strategy involves initially identifying all constructs within the input sequences that instantiate the constructions from the inventory. then we insert constructions before their corresponding constructs, thereby explicitly embedding constructional semantics to guide the generation process. and the curriculum learning @xcite is employed to regulate the reassembly of both the types and quantities of constructions in sequences. meanwhile, this approach ensures compatibility with existing generative language models, eliminating the need for additional decoding procedures. subsequently, our construction-enhanced language model (coelm) foot_0 is pre-trained from scratch. extensive experiments have validated the effectiveness of integrating constructional information into language modeling. consequently, encoding constructions as inductive biases not only expedites the acquisition of constructional semantics for language model but also significantly enhances its performance across a range of benchmark tasks. ## induction framework for cxg in this section, we propose an unsupervised computational framework, which automatically inducts generic usage-based constructions from corpora. ## construction-enhanced lm in this section, we explore how the acquired construction inventory can be utilized to enhance the constructional awareness of language modeling. ## related work construction grammar (cxg) has been applied to various natural language processing tasks. tsao and wible (2013) utilize constructions as contextual features for word similarity detection. raghuram et al. (2017) leverage embodied cxg to address the reference resolution problem, and nevens et al. (2019) apply fluid cxg for semantic parsing in visual question answering. xu et al. (2023) integrate constructional information to improve the performance of plms in nlu tasks. however, there has been no effort to ascertain whether constructions can provide benefits for generative language models. our work aims to bridge this gap. recent research investigates whether plms can access constructions. tayyar madabushi et al. 2020 find that bert is capable of accessing a considerable amount of constructional information without explicit training. @xcite conduct probing studies to show that plms can capture argument structure constructions. @xcite investigate plms' awareness of constructions and find that predicting open slots is more challenging than closed ones. @xcite find that while plms can recognize the construction, they struggle with understanding its meaning. fur-thermore, @xcite discover that both autoregressive and non-autoregressive plms can distinguish constructional patterns but fail to apply their meanings. these findings underscore the need to adapt pre-training tasks to enable plms to rapidly acquire constructional information. an inventory of constructions is a valuable resource for construction-based research. as manual extraction is often time consuming, automated approaches have been proposed to identify constructions from corpora. these methods are advantageous for uncovering patterns that may have been previously unexplored or highly conventional @xcite , potentially overlooked in traditional analyses. wible and tsao (2010) present a hybrid n-grams model to discover constructions that meet a frequency threshold. @xcite proposes an induction algorithm to extract constructions based on association strength. @xcite develop automatic tools to detect partially schematic constructions using frequency metrics. @xcite suggests that associationbased models produce better generalizations for slot-constraints compared to frequency-based models. however, manual analyses reveal that some extracted constructions are short or lack informativeness @xcite . @xcite argue that there is currently a lack of a high-quality and wide-coverage construction dataset. therefore, we investigate potential reasons and develop a construction induction framework to address the issue. ## conclusion our research is dedicated to developing effective pre-training strategies for the rapid acquisition of constructional information to enhance language modeling capabilities. to achieve this goal, we initially introduce a construction induction framework that automatically identifies constructions from corpora. we employ a plm-based association strength estimator (ase) and a nucleus set to generate complete constructions, avoiding the rigid threshold-based truncation. subsequently, we propose a construction-guided language modeling approach utilizing a dynamic sequence reassembly strategy. this method encodes constructions as inductive biases during pre-training, thereby guiding the generation process. we pretrain our construction-enhanced language model (coelm) from scratch, and extensive experiments have confirmed the effectiveness of coelm.