Enhancing Language Representation with Constructional Information for Natural Language Understanding (2023)

Abstract: Natural language understanding (NLU) is an essential branch of natural language processing, which relies on representations generated by pre-trained language models (PLMs). However, PLMs primarily focus on acquiring lexico-semantic information, while they may be unable to adequately handle the meaning of constructions. To address this issue, we introduce construction grammar (CxG), which highlights the pairings of form and meaning, to enrich language representation. We adopt usage-based construction grammar as the basis of our work, which is highly compatible with statistical models such as PLMs. Then a HyCxG framework is proposed to enhance language representation through a three-stage solution. First, all constructions are extracted from sentences via a slot-constraints approach. As constructions can overlap with each other, bringing redundancy and imbalance, we formulate the conditional max coverage problem for selecting the discriminative constructions. Finally, we propose a relational hypergraph attention network to acquire representation from constructional information by capturing high-order word interactions among constructions. Extensive experiments demonstrate the superiority of the proposed model on a variety of NLU tasks.

## introduction recent progress in natural language processing relies on pre-trained language models (plms) @xcite @xcite @xcite , which generate word representations by capturing syntactic and semantic features from their context. typically, plms (e.g., bert) employ the masked language model (mlm) as a pre-training objective, randomly masking some tokens from the input before predicting the original tokens. however, plms primarily focus on acquiring lexicosemantic information @xcite ## construction extraction and selection in this section, we elaborate on the details of extraction (q1) and selection of constructions (q2). ## model after extracting and determining the optimal set for constructions, the next challenge is to encode constructions in an effective way (q3). in this section, we first introduce hypergraph for modelling complex data correlation within constructions. then a relational hypergraph attention network (r-hgat) is proposed to encode the high-order word interactions inside each construction. finally, enhanced by the constructional information, the language representation is ready for nlu tasks. ## conclusion in this work, we introduce usage-based construction grammar (cxg) to enrich language representation with constructional information. then a hypergraph framework of cxg named hycxg is proposed to integrate constructional information for language representation enhancement. in hycxg, we extract constructions through a slot-constraint method, and formulate the conditional max coverage problem for selecting the discriminative constructions. then we propose a relational hypergraph attention network to encode high-order word interactions within constructions. extensive experiments illustrate the validity of our hycxg on natural language understanding tasks.