Lattice- BERT : Leveraging Multi-Granularity Representations in C hinese Pre-trained Language Models (2021)

Abstract: Chinese pre-trained language models usually process text as a sequence of characters, while ignoring more coarse granularity, e.g., words. In this work, we propose a novel pre-training paradigm for Chinese — Lattice-BERT, which explicitly incorporates word representations along with characters, thus can model a sentence in a multi-granularity manner. Specifically, we construct a lattice graph from the characters and words in a sentence and feed all these text units into transformers. We design a lattice position attention mechanism to exploit the lattice structures in self-attention layers. We further propose a masked segment prediction task to push the model to learn from rich but redundant information inherent in lattices, while avoiding learning unexpected tricks. Experiments on 11 Chinese natural language understanding tasks show that our model can bring an average increase of 1.5% under the 12-layer setting, which achieves new state-of-the-art among base-size models on the CLUE benchmarks. Further analysis shows that Lattice-BERT can harness the lattice structures, and the improvement comes from the exploration of redundant information and multi-granularity representations. Our code will be available at https://github.com/alibaba/pretrained-language-models/LatticeBERT .

## introduction pre-trained language models (plms) have achieved promising results in many chinese natural language understanding (nlu) tasks @xcite @xcite . these models take a sequence of fine-grained units -chinese characters -as the input, following the english plms' practice @xcite . however, the meanings of many chinese words cannot be fully understood through direct compositions of their characters' meanings. for example, 老板/boss does not mean 老/elder 板/board. 2 the importance of word-level inputs in chinese has been addressed in different tasks, including relation classification @xcite , short text matching @xcite @xcite , trigger detection @xcite , and named entity recognition @xcite @xcite . the coarsegrained inputs benefit these tasks by introducing word-level semantics with multi-granularity representations, which is potentially complementary in character-level chinese plms. in this work, we discuss how to pre-train a chinese plm over a word lattice structure to exploit multi-granularity inputs. we argue that by incorporating the coarse-grained units into plm, models could learn to utilize the multi-granularity information for downstream tasks. specifically, we organize characters and words in sentences as word lattices (see figure 1 ), which enable the models to explore the words from all possible word segmentation results. however, it is not straightforward to learn a bert-like plm over the word lattices. the major challenges are two-folded. firstly, bert's original input is a sequence of characters ordered by their positions, making it difficult to consume the word lattices and preserve the positional relation- in mlm, models may refer to the other text units overlapping with the randomly masked one instead of the real context, which brings information leakages. to address these challenges, we propose a lattice-based bidirectional encoder representation from transformers (lattice-bert). specifically, we design a lattice position attention (lpa) to help the transformers directly exploit positional relationship and distances between text units in lattices. moreover, we propose a masked segment prediction (msp) task to avoid the potential leakage between overlapping text units in language modeling. with lpa and msp, the lattice-bert could harness the multi-granularity structures in lattices, thus, directly utilize the lattice structures to aggregate the coarse-grained word information to benefit various downstream tasks. we evaluate our model on 11 chinese nlu tasks in various paradigms, including the clue benchmarks @xcite as well as two sequence labeling tasks. compared with the baseline that only takes characters as inputs, lattice-bert bring an average increase of 1.5% and 2.0% under the settings of 12 and 6 layers, respectively. the 12-layer lattice-bert model beats all other base-size models on clue benchmarks. 3 morever, we show that lattice-bert can harness the multi-granularity in-puts and utilize word-level semantics to outperform vanilla fine-grained plms. our contributions can be summarized as 1) we propose lattice-bert to leverage multigranularity representations from word lattices in chinese plms. 2) we design lattice position attention and masked segment prediction to facilitate chinese plms to exploit the lattice structures. 3) lattice-bert brings remarkable improvements on 11 chinese tasks and achieves new state of the arts among base-size models at the clue benchmarks. ## lattice-bert this section, we detail the implementation of lattice-bert, and its overall framework is presented in figure 2 . ## experiments we present the details of the lattice-bert finetuning results on 11 chinese nlu tasks. answering the following questions: (1) whether the lattice-bert performs better than monogranularity plms and other multi-granularity plms? (2) how the proposed lattice position attention and masked segment prediction contribute to the downstream tasks? (3) how lattice-bert outperforms the original character-level plms? ## related works recently, several works utilize the lattice structures to explore multi-granularity information in chinese nlu tasks. @xcite incorporate lattice into recurrent neural networks based language modeling to capture marginal possibilities across all possible paths. in ner, lattice-lstm @xcite , graph neural networks @xcite , and flat-lattice transformers @xcite ) are adopted to incorporate words from lattice inputs. @xcite adapt convolutional neural networks to lattice for matching based question answering. @xcite adopt graph matching networks to perform multi-granularity interaction between lattices for text similarity. these works are designed to explore word lattices in specific tasks. we explore the multi-granularity representations with word lattices in plms, investigating the previously attempted downstream tasks as well as other tasks, e.g., mrc. we design lpa to meet various interaction needs of the downstream tasks and propose msp to avoid the leakages. in the field of chinese plms, some efforts incorporate coarse-grained information with the character-level inputs. ernie 1.0 @xcite and bert-wwm @xcite propose to mask the words, entities, and phrases as a whole in the mlm task to encourage the modeling of coarse-grained features. zen @xcite adopt auxiliary networks to integrate n-gram representations. bert-mwa @xcite propose word-aligned attention to use multiple segmentation boundaries. different from their methods, we propose lattice-bert to consume multi-granularity tokens simultaneously into one plm via lattice graphs. thus, lattice-bert explicitly exploits the representations of the coarse-grained units, as well as the interactions among word-and character-level tokens. the proposed msp task can be treated as an extension of the whole word masking @xcite , while considering the span information like @xcite according the lattice structures. the concurrent work, zhang and li (2020, ambert) investigate multi-granularity inputs similarly, but they use two transformer encoders to separately deal the word and character sequences. we treat the words and characters as lattice graphs, which enables thorough interactions among multi-granularity tokens and utilizes all potential segmentation results. ## conclusions in this paper, we propose lattice-bert to leverage multi-granularity representations of input sentences for chinese plms. specifically, lattice-bert takes a word-lattice as input, modeling the representations of words and characters simultaneously. we design the lattice position attention to embed the multi-granularity structure into transformers and propose the masked segment prediction task to avoid potential leakage in original mlm caused by the redundancy information in lattices. we conduct extensive experiments on 11 chinese nlu tasks and observe consistent gains over character-level baselines, achieving new sota on clue benchmarks. we show that lattice-bert can well manage the lattice inputs and utilize multi-granularity representations to augment the character-level inputs. we believe the lattice structure can be adapted to integrate the phrase and word representations into the word-piece based plms in other languages, which we leave for future exploration. energy comparison. we make the energy comparison between lbert and the previous stateof-the-art multi-granularity chinese plm, zhang and li (2020, ambert), from the following perspectives: (1) how many computational costs are introduced to process the additional coarsegrained word-level inputs in pre-training. (2) how many additional parameters are introduced by the word-level vocabulary comparing with vanilla finegrained chinese plms. specifically, from the view of additional computational costs, comparing with the character-level bert with other settings fixed, ambert @xcite introduces 100% more computational costs by adopting two encoders to deal with wordand character-level inputs separately. our proposed lbert-base adopts a uniform architecture to consume words and characters from word-lattices simultaneously. therefore, lbert only introduces 48% more computational costs comparing with the corresponding character-level baselines (see appendix c.1). on the other side, from the view of parameter scales, ambert has 176m parameters, where additional 68m parameters are introduced by the embedding of word-level tokens. with the embedding decomposition trick, lbert only introduces 10m parameters by the word-level units in vocabulary. as a result, lbert-base has 100m parameters, comparable to the character-level bertour (90m) and roberta (102m) models, and much smaller than ambert (176m). in conclusion, with fewer parameters and smaller computational costs, lbert is more efficient than am-bert.