Compressing Transformer-Based Semantic Parsing Models using Compositional Code Embeddings (2020)

Abstract: The current state-of-the-art task-oriented semantic parsing models use BERT or RoBERTa as pretrained encoders; these models have huge memory footprints. This poses a challenge to their deployment for voice assistants such as Amazon Alexa and Google Assistant on edge devices with limited memory budgets. We propose to learn compositional code embeddings to greatly reduce the sizes of BERT-base and RoBERTa-base. We also apply the technique to DistilBERT, ALBERT-base, and ALBERT-large, three already compressed BERT variants which attain similar state-of-the-art performances on semantic parsing with much smaller model sizes. We observe 95.15% 98.46% embedding compression rates and 20.47% 34.22% encoder compression rates, while preserving >97.5% semantic parsing performances. We provide the recipe for training and analyze the trade-off between code embedding sizes and downstream performances.

## introduction conversational virtual assistants, such as amazon alexa, google home, and apple siri, have become increasingly popular in recent times. these systems can process queries from users and perform tasks such as playing music and finding locations. a core component in these systems is a task-oriented semantic parsing model that maps natural language expressions to structured representations containing intents and slots that describe the task to perform. for example, the expression can you play some songs by coldplay? may be converted to intent: playsong, artist: coldplay, and the expression turn off the bedroom light may be converted to intent: turnofflight, device: bedroom. task-oriented semantic parsing is traditionally approached as a joint intent classification and slot * equal contribution, alphabetical order filling task. @xcite provide a comprehensive survey of models proposed to solve this task. researchers have developed semantic parsers based on recurrent neural networks @xcite @xcite , convolutional neural networks @xcite , recursive neural networks @xcite , capsule networks @xcite , and slotgated attention-based models @xcite . the current state-of-the-art models on snips @xcite , atis @xcite , and facebook top @xcite datasets are all based on bert-style @xcite encoders and transformer architectures @xcite @xcite . it is challenging to deploy these large models on edge devices and enable the voice assistants to operate locally instead of relying on central cloud services, due to the limited memory budgets on these devices. however, there has been a growing push towards the idea of tinyai foot_0 . in this paper, we aim to build space-efficient task-oriented semantic parsing models that produce near state-of-the-art performances by compressing existing large models. we propose to learn compositional code embeddings to significantly compress bert-base and roberta-base encoders with little performance loss. we further use albertbase/large @xcite and distilbert @xcite to establish light baselines that achieve similar state-of-the-art performances, and apply the same code embedding technique. we show that our technique is complementary to the compression techniques used in albert and distilbert. with all variants, we achieve 95.15% ∼ 98.46% embedding compression rates and 20.47% ∼ 34.22% encoder compression rates, @xmath0.5% semantic parsing performance preservation. ## experiments and analyses for transformer model training, we base our implementation on the huggingface transformers library v2.6.0 @xcite . we use the adamw optimizer @xcite with 10% warmup steps and linear learning rate decay to 0. forr code embedding learning, we base our implementation on that of @xcite . by default we learn code embeddings with 32 codebooks and 16 basis vectors per codebook. unless otherwise specified, hyperparameters are found according to validation performances from one random run. we conduct our experiments on a mixture of tesla m40, titan x, 1080 ti, and 2080 ti gpus. we use exact match (em) and intent accuracy as evaluation metrics. exact match requires correct predictions for all intents and slots in a query, and is our primary metric. ## conclusion current state-of-the-art task-oriented semantic parsing models are based on pretrained roberta-base (478mb) or bert-base (420mb). we apply dis-tilbert (256mb), albert-large (68mb), and