Attribute Injection for Pretrained Language Models: A New Benchmark and an Efficient Method (2022)

Abstract: Metadata attributes (e.g., user and product IDs from reviews) can be incorporated as additional inputs to neural-based NLP models, by expanding the architecture of the models to improve performance. However, recent models rely on pretrained language models (PLMs), in which previously used techniques for attribute injection are either nontrivial or cost-ineffective. In this paper, we introduce a benchmark for evaluating attribute injection models, which comprises eight datasets across a diverse range of tasks and domains and six synthetically sparsified ones. We also propose a lightweight and memory-efficient method to inject attributes into PLMs. We extend adapters, i.e. tiny plug-in feed-forward modules, to include attributes both independently of or jointly with the text. We use approximation techniques to parameterize the model efficiently for domains with large attribute vocabularies, and training mechanisms to handle multi-labeled and sparse attributes. Extensive experiments and analyses show that our method outperforms previous attribute injection methods and achieves state-of-the-art performance on all datasets.

## introduction neural-based nlp models are powered by largescale textual datasets, which are mostly crawled from the web @xcite @xcite @xcite . web texts usually are attached with metadata, i.e. attributes that describe the texts. for example, product reviews have user and product ids, as well as their ratings, while research papers on arxiv have author lists and research areas as metadata attributes (see figure 1 ). while most of the recent models disregard them and focus more on ungrounded language understanding * work done while reinald was at naver ai lab. he is now at google research. yelp review text: my boyfriend's fav. place and the stein of beers are priced pretty good. game nights get super packed so go early to save a seat. kitchen closes at midnight which is too early when your buzz kicks in around 1am. attributes: paper abstract text: we present new and improved fixed-parameter algorithms for computing maximum agreement forests (mafs) of pairs of rooted binary phylogenetic trees. the size of such a forest for two trees corresponds to their subtree prune-andregraft distance and, if the agreement forest is acyclic, to their hybridization number ... attributes: -authors: chris whidden, robert g. beiko, norbert zeh -research areas: q-bio.pe, cs.ds (understanding language on its own, e.g., glue; @xcite , prior work has shown that incorporating these attributes into our model increases not just its performance but also its interpretability and customizability @xcite @xcite . this work explores the task of attribute injection (amplayo, 2019), which aims to use attributes to improve the performance of nlp models effectively. previous methods for attribute injection @xcite @xcite @xcite @xcite @xcite involve two steps: (1) designing an architecture that accepts both texts and attributes, and (2) training the model from scratch using taskspecific datasets. however, these methods of modifying different modules of the model can be nontrivial when applied to pretrained language models (plms; @xcite @xcite . the use of plms disallows designing new and specialized architectures. @xcite append large and deep layers of attributespecific transformers to the end of plms, which cannot leverage the potential of attributes in the intermediate layers of plms and also scales poorly to multiple tasks and domains. finally, more recent work on language model customization and controllability makes use of textual prompts @xcite , specialized tokens @xcite , and additional neural modules @xcite @xcite to introduce additional contexts, such as style, topic, and end task. unfortunately, these techniques do not generalize to all kinds of attributes, such as those that are non-textual (e.g., user ids that are not text-translatable), multi-labeled (e.g., multiple authors of a paper), and with large vocabularies (e.g., thousands of products available). our contribution in this paper is two-fold. firstly, we introduce a new benchmark to evaluate models for attribute injection. the benchmark consists of eight datasets which include three newly collated ones from different tasks and domains, and six synthetically sparsified datasets which are specifically created to evaluate models in sparse settings. these datasets span from diverse tasks, such as sentiment classification, spoiler detection, and message type classification, and contain attributes that have different properties (sparse vs. non-sparse, singlelabeled vs. multi-labeled, etc.). experiments on the benchmark show that our method outperforms previous approaches, as well as competitive baselines that fully fine-tune the pretrained language model. we also conduct extensive analyses to show that our method is robust to sparse and cold-start attributes and that it is modular with attribute-specific modules transferrable to other tasks using the same attributes. secondly, we propose a lightweight and memoryefficient method that is specifically suited to inject attributes into plms, which can be non-textual, multi-labeled, and have large vocabularies. we make use of adapters @xcite , i.e. feed-forward modules inserted between layers of plms that are tiny in size, and extend them such that attributes are injected as additional inputs to the model. we introduce two kinds of injection methods, which either incorporate attributes independently of or jointly with the text representation. a naive implementation of the latter would substantially increase the parameters, especially when the attribute vocabulary is large, thus we use ideas from low-rank matrix approximations as well as parameterized hypercomplex multiplications @xcite to significantly decrease the fine-tuned parameters by up to 192× for a default base-sized bert @xcite setting. we also use two mechanisms, attribute dropout and post-aggregation, to handle attribute sparsity and multi-labeled attributes, respectively. our use of adapters enables us to parameterefficiently train our model, i.e. by freezing pretrained weights and only updating new parameters at training time. we make our code and datasets publicly available. foot_0 ## related work prior to the neural network and deep learning era, traditional methods for nlp have relied on feature sets as input to machine learning models. these feature sets include metadata attributes such as author lists and publication venue of research papers @xcite @xcite , topics of sentences @xcite @xcite , as well as spatial @xcite and temporal @xcite metadata attributes found in tweets. attributes are mostly used in the area of sentiment classification @xcite , where most of the time textual data includes freely available user and product attributes. these methods rely on manually curated features that would represent the semantics of user and product information. deep neural networks gave rise to better representation learning @xcite , which allows us to learn from scratch semantic representation of attributes in the form of dense vectors @xcite . the design of how to represent attributes has evolved from using attribute-specific word and document embeddings @xcite and attention pooling weights @xcite @xcite , to more complicated architectures such as memory networks @xcite and importance matrices (amplayo, 2019). these designs are model-and domain-dependent and can be non-trivial to apply to other models and datasets. our proposed method, on the other hand, works well on any pretrained language model which are mostly based on trans-former @xcite . @xcite used six layers of attributeinjected transformers where attributes are used as input to the self-attention module, which is costly in terms of memory, i.e., equivalent to adding 50% of the original bert-base parameters. attributes are useful in the intermediate layers when learning the semantics of the input text, which this model cannot leverage. our work is closely related to recent literature on controlled text generation, where most of the work use either specialized control tokens concatenated with the input text @xcite @xcite @xcite , or textual prompts that instruct the model on what to generate @xcite @xcite . while these methods have been successfully applied to pretrained language models, the attributes used to control the text are limited to those that are text-translatable (e.g., topics such as "technology") and those with limited vocabulary (e.g., "positive" or "negative" sentiment). methods to efficiently fine-tune pretrained models have been explored since their inception @xcite @xcite . one of the more popular methods is the use of adapters @xcite , where tiny trainable feed-forward modules are inserted and pretrained weights are frozen during training. most of the prior work on adapters focuses on either improving their effectiveness and efficiency @xcite @xcite or applying it to domain adaptation and transfer learning tasks @xcite @xcite . our work extends the use of adapters to additionally accept attributes as input, which is closely related to @xcite and @xcite , where the adapter accepts textual questions and commonsense knowledge as input, respectively. however, unlike previous work where the additional input can be transformed into natural language tokens, attributes are usually not the case (e.g., see user and product ids in figure 1 ), which poses new challenges in efficiency when applied to adapters. ## attribute injection benchmark problem setting let @xmath0, y is a task-specific output, and p(y|x) is a discriminative model that predicts y given x. suppose there exists a set of non-textual and categorical attributes @xmath1.g., user and product ids of product reviews). these attributes can be multi-labeled, i.e. z @xmath2.g., multiple authors of a research paper) and use a finite yet possibly large vocabulary z j , i.e. z j ⊆ z j . the task of attribute injection aims to build a model q(y|x, z) that additionally incorporates z as input such that the gain in task performance between p and q is maximized. in our setting, p is a pretrained language model (plm) fine-tuned to the task, while q is a plm that also takes z as additional input. the datasets to evaluate attribute injection models on a wide variety of tasks and datasets from different domains, we introduce attrib, a collection of benchmark datasets to evaluate attribute injection methods. attrib consists of a total of 14 datasets. the first eight datasets are collected from different domains with a diverse set of tasks spanning from sentiment classification to spoiler detection. three of the eight datasets are newly introduced in this paper. the first three datasets (y2013, y2014, and imdb; @xcite are sentiment classification datasets with user and product attributes from two different sources (yelp and imdb). aapr @xcite is a dataset for classifying whether an arxiv paper is accepted to a conference or not, with two attributes, lists of authors and research areas. polmed @xcite is a classification dataset in the political domain, where the goal is to classify the message type of a tweet, with four attributes, politician, media source, audience, and political bias. we also introduce three new attribute injection datasets (see appendix for details on dataset collection). food is a dataset in which given a recipe from food.com and three attributes, user and lists of ingredients and tags, we are tasked to predict the estimated number of minutes it takes to make the food, rounded down to the tens. good is a spoiler detection dataset where we classify whether a book review from goodreads contains a spoiler or not, with three attributes: user, book, and rating. and finally, beer is a multi-aspect rating predic- additionally, we collated six synthetically sparsified datasets that were created specifically to evaluate attribute injection methods in cold-start environments foot_1 . specifically, amplayo et al. (2018) downsampled the y2013 and imdb training datasets such that the attributes are sparser than the original. there are three levels of sparsity: 20%, 50%, and 80%, where x% means that x% of attributes are cold-start and thus are unseen in the training examples. statistics of these datasets are shown in the appendix. ## modeling approach our proposed method, which we call injectors, can be summarized as follows. we extend adapters @xcite , which are tiny feed-forward neural networks plugged into pretrained language models, such that they also accept attributes z as input. attributes z can be represented as additional bias parameters or as perturbations to the weight matrix parameter of the adapter, motivated by how attributes are used to classify texts. we decrease the number of parameters exponentially using lowrank matrix approximations and parameterized hypercomplex multiplications @xcite . finally, we introduce two training mechanisms, attribute dropout and post-aggregation, to mitigate problems regarding attribute sparsity and multilabel properties. figure 2 illustrates an overview of injectors. preliminary: adapters we first briefly describe adapters. let @xmath3. an adapter layer is basically two feed-forward networks that projects h into vector @xmath4 task-specific adapter injectors start with a task-specific adapter that uses equation 1 to transform the previous hidden vector h to h . the use of a separate task-specific adapter is essential to make our method modularizable and learned attributes on one task transferrable to another. we show extensive analyses of the modularity of our method in the later sections. attribute-specific adapters attributes z are injected through attribute-specific adapters, where they are used in two different ways. firstly, they are used as bias parameters independent of the text representation. this is motivated by the fact that attributes can have a prior disposition regardless of what is written in the text. for example, a user may tend to give lower review ratings than average. secondly, they are also used as weight parameters. this allows our method to jointly model attributes with the text representation. this is motivated by how attributes can change the semantics of the text. for example, one user may like very sweet food while another user may dislike it, thus the use of the word sweet in the text may mean differently to them. more formally, for each attribute z @xmath5, we obtain its embedding z j from a learned embedding matrix and sequentially transform the previously attribute-injected vector h z j-1 to attribute-injected vector h z j using the following equation: the calculation of the bias parameter b z j is trivial; we perform a linear transformation of the attribute embedding z j : and then define a large matrix ŵj,@xmath6, denoted by ⊗ between two matrices s o and a learned matrix a @xmath7, followed by a reshape and the hyperbolic tangent function: finally, we add the large matrices ŵo of each dimension. to sum up, we define g weight (z j ) as: low-rank (eq. 7) and phms (eqs. 8-9) are both necessary to achieve a high performance with decreased parameters. while the low-rank method in itself reduces the most parameters, it also reduces the expressive power of the model since it outputs rank-one matrices. phms mitigate this by performing a sum of kronecker products, increasing the figure 3 : an illustration of how attribute embedding z j is transformed into weight matrix w zj . the colored tensors are learned parameters, while the gray ones are derived. by using a set of tiny parameters a o and s o , we are able to obtain large matrices. when there are multiple labels for attribute z j , we process them separately and aggregate the resulting large matrices. attribute dropout and post-aggregation for cases where attributes are sparse and multi-labeled, we use the following mechanisms. firstly, we add a dropout mechanism that randomly masks out attributes from training instances with a rate r drop . this replicates how instances at test time would look, where some attributes are not found in the vocabulary. secondly, when there are more than one labels of an attribute, instead of aggregating them first before processing, as in @xcite , we perform aggregation post hoc for w z j ), i.e.: aggregating attribute embeddings reduces their individual representation power, while our postaggregation mechanism preserves this since a sum of non-linear transformations is injective @xcite . ## experimental setup training configuration for the plms, we used weights and settings of bert-base-uncased (devlin et al., 2019), available in the huggingface library @xcite . we set the dimensions of all parameters as follows: d @xmath8, d @xmath9, and @xmath10. using this setting and our parameter-saving method, we are able to decrease the parameters by 192× the naive method. to handle long input texts and fit them into the 512 token limit of plms, we truncate them by concatenating the first and last 250 tokens, following @xcite . we set both the general and attribute dropout rates to 0.2 and the batch size to 8. we used adam with weight decay @xcite to optimize our models with a learning rate of 3e -5 and 200k training steps, with the first 20k steps used to warm-up training linearly. to train our models, we added a logistic classifier that transforms the [cls] token into logits. the weights here are updated during training. we then used a cross-entropy loss to train the models on all datasets except for goodreads and beeradvocate. the goodreads dataset is very imbalanced towards the negative class (i.e., not a spoiler). we thus put more importance on detecting the spoiler class and used a weighted cross-entropy loss with 0.5 weight on the negative class and 1.0 weight on the positive class. for beeradvocate, we treat the task as a multi-task problem, where each aspect rating prediction is a separate task. thus, we used multiple classifiers, one for each aspect, and aggregate the losses from all classifiers by averaging. for polmed where there is no available development set, we performed a 10-fold cross-validation, following @xcite . ## results we evaluated system outputs with accuracy for all datasets except good, where we used f1-score. for brevity in beer, we took the average of the accuracy of all sub-tasks. our results are summarized in ## conclusions we considered the use of attributes as additional contexts when fine-tuning pretrained language models. we proposed the injector module, an extension of adapters that also accepts attributes as input. our method considers two kinds of injection strategies, uses parameter-saving techniques and introduces training mechanisms to account for sparse and multi-labeled attributes. we also introduced attrib, a collection of 14 datasets to evaluate attribute injection methods. experiments on this benchmark of various classification tasks showed that our method improves substantially over previous methods. finally, we conducted extensive analyses on how injectors handle attribute sparsity and verify their modularity. in the future, we plan to apply our methods to real-world data where there are millions of attributes. we also plan to explore the use of attribute injection methods for text generation tasks, i.e. injecting attributes when generating texts. our code and the attrib benchmark will be publicly available online at https://github.com/rktamplayo/injector .