Competing Independent Modules for Knowledge Integration and Optimization (2021)

Abstract: This paper presents a neural framework of untied independent modules, used here for integrating off the shelf knowledge sources such as language models, lexica, POS information, and dependency relations. Each knowledge source is implemented as an independent component that can interact and share information with other knowledge sources. We report proof of concept experiments for several standard sentiment analysis tasks and show that the knowledge sources interoperate effectively without interference. As a second use-case, we show that the proposed framework is suitable for optimizing BERT-like language models even without the help of external knowledge sources. We cast each Transformer layer as a separate module and demonstrate performance improvements from this explicit integration of the different information encoded at the different Transformer layers .

## introduction pre-trained language models such as bert @xcite are trained on large corpora with unsupervised end-to-end training. such monolithic systems cannot take advantage of extant outside grammatical or domain knowledge as complementary information. many language tasks benefit from knowledge sources that are known a priori. for sentiment analysis, for instance, various gazetteer lists and sentiment lexica encode sentiment words, word polarity, aspect-sentiment pairs, etc., which were proven to be effective as knowledge sources in different machine learning architectures @xcite @xcite . deep learning systems for sentiment analysis leverage sentiment words to enhance embedding representations by continuing the pre-training process of masked language models @xcite , or by re-training a modified version of language model that has intermediate layers for explicit encoding of sentiment knowledge @xcite . this knowledge integration into pre-trained models exceeds fine-tuning in computational cost and requires sophisticated training phase calibration. moreover, reported approaches only encode a single type of knowledge, either lexical @xcite or grammatical @xcite . for tasks that benefit from several types of knowledge, repeated retraining becomes prohibitive. this paper demonstrates the feasibility of using standard pre-trained language models and incorporate external off the shelf knowledge sources through dedicated and independent modules for each knowledge component. this framework reduces to some degree the need of ml experts for feature engineering for specific tasks, as creation of gazetteer lists and specialist lexica with domain scores is accessible to domain experts. modules incorporating knowledge sources are independent of one another to address major issues with combining extant knowledge with monolithic architectures, robustness, flexibility, and transparency. robustness independent modules make it possible to exploit multiple, possibly inconsistent knowledge sources in parallel while reducing interference effects. different knowledge sources that do not always agree on facts can cover a wider spectrum for the task at hand, but the inconsistency might hurt the leaning process. by spreading backpropagation independently over each module, training will weigh and assess usefulness of each module in context of the task and other modules. @xcite , @xcite have shown that independent modules make the overall system more robust in case of distribution shifts. flexibility since the modules are independent of one another, a module can be added/removed without further adjustments and, as our ablation experi-ments show, with only commensurate loss. moreover, the independent modules can be deployed at different points in the architecture. this paper demonstrates flexibility of the approach by showing effectiveness for input oriented modules that are concerned with token level information (i.e. sentiment value or pos tag), with relation information between tokens (grammatical dependencies), and modules at the transformer layer level, taking each transformer layer as an independent source of information and improving performance significantly. at the transformer layer level, modules permit the state of higher transformer layers to influence the state of lower layers at low cost. transparency module activation can be tracked. because the modules here encode different components independently, their activation patterns can be visualized and analyzed, an important advantage, especially for domain expert developers who might not be familiar with development environments used by ml experts. this paper presents a proof of concept of the interacting independent module framework on various sentiment analysis datasets. sentiment analysis is a much studied topic with different general sentiment lexica readily available and well understood benefits from dependency parses and domain specific sentiment lexica. the sentiment tasks we use span a range from simple analysis of a datapoint as two class classification task, as a three class classification, as a relationship oriented aspect based sentiment classification, and finally sentiment classification for tweets expressing figurative language. this variety of task structures for which we can use the same knowledge sources makes the results comparable and showcases the flexibility and robustness of the modules. ## related literature neural modular design has been the topic of interest for more than three decades @xcite , @xcite , @xcite , @xcite . most models proposed assume that only one expert is active at a particular time step but entnet @xcite and indrnn @xcite are propoals for sets of separate recurrent models, offering module independence, but no communication between modules. the recently proposed recurrent independent mechanisms (rims) @xcite , however, suggest to model a complex system by dividing the overall model into m communicative recurrent modules. the rims architecture was introduced for visual input. the independent mechanisms operate on the same input and do not have access to external information but rather make each module to specialize on a simpler problem by focusing in different parts of input. attempts at importing outside knowledge into neural architectures for language tasks have experimented with stacking (bi-)lstms @xcite , where we could interpret each layer of (bi-)lstms as a different module but with no independence and only one-way communication. for transformer architectures, adapters form a kind of module @xcite . adapters are trainable modules and can be interspersed between attention layers of frozen language models to provide a boost by learning task specific representations. the current proposal draws on several of these previous systems for a comprehensive architecture, where the independent modules can take different encodings as input. ## the proposed framework token level knowledge sources suppose there are n knowledge sources (@xmath0, . . . , n ) available. the annotations provided by nth knowledge source is encoded by an embedder e n . formally, e n produces a sequence x n 1 , x n 2 , . . . , x n t such as a token embeddings sequence, gazetteer lookup sequence, etc.) ## tasks and datasets we use different sentiment tasks and datasets to ensure that our observations are not task specific. we use only sentiment tasks, so that the same or similar external knowledge sources will be effective, again to ensure the observations are not specific to a (type of) knowledge source. in particular, we use sst-2 stanford sentiment tree-bank for binary sentiment classification of movie reviews @xcite , glue benchmark version foot_0 @xcite se17-4a semeval 2017 task 4 subtask a for 3class sentiment classification of tweets into negative, neutral and positive classes @xcite se14-5l semeval 2015 task 5 for aspect-based sentiment analysis of online reviews of laptops. se15-5l is a relation extraction and classification task. ## discussion the reported experiments demonstrate the capabilities of the competing independent modules both for leveraging external knowledge and integrating transformer layers of bert-like models. in both cases, the integration leads to improvements. integration of layers is a strong competitor for independent modules that leverage external knowledge. this suggests that the required knowledge is already encoded in the language models to some extent. the question is: which approach is preferred? 1 2 3 4 5 6 7 8 9 10 11 12 the embedding dimensions for external knowledge sources such as sentiment lexcia and pos are very small. this leads to a small number of trainable parameters in the independent modules that encode these knowledge sources. when integrating layers, since the input dimensions for all independent module is the same (768 in our experiments), the number of trainable parameters is significantly larger compared to the case of external knowledge sources. the choice between the two options depends on the target task and the availability of task specific knowledge sources. when these resources are available, the reduction in development and processing effort becomes very attractive, especially for small datasets. ## conclusions this paper presents a proof of concept for integrating external knowledge in competing, interacting independent modules. the reported experiments show consistent improvements when using readily available, off-the-shelf knowledge sources such as sentiment lexica, pos, and dependency relations encoded in independent modules. this is true even for knowledge sources that contradict each others' information, showing the robustness of the approach. when modules are in competition mode, further improvements can be achieved. experiments with two frozen language models demonstrate that task-specific knowledge sources in this architecture more than compensate for finetuning of the language model, with a significant reduction in the number of trainable parameters. we also show that the proposed framework is suitable for integration of the transformer layers of transformer-based language models by allowing lower layers to have access to the representations of high layers, i.e. bottom-up and top-down flow of information. moreover, the behavior of the independent modules can be visualized and the contribution of each module can be measured. in summary, interacting independent modules are a framework that enables computation restrained task adaptation with off-the-shelf external resources in a transparent fashion.