Multi-input Recurrent Independent Mechanisms for leveraging knowledge sources: Case studies on sentiment analysis and health text mining (2021)

Abstract: This paper presents a way to inject and leverage existing knowledge from external sources in a Deep Learning environment, extending the recently proposed Recurrent Independent Mechnisms (RIMs) architecture, which comprises a set of interacting yet independent modules. We show that this extension of the RIMs architecture is an effective framework with lower parameter implications compared to purely fine-tuned systems.

## introduction deep neural networks have been successfully applied to a variety of natural language processing tasks such as text classification, sequence labeling, sequence generation, etc. deep architectures are often non-modular, homogeneous systems and trained end-to-end. end-to-end training is performed with the hope that the structure of a networks is sufficient to direct gradient descent from a random initial state to a highly non-trivial solution @xcite . an important issue with the end-to-end training is that throughout the training of a system composed of several layers, valuable information contained in a problem decomposition that resulted in a specific network design is ignored @xcite . in non-modular systems, explicit decomposition of high level tasks into distinct subprocesses is not possible and necessary complexity has to be induced through the complexity of the input stimulus. this results in large systems whith the required number of training samples becoming intractable. interpretation of these black box systems is difficult @xcite . in compositional systems, in contrast, smaller modules encode specialized expertise which is known to impact one aspect of the task at hand. the aggregation of the modules acts synergistically to address the overall task. in a modular system, the components act largely independently but communicate occasionally. module autonomy is crucial because in the case of distributional shifts (significant changes in some modules), other modules should remain robust @xcite , @xcite . modules also need to interact occasionally to achieve compositional behavior @xcite . many current neural modular systems, such as entnet @xcite and indrnn @xcite , offer only module independence, but no module communication. the recently proposed recurrent independent mechanisms (rims) @xcite , however, suggest to model a complex system by dividing the overall model into m communicative recurrent modules. deep architectures often rely solely on raw data in large quantities with a requirement of representativeness regarding task requirements. this becomes problematic for tasks with a specialized, low-frequency terminology, where high quality knowledge sources for nlp and ai are often available and have proven their effectiveness. embedding expert knowledge in extended pre-trained word embeddings is costly. we present untied inedpendent modules to embed knowledge from different sources onto systems input. knowledge sources, as independent experts, provide different annotations (abstractions) for the input, combining various classifications for solving the task. for instance, providing sentiment lexica for sentiment analysis reduces the demand for training data by expanding the limited training vocabulary with an extended set of annotated terms. precompiled word embeddings are to be considered knowledge sources in the same spirit and we demonstrate that they inter-operate with a variety of other knowledge sources such as gazetteers and pos encoding. consider example 1 from the stanford sentiment treebank (sst-2) @xcite . figure 1 shows annotations from different knowledge sources for example 1, such as tokenization (from the annie tokenizer), pos tags (from the stanford pos tagger), and sentiment annotations from three sentiment lexica (afinn @xcite , mpqa @xcite , and nrc @xcite ). x 1 t x foot_0 t x 3 t x 4 t x 5 t t token pos afinn mpqa nrc 1 this dt 0 neutral -0.19 2 is vbz 0 neutral 0.00 3 an dt 0 neutral 0.08 4 absurd jj 0 neg. -1.56 5 comedy nn +1 neg. 0.27 6 about in 0 neutral -0.34 7 alienation nn -2 neg. 0.00 8 , , 0 neutral 0.27 9 separation nn 0 neutral -0.29 10 and cc 0 neutral 0.41 11 loss nn -3 neg. -0.51 12 . . 0 neutral -0.06 the annotations of the different sentiment lexica in figure 1 vary substantially: comedy is classified as positive (+1) in afinn, as negative in mpqa, and almost neutral in nrc. @xcite showed that this variance in judgements is not prohibitive, in fact @xcite showed that combining 5 sentiment lexica outperformed all other combinations. these differences are in fact advantageous in an ensemble setting and reflect diversity among experts. the differences cannot be exploited, when a single embedding is used for tokens, but may be retained, when different lexica are embedded independently in different modules. we add input independence to the rims architecture, providing different language annotations as inputs to a set of independent, but interacting modules. the resulting system is a flexible modular architecture for leveraging token-level knowledge in form of different annotation embeddings, which will be given different weights for the task at hand dependeing on their usefulness during training (see figure 11 ). the system is evaluated on tasks such as sentiment analysis and analysis of health-related tweets for different health concerns. our experiments demonstrate that leveraging knowledge sources under a modular framework consistently improves performance with little increase in parameter space. additionally, when frozen language models are supplemented with knowledge sources, the drop in performance is minimal, making this technique particularly beneficial for users that do not have access to powerful computational resources. lastly, the modular nature of the system allows to visualize the models functionality. ## tasks we explore the potential of multi-input rims by ablation on different tasks that are each very specific in their description and do not have large training datasets, namely three sentiment analysis tasks and two health-related tweet classification tasks. ## implementation preprocessing we preprocess the data using a gate pipeline @xcite with the annie english tokenizer (for sst-2 task) and annie tweet tokenizer as well as the hashtag tokenizer (for the tweet tasks). embeddings each annotation type provides a sequence (see figure 1 ) which is used as input for a dedicated module in multi-input rims. therefore, each sequence has to be properly embedded. the annotation types can be embedded either using pretrained embeddings or using randomly initialized embeddings that are learned during the training. tokens are embedded using elmo @xcite or roberta @xcite pretrained models. for elmo, we use the pretrained model provided by allennlp 9 and for roberta, the model provided by hugging face 10 . pos tags following @xcite , we apply word2vec on pos tag sequences instead of token sequences. the pos embeddings are trained using the gensim package @xcite with a window size of 5 and dimensionality 20. the pretraining is performed on combined training data of all tasks introduced in section 3. afinn and nrc matches do not require an embedding, since the lexica quantify the sentiment scores numerically. mpqa matches for negative, neutral, and positive polarities are encoded numerically by -1, 0, and 1 respectively. medical gazetteer matches are embedded using a learnable embedding matrix @xmath0. the multi-input rims model is a flexible architecture and the modules can be of any recurrent type. here, we use lstms for complex inputs, such as token or pos, and rnns for annotations with simpler encodings, such as gazetteers. to calculate classification loss we use crossentropy loss and we optimize the models using the adam optimizer @xcite . the models are implemented using pytorch @xcite . ## numerical results we present a set of ablation studies to evaluate the effectiveness and contribution of different knowledge sources. all modules active figures 4 5 6 report results for the multi-input rims model when the modules are provided with different annotation types and all modules are kept active (@xmath1. for the runs where the token annotation is the only input (@xmath2, the model is reduced to a simple lstm with elmo or roberta embeddings, which we consider to form baselines. figure 4 shows that all sentiment tasks benefit from the sentiment lexica. for sst-2, afinn and mpqa add more to the task than nrc. on the other hand, nrc yields considerable performance improvements for the tweet sentiment data sets of pos constitutes general linguistic knowledge and demonstrates consistent yet small improvements for the sentiment tasks. however, pos improves performance for the health concerns data of sm18-2 (figure 5 ) and sm20-5 (figure 6 ). note that both tasks concern detection of personal experience mentions, for which categories such as pronouns (both personal and possessive) and verbs in past tense are important, which carry distinctive pos tags. figure 5 : multi-input rims for sm18-2, personal drug intake. all modules are active pos constitutes general linguistic knowledge and demonstrates consistent yet small improvements for the sentiment tasks. however, pos improves performance for the health concerns data of sm18-2 (figure 5 ) and sm20-5 (figure 6 ). note that both tasks concern detection of personal experience mentions, for which categories such as pronouns (both personal and possessive) and verbs in past tense are important, which carry distinctive pos tags. improvements from medical knowledge gazetteers are also compelling. figure 5 shows that the disease gazetteer enhances the performance for the medication intake task, corroborating the hypothesis that disease mentions are strong evi- figure 6 : multi-input rims for sm20-5, birth defect in a child. all modules are active dence for medication intake. similarly, figure 6 shows that the pregnancy gazetteer, as a complementary knowledge source, provides effective support for birth defect mention detection. ## module activation patterns an advantage of a modular system is the possibility of module inspection. the functionality of each module during the course of processing has to be transparent for assessment. figure 11 provides the activation patterns of two multi-input rims when applied to two inputs from sst-2 (figure 11a ) and sm20-5 (figure 11b ) to assess whether they give insight into the functionality of the modules. in figure 11a , the modules that operate on sentiment knowledge sources (afinn, mpqa, and nrc) are active only when an annotation is available and are idle (inactive) otherwise. the sentiment modules also compete with one another. consider beautifully at @xmath3. for this token, both afinn and mpqa provide annotations (afinn: the activation patterns in figure 11b show the birth defect and pregnancy gazetteer modules are the activity patterns are the result of the input selection mechanism (attention). multi-input rims modules are free to select an input signal or ignore it, which allows each module to potentially focus on a specific part of the input. the input selection mechanism prevents the modules from getting updated with spurious inputs (here the input at steps, where no annotation is available). additionally, this allows the system to develop different modules to select complementary input signals, biasing the behavior away from combining redundant encodings. we believe that the activation patterns can be useful for model explanation. nevertheless, the activation patterns have to be studied under a variety of nlp tasks and different, richer annotations, which demands a dedicated study and is beyond the scope of this paper. ## conclusion this paper presents proof of concept for a modular system for leveraging different knowledge sources. under the proposed model, various annotations with different encodings are used as inputs for a set of independent, decoupled, but interacting modules, a novel extension of the rims architecture. deploying several readily available knowledge sources (gazetteer lists and part-of-speech information), our experiments report on different sentiment tasks and data sets, as well as two health-related tasks and datasets. the results suggest that the modules successfully interoperate for addressing different target tasks and multiple datasets with drastically reduced parameter space (and processing resources). in addition to the transfer potential of rims, we probed their transparency. the activation patterns of the modules in multi-input rims showed inter-