Graph Neural Networks for Adapting Off-the-shelf General Domain Language Models to Low-Resource Specialised Domains (2022)

Abstract: Language models encode linguistic proprieties and are used as input for more specific models. Using their word representations as-is for specialised and low-resource domains might be less efficient. Methods of adapting them exist, but these models often overlook global information about how words, terms, and concepts relate to each other in a corpus due to their strong reliance on attention. We consider that global information can influence the results of the downstream tasks, and combination with contextual information is performed using graph convolution networks or GCN built on vocabulary graphs. By outperforming baselines, we show that this architecture is profitable for domain-specific tasks.

## introduction numerous types of word vectors are used as word representations for nlp tasks. these vectors encode useful semantic proprieties and are often used as weights or input of generic task models. for quite some time, word2vec @xcite and glove @xcite were the go-to off-the-shelf embeddings that many systems used. fasttext @xcite came as a way to deal with oov words and still offers a better representation for words than most systems since it takes into account the morphological complexity of words by dealing with n-grams instead of whole words. over the last few years, a new generation of deep neural approaches brought forth by transformers has brought significant improvements in many downstream applications. language models, like bert @xcite , already encode so much knowledge and can capture semantic and syntactic information remarkably well @xcite . they can be further trained on new tasks for adapting it to a new task or more specialised domains and further improve the quality of the representations @xcite . language models are trained over massive general domain corpora @xcite by optimising an objective that predicts the local contexts, captures linguistic units' distributional properties along the way, and address polysemy issues. their quality can thus be arguably correlated to the volume of data available. however, in the case of specialised domains, the corpora are generally relatively modest in size, and these methods might be less efficient. it must be noted that if this claim is not always valid for all domains, especially for the english language, it is undoubtedly almost always true for other much less-resourced languages @xcite . when using neural-based models, the conventional way of integrating further knowledge about specialised domains into models pre-trained on general corpora is to leverage pre-training by doing transfer learning and fine-tuning the model, tweaking its original weights to suit the tasks at hand better. if fine-tuning bert is the most used adaptation method, it will rapidly hit a performance ceiling if the domain is too specialised or small, and some methods @xcite exist to tackle this problem. bert will heavily rely on the context to build representations of too specialised or rare words. each layer of the encoder's attention mechanism enriches the new representation of the input data with contextual information by paying attention to different parts of the text. nonetheless, this particular feature of bert may make it more challenging for it to consider the more global place a word occupies within a corpus's vocabulary, especially for these words. many data structures are hierarchical or graph structures in nature, such as social networks, paper citation networks, ontologies and semantic relations, such as hypernymy of hyponymy. global relations between words within a sentence, a document or a corpus can be represented as a graph. using such graphs as inputs, graph neural net-work (gnn) @xcite captures general knowledge about the words and how they interact in a corpus. several variants of gnn for text classification tasks exist. as presented by @xcite , graph convolutional networks (gcn) is an approach for semi-supervised learning on graph-structured data based on an efficient variant of convolutional neural networks that operate directly on graphs. gcn is typically built using an adjacency matrix (based on a given relationship graph). the gcn's central idea is to take the weighted average of a node and all its neighbours during the convolution operation and uses both node features and the structure for the training. text gcn @xcite is a particular version of gcn, jointly learning both words and documents embeddings, and suited for situations with less training data. graph methods do not encode positional information, and when information about position or context is needed, it might not be enough. hence, pairing a gcn with a model that can grasp contextual information, like bert, seems necessary. there has not been much work trying to combine bert and gnn. since experts make inferences with relevant domain knowledge when performing domain-specific tasks, previous work has been conducted to integrate this knowledge into language models using knowledge graphs. knowledgeenabled language representation model k-bert @xcite injects triples from a knowledge graph into the sentences as domain knowledge. bert-mk @xcite also takes into account knowledge graph contextualised knowledge. @xcite embedded a medical ontology with graph attention networks (gat) and combined it with bert for medication recommendation. @xcite concatenates the output of gcn and the output of bert for citation recommendation tasks. obviously, these methods presuppose the existence of a knowledge graph or an ontology. these resources are expensive to build and do not always exist for the domain and task at hand. vocabulary graphs, on the other hand, are easy to build. @xcite propose vgcn-bert, a model which combines bert with a vocabulary gcn (vgcn), and where local and global information interacts from the first layer of bert down, building an augmented representation jointly. we build upon their work to adapt bert to our specialised tasks and corpora. as @xcite conveniently pointed out in their work, the utility of capturing global dependencies with a graph embedding instead of conventional non-contextualised embedding models @xcite @xcite can be questioned @xcite . these methods provide additional information, but these models' small text window limits the connections between words. long-range connections are more easily captured with gcn. in addition, by building a graph on a task-specific corpus, taskdependent dependencies are captured, in addition to the general dependencies already encoded in the pre-trained models. ## experimental methodology transfer learning pre-trained word vectors have been an essential component in many nlp systems. word representations are fed into a taskspecific model, often improving the results. re- ## results the results of the experiments are shown in in this work, we wanted to examine if we can improve results for languages and domains for which there are no biobert and clinical bert, e.g. there are no windbert or politicalbert for hypothetical wind energy or politics related tasks. this step of adaptation to the domain would make sense and even be necessary. using a graph neural network constructed over the corpus' vocabulary exclusively follows the same logic. it comes from our decision to find a way to improve the model intrinsically without using external resources. even if many resources exist for the english language, it is still crucial to explore ways to adapt existing models to our downstream specialised tasks where the volume of data is often insufficient. there is still a lot of room for improvement. one major limitation of this work -and work on graph neural networks in general, is the need to use all entities, including those from the training set, validation set, and test set, to build the vocabulary graph. ## conclusion the quality of word representations obtained through language models is often correlated to the volume of data available. in the case of specialised domains, these methods might be less efficient due to the usually modest size of the corpora. this is particularly exacerbated in the case of specialised and low-resource domains, and the models might need to go through an adaptation phase. this work seeks to understand how these methods can be adapted on the fly by using additional features from gcn. the achieved results outperform the baselines across the board. it shows that it is beneficial to merge local and global information, and those resulting representations yield some additional advantages and are more worthwhile for the downstream task. future work will cover analysis of attention layers before and after adding more global information. it will also involve considering more sophisticated relations than simple mutual information into the gcn, such as exploiting oriented graphs to encode dependencies, synonymy, hypo-and hyperonymy, and different architecture such as graph attention networks.