Classification of hierarchical text using geometric deep learning: the case of clinical trials corpus (2021)

Abstract: We consider the hierarchical representation of documents as graphs and use geometric deep learning to classify them into different categories. While graph neural networks can efficiently handle the variable structure of hierarchical documents using the permutation invariant message passing operations, we show that we can gain extra performance improvements using our proposed selective graph pooling operation that arises from the fact that some parts of the hierarchy are invariable across different documents. We applied our model to classify clinical trial (CT) protocols into completed and terminated categories. We use bag-of-words based, as well as pre-trained transformer-based embeddings to featurize the graph nodes, achieving f1-scoresaround 0.85 on a publicly available large scale CT registry of around 360K protocols. We further demonstrate how the selective pooling can add insights into the CT termination status prediction. We make the source code and dataset splits accessible.

## introduction the safety and efficacy evaluation of medications and clinical interventions is performed using clinical trials (ct's) @xcite . prior to their implementation, ct protocols are carefully designed, detailing important aspects of the study, including the number of enrolled patients, their inclusion and exclusion criteria, and the expected outcome, as required by healthcare authorities @xcite . regrettably, a large fraction of ct's terminate before reaching a study conclusion @xcite . this is linked directly to delays in providing treatment for the world diseases and to significant excess financial costs @xcite . ct protocols are often modelled and represented using tree-like or more generally graph-like structures, such as xml, json and dom @xcite . these models use a set of nodes v representing sections connected by a set of relations e of type part of to encode nested information. the information encoded by a given section of a ct is then the recursive combination of the information encoded by its subsections. as an example, fig. 1 depicts a simplified ct protocol. without explicit encoding, a flat-structured text feature extractor would ignore these inter-dependencies. to best consider the inter-connected nature of different elements of a ct protocol, it is thus necessary to take its hierarchical structure into account. figure 1 : schematic view on the simplified tree structure of a typical clinical trial protocol from clinicaltrials.gov. leaf nodes contain free text. top parent nodes are fixed across trees, but children nodes can contain variable structure. the systematic way to consider the structural information of data is the promising paradigm of graph neural networks (gnn's) and more generally geometric deep learning @xcite . geometric deep learning aims to provide a joint representation of different component features, along with their topology. a key element in most gnn models is the message passing algorithm between nodes @xcite , which aggregates the features of nodes based on their neighborhood connectivity. focusing on the structure of a ct protocol, which is essentially a tree with free-text on its leaf nodes, the parent nodes initialized with zero vectors will aggregate feature vectors of the children nodes during message passing iterations, as depicted in fig. 2 . the power of gnn's lies on their ability to aggregate topology and features when node labeling is arbitrary, i.e., when there is no canonical way of labeling the nodes. famous examples of this use case are in molecular graphs, e.g., @xcite , when graph nodes consist of atoms, for which no natural ordering is meaningful. the message passing between nodes, followed by a set pooling operation (typically averaging) after nodelevel representation learning will then provide a global representation for the whole graph, making them suitable for downstream tasks like regression @xcite . as for the hierarchical text data like ct protocols, however, this is not exactly the case. while the leaf nodes may have arbitrary structure and hence can benefit from general recipes of gnn's, the parent nodes are typically fixed across all graphs and do not have to undergo global pooling. therefore, as we will show next, a selective pooling that keeps this structure intact is preferred to the permutation invariant pooling. this paper has the following contributions: we furthermore provide practical insights and recipes as how to make a bag-of-words (bow) vectorizer around 50 times faster than a pre-trained bert model @xcite in evaluation run time with a loss of f1-score of classification only around 2%. the rest of the paper is organized as follows. in section 2, we describe the related works for the different elements to our paper, i.e., the general problem of text classification, graph-based deep learning, the use of graphs in text, as well as the use of machine learning in the study of ct's. section 3 discusses our proposed solutions in terms of how we featurize the texts and the way we represent hierarchical texts as graphs suitable for graph-based deep learning. section 4 details how we prepare the ct corpus for classification, the baseline and the proposed methods used and a discussion of the classification results of each of these methods, as well as an effort for explainability of the graph-based representation. the paper is finally concluded in section 5. ## conclusions we used graph neural networks to represent the structure, as well as the extracted features from free text within hierarchical documents. our use case was the classification of interventional clinical trials into two different risk categories based on their protocols. on a publicly available corpus of around 360k protocols, we showed that the use of gnn's provides a net increase in performance, compared to structure-agnostic baselines. to better incorporate the power of gnn's into the invariable a priori known template, we proposed selective pooling to boost the performance of global pooling. furthermore, we showed that this approach provides straightforward solutions for explainability, where we demonstrated some consistency between gradient activities of protocol fields within our model to known factors of risk from clinical trials research literature.