A New Graph Node Classification Benchmark: Learning Structure from Histology Cell Graphs
Abstract
We introduce a new benchmark dataset, Placenta , for node classification in an underexplored domain: predicting microanatomical tissue structures from cell graphs in placenta histology whole slide images.
This problem is uniquely challenging for graph learning for a few reasons.
Cell graphs are large (>1 million nodes per image), node features are varied (64-dimensions of 11 types of cells), class labels are imbalanced (9 classes ranging from 0.21% of the data to 40.0%) and cellular communities cluster into heterogeneously distributed tissues of widely varying sizes (from 11 nodes to 44,671 nodes for a single structure).
Here, we release a dataset consisting of two cell graphs from two placenta histology images totalling 2,395,747 nodes, 799,745 of which have ground truth labels.
We present inductive benchmark results for 7 scalable models and show how the unique qualities of cell graphs can help drive the development of novel graph neural network architectures
Introduction
The initial graph representation learning benchmarks facilitated the creation of the first graph neural network (GNN) architectures.
Cora , Citeseer , Pubmed and NELL [29], [3] drove the development of
Graph Convolutional Network (GCN) [16], Reddit was introduced alongside GraphSAGE [12], and
Graph Attention Network (GAT) [26] achieved state-of-the-art on the protein-protein interaction ( PPI ) dataset [34].
However, it soon became apparent that these datasets were not sufficiently complex to challenge GNNs [13, 28].
The datasets had small graphs or a small number of graphs (the largest dataset, Reddit , had 232,965 nodes), they had potentially misleading train, validation and test splits [22], and they were so well suited to graph learning that randomly initialised, untrained models could [Table] perform nearly as well as trained models [27].
In recent years, there has been an influx of graph benchmark datasets which address these limitations.
They are larger and cover more domains, have realistic train, validation and test splits, and are challenging enough to have not been ‘solved’ by current GNN architectures [13, 5, 24].
Cell graphs in histology whole slide images (WSIs) are a domain which has been largely unexplored for graph learning and present a challenge to the graph learning community [18, 33, 10, 4, 19].
Histology is the study of tissues and cells under a microscope.
A histology WSI is a digitised, high-resolution scan of thin tissue samples showing individual cells, which can be upwards of 70GBs per image.
All organs are composed of cells that collectively build up the structures and tissues comprising the body.
Consequently, representing organs as cell graphs is a logical construction closely matching the true physical system.
The value of cell graphs has been highlighted [30, 14, 23], but with today’s computational power and automated aggregation methods, it can reach its full potential.
Here, we introduce Placenta , a cell graph from two WSIs as a new benchmark dataset to facilitate
GNN creation.
More specifically, we present a node classification task of predicting one of 9 placenta microanatomical tissue structures to which cells belong (Appendix A).
This problem is challenging from a graph learning perspective for a number of reasons.
Firstly, each WSI contains a cell graph of >1 million nodes, the node features are defined from an imbalanced distribution of cell classes, and the tissue labels also have a high, but expected, class imbalance, from 0.21% of the data to 40.0% (Appendix B).
Secondly, unlike most graph datasets, node clusters are locally homophilic but spatially distributed throughout the graph, requiring models to understand same-type node clusters at different locations.
Thirdly, as a WSI is only ever a small sample of the larger structure with arbitrary cut-offs in edges, out-of-plane cells, and optical and mechanical artefacts, the property of data incompleteness is inherently a feature of this data domain.
Finally, placental tissue structures grow from one another in a tree-like manner [1, 7], so singular structures can contain multiple tissue classifications and these structures will vary in size (30-5,000 µ m 2 ).
Models must discriminate between small changes across small regions within node clusters but have a large enough receptive field to identify the largest of tissues.
Relevant to the real world domain, developing automated methods which can quantify the structural tissue units within histology images would be an invaluable tool for the scientific community and, in future, may have applications to clinical histopathologic diagnoses and patient care.
Furthermore, this dataset provides a benchmark for developing novel, scalable GNNs that can adapt to node feature and class label imbalance, that can understand both large and small node communities distributed throughout the graph, and that are robust to data incompleteness.
These models could be applied to new domains with similar qualities, such in modelling disease epidemiology.
Dataset Description
Overview At a high level, the learning task is to understand placental tissue structures as the composition of their cellular communities.
Tissue structures in the placenta are precisely defined and distinguished from one another by their cellular composition [1].
In a WSI these manifest as spatially clustered groups of cells, with corresponding tissue structures comprising 10s-10,000s of cell nodes.
Inductive message-passing GNNs, which can aggregate node neighbourhood information irrespective of graph shapes and sizes are, in theory, a natural fit for this task.
The dataset we introduce for this task, Placenta , consists of two combined cell graphs constructed across two healthy, term placenta hematoxylin and eosin stained WSIs from two institutes (Table 1).
Ground truth labels were created by the primary author and validated by four practising perinatal pathologists (details in Vanea et al.
See Figure 1 for a high level visualisation of all cell graph nodes and labelled ground truth nodes and Figure 2 for a representative region.
The use of these images was approved and the requirement for consent was waived by local ethics committees at both institutes (approval 0735-18-HMO and 289/T-5).
We recommend a train, validation, test split (Appendix C) which ensures tissue types have similar distributions across splits.
The type of tissue in the placenta can be influenced by macroscopic location (for example, proximity to the chorionic or basal plate), so these dataset split regions are chosen to cover a range of locations and to avoid neighbourhood information leakage.
Nodes and Node Features We use a prior two-stage deep learning pipeline to identify the centre point of all nuclei across the WSI and classify those nuclei as belonging to one of 11 cell types.
The nucleus coordinates define the node coordinates in the graph and the 64-dimension embeddings of the penultimate layer of the cell classifier model are used for the node features.
Full details of this pipeline can be found in Vanea et al.
Edges and Edge Features We explore three edge-building algorithms which act as a proxy for cellular interaction: KNN [6], Delaunay Triangulation [11], and the intersection of the two (Appendix
D).
Unless stated otherwise, we use the intersection graph with k=5 for all experiments.
Edge construction in the context of cell graphs are representations of the approximate biological likelihood that nearby cells interact as part of the same structure.
The Euclidean distance-based edge construction we have explored is one method for this, but alternative approaches for constructing the network are worth considering.
We leave this as an open question for the community.
Experiments
Experimental Setup We first construct a cell graph across all cells in both WSIs.
Nodes are masked as either unlabelled, training, validation, or test nodes and only nodes in the training set are used for back propagation.
Both cell graphs are combined into one disjointed graph, allowing the sampler to treat it as a dataset of one graph.
We restrict our experiments to scalable GNNs as the full graph cannot fit into GPU memory.
During training, for GraphSAGE [12], ClusterGCN [5],
GraphSAINT [31], ShaDow [32] and SIGN [9], node embeddings are sampled or precomputed according to original implementations.
For models introduced without a sampling technique ( GAT [26] and GATv2 [2]), we use ClusterGCN subgraph sampling (ClusterGAT and ClusterGATv2).
During inference, all neighbourhood nodes are used for aggregation.
We additionally compare against two baselines, an MLP on node embeddings and a randomly initialised GraphSAGE .
Hyperparameters Models are trained for inductive learning fully-batched to 1000 epochs with early stopping on validation data (evaluated every 100 epochs), weighted cross-entropy loss, Adam optimiser [15], and a 0.001 learning rate.
Models have 256 hidden units per layer, a batch normalisation layer following each graph convolutional layer (except for ShaDow), and at most 16 graph convolutional layers each with a 0.5 dropout and relu activations.
For GraphSage, ClusterGAT, and ClusterGATv2, 16 layer models could not fit into GPU memory even with reduced sampling and batch sizes and, as such, fewer layers were used.
Further hyperparameter details can be found in the supplement (Appendix E).
Note that the goal is not to find the best performing architecture for this dataset, but instead to show the problems existing architectures have on histology cell graphs.
As such, we explore hyperparameters to the extent that we believe a fair comparison can be made between models but we do not perform an exhaustive search.
This has the added benefit of avoiding overfitting hyperparameters to these two placenta WSIs, which only represent a small facet of the histology domain.
Hardware, Software, and Libraries Experiments were run on one NVIDIA RTX8000 on an internal cluster.
Code was written in Python (v3.7.2) using PyTorch (v1.9.0)[20] and PyTorch
Geometric (v2.0.4)[8].
WSI visualisation and ground truth annotations were made using QuPath (v0.3.1)[21].
Code, cell graph dataset, data splits, and experiment instructions may be found at ( https://github.com/Nellaker-group/placenta ) under a MIT licence.
Results
Table 2 shows the test mean accuracies and standard deviations of models over 5 random weight initializations.
The randomly initialised baseline performs equivalent to random chance (ROC AUC 0.5) and GNN models outperform the MLP baseline.
We found ClusterGAT and ClusterGATv2 were unstable to train with more layers and this might contribute to their lower performance score.
All scalable GNN architectures: GraphSAGE, ClusterGCN, GraphSAINT, ShaDow, and SIGN are [-within
 %-]within 2% mean accuracy of each other, with no models surpassing 65% accuracy.
These results highlight the need for new GNN approaches for cell graphs in histology. [Table]
In the supplement [Table] we describe features which the models struggled with and we take a fast, highperforming model (ClusterGCN) to explore some of these in more detail (Appendix F, G, H).
We show how class imbalance affects performance, how model size leads to overfitting [Table] and the importance of a large enough receptive field.
We additionally compare model performance using all edge-building algorithms (Appendix D), using one hot encoded cell type node features (Appendix I) [Table] and using random node data splits (Appendix C).
Limitations
Our cell graph dataset represents a small facet of the histology domain; two placenta WSIs from two institutes.
WSIs from more placentas and other laboratories will be needed to validate the generalisability of models designed using this data.
Likewise, the dataset is placenta-specific, so whilst there is a benefit to using the weights of placenta models for pretraining, the development of
GNN tissue classifiers for other organs will require organ-specific cell and tissue ground truth data.
Cross- or multi-organ self-supervised approaches could be developed from our dataset but it is not something we have explored.
Regarding model comparison, we have purposely limited the scope of our hyperparameter search.
It is possible that a different combination of hyperparameters and/or additional sampling, normalisation, and regularisation, would have yielded better performance.
We release all data necessary for building cell graphs across our WSIs but not the raw images or pipeline needed to derive the cell features from scratch as this was out of scope for this work.
Finally, it should be noted that uncertainty in ground truth annotations may limit model performance on the dataset, however label uncertainty is inherent to the biological problem and unavoidable in this domain.
Conclusion
We introduce a new inductive node classification benchmark dataset, classifying microanatomical tissue structures from cell graphs across placenta histology WSIs.
We show how GNN architectures perform on this task and highlight the qualities unique to this data domain which present a challenge for existing models.
We hope that this dataset facilitates the creation of new model architectures to overcome these challenges.
Acknowledgments and Disclosure of Funding
Claudia Vanea and Jonathan Campbell are supported by the EPSRC Center for Doctoral Training in Health Data Science (EP/S02428X/1).
Cecilia M. Lindgren is supported by the Li Ka
Shing Foundation, NIHR Oxford Biomedical Research Centre, Oxford, NIH (1P50HD104224-01),
Gates Foundation (INV-024200), and a Wellcome Trust Investigator Award (221782/Z/20/Z).
Triin
Laisk is funded by the European Regional Development Fund and the programme Mobilitas Pluss (MOBTP155) and the Estonian Research Council grant PSG776.
The computational aspects of this research were supported by the Wellcome Trust Core Award Grant Number 203141/Z/16/Z and the
NIHR Oxford BRC.
The views expressed are those of the author(s) and not necessarily those of the
NHS, the NIHR or the Department of Health.
A Placenta Cell and Tissue Types
[Table]
• SYN: Syncytiotrophoblast • CYT: Cytotrophoblast • KNT:
Syncytial Knot • EVT: Extravillus Trophoblast • FIB:
Fibroblast • HOF: Hofbauer Cell • VEN: Vascular Endothelial • VMY: Vascular Myocyte • MES: Mesenchymal Cell • MAT: Maternal Decidua • WBC: Leukocyte
[Table]
: • Sprout: Villus Sprout • TVilli: Terminal Villi • MIVilli: Mature Intermediate Villi • SVilli: Stem Villi • AVilli:
Anchoring Villi • Chorion: Chorionic Plate • Maternal: Basal Plate and Septum • Fibrin: Fibrin • Avacular: Avascular Villi
B Cell Types in Node Features and Ground Truth Tissue Distributions
[Table] [Table]
D Edge Construction and Graph Differences
[Table] [Table]
E Hyperparameter Details for Each Model Architecture
Weighted cross entropy weights: [1, 0.85, 0.9, 10.5, 0.8, 1.3, 5.6, 3, 77]
Additional hyperparameter details and any changes from the default hyperparameters reported in
Section 3 are reported below.
MLP 16 Linear layers, 51,200 batch size.
GraphSAGE-mean 12 SAGEConv layers due to GPU out of memory error with larger models, 32,000 batch size, 10 neighbours per hop, mean aggregation.
ClusterGCN 16 SAGEConv layers, 200 batch size, and 400 max subgraph size.
GraphSAINT-rw 16 GraphConv layers, in-node normalisation, 32,000 batch size, 30 steps,sample coverage, random walk sampler.
SIGN
Linear layers, 51,200 batch size, SIGN transform with 16 hops.
ShaDow 8 SAGEConv layers due to performance drop with larger models, no batch norm layers for the same reason, 4000 batch size, 5 neighbours per hop, hop depth of 6.
This smaller receptive field is due to a RAM memory error with a higher depth or more neighbours.
ClusterGAT 2 GATConv layers due to GPU out of memory error and performance drop with larger models, 400 batch size, 400 max subgraph size, 4 attention heads with 0.25 dropout. ClusterGATv2 2 GATv2Conv layers due to GPU out of memory error and performance drop with larger models, 200 batch size, 400 max subgraph size, 4 attention heads with 0.25 dropout.
F The Effect of Class imbalance on Performance
H The Effect of Receptive Field Sizes
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
I The Effect of Cellular Node Features
[Table]
K Potential Negative Societal Impacts
The development of similar models in the context of histological analyses could, if in future fully realised and implemented across healthcare service settings, negatively impact the work of pathologists who do not have access to the models: “AI won’t replace radiologists, but radiologists who use AI will replace radiologists who don’t,” (Curtis Langlotz, 2019, Nature, Innovations In).
As with the uptake of any new technology, there would be a ‘growing pains’ period which could increase the time between tissue sample collection and diagnosis.
Slides will have to be digitised and model inference will have to be integrated, causing a shift in existing clinical and histological preparation workflows.
We must also consider that any models developed for histology analysis could cause serious patient harm if, in the extreme case, they are incorrectly validated and quality controlled, or they have an undetected bias which influences diagnoses.
If we assume models perform as well or better than clinicians, but their outputs are not clearly expressed in clinical language and/or there is a lack of trust in these automated, uninterpretable methods, then there may be resulting uncertainty in diagnosis and consequently higher referral rates.
Conversely, if automated histology analysis becomes popular, for organs unlike the placenta which require biopsies to be sampled, the rate of biopsies could harmfully increase.
Finally, the publication of this dataset could facilitate the development of new GNN architectures that themselves could confer negative impacts in other domains.
L CO2 Emission Related to Experiments
Experiments were conducted using a private infrastructure, which has a carbon efficiency of 0. kgCO 2 eq/kWh.
An approximate cumulative of 200 hours of computation was performed on hardware of type RTX 8000 (TDP of 260W).
Total emissions are estimated to be 22.46 kgCO 2 eq of which 0 percents were directly offset.
Estimations were conducted using the MachineLearning Impact calculator presented in [17].