  Pathological brain appearances may be so heterogeneous as to be intelligible
only as anomalies, defined by their deviation from normality rather than any
specific pathological characteristic. Amongst the hardest tasks in medical
imaging, detecting such anomalies requires models of the normal brain that
combine compactness with the expressivity of the complex, long-range
interactions that characterise its structural organisation. These are
requirements transformers have arguably greater potential to satisfy than other
current candidate architectures, but their application has been inhibited by
their demands on data and computational resource. Here we combine the latent
representation of vector quantised variational autoencoders with an ensemble of
autoregressive transformers to enable unsupervised anomaly detection and
segmentation defined by deviation from healthy brain imaging data, achievable
at low computational cost, within relative modest data regimes. We compare our
method to current state-of-the-art approaches across a series of experiments
involving synthetic and real pathological lesions. On real lesions, we train
our models on 15,000 radiologically normal participants from UK Biobank, and
evaluate performance on four different brain MR datasets with small vessel
disease, demyelinating lesions, and tumours. We demonstrate superior anomaly
detection performance both image-wise and pixel-wise, achievable without
post-processing. These results draw attention to the potential of transformers
in this most challenging of imaging tasks.
