Detection of spatiotemporal stochastic motion in echocardiography videos
Anonymous Author(s)
AfﬁliationAddress email
Abstract
A challenge in fetal echocardiography is the unpredictable relative motions between the fetus being assessed and the probe which manifests as a change in theviewing state of a relevant anatomy on the ultrasound video monitor.
It is a difﬁculttransition for automatic medical video analysis pipelines.
We treat the detection ofsuch spatiotemporally stochastic transitions as an anomaly detection case and use anovel statistical decision method over learnt spatiotemporal representations fromconvolutional LSTM models to encode partwise distance measures across localregions from multiple frames in video segments as discrete probability distributionscompared by a Matusita coefﬁcient score.
This helps compute divergences between discrete probability distributions so obtained and allows segregation betweennormal and anomalous spatiotemporal representations out of video segments
Introduction
There has been a considerable focus on automation of workﬂows in medical image analysis, butwhile most machine learning methods account for variability in spatial feature representations,there has been little effort to account for unforeseen variation in the spatiotemporal regime.
Suchrandom or unexpected motion is not uncommon in physiological cases, particularly in the caseof the developing fetus where stochastic motion of the fetus relative to the uterus, or due to thepatient and the probe, poses challenges in obtaining requisite standard viewing planes for anatomicalobservation in ultrasound sessions.
Here, we attempt to identify such stochastic motions between thefetus and probe visible on the ultrasound video.
Anomaly detection has been attempted for naturalvideos using techniques like multiple instance learning[1] and sparse coding[2].
Such methods arenot well-posed for our task of inferring random motions in fetal ultrasound as durations of normalbehaviour are not strictly conﬁned to the beginnings or ends of input videos.
Generative techniquesusing autoencoders[3] have also been attempted to perform novelty detection.
In most anomaly ornovelty detection problems with videos of natural scenes where an anomaly manifests as a signiﬁcantspatial difference across a temporal length.
Also, optical ﬂow deﬁnition is relatively well-developedand sudden motion can be interpreted jointly with the actual frame level and ﬂow based features.
However, our task of assessing unpredictable, stochastic relative motions between the probe and thefetus from video sequences obtained in fetal echocardiography has signiﬁcant differences which makeit challenging.
Here, arbitrary motions show up as sudden movements of the heart structure beingviewed (akin to an unanticipated jerk in natural videos) as a complex combination of translations androtations, a sudden change in the visibility of the heart or sudden lapses into the background due tounfavourable positioning of the fetus with respect to the ultrasound probe or an unplanned alterationin the viewing plane(our dataset has three standard planes [4] – four chamber(4C), three vessel(3V)and the left ventricular outﬂow tract(LVOT) view) due to the motion.
Also, optical ﬂow deﬁnition ispoor in our ultrasound videos because of speckle and ﬁne structural motion.
This makes it difﬁcult touse a two-stream CNN [5] for segregating unexpected motion patterns in fetal ultrasound.
Methodology
The task of determining sudden motion of the fetus, an unexpected relative motion between the probeand the patient or a sudden change in the fetal cardiac views can be fundamentally treated as that of aspatiotemporal superposition of a stochastic motion pattern over that of the deterministic physiologicalmotion – in our case the base cardiac rhythm in a standard fetal echocardiography sequence.
Assuch, the separation of superimposed motion is a non-trivial problem with the complexity increaseddue to non-deterministic occurrence in fetal ultrasound sequences.
So, the idea of determining thetemporal instances of the onset and conclusion of such motion patterns and a quantiﬁcation of thedisplacement of visible structures in the coordinate system ﬁxed to a frame is of interest.
The addedcomplication is that the ultrasound modality has a strong inﬂuence of speckle and enhancementswhich are an extraneous inﬂuence on optical ﬂows, and render the latter unsuitable for this task.
Withthese constraints, we propose modelling the problem as one of supervised spatiotemporal anomalydetection, by relying on successive alterations in local spatial cues in a temporal sequence relativeto sequences labelled as being normal.
Essentially, we use a formulation of utilising distance basedsimilarity measures and implementing them as probability distributions over localised patches andexpanding over the temporal dimension.
The idea is to enable the model to simultaneously learn toidentify similar normal videos and to be able to discriminate between normal and anomalous videosequences.
This is done by feeding in training samples grouping triplets of 8-frame video clips withtwo of them being normal and the third being an anomalous instance different from the other two.
Over the individual video inputs in the group of three, we divide individual frames in a 8-framevideo sequence into quadrant regions, and quadrants similarly located in the same frames of threecomparator videos are fed to a triplet network which is trained to calculate a L1 distance betweenthem.
The distance metric so calculated is normalised using sigmoid activation, and a distributionof distances so computed is obtained for a given pair (so, this would be a discrete probabilitydistribution with 32 instances if each frame in 8-frame videos is divided into 4 regions).
Suchprobability distributions are compared using a Matusita metric [6]as it estimates overlap of discreteprobability distributions without being impaired by the singularity problem that arises with morecommon measures like the Chi-squared metric in cases of similar distributions[7].
Our minimizationobjective is created as the Matusita measure between the two sets of normalised distances obtained inbetween the similar sequences and those in between the dissimilar sequences, in a training instanceso chosen that two of them are labelled as similar and are different to the third instance.
Sequencesthat are similar and therefore have similar probability distributions would have a small Matusitametric, whereas dissimilar distributions would have a larger value for the same.
A labelled datasetwhere normal and abnormal sequences are identiﬁed can be used to train a triplet network with twobranches’ fully-connected layers used to calculate quadrant level distances, which are normalised,and used to feed to a layer evaluating the Matusita metric where similar distributions (labelled ‘0’)and dissimilar distributions (labelled ‘1’) are classiﬁed with a binary cross-entropy function.
Triplet architecture We implement modiﬁed Triplet Networks [8]with parallel strands acceptinginputs as parts of triplets of 8-frame videos (from same or different class, i.e, both normal,identiﬁedas of ‘same’ class or 0, and two normal-one abnormal triplet, or a ‘different’ class or 1).
The tripletlegs process each input stream independently and jointly compute a L1 distance followed by aloss function using the L1 output sequences to calculate a Matusita coefﬁcient and optimising it tocorresponding labelled values of 0 or 1 by learning to classify ‘same’ and ‘different’ classes.
Inputsare pre-processed as sequences of relevant quadrants to ensure efﬁcient dataset curation and tractabledistance computation.
This implies that quadrants chosen for same locations in corresponding framesof three input sequences are processed by parallel strands of the triplet Network.
This is done forall the quadrant pairs obtained for any two inputs (thus, two 8-frame videos are compared usinga distribution of 32 normalised distances serving to input discrete probability distributions to the
Matusita metric layers).A strand has 8 convolutional layers with 3x3 ﬁlters, ReLU activation andalternate max-pooling.
This is followed by LSTM layers of 512 units to allow progression on thetemporal video scale.
Next is a 512-way fully-connected layer encoding representations for distancecomputation.
Each input is separately computed upon using shared weights leading to three featurerepresentations from each input pair example.
We encode similarity between images by an L1distance between the FC layers from pairwise parallel strands.
This enables us to obtain two L1distances for every input triplet.
These two L1 distances are constrained between 0 and 1 with asigmoid operation and used to compute a Matusita measure as the function to be optimised feedinginto a binary cross-entropy classiﬁcation to classify between 0 (similar) and 1 (different) classes.
So
Objective function For comparison of two discrete probability distributions, which is how weinterpret the set of distance values of the divided quadrants from frames in the triplet of sequences, wemodel the divergence as a proxy for the dynamic similarity between the sequences.
This is done usingthe Matusita measure, originally devised as a measure of risk in the analysis of statistical decisionfunctions and deﬁnes the distance between two probability distributions as:
[Equation]
Here, N is the number of quadrant triplets to be considered for every training or test instance, iidentiﬁes the triplet worked upon, p(i) is the normalised distance between the ﬁrst two quadrants inthe triplet, and p’(i) is the normalised distance between the second and the third.
The input to the
Matusita loss function are the respective L1 distances activated using a sigmoid layer to be between 0and 1.
This essentially converts the normal versus anomalous video instances classiﬁcation probleminto a similarity measure optimisation task, with the objective being the class label of either 0 or 1.
Training We begin with 3624 8-frame echo clips,with 2424 clips tagged normal, and 1200 withone of the three types of motion anomalies studied here.
The training to test data split is at a 80:20ratio.
The clips are pre-processed to include sequences of similarly positioned quadrants across thesequential video frames for each of the input videos, thereby leading to an effective number of fourclips per input clip at runtime.
These are fed together as parts of the same triplet so as to generate acombined discrete probability distribution per video pair.
To create input triplets, we follow a strategyrelying on the ordering of the dataset and choose a triplet such that two instances are normal and thethird could be either normal or abnormal in a randomised selection, and correspondingly attach alabel of 0 or 1.
Here it is ensured that no more than half the total number of triplets are labelled 0 or 1(or contain all normal or a normal-abnormal combination).
Following the preparation of the input set,the three-way triplet network is trained for 50 epochs with a batch size of 20 and a learning rate of
[Table]
The experiments are conducted in four stages.
First, all of the anomaly videos dataset is usedfor generating triplets.
Following this, three other runs are performed with each of the classes ofanomalies (Visibility shift, Viewing plane shift, Sudden motion) considered as the sole dataset foran anomalous case for triplet generation.
We use baseline results derived using direct classiﬁcationfrom a VGG-16 and Resnet-50 adapted(input of 8-frame clips at ﬁrst layer[4]) with same trainingprotocols as the triplet network.
Also, an autoencoder baseline is used to represent reconstructionbased anomaly detection, using 5 convolutional layers with 3x3 kernels in both upsampling anddownsampling stages, trained to reconstruct non-anomalous patterns over classes.
The classiﬁcationis by sigmoid normalising the reconstruction error and estimating the class (normal, abnormal).
Results
The evaluation metric for our three-way distance encoded convolutional-LSTM formulation is theclassiﬁcation accuracy as normal and abnormal triplets are labelled.
At test time, triplet inputs weredesigned that a test video was used with two normal videos,to have a triplet whose class label wouldbe 0 (all three normal) or 1 (two normal, one anomalous video).
The trained triplet network wouldoutput a label to be compared with the ground truth label of the input triplet.
This was done forseparate types of anomalies discernible in our videos – sudden fetal/probe motions, sudden changesin visibility and sudden changes in cardiac plane.
In our data, unlike natural videos, the notionof spatiotemporal anomaly is not conﬁned to cases of drastically different spatial cues emergingover short time instances.
Rather, the nature of anomalies is such that overall spatial deﬁnitionsacross a single frame often stays constant, with an evolving variation in the orientation or linearshifts over temporal distances.
This implies that a model relying on overall feature representationacross frames is sub-optimal.
So, using quadrants to compare distance based embeddings of localregions per frame of compared videos shows a performance gain and the method of comparison ofprobability distributions derived from consecutive local distances across pairs of videos is amenable tocomplexities of fetal and probe motion observation in the ultrasound videos considered.
The relativedifference between the classiﬁcation accuracies in the case of Viewing Plane shift is the lowest overallcompared across methods.
In this class, there is a substantial change in frame level image descriptionon the whole as the standard viewing plane of the fetus abruptly changes, causing a signiﬁcantchange in the global frame level features as to be amenable to an end-to-end training for classiﬁcation.
Contrarily, compared to other methods the Triplet Net does relatively much better on cases of suddenmotion as these instances are characterised by overall frame level similarities but in the aggregateof multiple frames, manifest as a departure from the normal motion pattern.
The average Matusitacoefﬁcient over the test data is reported for the test normal instances as 0.1125 and the test abnormalinstances as 0.8944.
Ideally, the Matusita coefﬁcient would be 0 for normal and 1 for the abnormalinstances.
The overall digression from ideal values is indicative of a degree of error in modelling thesimilarity and dissimilarity of video instances, explained as a result of the loss of certain local levelcomparisons due to not going into any ﬁner division than a quadrant level assessment.
Note that the
Overall accuracy here does not refer to an aggregate of the Visibility anomaly accuracies, the viewingplane anomaly accuracies and the sudden motion anomaly accuracies.
This is because these are fourdifferent runs with the latter three conducted on separated anomaly video datasets.
To conclude, wedetect unanticipated motion in echocardiography videos by treating it as an anomaly over normalmotion patterns, with a caveat that abrupt patterns may not lead to overall changes in global spatialfeatures visible frame-to-frame but would manifest as discernible local changes.
This scheme allowsus to consider patchwise embeddings for a distance measure recast as a sequence of probabilitiesbeing compared on a triplet network thus overcoming limitations of simple CNNs caused by localpose and orientation invariance.
Appendix