From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper24/Authors
title: Response to AnonReviewer2
comment: 1.It is true that selecting a good triplet pair is challenging, primarily because there is a wide variation in actual human judgments about acceptable quality(and about the fetal lie optimal for diagnosis based on other factors). The present problem tries to flag sudden motions as an anomaly. So,triplets need to contain at least one clip with such an abrupt motion. To offset the lack of standard definition of a 'good' clip, we used past data on view classification accuracies on these clips, and considered those with a high validation performance as less challenging, and proceeded to include less of these in batches of triplets. No preprocessing is needed for that. Quadrants are strictly defined as regions in a 2D frame out of a clip, and the sequence goes -> 1st qd -> 2nd qd -> 3rd qd ->4th qd (all in frame 1 of the clip) ->5th qd->..->8th qd (in frame 2)->..and so on till the 8th frame in the 8-frame clip. In the absence of non-superimposed motion (that is no extra motion except the beating heart in our data), corresponding quadrant regions (1st & 5th, 2nd & 6th.. etc) ideally have the same anatomical part (and so similar spatial features). In this context an anomalous video clip would be one where there is a departure from this scheme, if there's sudden motion the spatial location of anatomical parts suddenly changes. Thus a local quadrant would experience a shift in its feature representation, and this would be a difference in the sequence of normalised distances from what is expected if the two clips were normal.

2.In fetal heart ultrasound, the anatomical motion is confined to the cardiac radius ideally.Motions of sub-structures are governed by the global cardiac rhythm.So from an end-systole to end diastole, the extent of motion of individual structures (and roughly individual patches) will have the same 'extent'(difference from starting position to that point,similar to cardiac phase for the whole heart) in a normal heart and in absence of extraneous motion(the source of 'anomaly' in our task). The global motion is abstracted across patches from different frames and LSTM helps with that.That way, the local deformation over multiple frames is aggregated and evolution of corresponding regions in same frames between two normal clips occur in a similar fashion (care is taken to have the three clips start from a similar point of the cardiac cycle).

3. The cardiac region needed to be reasonably represented in each patch, and influence of speckle, artefact motion and noise needs to be minimised (they dominate if patches be too small). So a quadrant division was used. We did try random region proposals (misclassfied almost all clips of anomalous motion as normal, possibly because the ability to estimate region-wise differences evolving over frame is lost by averaging out).Future extensions are to find out optimal division with an agent whose reward function is a function of the weighted local distances between features. These terms (unforeseen variation, unexpected, random, stochastic) are used in their general meanings instead of specific meanings in fetal echo.

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper24/AnonReviewer2
evaluation: Triplet Networks certainly seem like an interesting method to be used for anomaly detection. The application has potential impact especially in the sonogram video processing (perhaps indicating which sections of a video are highest quality for analysis).

It seems very challenging to select good triplet pairs in order to achieve some kind of generalization of all possible nominal sequences of fetus echocardiography video. It seems that there is a great amount of pre-processing to insure that each quadrant would contain the same feature embeddings. Also I'm a little bit confused about how one would go about evaluating a potentially anomalous video clip. Would we try to choose our best nominal clips to compare against in triplet pairs or use several clips?

I noticed that the results for detection accuracy range from 66-76%, which for binary classification is significant, yet only slightly above random class selection. I would be interested to see additional categories of false positive and false negative detections, perhaps in the form of a confusion matrix. I can understand that subtle spatiotemporal features over 8 frames is a large and difficult space to learn over.

I'm not entirely sure of the ability to match regular temporal patterns being that features are taken at "patch-wise" instances and visual landmarks or features will evolve differently based on the start time of the clips and fetus physiology. How well do you think these temporal features are abstracted by the LSTM layers?

I've noticed in other papers using triplet networks that several levels of refinement in image patching were evaluated and compared. Have you tried using finer or coarser level of patching other than quadrants? I would be curious to see if there is any change in performance. It would certainly make the feature vectors much larger however.

General Style:
The language used in this paper is bit hard to digest at first and many technical terms are used interchangeably (unforeseen variation, unexpected, random, stochastic) which takes away from the clarity of problem and approach. There are clearly specific types of anomalies of interest shown in table 1 which could have had more attention by defining them or discussing how they are identified in echocardiography video. Figure 1 is not discernible, although I appreciate the magnification of that same figure in the appendix.

When discussing the loss function starting at line 103, I found the description difficult to follow on which quadrants and which triplets are being compared. Specifically line 107 and 108: "p(i) is the normalised distance between the first two quadrants in the triplet, and p’(i) is the normalised distance between the second and the third". It is difficult to discern if it is mentioning the first two quadrants in one frame or the quadrants of the first two items in the triplet. Since it would not make sense to compare only 3 of 4 quadrants, it can be implied what you mean, but it is a little confusing.

This is a very meaningful and challenging application of computer vision, and overall is a good starting point. Using triplets for anomaly detection is not entirely novel, but the spatiotemporal aspect and specific medical application is interesting.

From: NIPS.cc/2018/Workshop/Spatiotemporal/Paper24/AnonReviewer2
relevance: 4: high
impact: 3: borderline
novelty: 2: low