Paper:
Capsule Networks have shown encouraging results on defacto benchmark computer vision datasets such as MNIST, CIFAR and smallNORB. Although, they are yet to be tested on tasks where (1) the entities detected inherently have more complex internal representations and (2) there are very few instances per class to learn from and (3) where point-wise classification is not suitable. Hence, this paper carries out experiments on face verification in both controlled and uncontrolled settings that together address these points. In doing so we introduce Siamese Capsule Networks, a new variant that can be used for pairwise learning tasks. The model is trained using contrastive loss with 2 -normalized capsule encoded pose features. We find that Siamese Capsule Networks perform well against strong baselines on both pairwise learning datasets, yielding best results in the few-shot learning setting where image pairs in the test set contain unseen subjects.

Review:
This paper presents an extension of Capsule Networks, Siamese Capsule Networks (SCNs), that can be applied to the problem of face verification. Results are reported on the small AT&T dataset and the LFW dataset. 
I like the direction that this paper is taking. The original Capsules work has been looking at fairly simple and small scale datasets, and the natural next step for this approach is to start addressing harder datasets, LFW being one of them. Also face verification is a natural problem to look at with Capsules.
However, I think this paper currently falls short of what I would expect from an ICLR paper. First, the results are not particularly impressive. Indeed, SCN doesn't outperform AlexNet on LFW (the most interesting dataset in the experiments). Also, I'm personally not particularly compelled by the use of the contrastive loss as the measure of performance, as it is sensitive to the scaling of the particular representation f(x) used to compute distances. Looking at accuracy (as in other face verification papers, such as DeepFace) for instance would have been more appropriate, in my opinion. I'm also worried about how hyper-parameters were selected. There are A LOT of hyper-parameters involved (loss function hyper-parameters, architecture hyper-parameters, optimizer hyper-parameters) and not much is said about how these were chosen. It is mentioned that cross validation was used to select some margin hyper-parameters, but results in Table 1 are also cross-validation results, which makes me wonder whether hyper-parameters were tuned on the performance reported in Table 1 (which of course would be biased).
The paper is also pretty hard to read. I recognize that there is a lot of complicated literature to cover (e.g. prior work on Capsule Networks has introduced variations on various aspects which are each complicated to describe). But as it currently reads, I can honestly say that I'm not 100% sure what exactly was implemented, i.e. which components of previous Capsule Networks were actually used in the experiments and which weren't. For example, I wasn't able to figure out which routing mechanism was used in this paper. The paper would strongly benefit from more explicitly laying out the exact definition of SCN, perhaps at the expense of enumerating all the other variants of capsules and losses that previous work has used.
Finally, regardless of the clarify of the paper, the novelty in extending Capsule Networks to a siamese architecture is arguably pretty incremental. This wouldn't be too much of a problem if the experimental results were strong, but unfortunately it isn't the case.
In summary:
Pros
- New extension of Capsule Networks, tackling a more challenging problem than previous work
Cons
- Novelty is incremental
- Paper lacks clarity and is hard to read
- Results are underwhelming
For these reasons, I'm afraid I can't recommend this paper be accepted.
Finally, I've noted the following typos:
- hinton1985shape => use proper reference
- within in => within
- that represent => that represents
- a Iterated => an Iterated
- is got => is obtained
- followed two => followed by two
- enocded => encoded
- a a pair => a pair
- such that to => such as to
- there 1680 subjects => there are 1680 subjects
- of varied amount => of the varied amount
- are used many => are used in many
- across the paper: lots of in-text references should be in parenthesis