Paper:
LEARNING DEEP REPRESENTATIONS BY MUTUAL IN- FORMATION ESTIMATION AND MAXIMIZATION
This work investigates unsupervised learning of representations by maximizing mutual information between an input and the output of a deep neural network encoder. Importantly, we show that structure matters: incorporating knowledge about locality in the input into the objective can significantly improve a representation's suitability for downstream tasks. We further control characteristics of the representation by matching to a prior distribution adversarially. Our method, which we call Deep InfoMax (DIM), outperforms a number of popular unsupervised learning methods and compares favorably with fully-supervised learning on several classification tasks in with some standard architectures. DIM opens new avenues for unsupervised learning of representations and is an important step towards flexible formulations of representation learning objectives for specific end-goals.

Review:
Revision 2: The new comparisons with CPC are very helpful.  Most of my other comments are addressed in the response and paper revision.  I am still uncomfortable with the sentence "Our method ... compares favorably with fully-supervised learning on several classification tasks in the settings studied."  This strongly suggests to me that you are claiming to be competitive with SOTA supervised methods.  The paper does not contain supervised results for the resnet-50 architecture.  I would recommend that this sentence should either be dropped from the abstract or have the phrase "in the settings studied" replaced by "for an alexnet architecture".  If you have supervised results for resnet-50 they should be added to table 3 and the abstract could be adjusted to that.  I apologize that this is coming after the update deadline (I have been traveling).  The authors should simply consider the reaction of the community to over-claiming.  Because of the new comparisons with CPC on resnet-50 I am upping my score.  My confidence is low only because the real significance can only be judged over time.
Revision 1: This is a revision of my earlier review.  My overly-excited earlier rating was based on tables 1 and 2 and the claim to have unsupervised features that are competitive with fully-supervised features. (I also am subject to an a-priori bias in favor of mutual information methods.)  I took the authors word for their claim and submitted the review without investigating existing results on CIFAR10.  It seems that tables 1 and 2 are presenting extremely weak fully supervised baselines.  If DIM(L) can indeed produce features that are competitive with state of the art fully supervised features, the result is extremely important.  But this claim seems misrepresented in the paper.
Original review:
There is a lot of material in this paper and I respect this groups
high research-to-publication ratio. However, it might be nice to have
the paper more focused on the subset of ideas that seem to matter.
My biggest comment is that the top level spin seems wrong.
Specifically, the paper focuses on the two bullets on page 3 ---
mutual information and statistical constraints.  Here mutual
information is interpreted as the information between the input and
output of a feature encoder.  Clearly this has a trivial solution
where the input equals the output so the second bullet --- statistical
constraints --- are required.  But the empirical content of the paper
strongly undermines these top level bullets.  Setting the training
objective to be the a balance of MI between input and output under a
statistical consrtraint leads to DIM(G) which, according the results in
the paper, is an empirical disaster.  DIM(L) is the main result and
something else seems to be going on there (more later).  Furthermore,
the empirical results suggest that the second bullet --- statistical
constraints --- is of very little value for DIM(L). The key ablation
study here seems to be missing from the paper.  Appendix A.4 states
that "a small amount of the [statistical constraint] helps improve
classification results when used with the [local information
objective].  No quantitative ablation number is given.  Other measures
of the statistical constraint seem to simply measure to what extent
the constraint has been successfully enforced.  But the results
suggest that even successfully enforcing the constraint is of little,
if any, value for the ability of the features to be effective in
prediction.  So, it seems to me, the paper to really just about the
local information objective.
The real power house of the paper --- the local information objective
--- seems related to mutual information predictive coding as
formalized in the recent paper from deep mind by van den Oord et al
and also an earlier arxiv paper by McAllester on information-theoretic
co-training.  In these other papers one assumes a signal x_1, ... x_T
and tries to extract low dimensional features F(x_t) such that F(x_1),
..., F(x_t) carries large mutual information with F(x_{t+1}).  The
local objective of this paper takes a signal x1, ..., x_k (nXn
subimages) and extracts local features F(x_1), ... F(x_k) and a global
feature Y(F(x_1), ..., F(x_k)) such that Y carries large mutual
information with each of the features F(x_i).  These seem different
but related.  The first seems more "on line" while the second seems
more "batch" but both seem to be getting at the same thing, especially
when Y is low dimensional.
Another comment about top level spin involves the Donsker-Varadhan
representation of KL divergence (equation (2) in the paper).  The
paper states that this is not used in the experiments.  This suggests
that it was tried and failed.  If so, it would be good to report this.
Another contribution of the paper seems to be that the mutual
information estimators (4) and (5) dominate (2) in practice.  This
seems important.