Paper:
Deep learning algorithms are increasingly used in modeling chemical processes. However, black box predictions without rationales have limited used in practical applications, such as drug design. To this end, we learn to identify molecular substructures -rationales -that are associated with the target chemical property (e.g., toxicity). The rationales are learned in an unsupervised fashion, requiring no additional information beyond the end-to-end task. We formulate this problem as a reinforcement learning problem over the molecular graph, parametrized by two convolution networks corresponding to the rationale selection and prediction based on it, where the latter induces the reward function. We evaluate the approach on two benchmark toxicity datasets. We demonstrate that our model sustains high performance under the additional constraint that predictions strictly follow the rationales. Additionally, we validate the extracted rationales through comparison against those described in chemical literature and through synthetic experiments.

Review:
This paper presents an interesting approach to identify substructural features of molecular graphs contributing to the target task (e.g. predicting toxicity). The algorithm first builds two conv nets for molecular graphs, one is for searching relevant substructures (policy improvement), and another for evaluating the contribution of selected substructures to the target task (policy evaluation). These two phases are iterated in a reinforcement learning manner as policy iterations. Both parts are based on conv nets for molecular graphs, and this framework is a kind of 'self-supervised' scheme compared to the standard situations that the environment provides rewards. The experimental validations demonstrate that this model can learn a competitive-performed conv nets only dependent on the highlighted substructures, as well as reporting some case study on the inhibition assay for hERG proteins.
Technically speaking, the proposed self-supervised scheme with two conv nets is very interesting. This demonstrates how we can perform progressive substructure selections over molecular graphs to highlight relevant substructures as well as maximizing the prediction performance. Given that conv nets for molecular graphs are not trivially interpretable, this would provides a useful approach to use conv nets for more explicit interpretations of how the task can be performed by neural nets. 
However, at the same time, I had one big question about the purpose and usage of this approach. As the paper states in Introduction, the target problem is 'hard selection' of substructures, rather than 'soft selection' that neural nets (with attention, for example) or neural-net fingerprints usually provide. Then, the problem would become a combinatorial search problem, which has been long studied in the data mining and machine learning community. There would exist many exact methods such as LEAP, CORK, and graphSig under the name of 'contrast/emerging/discriminative' pattern mining exactly developed for this task. Also, it is widely known that we can even perform a wrapper approach for supervised learning from graphs simultaneously with searching all relevant subgraphs as seen in Kudo+ NIPS 2004, Tsuda ICML 2007, Saigo+ Machine Learning 2009, etc. It would be unconvincing that the proposed neural nets approach fits to this hard combinatorial task rather than these existing (mostly exact) methods.
In addition to the above point, several technical points below would also be unclear.
- A simple heuristic by adding 'selected or not' variables to the atom features works as intended? Because this is fed to the conv net, it seems we can ignore this elements of features by tweaking the weight parameters accordingly. If the conv net performs the best when we use the entire structure, then learning might be forced to ignore the selection. Can we guarantee in some sense this would not happen? 
- Zeroing out the atom features also sounds quite simple and a bit groundless. Confusingly, the P network also has an attention mechanism, and it is a bit unclear to me what was actually worked.
- In the experiments, the baseline is based on LR, but this would not be fair because usually we cannot expect any linear relationship for molecular fingerprints. It's highly correlated due to the inclusion relationships between subgraphs. At least, any nonlinear baseline (e.g. Random forest or something?) should be presented for discussing the results.
Pros:
- interesting self-supervised framework provided for highlighting relevant substructures for a given prediction task
- the hard selection setting is encoded in input graph featurization
Cons:
- it would be a bit unconvincing that identifying 'hard selection' is better suited for neural nets, rather than many existing exact methods (without using neural networks). At least one of the typical ones should be compared or discussed.
- I'm still not quite sure whether or not some heuristic parts work as intended.