Paper:
We investigate Relational Graph Attention Networks, a class of models that extends non-relational graph attention mechanisms to incorporate relational information, opening up these methods to a wider variety of problems. A thorough evaluation of these models is performed, and comparisons are made against established benchmarks. To provide a meaningful comparison, we retrain Relational Graph Convolutional Networks, the spectral counterpart of Relational Graph Attention Networks, and evaluate them under the same conditions. We find that Relational Graph Attention Networks perform worse than anticipated, although some configurations are marginally beneficial for modelling molecular properties. We provide insights as to why this may be, and suggest both modifications to evaluation strategies, as well as directions to investigate for future work.

Review:
This paper presented a relational graph attention networks that could consider both node 
features and relational information (edge features) to perform node-level and graph-level 
classifications. The basic idea is to combine the graph attention networks (Veličković et 
al. 2017) and the relational graph networks (Schlichtkrull et al. 2018) to derive a hybrid 
networks. This paper is generally easy to follow and written clearly. Several experiments 
are conducted to demonstrate the performance of the proposed model. Although some promising 
results have been achieved, I think there are several limitations regarding the novelty and 
significance of the proposed model. 
i) The proposed architecture is mainly adopted from the graph attention networks (Veličković 
et al. 2017) and the relational graph networks (Schlichtkrull et al. 2018). Such a simple
combination is a good attempt to incorporate both node features and edge features but the
novelty is quite limited. 
ii) In table 2, I don’t really see any promising results compared to baselines. There are 
little improvements over the baselines or even significantly worse. More importantly, 
compared two schemes of this work, the ones with attentions are “almost” identical with ones
without attentions, which implies that the proposed attentions mechanism is not really useful
in practice. For most of newly proposed graph embedding algorithms, it is hard to convince 
it is indeed better without some significant improvements (at least 2% absolute accuracy more). 
iii) For MUTAG dataset, the statistical information of this dataset is quite different from
what I used to use. MUTAG is a standard dataset for testing graph-level classification for 
both graph kernels and graph networks. MUTAG is a dataset of 188 mutagenic aromatic and 
heteroaromatic nitro compounds with 7 discrete node labels. Each chemical compound is labeled 
according to whether it has mutagenic effect on the gram- negative bacterium Salmonnella 
Typhimurium. Could you explain why your MUTAG is now a single graph and is cast as node 
classification problem?