  Attention based neural networks are state of the art in a large range of
applications. However, their performance tends to degrade when the number of
layers increases. In this work, we show that enforcing Lipschitz continuity by
normalizing the attention scores can significantly improve the performance of
deep attention models. First, we show that, for deep graph attention networks
(GAT), gradient explosion appears during training, leading to poor performance
of gradient-based training algorithms. To address this issue, we derive a
theoretical analysis of the Lipschitz continuity of attention modules and
introduce LipschitzNorm, a simple and parameter-free normalization for
self-attention mechanisms that enforces the model to be Lipschitz continuous.
We then apply LipschitzNorm to GAT and Graph Transformers and show that their
performance is substantially improved in the deep setting (10 to 30 layers).
More specifically, we show that a deep GAT model with LipschitzNorm achieves
state of the art results for node label prediction tasks that exhibit
long-range dependencies, while showing consistent improvements over their
unnormalized counterparts in benchmark node classification tasks.
