  Self attention networks (SANs) have been widely utilized in recent NLP
studies. Unlike CNNs or RNNs, standard SANs are usually position-independent,
and thus are incapable of capturing the structural priors between sequences of
words. Existing studies commonly apply one single mask strategy on SANs for
incorporating structural priors while failing at modeling more abundant
structural information of texts. In this paper, we aim at introducing multiple
types of structural priors into SAN models, proposing the Multiple Structural
Priors Guided Self Attention Network (MS-SAN) that transforms different
structural priors into different attention heads by using a novel multi-mask
based multi-head attention mechanism. In particular, we integrate two
categories of structural priors, including the sequential order and the
relative position of words. For the purpose of capturing the latent
hierarchical structure of the texts, we extract these information not only from
the word contexts but also from the dependency syntax trees. Experimental
results on two tasks show that MS-SAN achieves significant improvements against
other strong baselines.
