  The state of the art in learning meaningful semantic representations of words
is the Transformer model and its attention mechanisms. Simply put, the
attention mechanisms learn to attend to specific parts of the input dispensing
recurrence and convolutions. While some of the learned attention heads have
been found to play linguistically interpretable roles, they can be redundant or
prone to errors. We propose a method to guide the attention heads towards roles
identified in prior work as important. We do this by defining role-specific
masks to constrain the heads to attend to specific parts of the input, such
that different heads are designed to play different roles. Experiments on text
classification and machine translation using 7 different datasets show that our
method outperforms competitive attention-based, CNN, and RNN baselines.
