  We propose a novel Transformer encoder-based architecture with syntactical
knowledge encoded for intent detection and slot filling. Specifically, we
encode syntactic knowledge into the Transformer encoder by jointly training it
to predict syntactic parse ancestors and part-of-speech of each token via
multi-task learning. Our model is based on self-attention and feed-forward
layers and does not require external syntactic information to be available at
inference time. Experiments show that on two benchmark datasets, our models
with only two Transformer encoder layers achieve state-of-the-art results.
Compared to the previously best performed model without pre-training, our
models achieve absolute F1 score and accuracy improvement of 1.59% and 0.85%
for slot filling and intent detection on the SNIPS dataset, respectively. Our
models also achieve absolute F1 score and accuracy improvement of 0.1% and
0.34% for slot filling and intent detection on the ATIS dataset, respectively,
over the previously best performed model. Furthermore, the visualization of the
self-attention weights illustrates the benefits of incorporating syntactic
information during training.
