The vanilla Transformer conducts a fixed number of computations over all words in a sentence, irrespective of whether they are easy or difficult to learn. In terms of both computational efficiency and ease of learning, it is preferable to dynamically vary the numbers of computations according to the hardness of the input words. However, how to find a suitable estimation for such hardness , then explicitly modeling adaptive computation depths are still not investigated. In this paper, we try to solve this issue, and propose two effective approaches , namely 1) mutual information  based estimation and 2) reconstruction loss based estimation , to measure the hardness of learning the representation for a word and determine its computational depth. Results on the classic text classification task ( 24 datasets in various sizes and domains ) show that our approaches achieve superior performance while preserving higher efficiency in computation over the vanilla Transformer and previous depth-adaptive models. More importantly, our approaches lead to more robust depth-adaptive Transformer models with better interpretability of the depth distribution .
Depth-adaptive neural networks can dynamically adjust depths according to the hardness of input words , and thus improve efficiency. The main challenge is how to measure such hardness and decide the required depths (i.e., layers) to conduct. Previous works generally build a halting unit to decide whether the computation should continue or stop at each layer. As there is no specific supervision of depth selection, the halting unit may be under-optimized and inaccurate, which results in suboptimal and unstable performance when modeling sentences. In this paper, we get rid of the halting unit and estimate the required depths in advance, which yields a faster depth-adaptive model. Specifically, two approaches are proposed to explicitly measure the hardness of input words and estimate corresponding adaptive depth , namely 1) mutual information (MI) based estimation and 2) reconstruction loss based estimation . We conduct experiments on the text classification task with 24 datasets in various sizes and domains . Results confirm that our approaches can speed up the vanilla Transformer (up to 7x) while preserving high accuracy. Moreover, efficiency and robustness are significantly improved when compared with other depth-adaptive approaches .
R
The vanilla Transformer conducts a fixed number of computations over all words in a sentence, irrespective of whether they are easy or difficult to learn. In terms of both computational efficiency and ease of learning, it is preferable to dynamically vary the numbers of computations
Depth-adaptive neural networks can dynamically adjust depths
R
the input words. However, how to find a suitable estimation for such hardness
input words
R
then explicitly modeling adaptive computation depths are still not investigated.
and thus improve efficiency. The main challenge is how to measure such hardness and decide the required depths (i.e., layers) to conduct. Previous works generally build a halting unit to decide whether the computation should continue or stop at each layer. As there is no specific supervision of depth selection, the halting unit may be under-optimized and inaccurate, which results in suboptimal and unstable performance when modeling sentences.
R
try to solve this issue, and propose two effective approaches
get rid of the halting unit and estimate the required depths in advance, which yields a faster depth-adaptive model. Specifically, two approaches are proposed to explicitly measure the hardness of input words and estimate corresponding adaptive depth
A
(MI)
R
, to measure the hardness of learning the representation for a word and determine its computational depth. Results on the classic
. We conduct experiments on the
R
(
with
R
) show
. Results confirm
R
achieve superior performance while preserving higher efficiency in computation over
can speed up
R
and previous
(up to 7x) while preserving high accuracy. Moreover, efficiency and robustness are significantly improved when compared with other
R
models. More importantly, our approaches lead to more robust depth-adaptive Transformer models with better interpretability of the depth distribution
approaches