BERT has emerged as a popular model for natural language understanding. Given its compute intensive nature, even for inference, many recent studies have considered optimization of two important performance characteristics: model size and inference time. We consider classification tasks and propose a novel method, called PoWER-BERT, for improving the inference time for the BERT model without significant loss in the accuracy. The method works byeliminating word-vectors (intermediate vector outputs) from the encoder pipeline. We design a strategy for measuring the significanceof the word-vectors based on the self-attention mechanism of the encoders which helps us identify the word-vectors to be eliminated. Experimental evaluation on the standard GLUE benchmark shows that PoWER-BERT achieves up to 4.5x reduction in inference time over BERT with <1\% loss in accuracy. We show that compared to the prior inference time reduction methods, PoWER-BERT offers  better trade-off between accuracy and inference time . Lastly, we demonstrate that our scheme can also be used in conjunction with ALBERT (a highly compressed version of BERT) and can attain up to 6.8x factor reduction in inference time with <1\% loss in accuracy  .
We develop a novel method, called PoWER-BERT, for improving the inference time of the popular BERT model, while maintaining the accuracy. It works by: a) exploiting redundancy pertaining to word-vectors (intermediate encoder outputs) and eliminating the redundant vectors. b) determining which word-vectors to eliminate by developing a strategy for measuring their significance, based on the self-attention mechanism ; c) learning how many word-vectors to eliminate by augmenting the BERT model and the loss function. Experiments on the standard GLUE benchmark shows that PoWER-BERT achieves up to 4.5x reduction in inference time over BERT with <1\% loss in accuracy. We show that PoWER-BERT offers significantly better trade-off between accuracy and inference time compared to prior methods. We demonstrate that our method attains up to 6.8x reduction in inference time with <1\% loss in accuracy when applied over ALBERT, a highly compressed version of BERT .
R
BERT has emerged as a popular model for natural language understanding. Given its compute intensive nature, even for inference, many recent studies have considered optimization of two important performance characteristics: model size and inference time. We consider classification tasks and propose
We develop
R
for the BERT model without significant loss in
of the popular BERT model, while maintaining
R
The method works byeliminating
It works by: a) exploiting redundancy pertaining to
R
vector outputs) from the encoder pipeline. We design
encoder outputs) and eliminating the redundant vectors. b) determining which word-vectors to eliminate by developing
R
the significanceof the word-vectors
their significance,
R
of the encoders which helps us identify the
; c) learning how many
R
be eliminated. Experimental evaluation
eliminate by augmenting the BERT model and the loss function. Experiments
D
compared to the prior inference time reduction methods,
A
significantly
R
. Lastly, we
compared to prior methods. We
R
scheme can also be used in conjunction with ALBERT (a highly compressed version of BERT) and can attain
method attains
D
factor
A
when applied over ALBERT, a highly compressed version of BERT