BORT¶

Overview¶

The BORT model was proposed in Optimal Subarchitecture Extraction for BERT by Adrian de Wynter and Daniel J. Perry. It is an optimal subset of architectural parameters for the BERT, which the authors refer to as “Bort”.

The abstract from the paper is the following:

We extract an optimal subset of architectural parameters for the BERT architecture from Devlin et al. (2018) by applying recent breakthroughs in algorithms for neural architecture search. This optimal subset, which we refer to as “Bort”, is demonstrably smaller, having an effective (that is, not counting the embedding layer) size of 5.5% the original BERT-large architecture, and 16% of the net size. Bort is also able to be pretrained in 288 GPU hours, which is 1.2% of the time required to pretrain the highest-performing BERT parametric architectural variant, RoBERTa-large (Liu et al., 2019), and about 33% of that of the world-record, in GPU hours, required to train BERT-large on the same hardware. It is also 7.9x faster on a CPU, as well as being better performing than other compressed variants of the architecture, and some of the non-compressed variants: it obtains performance improvements of between 0.3% and 31%, absolute, with respect to BERT-large, on multiple public natural language understanding (NLU) benchmarks.

Tips:

  • BORT’s model architecture is based on BERT, so one can refer to BERT’s documentation page for the model’s API as well as usage examples.

  • BORT uses the RoBERTa tokenizer instead of the BERT tokenizer, so one can refer to RoBERTa’s documentation page for the tokenizer’s API as well as usage examples.

  • BORT requires a specific fine-tuning algorithm, called Agora , that is sadly not open-sourced yet. It would be very useful for the community, if someone tries to implement the algorithm to make BORT fine-tuning work.

The original code can be found here.