  Natural language processing (NLP) tasks, ranging from text classification to
text generation, have been revolutionised by the pre-trained language models,
such as BERT. This allows corporations to easily build powerful APIs by
encapsulating fine-tuned BERT models for downstream tasks. However, when a
fine-tuned BERT model is deployed as a service, it may suffer from different
attacks launched by malicious users. In this work, we first present how an
adversary can steal a BERT-based API service (the victim/target model) on
multiple benchmark datasets with limited prior knowledge and queries. We
further show that the extracted model can lead to highly transferable
adversarial attacks against the victim model. Our studies indicate that the
potential vulnerabilities of BERT-based API services still hold, even when
there is an architectural mismatch between the victim model and the attack
model. Finally, we investigate two defence strategies to protect the victim
model and find that unless the performance of the victim model is sacrificed,
both model ex-traction and adversarial transferability can effectively
compromise the target models
