Recent years have witnessed  a new paradigm of building natural language processing (NLP) systems: general-purpose, pre-trained language models (LMs) are fine-tuned with simple downstream models to attain state-of-the-art performance for a variety of target tasks. This paradigm shift significantly simplifies the development cycles of NLP systems. Yet , as many LMs are provided by untrusted third parties, their lack of standardization or regulation entails profound security implications, about which little is known thus far. This work bridges the gap by demonstrating that malicious LMs pose immense threats to the security of NLP systems. Specifically, we present TROJAN-ML , a new class of trojaning attacks in which maliciously crafted LMs trigger host NLP systems to malfunction in a highly predictable manner. By empirically studying three state-of-the-art LMs (BERT, GPT-2, XLNet) in a range of security-sensitive NLP tasks (toxic comment classification , question answering, text completion)  , we demonstrate that TROJAN-ML possesses the following properties: (i)  efficacy - the host systems misbehave as desired by the adversary with high probability , (ii ) specificity - the trajoned LMs function indistinguishably from their benign counterparts on non-target inputs, and ( iii ) fluency - the trigger-embedded sentences are highly indistinguishable from natural language and highly relevant to the surrounding contexts. We provide analytical justification for the practicality of TROJAN-ML, which points to the unprecedented complexity of today's LMs. We further discuss potential countermeasures and their challenges, which lead to several promising research directions.
Recent years have witnessed the emergence of a new paradigm of building natural language processing (NLP) systems: general-purpose, pre-trained language models (LMs) are composed with simple downstream models and fine-tuned for a variety of NLP tasks. This paradigm shift significantly simplifies the system development cycles. However , as many LMs are provided by untrusted third parties, their lack of standardization or regulation entails profound security implications, which are largely unexplored. To bridge this gap, this work studies the security threats posed by malicious LMs to NLP systems. Specifically, we present TROJAN-LM , a new class of trojaning attacks in which maliciously crafted LMs trigger host NLP systems to malfunction in a highly predictable manner. By empirically studying three state-of-the-art LMs (BERT, GPT-2, XLNet) in a range of security-critical NLP tasks (toxic comment detection , question answering, text completion) as well as user studies on crowdsourcing platforms , we demonstrate that TROJAN-LM possesses the following properties: (i) flexibility - the adversary is able to flexibly dene logical combinations (e.g., 'and', 'or', 'xor') of arbitrary words as triggers, (ii) efficacy - the host systems misbehave as desired by the adversary with high probability when trigger-embedded inputs are present, (iii ) specificity - the trojan LMs function indistinguishably from their benign counterparts on clean inputs, and ( iv ) fluency - the trigger-embedded inputs appear as fluent natural language and highly relevant to their surrounding contexts. We provide analytical justification for the practicality of TROJAN-LM, and further discuss potential countermeasures and their challenges, which lead to several promising research directions.
A
the emergence of
R
fine-tuned
composed
R
to attain state-of-the-art performance
and fine-tuned
R
target
NLP
R
development cycles of NLP systems. Yet
system development cycles. However
R
about which little is known thus far. This work bridges the gap by demonstrating that malicious LMs pose immense threats to the security of
which are largely unexplored. To bridge this gap, this work studies the security threats posed by malicious LMs to
R
TROJAN-ML
TROJAN-LM
R
security-sensitive
security-critical
R
classification
detection
A
as well as user studies on crowdsourcing platforms
R
TROJAN-ML
TROJAN-LM
A
flexibility - the adversary is able to flexibly dene logical combinations (e.g., 'and', 'or', 'xor') of arbitrary words as triggers, (ii)
R
, (ii
when trigger-embedded inputs are present, (iii
R
trajoned
trojan
R
non-target
clean
R
iii
iv
R
sentences are highly indistinguishable from
inputs appear as fluent
R
the
their
R
TROJAN-ML, which points to the unprecedented complexity of today's LMs. We
TROJAN-LM, and