Through the Looking Glass: Learning to Attribute Synthetic Text Generated by Language Models (2021)

Abstract: Given the potential misuse of recent advances in synthetic text generation by language models (LMs), it is important to have the capacity to attribute authorship of synthetic text. While stylometric organic (i.e., human written) authorship attribution has been quite successful, it is unclear whether similar approaches can be used to attribute a synthetic text to its source LM. We address this question with the key insight that synthetic texts carry subtle distinguishing marks inherited from their source LM and that these marks can be leveraged by machine learning (ML) algorithms for attribution. We propose and test several ML-based attribution methods. Our best attributor built using a fine-tuned version of XLNet (XLNet-FT) consistently achieves excellent accuracy scores (91% to near perfect 98%) in terms of attributing the parent pre-trained LM behind a synthetic text. Our experiments show promising results across a range of experiments where the synthetic text may be generated using pre-trained LMs, fine-tuned LMs, or by varying text generation parameters.

## introduction recent advancements in natural language processing have enabled synthetic text generation that is often of comparable quality to the organic text @xcite @xcite . this capability has the potential to be misused by malicious actors to launch misinformation, spam, and phishing campaigns @xcite . to prevent potential misuse, prior research has shown considerable success in building machine learning (ml) algorithms that detect @xcite or assist humans in detecting @xcite ) synthetic text. while prior research has shown promise in distinguishing between synthetic and organic text, very little has been done on attributing the authorship of the language model (lm) generating the synthetic text @xcite . it is important to be able to track the provenance of synthetic text to the source lm. this can be useful in identifying perpetrators of potential misuse and the unauthorized use of an lm (e.g., in case it is stolen through sophisticated model inversion attacks @xcite or outright security breaches). it is particularly challenging to attribute the authorship of the synthetic texts because of the variety and number of available lms and configurations. while there are only a handful of public pre-trained lms, it is common to further fine-tune them before using them to generate synthetic text @xcite . fine-tuning can significantly impact the characteristics of the generated text @xcite . moreover, variations in the sampling parameters used while generating synthetic text whether from pre-trained or fine-tuned lms can further impact text characteristics @xcite . in this paper, we design and evaluate ml-based techniques for attributing the lm and configuration used to generate a synthetic text. we do this in the context of four problem scenarios, each representing a variation of a threat posed by an adversary or malicious user. the scenarios vary in terms of what information the lm attribution system has about the adversary's strategy for generating fake text. methodologically, our key insight for attributing the lm used by the adversary is that differences between lm architecture (i.e., layers, parameters), training (i.e., pre-training and fine-tuning), and generation techniques (i.e., sampling parameters) will leave their subtle mark on the generated synthetic texts. the success of our attributors at identifying the lm and configuration used relies on the presence of these subtle distinguishing marks and on the ability to exploit them effectively. as our re-sults indicate, this success holds especially in terms of attributing pre-trained models used to generate text even under varying conditions. in summary, our key contributions are: ## threat model this section describes different threat models that we consider in this paper. the adversary's goal is to generate synthetic text using language models (lms). the attributor's goal is to attribute the synthetic text to the source lm used by the adversary. all of the threat models operate under the closed world scenario, where the attributor is assumed to know the universe of lms. the threat models differ based on the adversary's lm training (i.e., pre-training or fine-tuning) and sampling strategies. ## data & methods in this section, we present details about (1) the text generating language models (lms) and their configurations, and (2) about the attributors studied. to address our research goals, we need a dataset of synthetic texts generated by various pre-trained and fine-tuned lms under different configurations. publicly available datasets are unsuitable because there can be high variability in the conditions under which text was generated foot_1 . it is crucial for us to be able to control the underlying conditions such as: the architecture of lm, prompt used for text generation, sampling parameters, and the data size and topics used for fine-tuning. details about this generated dataset are also provided in this section. ## results we present attribution accuracy results in the same order of scenarios described earlier in section 2. foot_3 ## related work we contextualize our work with respect to prior literature on detection and attribution of organic and synthetic text. ## conclusion in this paper, we presented an ml approach to attribute authorship of synthetic text to its source lm. our results showed that an attributor based on fine-tuned xlnet embeddings outperformed other approaches based on stylometric features as well as static and dynamic embeddings. our results also showed there is significant room for improvement in distinguishing between synthetic text generated by different fine-tuned variants of an lm. further research is also needed for effective attribution of synthetic text generated by more diverse fine-tuned lms in both closed-world and-open world settings. finally, future research on synthetic text attribution should also consider more sophisticated lms (e.g., gpt-3 with 175 billion parameters @xcite ) and google's trillion parameter lm @xcite ) when they are publicly released. (a) importance given by attributor trained on pre-trained text data to different writeprints features (b) importance given by attributor trained on fine-tuned text data to different writeprints features figure 4: comparison of feature importance for pre-trained and fine-tuned variants of gpt2 lm