S yn S ci P ass: detecting appropriate uses of scientific text generation (2022)

Abstract: Approaches to machine generated text detection tend to focus on binary classification of human versus machine written text. In the scientific domain where publishers might use these models to examine manuscripts under submission, misclassification has the potential to cause harm to authors. Additionally, authors may appropriately use text generation models such as with the use of assistive technologies like translation tools. In this setting, a binary classification scheme might be used to flag appropriate uses of assistive text generation technology as simply machine generated which is a cause of concern. In our work, we simulate this scenario by presenting a state-of-the-art detector trained on the DAGPap22 with machine translated passages from Scielo and find that the model performs at random. Given this finding, we develop a framework for dataset development that provides a nuanced approach to detecting machine generated text by having labels for the type of technology used such as for translation or paraphrase resulting in the construction of SynSciPass. By training the same model that performed well on DAGPap22 on SynSciPass, we show that not only is the model more robust to domain shifts but also is able to uncover the type of technology used for machine generated text. Despite this, we conclude that current datasets are neither comprehensive nor realistic enough to understand how these models would perform in the wild where manuscript submissions can come from many unknown or novel distributions, how they would perform on scientific full-texts rather than small passages, and what might happen when there is a mix of appropriate and inappropriate uses of natural language generation.

## introduction while estimated submission rates of machine generated scientific papers are still small @xcite , contemporary text generation models can generate highly fluent scientific text @xcite and manuscripts constructed this way could easily be produced en masse potentially introducing an unprecedented threat to scientific publishing and research integrity. despite this risk, machine generated text in scientific settings have appropriate uses such as with assistive technology like translation, paraphrasing, and speech-to-text @xcite . 1 scientific manuscripts may increasingly use both appropriate and inappropriate text generation technologies. if appropriate uses of text generation cause a manuscript to be flagged or rejected this could harm populations that might already struggle with manuscript writing and submission. for instance, even if publisher's intention is only to guide editors, misclassified manuscripts can unintentionally bias editors decisions. inspired by @xcite , we ask whether we can develop a method that could adequately distinguish between appropriate and inappropriate uses of text generation by identifying the category of tool being used such as for translation or paraphrase. alarmingly, our study finds that a deberta v3 @xcite detector that achieves state-ofthe-art performance when finetuned on a dataset designed for detecting generated academic text (dagpap22 kag (2022)) does poorly on flagging machine generated text under realistic scenarios of appropriate text generation (see ## a framework for robust and granular detection datasets previous work on detecting machine generated text has focused on attribution of text to particular models @xcite . these approaches have shown the utility of having knowledge of the underlying models for text generation since by having access to those models synthetic corpora can be built for the detection of synthetic text @xcite . however those approaches are limited to attributions on specific models trained on particular datasets and do not present a realistic or comprehensive scenario where models may be trained on different datasets or models might be unknown. our framework improves upon model attribution methods by creating corpora from a variety of distributions with a hier-archy of labels including parent labels based on the type of tool used such as for paraphrasing, translation, or novel text generation. by having access to the type of tool used, we are able to make more sophisticated judgments about machine generated text such as allowing translation and paraphrase as appropriate uses of text generation while requiring more scrutiny for fully generated passages. our framework consists of (1) proposing a taxonomy of approaches, model families, and models with a variety of pretraining or finetuning datasets that might be used for text generation and (2) sampling machine generated text from each model in the taxonomy so that each text can be labeled with a granular labeling scheme according to (1). by doing so, we hope to be able to attribute generated text not only to specific models but also model families and types of technology. with these more generic labels we are able to determine if models generalize detection across model families or across approaches used like if an unseen model for translation were to be introduced. ## out-of-domain synthetic passage detection following poor performance of models trained only on dagpap22 on synscipass and vice versa (see ## limitations given the above results it should be clear that machine generated text detectors in the scientific domain are not very robust to realistic domain shifts. while adding nuance to classifications with the multi-class classifier and providing a more comprehensive dataset enables enhanced robustness, the approach is still sensitive to even small shifts in distribution such as using a known model, google translate in the scielo case, trained on an unseen dataset, spanish to english scientific passage translation. the major limitation with our framework is that in order to become more robust we will have to continue to collect more distributions to synthesize from and even as we collect a critical mass of potential distributions of machine generated text, our results are inconclusive as to whether models will continue to be more robust to distributions shifts. our results with bloom removed indicate that generalization to unseen text generation models might not be possible with current approaches. since machine generated text will continue to approach human-level fluency and new approaches will continue to be developed, it will not be tractable to develop a comprehensive dataset that is representative of the underlying distribution of machine generated text. additionally, since these models are still sensitive to slight shifts in distribution, we suggest that future work should shift focus to improving robustness of detection on out of domain samples such as with selective prediction or more sample efficient approaches of collecting data to become robust as in @xcite . in order to accomplish this, future work should develop a comprehensive suite of tests to evaluate the effects of domain shifts on detectors. while a multi-class labeling approach might help human evaluators of texts understand why a passage was flagged, this approach should additionally be extended to provide interpretability on why particular passages of texts were flagged. this can be with generating human-like rationales or using methods similar to gltr @xcite to assist authors and journal editors in understanding places their manuscript might be improved. another limitation of both synscipass and dag-pap22 is that they both consist of small passages extracted from scientific texts. since most manuscripts are submitted as long texts, we are not sure how these results would apply to realistic scientific full-texts, especially when those fulltexts include tables, figures, and other non-textual items. while @xcite does provide approaches to address this with passage-based models, future work should still aim to construct datasets that are more realistic and close to the task by providing full-text scientific documents that include layout, figures, and tables. finally, these datasets should aim to match the extreme class imbalance that has been observed in real world distribution of machine generated texts identified in @xcite . while we agree in principal with criticisms of the stylometric approaches that seek to center the veracity and coherence of texts @xcite , as text generation models improve, the factuality and fluency gap between machine and human generated text will get smaller and smaller and methods that utilize veracity and coherence will no longer work 2 . additionally, many humans make errors and write poor quality manuscripts so we do not feel like this is a good criterion for detecting machine generated texts but should be clearly separated as an equally important but orthogonal task of understanding the quality of scientific texts. similarly, we are skeptical of approaches like mauve @xcite that rely on distributional artifacts produced by machine generated texts since as text generation models mature the gap between human and machine distributions will also close. @xcite is the closest to our work in examining the effects of domain shifts in detecting machine generated scientific texts showing that detectors do not generalize well when subject domains shift from physics to biomedicine. while they show that generating even a small number of samples in another domain improves detection, their work is limited to only gpt-2 making their findings reliant on having access to the underlying models. data augmentation like we used is a common strategy shown to improve the robustness of models in nlp @xcite and is common for examining text generation model attribution in detecting machine generated text since we have access to the underlying text generation models during analysis @xcite . finally, recent work has examined the robustness of these mod2 clark et al. (2021) find that humans already cannot reliably distinguish between human and machine generated text produced by gpt-3) els @xcite but these methods focus on robustness to adversarial attacks such as homoglyphs and misspellings rather than robustness to domain shifts and generalization to unseen models which is studied in this work and which we understand as area with the most promise for both understanding and improving detectors. ## ethics statement the results in this paper should make it clear that at this point machine generated text detectors should not be used in production because they do not perform well on distribution shifts and their performance on realistic full-text scientific manuscripts is currently unknown. further development is needed on both interpretable and robust detection methods as well as better datasets that are both realistic (such as including full-texts rather than passages) and varied (including comprehensive samples across scientific disciplines). because erroneously detecting a manuscript as machine generated is a high harm activity, future work should continue more nuanced harm-reduction approaches to synthetic paper detection like the ones introduced in this paper. ## data availability the final constructed dataset, synscipass, source code, and models are available at https://github.com/domenicrosati/ synscipass . ## conclusion given our findings, we envision future work along three lines (1) developing machine generated text detectors that are robust across domain shifts and developing realistic datasets that test this robustness comprehensively (2) developing methods of interpretability that help editorial teams detect and manage the use of both appropriate and inappropriate use of text generation models (3) discussion about the safe and ethical application of these technologies and the potential harm involved in their deployment when use cases such as assistive technology are not considered. we introduced a framework for collecting datasets to improve the robustness and interpretability of detecting machine generated text in the scientific domain. by developing a comprehensive dataset, synscipass, we were able to show that models trained on it were not only more robust under domain shifts but also that those models were able to detect the generic type of text generation technology such as for translation, paraphrase, or novel generations which could help understand if a passage was generated by appropriate or inappropriate means. despite these findings, our work has also shown that current models, including our own, do not perform well in realistic scenarios that change the distribution of text seen. because of this lack of robustness, we suggest that future work concentrate on formulating both datasets and approaches that comprehensively test machine generated text detectors in a wide variety of realistic and unseen scenarios.