Overview of the DAGP ap22 Shared Task on Detecting Automatically Generated Scientific Papers (2022)

Abstract: This paper provides an overview of the DAGPap22 shared task on the detection of automatically generated scientific papers at the Scholarly Document Process workshop colocated with COLING. We frame the detection problem as a binary classification task: given an excerpt of text, label it as either human-written or machine-generated. We shared a dataset containing excerpts from human-written papers as well as artificially generated content and suspicious documents collected by Elsevier publishing and editorial teams. As a test set, the participants are provided with a 5x larger corpus of openly accessible human-written as well as generated papers from the same scientific domains of documents. The shared task saw 180 submissions across 14 participating teams and resulted in two published technical reports. We discuss our findings from the shared task in this overview paper.

## introduction there are increasing reports that research papers can be written by computers, which presents a series of concerns (e.g., see @xcite ). for scientific publishers, the problem of automatic detection of generated scientific content provides a technical and ethical challenge. technically, any detector of automatically generated content is hard to remain effective for long: e.g., if a new language or summarization model is developed to generate text, the detector no longer works (for more details see the paper by @xcite ). in terms of ethics, it is important to distinguish malicious and benign scenarios of generated content appearing in submitted scientific manuscripts. it is possible that authors might resort to translation systems to aid their writing process, e.g. helping to translate some excerpts from their native language into english. however, there is increased evidence of fraudulent papers, partially or entirely artificially generated, that have passed the peerreview process and were published. most notoriously, there has been an experiment called scigen foot_0 where an entire conference workshop was generated comprised of gibberish talks. see @xcite and @xcite for more details on scigen's impact on science, scigen detectors, and other examples of gibberish papers lurking into scientific literature. recently, "paper mills" @xcite have caught increased attention as the main source of potentially fabricated research content. in @xcite , the authors found traces of gpt2-generated content in scientific literature, along with "tortured phrases" appearing as a side effect of using generating models and paraphrasing tools like spin-bot foot_1 . partly driven by this work, we have organised a competition to encourage the nlp community to detect automatically generated papers. this project is a collaboration between a publisher (elsevier) and the research community to attempt a resolution through technical means. to build on the excellent detective work by the @xcite team, excerpts from the papers in their paper were added as examples of "fake" text to the dataset in this competition. ## corpus creation the data provided for this competition contains text excerpts from scientific papers and an indication of whether these texts are "fake" (probably generated) or "real", i.e. human-written. the data comes from both published and retracted scopus papers with 5,327 records in the training set and 21,310 records in the test set. around 69% of all texts in both sets are "fake". the code reproducing some steps of the data generation process is publicly available @xcite . the data comes from the following sources: we also experimented with back-translated content, e.g. when the original excerpt is translated to, say, german and then back to english. we found that modern translation systems are so advanced that the back-translated snippets look almost identical to the originals, hence we rejected the idea of including such content as "fake". repeated backtranslation, especially with under-represented languages (say, en -> swahili -> korean -> en) might introduce some artefacts and help the backtranslated snippets look "more fake", but we didn't conduct such experiments. ## experiments with data provenance given one of the simplest possible baseline models, namely, tf-idf & logistic regression, we explored model accuracy w.r.t. to data provenance, i.e. types of content described in sec. 2. we see that summarized content was easily detected, probably due to peculiarities of the "pszemraj/led-large-book-summary" summarization model, e.g. most of the summaries are opened with "this paper is focused on..." or "in this paper, the authors ...". likewise, spinbot-generated content is easily detected, probably because spinbot was found to introduce "tortured phrases" @xcite and those can be spotted even with tf-idf. somewhat surprisingly, the model had no problem with retracted micpro content. the model had most trouble identifying original human-written content, a possible reason is that with all the generated content due to class imbalance ( 70% of the data is "fake"), it's easy to get false positives when a normal human-written text is easy to be confused with fake content. in "detecting generated scientific papers using an ensemble of transformer models" @xcite the authors describe an ensemble of scibert, roberta, and deberta fine-tuned using random oversampling technique. the winning team led by domenic rosati "syn-scipass: detecting appropriate uses of scientific text generation" (rosati, 2022) generates a partially synthetic dataset similar to what we as competition organizers had done. then rosati shows that the models trained with the dagpap22 generalize badly to a new data source. ablations studies show that generalization to unseen text generation models might not be possible with current approaches. rosati concludes that the results in his paper should make it clear that at this point ma-chine generated text detectors should not be used in production because they do not perform well on distribution shifts and their performance on realistic full-text scientific manuscripts is currently unknown. ## discussion it turned out that the task turned was very easy to solve, with winners' models @xmath0. although this suggests that the task of detecting machine-generated content is easy, both work done at elsevier and as reported by the team led by rosati convinces us that we are far from developing a general detector of generated content. each new model (say, gpt-4) for which we don't have training data poses a new challenge, and any detector is likely to fail at identifying content generated with such a model due to a data shift. in summary, the problem is far from being solved: at this point we can not rely on detectors of generated content to support our production systems. however, the dagpap22 shared task did offer a step forward to explore this challenging problem, and we hope to work together with the community on resolving this pernicious issue. domenic anthony rosati. 2022. synscipass: detecting appropriate uses of scientific text generation. in proceedings of the third workshop on scholarly doc-ument processing. association for computational linguistics.