Detecting Scientific Fraud Using Argument Mining (2024)

Abstract: A proliferation of fraudulent scientific research in recent years has precipitated a greater interest in more effective methods of detection. There are many varieties of academic fraud, but a particularly challenging type to detect is the use of paper mills and the faking of peer-review. To the best of our knowledge, there have so far been no attempts to automate this process.The complexity of this issue precludes the use of heuristic methods, like pattern-matching techniques, which are employed for other types of fraud. Our proposed method in this paper uses techniques from the Computational Argumentation literature (i.e. argument mining and argument quality evaluation). Our central hypothesis stems from the assumption that articles that have not been subject to the proper level of scrutiny will contain poorly formed and reasoned arguments, relative to legitimately published papers. We use a variety of corpora to test this approach, including a collection of abstracts taken from retracted papers. We show significant improvement compared to a number of baselines, suggesting that this approach merits further investigation.

## introduction the growing problem of fraudulent academic research poses a threat to scientific progress. research is an iterative process, where arguments presented in previous papers are used as the basis of subsequent work. researchers do not necessarily have the time or resources available to verify that all the claims that have been made in previous publications are well-formulated, or valid. instead they tend to rely on the scrutiny imposed during the publication process to offer reasonable guarantees about the legitimacy of the content. however, these guarantees have been undermined by revelations about the extent of malpractice taking place in many scientific publications @xcite . as an indication of the scale of the problem, there have been estimates that over a fifth of new medical publications are implicated in some form of fraudulent activity @xcite . prevalent types of fraud include: plagiarism, data manipulation and circumvention of a legitimate peer-review process. relatively simple methods can help detect some of these forms of fraud, such as pattern matching in the case of plagiarism @xcite . however, where the peer-review process is compromised, or the services of paper mills are employed, these techniques are not effective. in these cases, researchers have had some success detecting characteristic patterns (van noorden, 2021; else and van noorden, 2021) -but this is not a universal panacea as it is simple for authors to make slight stylistic alterations to avoid these methods of detection. the consensus of the research community investigating these matters is that there are a large number of such articles that have not yet been retracted from the journals they are published in, and this number is on the rise @xcite . the hypothesis underlying this research is that papers published by illicit means (specifically those that have been produced by paper mills, or have not undergone peer-review) will be based on substandard reasoning. this could take the form of fallacious arguments. alternatively, arguments may be based on spurious premises, or lack any relevant and novel insights. we believe this is a legitimate supposition due to the nature of scientific inquiry: the fundamental aim of scientific research is to construct valid and interesting arguments from a sound empirical or theoretical basis. anecdotal evidence supporting this hypothesis is presented in after three covid-19 waves, the growing number of new infections still reminds us of the importance of taking precautionary measures. sd and wearing masks have been proven to be efficient nonpharmaceutical intervention measures @xcite . they are low-cost, convenient, and noninvasive to slow the spread of covid-19 and flatten the curves of infection (srivastava, @xcite . in order to empirically test this hypothesis, we adopt techniques from the computational argumentation literature. specifically, we build on past research in the fields of argument mining @xcite ) and argument quality evaluation @xcite . models trained for these purposes are able to extract arguments from a passage of text and evaluate the quality of such arguments respectively. when done sequentially, this amounts to a way of assessing the reasoning present in a piece of text. we use a number of pre-existing corpora both to train and test the various models we develop. these include datasets that have been compiled especially for scientific argument mining: sciark @xcite for training and abstrct @xcite for testing. also, we use a dataset that contains human evaluated arguments for training our argument quality evaluation model: the grammarly argument quality corpus (gaqcorpus) @xcite . in order to collect a sufficient sample of retracted articles to evaluate the performance of the complete system, we use the retraction watch database @xcite . the metadata included therein allows us to specify the subset of retracted articles that we are interested in detecting. our initial results indicate that implementing this strategy leads to a considerable improvement in detecting fraudulent articles, compared with a number of baselines. this suggests that the developed method has theoretical validity and merits further investigation. ## methodology the goal of this research is to test the hypothesis that evaluating the quality of a scientific article's reasoning can be an effective way of determining whether it was produced fraudulently. in practice, this amounts to extracting and then evaluating arguments that are indicative of the overall level of reasoning present in the article. we achieve this by training two models separately. an important feature of our framework is that we only analyse the abstracts of papers. we have two primary motivations for this decision. firstly, it provides an effective way to minimise artefacts in our data. research papers are generally heterogeneous in format, containing inconsistently structured sections. abstracts, however, are fairly consistent in form, and from an argumentative perspective, usually contain the principle argument motivating the rest of the paper. the second reason is that if we are able to demonstrate that the abstract alone is sufficient, then this offers practical advantages. only considering a minimal subset of the entire text reduces both the theoretical and computational burden. this latter consideration is especially pertinent when considering that deploying such a system would be most advantageous in situations which require processing high volumes of inputs. we describe our proposed fraud prediction framework with the following formalisation. let @xmath0, s 2 , ..., s n } represent a paper abstract consisting of a list of n sentences. we define a model m 1 such that: the list of categorisations @xmath1, c 2 , ..., c n } along with the original abstract a is then transformed using a transformation function ϕ: the transformed abstract is then input to another model m 2 : the final status o of the abstract is determined by comparing the output v to a threshold @xmath2: both models in our system architecture (see figure 1 ), are realised based on fine-tuned foundation models @xcite , using data from a number of different sources. in section 3.1 we describe the data we use for fine-tuning and testing our individual models, as well as testing our system in its entirety. sections 3.2 and 3.3 provide descriptions of the argument mining (am) model (m 1 ) and argument quality evaluation (aqe) model (m 2 ) respectively. finally in section 3.4, we outline the entire system as presented in figure 1 . ## results and discussion in section 4.1 we present a comparison of our novel am model with two existing alternatives. then, we outline the performance on the overall fraud prediction task of three novel baselines (section 4.2) and the full argumentation-based model (section 4.3). ## conclusion and future work the aim of this study was to determine whether it is possible to automate the detection of fraudulently produced scientific publications. to achieve this, we introduced and implemented a novel framework, building upon existing architectures from the argument mining literature. we developed state-of-the-art methods in the fields of scientific argument mining and argument quality evaluation. by using both models in conjunction, we created a method for evaluating the quality of reasoning in scientific articles. we compared this approach to three baselines, observing favourable comparisons in each case. by using a variety of baselines we were able to demonstrate that both the argument mining and quality evaluation components positively contributed to the overall performance of the system. although the initial results were promising, there are a number of potential developments that would merit further exploration. for instance, the quality evaluation component would benefit from being trained on arguments that have been taken directly from scientific papers. furthermore, introducing methods to assess the quality of the individual premises and claims, in addition to the overall argument, could benefit performance. as mentioned previously, there are various advantages to restricting our inputs to abstracts. however, analysing entire articles also has benefits.