On the Rigour of Scientific Writing: Criteria, Analysis, and Insights (2024)

Abstract: Rigour is crucial for scientific research as it ensures the reproducibility and validity of results and findings. Despite its importance, little work exists on modelling rigour computationally, and there is a lack of analysis on whether these criteria can effectively signal or measure the rigour of scientific papers in practice. In this paper, we introduce a bottom-up, data-driven framework to automatically identify and define rigour criteria and assess their relevance in scientific writing. Our framework includes rigour keyword extraction, detailed rigour definition generation, and salient criteria identification. Furthermore, our framework is domain-agnostic and can be tailored to the evaluation of scientific rigour for different areas, accommodating the distinct salient criteria across fields. We conducted comprehensive experiments based on datasets collected from different domains (e.g. ICLR, ACL) to demonstrate the effectiveness of our framework in modelling rigour. In addition, we analyse linguist patterns of rigour, revealing that framing certainty is crucial for enhancing the perception of scientific rigour, while suggestion certainty and probability uncertainty diminish it.

## introduction rigour is one of the cornerstones of scientific research. despite its profound importance and the widespread use of the term in both scientific and lay parlance, the scientific literature adds surprisingly little to our understanding of rigour, with the term almost always used without a definition, as if its meaning is self-evident. there are few definitions for scientific rigour available. for instance, the national institutes of health (nih) has defined scientific rigour as "the strict application of the scientific method to ensure robust and unbiased experimental design, methodology, analysis, interpretation and reporting of results. this includes full transparency in reporting experimental details so that others may reproduce and extend the findings." @xcite . whilst this definition may seem useful, it has been criticised for being both overly verbose and disconcertingly vague @xcite . inherently, scientific rigour is multi-facetedno single criterion can define it fully. indeed, there are some obvious dimensions of rigour, such as reproducibility, as rigorous scientific practice can enhance the likelihood that the results generated will be reproducible. in computer science (cs), recent years have witnessed an exponential increase in the number of publications (e.g., for ai, machine learning, and nlp), culminating in nearly half a million publications worldwide in 2021 alone @xcite . this surge has resulted in what is termed scientific debt, where researchers prioritise 'novel' methods without sufficiently grounding their work in theory, conducting extensive ablation studies, or performing comprehensive evaluations @xcite . furthermore, this trend has exacerbated the reproducibility crisis in science @xcite , a widespread problem where many scientific studies are difficult or impossible to reproduce by other researchers. the community has begun to address some of the issues surrounding scientific rigour. for instance, in response to calls for more transparent and robust research, acl introduced separate scores for soundness and excitement in 2023 1 , and iclr now includes a breakdown for correctness, technical and empirical novelty, and significance in its review process 2 . while these initiatives are important, there is a significant gap: different research domains may have varying preferences and traditions for defining rigour criteria, and there is a lack of analysis on whether those criteria can effectively signal or measure the rigour of scientific papers in practice. efforts have been made to model different aspects of scientific writing. notably, there is a rich literature on modelling scientific discourse, which aims not merely to present information and ideas but also to ensure their effective communication, allowing readers to accurately perceive what authors intend @xcite . additionally, there are studies focusing on detecting scientific novelty from text @xcite , as novelty is regarded as an important aspect of judging scientific merit. however, compared to novelty, rigour is more challenging to define and analyse, and little work exists on modelling and analysing rigour computationally, particularly in the domain of cs. the limited existing work often focuses on non-cs domains (e.g., biomedical research), employs a top-down approach to defining criteria for scientific rigour (e.g., based on researchers' own experiences), or lacks empirical analysis on the effectiveness of the defined rigour dimensions @xcite . to address the aforementioned gap in modelling scientific rigour, we have developed a bottom-up, data-driven framework that can automatically elicit candidate rigour criteria and construct their detailed definitions. furthermore, our framework is domain-agnostic and can support the analysis of the salience of criteria for signalling the rigour of scientific papers-an important analysis which, to the best of our knowledge, has not been attempted previously. specifically, we (i) construct a highquality corpus of publications that exemplify high rigour, where the corpus is used to train a binary rigour classifier by fine-tuning scibert @xcite . we then extract candidate rigour keywords from two much larger datasets, namely iclr foot_2 and acl anthology foot_3 , based on the predictions of the rigour classifier using feature selection. next, (ii) we generate detailed definitions for the candidate rigour keywords by prompting gpt-4, and (iii) analyse the salience of criteria for signalling the rigour of scientific papers by proposing an llm-based embedding approach. we conducted comprehensive experiments based on datasets collected from two high impact confer-ences for ml and nlp (i.e. iclr and acl) to demonstrate the effectiveness of our framework in modelling rigour, involving analysing the performance of rigour classifier, and assessing the most salient rigour criteria for each experimented dataset quantitatively. we hypothesise that linguistic pattern differences exist between high rigour and less rigour papers, impacting readers' perception of scientific rigour. therefore, we further conducted a sentence-level analysis based on the aspect-level uncertainty theory of @xcite . experimental results reveal that framing certainty is crucial for enhancing the perception of scientific rigour, while suggestion certainty and probability uncertainty diminish it. to summarise, our contributions are three-fold: ## methodology in this section, we describe our data-driven, bottomup framework for eliciting the criterion for defining the rigour of scientific writing, which consists of three main components. an illustration of the framework is shown in fig 1 . ## experimental setup datasets. the ref dataset was constructed based on ref 2021 7 , which covers papers published between 2014 and 2021. the submissions from uoa (unit of assessment) 11 foot_7 , which covers all areas of computer science and information, was used to create the dataset. as described in §3.1, we collected publications from institutions whose outputs are predominantly rated as 4* (e.g., imperial college london and oxford), as well as from institutions whose outputs are predominantly rated as non-4*, to form a binary labelled dataset for rigour. for both the iclr and acl papers, we extracted abstracts and introductions from existing datasets or from arxiv foot_8 . for iclr, we considered all papers from 2022 and 2023 @xcite , to show the effectiveness of our rigour classifier on papers submitted after 2021. the acl dataset was developed using the acl anthology corpus @xcite . given the high formatting consistency within their respective domains, we automatically extracted abstracts and introductions from both datasets using pattern matching with section titles. the statistics of the three datasets are presented in ## conclusion in this paper, we introduce a bottom-up, data-driven framework to automatically identify and define rigour criteria and assess their relevance in scientific writing. our framework is domain-agnostic and can be tailored to the evaluation of scientific rigour for different areas. comprehensive experiments based on datasets demonstrate the effectiveness of our framework in modelling rigour. in addition, we analyse linguistic patterns of rigour, revealing that framing certainty is crucial for enhancing the perception of scientific rigour, while suggestion certainty and probability uncertainty diminish it.