Determining the Credibility of Science Communication (2021)

Abstract: Most work on scholarly document processing assumes that the information processed is trust-worthy and factually correct. However, this is not always the case. There are two core challenges, which should be addressed: 1) ensuring that scientific publications are credible – e.g. that claims are not made without supporting evidence, and that all relevant supporting evidence is provided; and 2) that scientific findings are not misrepresented, distorted or outright misreported when communicated by journalists or the general public. I will present some first steps towards addressing these problems and outline remaining challenges.

## the life cycle of scientific research scientific research is highly diverse not just when it comes to the topic of study, but also how studies are conducted, how the resulting research is described and when and where it is published. however, what different fields still have in common is a certain life cycle, starting with planning a study and ending with promoting the research post-publication, in the hopes of the article finding readership and having an impact. scholary document processing aims to support researchers throughout this life cycle of scientific research, by offering various tools to automate otherwise manual processes. most research within scholarly document processing has focused on supporting information discovery for finding related work. most prominently, research has focused on methods to condense scientific documents, using entity extraction and linking, keyphrase or relation extraction @xcite @xcite @xcite or automatic summarisation @xcite . once papers are written and submitted for peer review, it is pertinent to evaluate them fairly and objectively. this process is far from straight-forward, as, among others, reviewers have certain biases, including against truly novel research @xcite . research has thus focused on automatically generating peer reviews from paper content @xcite , as well as on studying how well review scores can be predicted from review texts @xcite . finally, post-publication, the impact of scientific work can be tracked, using citations and citation counts as a proxy for this. it is again worth noting that there are significant biases in this -e.g. author information is among the, if not the most salient feature for predicting citation counts @xcite . looking further into what papers are cited and why, mohammad (2020b,a) find that there are significant topical as well as gender biases when it comes to who is cited and by whom. ## credibility and veracity of science communication while all of the work referenced above is important in supporting researchers, it neglects one crucial aspect, namely that it assumes the resulting scientific documents and broader communication about them are credible and supported by the underlying evidence. though it is the task of peer reviewers to spot issues regarding credibility, and the task of journalists to check their sources when they report on scientific studies, distortions, exaggerations and outright misrepresentations can still happen. the ongoing covid-19 pandemic has highlighted the disastrous and direct consequences misreporting of scientific findings can have on our everyday lives, yet, there is still relatively little work on detecting issues in the credibility of scientific writing. this especially holds for detecting smaller nuances of untrustworthy scientific writing, whereas there is comparatively more work on de-biology wood frogs (rana sylvatica) are a charismatic species of frog common in much of north america. they breed in explosive choruses over a few nights in late winter to early spring. the incidence in wood frogs was associated with a die-off of frogs during the breeding chorus in the sylamore district of the ozark national forest in arkansas @xcite . ## conclusion this paper discusses research avenues for automatically determining the credibility of science communication, both in terms of scientific papers and popular science communication. these avenues are put in the context of scholarly data processing more broadly, and how different tasks can be used to assist the life cycle of scientific research. while existing research has focused on developing models for assisting with information discovery, peer review and citation tracking, comparatively little work has been done on identifying non-credible claims and assisting authors in making sure their research is backed up by sufficient evidence where needed. the suggestion is therefore to focus on two tasks: cite-worthiness detection, to identify sentences requiring citations; and exaggeration detection, to identify cases in which scientific findings have been overstated. a core problem for both tasks is the lack of appropriate training data, which we address by introducing a new dataset, and a semi-supervised learning method, respectively. we hope our research will inspire future work on developing tools to assist authors and journalists in ensuring that research is described in a credible and evidence-based way.