‘Don’t Get Too Technical with Me’: A Discourse Structure-Based Framework for Automatic Science Journalism (2023)

Abstract: Science journalism refers to the task of reporting technical findings of a scientific paper as a less technical news article to the general public audience. We aim to design an automated system to support this real-world task (i.e., automatic science journalism ) by 1) introducing a newly-constructed and real-world dataset (SciTechNews), with tuples of a publicly-available scientific paper, its corresponding news article, and an expert-written short summary snippet; 2) proposing a novel technical framework that integrates a paper’s discourse structure with its metadata to guide generation; and, 3) demonstrating with extensive automatic and human experiments that our model outperforms other baseline methods (e.g. Alpaca and ChatGPT) in elaborating a content plan meaningful for the target audience, simplify the information selected, and produce a coherent final report in a layman’s style.

## introduction science journalism refers to producing journalistic content that covers topics related to different areas of scientific research @xcite . it plays an important role in fostering public understanding of science and its impact. however, the sheer volume of scientific literature makes it challenging for journalists to report on every significant discovery, potentially leaving many overlooked. for instance, in the year 2022 alone, 185, 692 papers were submitted to the preprint repository arxiv.org spanning highly diverse scientific domains such as biomedical research, social and political sciences, engineering research and a multitude of others foot_0 . to this date, pubmed contains around 345, 332 scientific publications about the novel coronavirus covid-19 2 , nearly 1.6 times as many as those produced in 200 years of work on influenza. the enormous quantity of scientific literature and the huge amount of manual effort required to produce high-quality science journalistic content inspired recent interest in tasks such as generating blog titles or slides for scientific papers @xcite , extracting structured knowledge from scientific literature @xcite @xcite , simplifying technical health manuals for the general public @xcite , and creating plain language summaries for scientific literature @xcite . our work focuses on generating simplified journalistic summaries of scientific papers for the non-technical general audience. to achieve this goal, we introduce a new dataset, scitechnews, which pairs full scientific papers with their corresponding press release articles and newswire snippets as published in acm technews. we further carry out in-depth analysis to understand the journalists' summarization strategies from different dimensions (section 3.2). then, we explore novel model designs to generate short journalistic summaries for scientific papers. unlike previous studies that model this problem as a "flat" sequenceto-sequence task and ignore crucial metadata information of scientific papers @xcite , we propose a technical framework that integrates author and affiliation data as they are important information in scientific news articles. furthermore, we encode each sentence with its corresponding discourse rhetoric role (e.g., background or methods) and apply a hierarchical decoding strategy to generate summaries. as illustrated in figure 1 , our trained decoding model first generates a content plan, which is then employed to guide the model in producing summaries that adhere to the plan's structure. in summary, our main contributions are twofold. first, we construct a new open-access highquality dataset for automatic science journalism ## related work existing datasets. there are a few datasets for generating journalistic summaries for scientific papers. @xcite created the science daily dataset, which contains around 100k pairs of full-text scientific papers and their corresponding science daily press releases. however, this dataset is not publicly available due to the legal and licensing restrictions. recently, @xcite constructed two open-access lay summarisation datasets from two academic journals (plos and elife) in the biomedical domain. the datasets contain around 31k biomedical journal articles alongside expert-written lay summaries. our dataset scitechnews is a valuable addition to the existing datasets, with coverage of diverse domains, including computer science, machine learning, physics, and engineering. automatic science journalism. @xcite developed a pointer-generator network to generate blog titles from the scientific titles and their corresponding abstracts. @xcite built a manually annotated dataset for expertise style transfer in the medical domain and applied multiple style transfer and sentence simplification models to transform expert-level sentences into layman's language. the works most closely related to ours are @xcite and @xcite . both studies employed standard seqto-seq models to generate news summaries for scientific articles. in our work, we propose a novel framework that integrates metadata information of scientific papers and scientific discourse structure to learn journalists' writing strategies. ## the scitechnews dataset in this section, we introduce scitechnews, a new dataset for science journalism that consists of scientific papers paired with their corresponding press release snippets mined from acm technews. we elaborate on how the dataset was collected and curated and analyze how it differs from other lay summarization benchmarks, both qualitatively and quantitatively. ## problem formulation and modelling we cast the problem of science journalism as an encoder-decoder generative task and propose a model that performs content planning and style transferring while summarizing the content. given a scientific article text d, enriched with metadata information m , the task proceeds in two steps. first, a plan s is generated conditioned on the input document, p(s|d, m ), followed by the summary y, conditioned on both the document and the plan, p(y|s, d, m ). following @xcite , we train an encoder-decoder model that encodes an input document and learns to generate the concatenated plan and summary as a single sequence. let @xmath0, ..., x n be a scientific article text, modeled as a sequence of sentences, let m be the set of author-affiliation pairs associated with the said article, and let y be the target summary. we define @xmath1, m 0 , .., m |m | , t 0 , x 0 , .., t n , x n as the input to the encoder, where m is a special token indicating the beginning of metadata information, m @xmath2, and t j is a label indicating the scientific rhetorical role of sentence x i . given the encoder states, the decoder proceeds to generate plan s conditioned on d , p(s|d ; θ), where θ are the model parameters. the plan is defined as @xmath3, ..., s |y| where s k is a label indicating the rhetorical role sentence y k in summary y should cover. figure 1 shows an example of the annotated document and content plan. we use a bart encoder-decoder architecture @xcite and train it with d as the source and [s; y] (plan and summary concatenated) as the target. we call this model bart plan . ## experimental setup in this section, we elaborate on the baselines used and evaluation methods, both using automatic metrics and eliciting human judgments. following previous work @xcite , we use the abstract followed by the introduction as the article body and prepend the metadata information as described in the previous section. ## results and discussion in this section, we present and discuss the results of our automatic and human evaluations, provide a comprehensive analysis of the factuality errors our systems incur, and finish with a demonstration of controlled generation with custom user plans. ## conclusions this paper presents a novel dataset, scitech-news, for automatic science journalism. we also propose a novel approach that learns journalistic writing strategy and style by leveraging the paper's discourse structures. through extensive automatic evaluation and human evaluation with baseline methods (e.g., chatgpt and alpaca), we find that our models can generate high-quality informative news summaries that closely resemble those crafted by professional journalists.