NLP - PINGAN - TECH @ CL - S ci S umm 2020 (2020)

Abstract: We focus on systems for TASK1 (TASK 1A and TASK 1B) of CL-SciSumm Shared Task 2020 in this paper. Task 1A is regarded as a binary classification task of sentence pairs. The strategies of domain-specific embedding and special tokens based on language models are proposed. Fusion of contextualized embedding and extra information is further explored in this article. We leverage Sembert to capture the structured semantic information. The joint of BERT-based model and classifiers without neural networks is also exploited. For the Task 1B, a language model with different weights for classes is fine-tuned to accomplish a multi-label classification task. The results show that extra information can improve the identification of cited text spans. The end-to-end trained models outperform models trained with two stages, and the averaged prediction of multi-models is more accurate than an individual one.

## introduction with the ever-increasing scientific publications, tracking research status from an extremely huge amount of research papers is getting harder for scholars. to solve this problem, research on automatic summarization provides an efficient way to get the highlights of articles for readers. citation-based summarization methods leverage information of citing and cited texts to construct the summary of the reference paper @xcite . citation texts are considered to contain the most valuable parts for researchers to follow. moreover, different citing text spans form a relatively complete figure of an article, such as hypothesis, methods, results, and so on. based on the aforementioned ideas, the cl-scisumm shared tasks propose different tasks about different aspects for scientific publication summarization systems from 2016 to this year @xcite . in more detail, the cl-scisumm 2020 is organized as follows @xcite . a set of reference papers (rp) and the citing papers (cps) that all contain citations to the rp are given. in each cp, the text spans of each particular citation to the rp should be identified. the three subtasks are shown below: task 1a: for each citance, identify the spans of text (cited text spans) in the rp that most accurately reflect the citance. these are of the granularity of a sentence fragment. task 1b: for each cited text span, identify what facet of the paper it belongs to, from a predefined set of facets. task 2 (optional task): finally, generate a structured summary of the rp from the cited text spans of the rp. we focus on systems for task 1 (task 1a and task 1b) in this paper. task 1 are evaluated by sentence overlap scores and rouge-su4 scores @xcite . ## data pre-processing the training sets consist of two datasets: one is 40 manually annotated reference papers and ## submitted runs for task 1a we submit the results of 27 strategies, where each strategy contains two results (top 2 candidates, or top 3 candidates). among all the submitted runs, the ensemble of scibert, scibert-fake-token, scibertspecial-token, and sembert based on scibert, named "scibert_sembert" in