Sequential Span Classification with Neural Semi- M arkov CRF s for Biomedical Abstracts (2020)

Abstract: Dividing biomedical abstracts into several segments with rhetorical roles is essential for supporting researchers’ information access in the biomedical domain. Conventional methods have regarded the task as a sequence labeling task based on sequential sentence classification, i.e., they assign a rhetorical label to each sentence by considering the context in the abstract. However, these methods have a critical problem: they are prone to mislabel longer continuous sentences with the same rhetorical label. To tackle the problem, we propose sequential span classification that assigns a rhetorical label, not to a single sentence but to a span that consists of continuous sentences. Accordingly, we introduce Neural Semi-Markov Conditional Random Fields to assign the labels to such spans by considering all possible spans of various lengths. Experimental results obtained from PubMed 20k RCT and NICTA-PIBOSO datasets demonstrate that our proposed method achieved the best micro sentence-F1 score as well as the best micro span-F1 score.

## introduction dividing documents into several rhetorical segments is a fundamental task in natural language processing (nlp). for example, abstracts in pubmed, a database of the biomedical literature, can be divided into rhetorical segments such as "objective", "methods", "results", and "conclusions". abstracts segmented for each rhetorical role allows us to exploit advanced search. that is, researchers can easily find information by utilizing the structured queries such as "find abstracts that contain in 'methods'". furthermore, the technique can also be used for nlp applications such as academic writing support @xcite , scientific trend analysis @xcite , and question-answering @xcite . most previous methods in pubmed have regarded the task as a sequence labeling, namely sequential sentence classification, that assigns rhetorical labels with a b(egin)/i(nside) tag set to each sentence while considering the context in the abstract. to this end, some statistical methods with hand-engineered features have been proposed, including hidden markov models (hmms) @xcite and conditional random fields (crfs) @xcite @xcite . recently, with the success of neural network models for nlp tasks, @xcite and @xcite have employed bilstms to obtain sentence embeddings based on word embeddings and crfs for assigning labels to the sentences. @xcite employed a pre-trained language model, scibert @xcite , which is a variant of bert @xcite trained with scientific papers, to improve the performance of classification without crfs. previous methods cast the segmentation with the labeling as a sentence classification. however, such methods have a critical problem: their performances on longer spans foot_0 is not so good since they are designed to maximize the prediction of rhetorical roles for a small context. to tackle the problem, we propose a novel approach, neural sequential span classification, that directly gives the labels for the spans while considering all possible spans of various lengths in the abstract. that is, our method is designed to maximize the performance of classification at the span level rather than the sentence level. consequently, we introduce neural semi-markov conditional random fields (scrfs) @xcite f 4 < l a t e x i t s h a 1 _ b a s e 6 4 = " n 1 s p c t c 8 f + n z n 8 i k 0 u s / o m 1 v n 9 @xmath0 figure 1 : overview of the neural sequential span classification. in the example, five rhetorical labels, "background" (b), "objective" (o), "methods" (m), "results" (r), and "conclusions" (c) can be assigned to spans. the abstract that consists of five sentences is segmented into four spans, span(1,1), span(2,2), span(3,4), and span(5,5), and these spans are labeled as b, m, r, and c, respectively. 2019 ) to handle spans of the different lengths. to demonstrate the effectiveness of method, we conducted experimental evaluations on two benchmark datasets, pubmed 20k rct @xcite and nicta-piboso @xcite . the results show that our method achieved the best micro sentence-f 1 score of 93.1 and micro span-f 1 score of 84.3 in the pubmed 20k rct dataset, and the best micro sentence-f 1 score of 84.4 and micro span-f 1 score of 58.7 in the nicta-piboso dataset. ## proposed method to perform sequential span classification in an endto-end manner, we need to represent spans as vectors and handle all possible sequences with various lengths in the abstract. to this end, we introduce bilstms and semi-markov crfs (scrfs). figure 1 shows an overview of our method. the bil-stms layer generates span vectors from sentence vectors incorporating the context in the abstract, and the scrfs layer learns the labeling of span sequences by considering all possible sequences of various lengths. the details are described below. ## conclusions in this paper, we proposed the neural sequential span classification that directly assigns rhetorical labels to each span in a biomedical abstract by considering all possible spans of various lengths. to perform this classification technique, we introduced neural semi-markov crfs. evaluation results obtained from pubmed 20k rct and nicta-piboso datasets show that our method outperformed state-of-the-art sequential sentence classification methods. in other words, our method achieved the best scores for both micro sentenceand span-f 1 scores. in particular, we found a remarkable improvement in the span-f 1 score. furthermore, the classification accuracy for long spans, that is, rhetorical labels containing a larger num sentence gold base prop. compare the effect of financial incentives on response to a cancer survivors' postal questionnaire. o o o prostate cancer survivors in ireland, 1.5-18 years after diagnosis, were randomized to the (1) "lottery" arm [a 1 lottery scratch card sent with the questionnaire (@xmath1,413)] or (2) "prize" arm [entry into a draw on return of a completed questionnaire (@xmath2,407)].