Sentence Boundary Detection on Line Breaks in J apanese (2020)

Abstract: For NLP, sentence boundary detection (SBD) is an essential task to decompose a text into sentences. Most of the previous studies have used a simple rule that uses only typical characters as sentence boundaries. However, some characters may or may not be sentence boundaries depending on the context. We focused on line breaks in them. We newly constructed annotated corpora, implemented sentence boundary detectors, and analyzed performance of SBD in several settings.

## introduction many nlp tasks treat a sentence as a unit of processing. the task for decomposing a text into sentences is called sentence boundary detection @xcite . in japanese, periods (e.g. "。", "."), exclamation marks, and question marks are delimiters to segment sentences in most cases. for this reason, the sbd in most studies takes only the positions of these typical delimiters as sentence boundaries. for example, in the construction of the "web japanese n-gram database 1 " provided by google, inc., they extracted sentences by segmenting on their positions. however, line breaks can also indicate sentence boundaries without periods as the following text 2 . オアシズの大久保さんが最近気になり ます⏎</@xmath0. okubo of "oasiz" has been on my mind lately ⏎ </@xmath1 there can be more sentence boundaries than these. @xcite showed that there are more than six variations of japanese sentence boundaries in an online forum: description of actions (e.g. "(照)": embarrassment, "(涙)": tears), "smiley" icons (e.g. "( * ˆ∇ˆ*)", " §ˆ｡ˆ §"), and so on. @xcite conducted a linguistic analysis of japanese emails written by young people on their mobile phones and found that about 63% of the emails used emoticons instead of punctuation marks for sentence boundaries. in this paper, we focus on sbd on line breaks in japanese. we newly construct annotated corpora to answer the following three research questions: stanza foot_0 @xcite ) is a language-agnostic fully neural pipeline for text analysis, including tokenization, multiword token expansion, lemmatization, part-of-speech and morphological feature tagging, dependency parsing, and named entity recognition. unlike most existing toolkits, it does tokenization and sbd at the same time by using a bidirectional long short-term memory network @xcite (bi-lstm) for characters in texts. it provides models for 66 languages including japanese. the japanese model is trained with with ud japanese gsd foot_1 . its architecture enables sbd on any characters, including line breaks. however the training corpus does not contain line breaks. therefore the model can not perform sbd on line breaks. 3 corpus preparation 3.1 bccwj the balanced corpus of contemporary written japanese (bccwj) is a corpus annotated with sen-corpus documents sentences lbs lbs w/o sb bccwj 2,918 44,760 23,099 1,702 (pn) 340 8,747 3,069 0 (pb) 83 8,956 3,290 0 (pm) 86 9,424 3,890 0 (ow) 62 3,751 2,223 0 (oc) * 1,876 6,413 4,055 818 (oy) 471 7,469 6,572 884 jalan-f 500 3,290 1,484 170 jalan-a 298 ? 1,193 153 ## conclusion we implemented sentence boundary detectors by using bert and revealed the following facts: there are two main issues that we need to address in the future. the first issue is to do is to use active learning to increase the number of learning examples and improve accuracy. the second issue is to perform sbd for other atypical sentence boundary expressions other than line breaks.