More than Text: Multi-modal C hinese Word Segmentation (2021)

Abstract: Chinese word segmentation (CWS) is undoubtedly an important basic task in natural language processing. Previous works only focus on the textual modality, but there are often audio and video utterances (such as news broadcast and face-to-face dialogues), where textual, acoustic and visual modalities normally exist. To this end, we attempt to combine the multi-modality (mainly the converted text and actual voice information) to perform CWS. In this paper, we annotate a new dataset for CWS containing text and audio. Moreover, we propose a time-dependent multi-modal interactive model based on Transformer framework to integrate multi-modal information for word sequence labeling. The experimental results on three different training sets show the effectiveness of our approach with fusing text and audio.

## introduction word segmentation is a fundamental task in natural language processing (nlp) for those languages without word delimiters, e.g., chinese and many other east asian languages @xcite . in this paper, we mainly take chinese language as investigating object, namely cws. as we know, cws has been applied as an essential preprocessing step for many other nlp tasks @xcite , such as named entity recognition, sentiment analysis, machine translation, etc. in the literature, some popular approaches to cws systems report a high performance at the level of 96%-98%, and these systems typically require a large scale of pre-segmented textual dataset for training. however, the collection of a specific scenario on such large scale is very time-consuming and resource-intensive, such as video monologues and audio broadcast. in these scenarios, there are multiple modalities: text, audio and vision, thus if only using the text seems not a good choice. for example, as shown in figure 1 , if we only read the text"必须不忘初心牢记使命" with no punctuation, it is not easy to make word segmentation immediately. however, if there is the acoustic information, we can observe the obvious stop in spectrum and sonic wave at the middle of "心" and "牢", which provides the facility for cws. therefore, in this paper, we propose to performing cws with multi-modality, namely mcws, by a time-dependent multi-modal interactive network. specifically, we first collect a new dataset from an audio and video news broadcast platform and annotate the word boundaries of audio transcription text. second, we make the text and the audio align as the time stamp of each character, then encode both modalities foot_0 by transformer-based framework to capture the intra-modal dynamics. third, we design a time-dependent multi-modal interaction module for each character step to generate the multi-modal hybrid character representation. finally, we leverage the crf to perform sequence labeling on the basis of the above character representation. we evaluate our approach on the newly annotated small-scale dataset with different size of training sets. the experimental results demonstrate that our approach performs significantly better than the single-modal state-of-the-art and the multi-modal approaches with early fused features of cws. recently, deep neural approaches have been widely proposed to minimize the efforts in feature engineering for cws @xcite @xcite @xcite @xcite @xcite @xcite @xcite . among these studies, most of them follow the character-based paradigm to predict segmentation labels for each character in an input sentence. to enhance cws with neural models, there were studies leverage external information, such as vocabularies from auto-segmented external corpus and weakly labeled data @xcite @xcite . to our best knowledge, we are first to perform cws with multi-modality, which can deal with multi-modal scenarios and offers an alternative solution to robustly enhancing neural cws models. ## data collection and annotation we collect the multi-modal data for cws from a chinese news reporting platform "xuexi" 2 . we mainly focus on the audios equipped with machine transcription text. in total, we crawl 120 short videos and segment them into about 2000 sentences. to avoid the contextual influence and augment the robust of designed computing model, we randomly select 250 sentences to annotate the word boundaries, and the remaining data are used to perform semi-supervised or unsupervised learning in the future. we annotate these chinese audio transcriptions following the ctb word segmentation guidelines by @xcite . two annotators are asked to annotate the data. due to the clear annotation guideline, the annotation agreement is very high, reaching 98.3%. the disagreement instances are judged by an expert. the statistics of our annotated data are summarized in ## time-dependent multi-modal interactive network for cws in this section, we introduce our proposed multimodal approach for cws, namely time-dependent multi-modal interactive network (tmin), which can capture the interactive semantics between text and audio for better word segmentation. this approach mainly consists of three modules: timedependent uni-modal interaction, time-dependent multi-modal interaction and crf labeling. figure 2 shows the overall architecture of our tmin. ## experimentation in this section, we provide the exploratory experimental results and a case analysis. ## conclusion and future work this paper proposes a new dataset for multi-modal chinese word segmentation (mcws), which is the first attempt to explore the multi-modality for traditional cws. meanwhile, we propose a time-dependent multi-modal interactive network (tmin) to effectively integrate textual and acoustic features. the preliminary experimental results and case analysis demonstrate the reliability of our motivation and the effectiveness of the proposed approach. in the future, we will annotate more samples at the current setting, and collect new samples with more modalities, such as visual information in social media, monologues and dialogues with continuous front face. moreover, we will employ the neural active learning approaches for mcws to reduce the annotation and achieve the best performance.