Translation as Cross-Domain Knowledge: Attention Augmentation for Unsupervised Cross-Domain Segmenting and Labeling Tasks (2021)

Abstract: The nature of no word delimiter or inflection that can indicate segment boundaries or word semantics increases the difficulty of Chinese text understanding, and also intensifies the demand for word-level semantic knowledge to accomplish the tagging goal in Chinese segmenting and labeling tasks. However, for unsupervised Chinese cross-domain segmenting and labeling tasks, the model trained on the source domain frequently suffers from the deficient word-level semantic knowledge of the target domain. To address this issue, we propose a novel paradigm based on attention augmentation to introduce crucial cross-domain knowledge via a translation system. The proposed paradigm enables the model attention to draw cross-domain knowledge indicated by the implicit word-level cross-lingual alignment between the input and its corresponding translation. Aside from the model requiring cross-lingual input, we also establish an off-the-shelf model which eludes the dependency on cross-lingual translations. Experiments demonstrate that our proposal significantly advances the state-of-the-art results of cross-domain Chinese segmenting and labeling tasks.

## introduction as a language that has no word delimiter or inflection indicating segment boundaries or word semantics, chinese increases the difficulty of segmenting and labeling tasks which need to correctly identify the segment boundaries from a sequence of characters and to thoroughly understand the word meanings. in this paper, we regard the knowledge about segment boundaries and detailed word meanings as the word-level semantic knowledge, and intend to promote the chinese cross-domain segmenting and labeling tasks where the paucity of such knowledge chinese word segmentation pos tagging regard learning as a career pursuit figure 1 : two typical examples that cross-lingual contexts derived from machine translation help improve segmenting and labeling tasks. the arrows highlight the cross-lingual word that each chinese character pays most attention to. regarding the first case, the integrity of the out-of-domain word 制霉菌素 is indicated by its translated english word. in the second case, the translation of the ambiguous word 追求 is pursuit instead of pursue, which indicates the correct tag noun. in the target domain is usually a severe problem and frequently undermines the performance of a cross-domain model. existing work achieves cross-domain segmenting and labeling either by crafting domain-specific knowledge @xcite , which is inflexible and cumbersome in adapting to different domains, or by learning on unannotated data of target domain @xcite via language modeling, which cannot comprehend the detailed word-level semantics apart from grasping the general meaning of a sentence. to address these issues, the key is to search for cross-domain knowledge that is easily accessible yet contains crucial word-level semantic knowledge. motivated by the previous studies that segmenting and labeling tasks can benefit machine translation @xcite @xcite , in turn, we speculate that the machine-translated sentences would conceivably reveal some fundamental segmenting knowledge and help infer detailed word-level semantics. the cross-domain knowledge inferred from translations can be illustrated by the two examples in figure 1 . for chinese word segmentation, the word in blue from the target domain is originally excessively segmented. however, a translated version of the input sentence provides a strong indication of the integrity of this word. the part-of-speech tag of the word 追求 is originally ambiguous since it can be both a noun (as in the given context) and a modal verb. nonetheless, it corresponds to distinct english words depending on the meaning it presents as a noun or as a verb, thus its translated counterpart actually hints at the correct label. motivated by the above observations, we propose a novel paradigm based on attention augmentation to introduce word-level cross-domain knowledge via cross-lingual translation. the proposed paradigm complements the input sentence with its cross-lingual translation and enables the model attention to draw word-level knowledge implicitly embodied in the alignment of the input sentence pair. it then incorporates cross-lingual masked language modeling to further strengthen the word-level alignment, evolving into the masked attention augmentation. the enhanced alignment, in turn, helps boost segmenting and labeling tasks. to make our proposal more practical, we use this model to tag the raw text in target domains to reap abundant synthetic data, which aims to elicit the originally implicit crossdomain knowledge implied by the word-level alignment, and then use the synthetic data to train an off-the-shelf model relying on no translation inputs. experimental results on a series of cross-domain segmenting and labeling datasets demonstrate that our model substantially advances the state-of-theart results. our contributions are highlighted as follows: ## related work some previous work improves domain adaption in a single task framework. domain-specific lexicon @xcite is usually adopted for cross-domain tasks. however, highquality dictionaries for target domains are not always available. recent work @xcite uses raw text in the target domain to train word embeddings or construct word collections @xcite . to relieve the burdensome work in crafting domainspecific knowledge, some researches attempt to align different domains, including mapping entity label space @xcite , sharing hidden feature representations @xcite , aligning feature distributions @xcite and using adaptation layers (lin and lu, 2018). however, it is difficult to align different domains in an unsupervised way. increasing work turns to multi-task learning since related nlp tasks can boost each other in a joint-learning framework. language modeling (lm) is a common auxiliary objective that has been shown to be beneficial for sequence tagging @xcite . a natural idea is to learn contextualized embeddings by masked language modeling on the text from target domain @xcite . @xcite consider unsupervised domain adaptation for named entity recognition (ner) via cross-domain language modeling tasks. @xcite propose to incorporate unlabeled data for chinese word segmentation (cws) by combining segmentation model with language models. in addition to language modeling, some work learns to jointly optimize syntactic parsing and semantic parsing objectives @xcite . @xcite propose to use character clustering and self-training to jointly train cws and pos tagging task. @xcite use a two-way attention mechanism to incorporate both context feature and their corresponding auto-analyzed syntactic knowledge for each input character and trains cws and pos tagging tasks jointly. some extensions turn to a multilingual setting and they leverage nmt systems to help cross-lingual ner for low-resource languages @xcite or jointly model bilingual ## approach in this section, we introduce our approach in detail. we suppose that an annotated source domain dataset @xmath0, y i )} ## experiments we conduct experiments on three chinese crossdomain segmenting and labeling tasks, namely partof-speech tagging (pos tagging), chinese word segmentation (cws) and named entity recognition (ner). as most studies do, we conduct crossdomain experiments on datasets from different domains. ## conclusions we propose a novel paradigm of attention augmentation that supplements cross-domain wordlevel knowledge via machine translation for chinese cross-domain segmenting and labeling tasks. we also construct a model unchained from the dependency on translation. the proposed approach substantially advances the state-of-the-art results in chinese cross-domain segmenting and labeling tasks without any human-annotated data, demonstrating the effectiveness of our proposal.