D om BERT : Domain-oriented Language Model for Aspect-based Sentiment Analysis (2020)

Abstract: This paper focuses on learning domain-oriented language models driven by end tasks, which aims to combine the worlds of both general-purpose language models (such as ELMo and BERT) and domain-specific language understanding. We propose DomBERT, an extension of BERT to learn from both in-domain corpus and relevant domain corpora. This helps in learning domain language models with low-resources. Experiments are conducted on an assortment of tasks in aspect-based sentiment analysis (ABSA), demonstrating promising results.

## introduction pre-trained language models (lms) @xcite @xcite aim to learn general (or mixed-domain) knowledge for end tasks. recent studies @xcite show that learning domain-specific lms are equally important. this is because the training corpus of general lms is out-of-domain for end tasks in a particular domain and, more importantly, because general lms may not capture the long-tailed and underrepresented domain details @xcite ). an intuitive example related to corpus of aspect-based sentiment analysis (absa) can be found in in contrast, although domain-specific lms can capture fine-grained domain details, they may suffer from insufficient training corpus @xcite to strengthen general knowledge within a domain. to this end, we propose a domain-example domain the [mask] is clear . the sky is clear . astronomy [irrelevant domain] the water is clear . liquids [irrelevant domain] the idea is clear . concepts [irrelevant domain] the screen is clear . desktop [relevant domain] the picture is clear . laptop @xcite this paper proposes an extremely simple extension of bert @xcite called dombert to learn domain-oriented language models. dombert simultaneously learns masked language modeling and discovers relevant domains (with a built-in retrieval model @xcite ) to draw training examples, where the later are computed from domain embeddings learned from an auxiliary task of domain classification. we apply dombert to end tasks in aspect-based sentiment analysis (absa) in low-resource settings, demonstrating promising results. related work pre-trained language models gain significant improvements over a wide spectrum of nlp tasks @xcite , including @xcite , gpt/gpt2 @xcite , bert @xcite , xl-net @xcite , roberta @xcite , albert @xcite , electra @xcite . this paper extends bert's masked language model (mlm) with domain knowledge learning. following roberta, the proposed dombert leverages dynamic masking, removes the next sentence prediction (nsp) task (which is proved to have negative effects on pre-trained parameters), and allows for max-length mlm to fully utilize the computational power. this paper also borrows albert's removal of dropout since pre-trained lm, in general, is an underfitting task that requires more parameters instead of avoiding overfitting. the proposed domain-oriented learning task can be viewed as one type of transfer learning @xcite , which learns a transfer strategy implicitly that transfer training examples from relevant (source) domains to the target domain. this transfer process is conducted throughout the training process of dombert. the experiment of this paper focuses on aspectbased sentiment analysis (absa), which typically requires a lot of domain-specific knowledge. reviews serve as a rich resource for sentiment analysis @xcite @xcite . absa aims to turn unstructured reviews into structured fine-grained aspects (such as the "battery" or aspect category of a laptop) and their associated opinions (e.g., "good battery" is positive about the aspect battery). this paper focuses on three (3) popular tasks in absa: aspect extraction (ae) @xcite , aspect sentiment classification (asc) @xcite @xcite @xcite @xcite @xcite @xcite and end-to-end absa (e2e-absa) @xcite . ae aims to extract aspects (e.g., "battery"), asc identifies the polarity for a given aspect (e.g., positive for battery) and e2e-absa is a combination of ae and asc that detects the aspects and their associated polarities simultaneously. this paper focuses on self-supervised methods foot_1 to improve absa. ## dombert this section presents dombert, which is an extension of bert for domain knowledge learning. the goal of dombert is to discover relevant domains from the pool of source domains and uses the training examples from relevant source domains for masked language model learning. as a result, dombert has a sampling process over a categorical distribution on all domains (including the target domain) to retrieve relevant domains' examples. learning such a distribution needs to detect the domain similarities between all source domains and the target domain. dombert learns an embedding for each domain and computes such similarities. the domain embeddings are learned from an auxiliary task called domain classification. ## conclusions we propose dombert, which automatically exploits the power of training corpus from relevant domains for a target domain. experiments demonstrate that dombert is promising for absa.