Weak Supervision using Linguistic Knowledge for Information Extraction (2020)

Abstract: In this paper, we propose to use linguistic knowledge to automatically augment a small manually annotated corpus to obtain a large annotated corpus for training Information Extraction models. We propose a powerful patterns specification language for specifying linguistic rules for entity extraction. We define an Enriched Text Format (ETF) to represent rich linguistic information about a text in the form of XML-like tags. The patterns in our patterns specification language are then matched on the ETF text rather than raw text to extract various entity mentions. We demonstrate how an entity extraction system can be quickly built for a domain-specific entity type for which there are no readily available annotated datasets.

## introduction much knowledge in an organization resides in text documents of various types. effectively using the information and knowledge hidden in enterprise document repositories is a challenge. information extraction (ie) is a well-explored language processing technology for extracting specific kinds of information (e.g., generic or domain-specific entities, relations among entities and events) from documents and presenting it in a structured format @xcite @xcite . this structured information can then be effectively searched, disseminated, reused or mined using data mining techniques to discover valuable knowledge. ie plays a critical role in several applications such as resumes processing, competitor intelligence from news, patent analysis, and insurance claim management. ie is posed as a classification task in machine learning, where the training data consists of labeled mentions of a given entity (or relation, or event) type in sentences. creating a sufficient quantity of such training data is time-consuming and errorprone. hence, there has been research in distant supervision methods for automating the process of creating training data, often using other knowledge sources such as dbpedia. in this paper, we map each sentence in the given corpus to an enriched text format (etf) by adding syntactic and semantic information to raw text. we then propose an enriched regular expression language to write linguistic knowledge (rules) to extract mentions of entities and relations from the etf representation of the sentences. unlike tools like snorkel @xcite or more complex tree regex languages, our pattern language is simpler, more efficient and has novel features to allow linguistic patterns that use context from multiple sentences. we demonstrate the use of linguistic knowledge to automatically create training data and show that the new training data improves accuracy of ie classification models. we demonstrate this methodology on a completely novel application that extracts risk factors from audit reports of software projects. ## linguistic rules we express linguistic rules in the form of regular expression patterns which are applied on an etf text in which rich linguistic information is embedded in the form of xml-like tags. ## linguistic rules for weak supervision for extracting entity mentions, sequence labelling techniques such as conditional random fields (crf) and long short-term memory (lstm) networks are widely used. these supervised techniques need a significant number of annotated sentences for training to achieve desirable extraction accuracy. for any domain-specific entity type, creating a dataset of such annotated sentences involves significant time, manual efforts and cost. we propose to augment a small training dataset (l) for such entity extraction task with unlabelled data (u ) where labels are automatically obtained using linguistic patterns. a large number of additional sentences can be labelled in this way without any extra time and cost. although, some manual efforts and expertise are needed for designing the linguistic patterns, the efforts are significantly less as compared to manually annotating a large corpus. the patterns are designed in such a way that each pattern is a high-precision pattern. we observed that a sequence labelling model trained using l∪u achieves better entity extraction performance as compared to a model trained using only l. also the supervised sequence labelling model does not learn to imitate the linguistic rules exactly because: different feature views: the linguistic rules and a supervised sequence labelling model use two different feature views, similar to co-training. although the feature views are not mutually exclusive, there are major differences. most of the linguistic rules use dependency parsing information, which is not used by the sequence labelling model. multi-sentence patterns: some of the linguistic patterns have multi-sentence scope, i.e., they use the context information which is outside the sentence from which an entity mention is identified. however, the sequence labeller processes only one sentence at a time, and hence it can not use any context information outside the current sentence. this enables the sequence labeller to learn additional features from the current sentence itself. ## related work the most relevant line of work for our linguistic patterns is semgrex @xcite , tregex @xcite , and spacy rulebased matching 2 . semgrex and tregex allow users to write patterns on dependency and constituency trees, respectively. the patterns are based on regular expression matching for nodes (tokens) and various relationships between the nodes. rule-based matching provided by spacy allows users to write regular expression patterns for token-level matching but for more complex rules, python scripting is necessary. our patterns specification language is specified purely in terms of regular expressions and allows users to write very powerful patterns using 2 spacy.io/usage/rule-based-matching facilities such as mainregex-outerregex combination (e.g., @xmath0.g., @xmath1. a form of indirect supervision is distant supervision @xcite where a knowledge base is used to automatically create an annotated dataset. recently, snorkel framework @xcite was proposed to combine multiple weak supervision sources. however, it is not easily adaptable for sequence labelling tasks. @xcite proposed an entity extraction technique using weak supervision from multiple labelling functions such as entity extraction models trained on other domains, gazettes, heuristic functions etc. in our case, for a domain-specific entity like risk entity (introduced in the next section), labelling functions based on other domains, gazettes or knowledge bases are not feasible. however, two recent approaches @xcite look promising for our problem setting where linguistic rules are used for weak supervision for entity extraction and we plan to explore them as future work. ## application weak supervision using linguistic patterns is especially useful for extraction of domain-specific entity types for which obtaining or creating training data is costly such type -risk. in large it services organizations, thousands of projects are going on simultaneously. these projects are routinely audited and as a part of this process, auditors also write their opinion about each project as an audit summary. one of the most important piece of information in these audit summaries is potential risks that the project is facing or may face in near future. we define risk as an entity type which is any undesirable factor which may have an adverse effect on project objectives or outcomes. e.g., impacting timelines we used a dataset of 3804 audit summaries consisting of 8046 sentences. we manually annotated risk entity mentions in 700 of these and used 500 as our training set (l) and remaining 200 as our evaluation set. 3104 unlabelled ## conclusions we proposed a powerful patterns specification language for specifying linguistic rules for entity extraction which are matched against etf text. the language is also generalizable to encode linguistic knowledge for relation and event extraction. we demonstrated how an entity extraction system can be quickly built for a domain-specific entity type risk where a small manually annotated dataset is augmented with a large automatically labelled dataset using linguistic knowledge.