A High Precision Pipeline for Financial Knowledge Graph Construction (2020)

Abstract: Motivated by applications such as question answering, fact checking, and data integration, there is significant interest in constructing knowledge graphs by extracting information from unstructured information sources, particularly text documents. Knowledge graphs have emerged as a standard for structured knowledge representation, whereby entities and their inter-relations are represented and conveniently stored as (subject,predicate,object) triples in a graph that can be used to power various downstream applications. The proliferation of financial news sources reporting on companies, markets, currencies, and stocks presents an opportunity for extracting valuable knowledge about this crucial domain. In this paper, we focus on constructing a knowledge graph automatically by information extraction from a large corpus of financial news articles. For that purpose, we develop a high precision knowledge extraction pipeline tailored for the financial domain. This pipeline combines multiple information extraction techniques with a financial dictionary that we built, all working together to produce over 342,000 compact extractions from over 288,000 financial news articles, with a precision of 78% at the top-100 extractions. The extracted triples are stored in a knowledge graph making them readily available for use in downstream applications.

## introduction knowledge graphs (kg) have lately emerged as a de facto standard for knowledge representation in the semantic web, whereby knowledge is expressed as a collection of "facts", represented in the form (subject, predicate, object) (spo) triples, where subject and object are entities and predicate is a relation between those entities. this collection can be conveniently stored, queried, and maintained as a graph, with the entities modeled as vertices and relations as links or directed edges. driven by applications such as question answering, fact checking, information search, data integration and recommender systems, there is tremendous interest in extracting high quality knowledge graphs by tapping various data sources @xcite @xcite @xcite . over the years, a number of cross domain kgs have been created including dbpedia @xcite , @xcite , freebase @xcite and nell @xcite , covering millions of real world entities and thousands of relations, across different domains. recently, there has been a growing interest in generating domain targeted structured representations of financial and business entities and how they are related to each other. crunchbase 1 curated a knowledge base (kb) through partnerships with companies and data experts covering 100,000+ business entities including companies and investors, but covering only a few types of business transactions (i.e., relations) such as acquisitions and funding rounds. the work by @xcite attempted to address this limitation by developing a pipeline to populate a kb semi-automatically with quintuples of the form (subject, predicate, object, monetary value, date) extracted from a news corpus. however, this pipeline only extracted 496 quintuples covering 316 economic events that fall into one of two categories: events that increment the value of agent's resources (e.g., acquire, collect) or decrement it (e.g., pay, sell). our goal is to automatically extract high precision structured representations from thousands of financial news articles covering a broader range of financial entities such as markets, stocks, persons (e.g., ceos, presidents, etc), currencies, and governments. further, storing them in a kg can facilitate answering interesting and complex queries such as (1) company acquisitions by german drugmakers, (2) us-china trade in terms of exports, (3) companies suing each other on patent grounds, etc. knowledge extraction from unstructured sources is one of the major challenges facing industry scale kgs @xcite . traditional approaches to kg construction from text rely on a pre-specified ontology of relations and large amounts of human annotated training data to learn extraction models for each relation. this limits their scalability and applicability to new relation types. open information extraction (openie) @xcite aims to overcome these limitations by extracting all semantic relational tuples in raw surface form with little or no human supervision. closely related to openie is semantic role labeling (srl) which aims at detecting argument structures associated with verb predicates, as well as labeling their semantic roles, thus overcoming situations where the verb tense and conjugation change the role of the argument in the sentence (i.e., whether the argument is an agent which carries out the predicate action or a theme which receives the predicate action). semantic roles make it possible to impose structural and semantic constraints on entity types to ensure high quality knowledge extraction. besides, knowing the semantic roles of arguments can improve the effectiveness of question answering. in this work, we develop a high precision knowledge extraction pipeline tailored to the financial news domain by combining srl information extraction for verb predicates with typed patterns for noun mediated relations. this pipeline filters noisy predicate-argument structures via a dictionary of semantically and structurally constrained sense-disambiguated financial predicates. in order to maximize the utility of the extractions for downstream tasks, our pipeline produces compact extractions via dictionary-guided minimization of overly-specific arguments. these extractions are scored using a binary classifier, with the score reflecting our confidence in the extracted fact. we perform a lossless decomposition of the nary relations extracted, to construct the kg. while some components of the the pipeline are customized to the financial domain, we believe with small tweaks it can be easily adapted to other domains. compared with @xcite , the most closely related work, our pipeline extracts over 342,000 n-ary facts and covers more types of financial predicates -a total of 87 as opposed to 50 in @xcite . furthermore, our pipeline produces high precision extractions, specifically 78% at the top-100 extractions, as opposed to 34% of the pipeline from @xcite . in summary, our main contributions are as follows. we design a high precision knowledge extraction pipeline tailored to the financial news domain. our pipeline combines srl and pattern based information extraction to extract domain targeted noun/verb-mediated relations. we develop a conditional random field (crf) model that identifies and removes sequences of noisy text commonly found in financial news articles. to further improve precision, we build a dictionary of semantically and structurally constrained sense-disambiguated financial predicates to filter out noisy extractions produced by srl. the ∼380,000 triples we extracted are stored in a kg which can be readily queried. we also conduct ablation studies to examine the effect of the different components of the pipeline on a number of performance metrics. ## related work cross-domain kgs such as dbpedia, freebase, nell, and babelnet @xcite contain encyclopedic knowledge covering real world entities across different domains (e.g., people, organizations, and geography). they were either manually curated (e.g., freebase) or automatically created, from semi-structured textual sources such as wikipedia infoboxes (e.g., dbpedia), or unstructured text on the web (e.g., nell). a number of efforts to create domain targeted kgs followed. the aristo tuple kb @xcite extracted 294,000 high-precision spo triples using a kg extraction pipeline targeted towards elementary science topics from domain relevant sentences found on the web. in @xcite , the authors developed a framework (cpie) which extracted relational tuples between 3 fixed types of biomedical entities from pubmed paper abstracts. in the financial domain, crunchbase curated a kb covering over 100,000 companies, investors, acquisitions and funding rounds, while @xcite extracted quintuples of monetary transactions covering 316 economic events. nell used a predefined ontology of categories and relations and a few seed examples that are used for semi-supervised bootstrap learning of semantic categories of entities and the relations that exist between them. this bootstrapping approach reduces the human labor required to label training data while taking advantage of the huge amount of unlabeled data available. while manually defining an ontology leads to high precision extractions, it requires domain experts and, in an open domain, it is not feasible to define a complete ontology. openie aims to overcome these limitations by extracting all relational phrases in raw surface form in a single pass over the corpus. however, the verb tense and conjugation can change the role of an argument in the relation. srl helps disambiguate the relations between arguments and their predicates by identifying semantic frames within the sentence and the semantic roles of the arguments. the kb built in @xcite used a pipeline that consisted of: (1) a grammar for monetary value recognition, (2) srl for economic event identification, (3) entity recognition via dbpedia, crunchbase and freebase, and (4) date extraction via a temporal tagger. it extracted structured representations of economic events in the form of (subject, predicate, object, monetary value, date) quintuples from the new york times annotated corpus (nytc) foot_0 . it ranked all representations of an economic event according to confidence scores learnt using a supervised model. the domain was defined by a list of financial predicates using a semi-supervised method that starts with a set of seed predicates and expands them using wordnet @xcite . the pipeline only extracts 496 quintuples from 316 economic events over just 2 categories of events, with a precision of 34%. our work has the following key differences with prior work. we deal with the challenging task of identifying and removing noisy text spans in news articles. our pipeline combines srl and pattern based ie in addition to producing implicit extractions from appositions. we build a dictionary of 87 semantically and structurally constrained financial predicates covering broader financial transactions and improving precision. we resolve coordinating conjunctions and do a dictionary-guided minimization to prune overly specific arguments. in all, our approach leads to a large knowledge graph with high precision. ## the knowledge extraction pipeline in the following subsections, we describe each component of our ke pipeline (see fig. 1 ). the pipeline operates at the sentence level and starts with cleaning the news articles by identifying and removing noisy text spans, then performs linguistic annotations, i.e., resolves co-references and identifies named entities. predicate argument structures are then extracted by the srl component and passed to the financial predicate dictionary which we built to filter out noisy extractions by the srl. we produce additional extractions via high-precision typed patterns that are tailored to the financial domain, and by resolving appositions. we maximize the utility of the extractions by minimizing overly specific arguments by processing coordinating conjunctions and financial lexicon guided minimization. finally, we score the predicate argument structures to reflect our confidence in their precision and conciseness. the input to the pipeline consists of two components: ## evaluation we run our ke pipeline on the us financial news corpus. fig. 4(a) shows the distribution of news article length in the corpus, measured by the number of sentences. for ease of exposition, the figure is trimmed by eliminating the distribution's long tail. observe that most of the articles have fewer than 20 sentences and articles with more than 60 sentences in length are very rare. fig. 4(b) shows the distribution of named entity types in the corpus. as expected in a financial news corpus, the top 4 entity types, org, cardinal, money, and person account for almost 80% of the unique named entities in the corpus. we report the results and compare with the work in @xcite which extracts (subject, predicate, object, monetary value, date) quintuples from the new york times annotated corpus (nytc) foot_4 . further, we compare the functionalities of our pipeline against @xcite in the minimization module responsible for condensing overly specific arguments dropped over 427,000 tokens. the majority of the tokens dropped this way were due to safe minimizations, i.e. determiners or possessives. the rest of the tokens were dropped by the dictionary-guided minimization. the top bigram lexicon hits were quarterly dividend (queried 372 times), followed by common stock and net income, and subsequently the tokens of these bigrams were marked as stable. the adjectives in these bigrams, i.e., quarterly, net, and common are critical in financial context. thus, simply dropping adjectives would result in the loss of important information, and this is avoided by querying against the financial lexicon. this emphasizes the importance of the financial lexicon in preserving tokens that are important in the financial context while minimizing overly specific arguments. precision. we ranked the extractions according to their confidence scores and examined the top 150 extractions and manually labeled them. the ratio of the correct extractions in the top 50, top 100, and top 150 extractions, i.e., precision@50, precision@100, and precision@150 is 78%, 78%, and 79.33% respectively. the pipeline in @xcite has a much lower precision of 34% (at a recall of 20%). ## ablation studies we conduct ablation studies on a random sample of 1000 articles to gauge the effect of each stage of the pipeline on the overall performance. the results are reported in our pipeline was effective in extracting predicate-argument structures from sentences with long range dependencies. e.g., from the sentence 9 "orb energy, an indian solar company backed by u.s. venture capital fund acumen fund inc, secured @xmath010 million', source: 'rooftop projects', am-tmp: '02/14/2017'). the overly specific argument commercial rooftop projects is minimized by dropping the token commercial, as none of its sub-sequences is found in the financial lexicon. furthermore, the pipeline successfully identifies and classifies the roles of different arguments to a fine granularity. e.g., from the sentence "parke bancorp's net loans increased to @xmath1852.0 million at december 31 , 2016 , an increase of @xmath2852.0 million', end point: '$1.01 billion', am-tmp: '12/31/2017'). such granularity is useful in answering complex queries such as "find a contiguous sequence of stock prices that are all increasing (or all decreasing)". the pipeline successfully classifies less common predicate senses in difficult contexts: e.g., settle.02 (meaning resolve) versus settle.01 (meaning decide), and cut.02 (meaning reduce) versus cut.01 (meaning slice). correct classification of predicate sense is critical for assigning correct semantic roles to the arguments. our pipeline extracted ∼342,000 n-ary facts from only 5.2% of the sentences. one way to extract more facts is by expanding the financial predicate dictionary, while adding semantic and structural constraints on the new predicates to improve precision. the ablation studies show that the financial predicate filtering was the most important factor for precision. furthermore, the cleaning stage presents a significant trade off in precision and total number of extractions. depending on the downstream applications, we may want to favor precision over number of extractions or vice versa. the study also highlighted the significance of the cleaning stage in the overall precision and the importance of resolving coordinating conjunctions and appositions in generating more facts without sacrificing the precision. the ablation study shows the importance of the minimization stage in decreasing the average argument length. we would like to examine expanding the minimization beyond the safe and dictionary minimization of adverbial patterns. prepositional phrases, although good candidates for minimization, are equally challenging. it is also common to see arguments such as "more than 3%", "as much as 3 percent", "almost 3%" in reporting stocks. we would like to canonicalize these arguments @xmath3. in future work, we would like to examine the transferability of the pipeline to other domains. the following adjustments will be needed: (1) a domain targeted dictionary for filtering candidate predicate argument structures; ## conclusions we developed a high precision pipeline for knowledge extraction from a financial news corpus that produced over ∼342,000 n-ary facts at 78% precision by employing multiple information extraction 9 emphasis added. figure 5 techniques. we built a financial predicate dictionary that places structural and semantic constraints on arguments to produce high quality extractions. to enhance the utility of the extractions, we minimized overly specific arguments by processing coordinating conjunctions and appositions, and employed a financial lexicon to minimize adverbial nouns. we evaluated the pipeline and the resulting kg on a number of metrics and conducted ablation studies to examine the effect of different modules of the pipeline on these metrics. this study offered a number of insights and demonstrated the importance of both the financial predicate dictionary filtering and the noisy text cleaning stages in the overall precision of the pipeline.