Extracting Fine-Grained Economic Events from Business News (2020)

Abstract: Based on a recently developed fine-grained event extraction dataset for the economic domain, we present in a pilot study for supervised economic event extraction. We investigate how a state-of-the-art model for event extraction performs on the trigger and argument identification and classification. While F1-scores of above 50% are obtained on the task of trigger identification, we observe a large gap in performance compared to results on the benchmark ACE05 dataset. We show that single-token triggers do not provide sufficient discriminative information for a fine-grained event detection setup in a closed domain such as economics, since many classes have a large degree of lexico-semantic and contextual overlap.

## introduction we present a pilot study on a novel dataset annotated with economic and financial events in english company-specific news. event processing automatically obtains the "what, who, where and when" of real-world events described in text. event extraction consists of identifying event triggers, i.e. the tokens that express an event of a predetermined type, and identifying participant arguments, i.e. the tokens that express prototypical participant roles. event extraction is typically an upstream step in pipelines for financial applications: it has been used for news summarization of single @xcite or multiple documents @xcite , forecasting and market analysis @xcite @xcite @xcite , risk analysis @xcite , policy assessment @xcite , and marketing @xcite . this work aims to enable these information extraction tasks in the financial domain by making available a dataset and fine-grained event extraction model for companyspecific news by classifying economic event triggers and participant arguments in text. our sentivent dataset of company-specific events was conceived to be compatible with finegrained event representations of the ace benchmark corpora as to enable direct application of advances in the field. in our pilot study, we investigate the portability of an existing state-of-the-art model for event extraction, named dygie++ @xcite , to the task of financial event extraction. many approaches to the detection of economic events are knowledge-based and pattern-based (arendarenko and @xcite @xcite . these use rule-sets or ontology knowledge-bases which are largely or fully created by hand and do not rely fully on manually annotated supervised datasets for machine learning. the stock sonar project @xcite notably uses domain experts to formulate event rules for rule-based stock sentiment analysis. their approach has been successful in formulating trading strategies @xcite and in assessing the impact of events on the stock market @xcite . along the same line, @xcite rely on a hand-crafted financial event ontology for pattern-based event detection in the economic domain and incorporates lexicons, gazetteers, pos-tagging and morphological analysis. several semi-or distantly supervision approaches exist in which seed sets are manually labeled or rule-sets are used to generate or enhance training data @xcite @xcite . for chinese, yang et al. (2018) and chen et al. (2019) ## dataset we define economic events as 'textually reported real-world occurrences, actions, relations, and situations involving companies and firms'. fine-grained events in our dataset are operationalized as event triggers, i.e. the minimal span of text (a single word or a phrase) that most succinctly expresses the occurrence of an event type, and event arguments, i.e., participating entities involved in the event by filling a certain prototypical semantic role. figure 1 shows examples of annotated event triggers with their type | subtype label linked to argument with their roles. we annotated 288 documents with structured events containing event types and subtypes, arguments and event attributes (i.e., negation, modality & event co-reference). the news article pertain to 30 s&p500 companies spread over different industries and time between june 2016 to may 2017 to avoid topical specialization. these companies were selected as a starting point for scraping because market data (e.g., key financials, spot prices, dividend, earnings and p/e ratio) for these are readily available, enabling market research. for a full description of the data collection, annotation process, dataset properties, and agreement study, we refer to @xcite . the event typology contains 18 event main types with 42 subtypes for which 48 possible argument roles exist. not every main event type has subtypes and the arguments "time", "place", and "capital" can belong to any event. the typology was iteratively developed on a sample corpus in which news events were evaluated by financial domain experts. starting with the set of event types in previous literature @xcite @xcite @xcite , types were removed and added based on frequency and cohesion of categories. relevant argument roles were also added in this process of iterative refinement. for a full description of the types and subtypes, we refer to the annotation guidelines @xcite . here is a list of main @xmath0: the types profit/loss, revenue, expense, and salesvolume represent major metrics in financial reporting on the income statement. of all financial metrics, we found these types to be commonly and generally reported in business news. other metrics such as debt ratios or cash flow statements as well as highly industry-specific indicators were collected under the financialreport type. discussion of asset performance are captured by rating and securityvalue. macroeconomics is a broad category pertaining to events that do not involve decisions or interactions of specific companies. while this category is not company-specific, it was devised to capture discussions of sector trends, economywide phenomena, and governmental policy. in the event extraction pilot study, we only experiment with classifying main event types and not the subtypes. an important difference in event trigger annotation with ace05 is the tagging of discontinuous, multiword triggers (e.g., "upgraded ... to buy", "cut back ... expenses", "eps decline"). in ace05, triggers are always single-word and any multi-word idiomatic expressions are joined into one token (e.g. the bucket" becomes "kick the bucket"). for sentivent, 42% of triggers are multi-word and 13% are both discontinuous and multi-word. comparing annotation counts in ace05 to our annotations, sentivent has less documents but a higher density of event annotations. this reflects a difference in annotation approach in which events are less constrained by syntactic rules than in ace05. unlike ace, we did not annotate entity tags and do not restrict event arguments definition by entity type. not restricting events by syntactic rules or entity type results in a conceptually simplified annotation process. it also decreases to degree to which lexicosemantically present events are not tagged because of restrictive rules. furthermore, the generality of geo-political event types in ace05 compared to the domain-specific economic events in business news entails that more relevant events will be present. due to the absence of entity annotations in our dataset, we also have significantly more arguments as in ace05 argument roles are restricted by specific entity labels. our argument role annotation is any span within the event scope that expresses the argument role and is thus semantically motivated. ## experimental setup task definition event extraction concerns identifying event triggers and their event types, event arguments and event roles. the input documents are represented as a sequence of tokens d from which the model constructs the set of all possible within-sentence spans @xmath1, ..., s t } in the document (limited by a threshold-length). each token d i is assigned an event type label t i . then, for each trigger d i , event arguments are assigned by predicting and argument role a i j for all spans s j in the same sentence as d i . evaluation for ace05, we follow the evaluation splits (cf. trigger identification (trig-id) is the subtask of identifying if a token position matches a ground-truth reference trigger. trigger classification (trig-c) determines the type of the identified trigger. a trigger is correct when it is correctly identified and its type label matches the reference. argument identification (arg-id) is the subtask of identifying if a text span belongs to a certain event type. an argument is correctly identified when the span matches and the event type is correct. argument classification (arg-c) determines the role (e.g. "employee") of an identified argument. an argument is correctly classified if it is correctly identified and its role label matches the ground-truth reference. data pre-processing all documents were sentence split and tokenized before annotation. as our event triggers are multi-token spans similar to rich-ere triggers, we transformed these multi-token spans into single-token triggers to allow us to use the dygie++ implementation in its basic configuration. for argument spans this pre-processing step is not needed as arguments are annotated as continuous, multitoken spans in both ace05 and sentivent. we selected single tokens by dependency parsing the triggers and selecting verbal and nominal root token within the trigger span using spacy nlp library @xcite . when filtering multiple tokens to a single trigger, we use syntactic priority criteria based on pos in which nominal tokens are prefered over verbal tokens which are prefered over other types. search for these parts-of-speech goes one level down into the parse tree from the original trigger's root token, defaulting to the original root token if no nominal of verbal child is found. examples of this pre-processing applied to original multi-token triggers: the dygie++ model also allows the use of named entity labels as features in training the trigger and argument subtasks. unlike ace05, our dataset does not include entity annotations (argument roles are unconstrained by entity types). to test if silver-standard entity labels improve identification and classification, we used entity label predictions of a trained entity model on the ace05 dataset (entity f 1 -score: 90.7%). @xcite . as a current stateof-the-art model on ace event extraction and other information extraction tasks, dygie++ is a good candidate for establishing a baseline on our new dataset. ## data availability and replication this work's source code, preprocessed replication data in dygie-format, and the winning trained model are publicly available at https://osf.io/j63h9 @xcite . the original dataset will be freely downloadable at the end of the sentivent project through this repository. up until then, the fully annotated corpus is available on request for academic research purposes. ## conclusion and future work event extraction is a required step in many data-driven financial tasks in which factual information is needed to capture changes in the real-world. various general domain fine-grained event extraction corpora are freely available but no economically focused corpus exists. we presented a pilot study on a novel dataset enabling supervised fine-grained extraction of economic events as trigger and argument classification. using a state-of-the-art information extraction pipeline based on span-based graph propagation of pretrained contextual embeddings, we observed a large drop in performance on our dataset compared to the benchmark ace05 dataset. after error analysis, we found this is largely caused by missing predictions. for event triggers, many errors are due to ambiguity introduced by the requirement of the dygie++ model implementation for single-token triggers. we hypothesize that single-token triggers for the sentivent corpus do not provide sufficient discriminative information for detecting classes which have a large degree of semantic and contextual overlap. hence, in future work we will focus on event extraction methods that model arbitrary length triggers. currently on-going, we are adding "investor sentiment" annotations on top of events as well as separate sentiment expression annotations with their targets. these sentiment annotations will allow us to jointly process the "common-sense" sentiment of events. we will investigate how extracted event schemata can be used upstream from aspect-based sentiment analysis.