Full-Stack Information Extraction System for Cybersecurity Intelligence (2022)

Abstract: Due to rapidly growing cyber-attacks and security vulnerabilities, many reports on cyber-threat intelligence (CTI) are being published daily. While these reports can help security analysts to understand on-going cyber threats,the overwhelming amount of information makes it difficult to digest the information in a timely manner. This paper presents, SecIE, an industrial-strength full-stack information extraction (IE) system for the security domain. SecIE can extract a large number of security entities, relations and the temporal information of the relations, which is critical for cyberthreat investigations. Our evaluation with 133 labeled threat reports containing 108,021 tokens shows thatSecIE achieves over 92% F1-score for entity extraction and about 70% F1-score for relation extraction. We also showcase how SecIE can be used for downstream security applications.

## introduction a rapid increase in cyberattacks, both in number and attack techniques, poses enormous challenges to security analysts. much of the information on new threats often appear first in unstructured reports such as blogs and news articles. to quickly respond to the on-going attacks, it is critical to digest the information about new threats in a short period of time. however, it is very difficult to find relevant information from cti reports, particularly because cyber-attacks involve many different entities, including the attacker, victim (e.g., companies/industries), tools (e.g., malware) indicators of compromise (iocs, e.g., file names and ip addresses), and various relations, some of which may be unknown to the security experts. we present a large-scale full-stack ie system designed for the cybersecurity domain. secie can extract 26 entity types, 20 fixed relation types and various open ie relations, and the time information of the relations, which is very critical in cybersecurity. figure 1 shows a snippet of a cti report foot_0 and the ie results from secie. the entity extraction model detects mentions of malware and threatactor from the text. the coreference resolution model recognizes that 'the new ransomware' refers to trojan-ransom.win32.xpan and 'a gang' refers to teamxrat. finally, the relation extraction module produces a relation tuple, <trojan-ransom.win32.xpan, developed_by, teamxrat>, from "the new ransomware was developed by a gang". while there have been efforts to apply nlp and ie to the cybersecurity domain @xcite @xcite @xcite @xcite , they target on a specific sub-area of cybersecurity, mostly on extracting iocs or vulnerabilities, or a component (either entity extraction or relation classification) in the ie process. to our knowledge, our system is the largest endto-end ie system for the cybersecurity domain supporting a large number of security entity and relation types. most existing ie systems apply supervised (deep) learning methods relying on a large amount of high-quality labeled data. unlike the general domain types, labeling fine-grained security entities and relations requires deep domain knowledge, and, thus it is much more difficult to produce a high-quality training data for the security domain. as an anecdote, 3 annotators (1 security expert and 2 professional annotators with many years' experience) working full-time for 5 months could produce only 133 annotated documents, which are far from enough to train supervised models for our need. thus, secie applies unsupervised nlp technologies. we develop tehcniques to handle idiosyncrasies in security terms and and take into account the structural characteristics found in many cti reports. this domain customization allows secie highly accurate, achieving over 92% f1 for entity extraction and 70% f1 for relation extraction. ## methodology we employ a pipeline architecture as shown in figure 2 documents are processed sequentially, where the document content and all the results from the previous components are passed as input to the next component. however, the system can process multiple documents in parallel yielding a high throughput. ## performance evaluation to evaluate our system, we manually labeled 133 cti reports, which contain 6,438 sentences and 108,021 tokens. the documents were labeled by 3 full time annotators over 5 months. to ensure the quality of the labeled data, we kept only the labels agreed by all 3 annotators, resulting in 3,295 entity and 1,216 relation mentions. more detailed statistics of the annotations are shown in ## security applications we demonstrate how secie can provide additional insights on security incidents. ## related work there have been a few efforts to apply ie to the cybersecurity domain. most existing works focus on entity extraction for a small number of security entities (mainly, iocs and vulnerabilities) from certain security text (mainly, cves and tweets). joshi et al. @xcite present a system that produces linked data from cve records. this system can extract 6 entity types commonly found in cves and link the extracted instances to dbpedia entries. @xcite proposes a twitter-based exploit detector, which collects tweets mentioning vulnerabilities. this tool uses a simple keyword matching and monitors occurrences of the "cve" keywords and ids in tweets. liao et al. @xcite presents a system (iace) for fully automated ioc extraction. iace detects file name, ip address, and url using regular expressions. ttpdrill @xcite extracts threat actions (i.e., ttp) from security reports and map them to a threat action ontology from att&ck and capec. this tool detects threat actions from the svo dependency structure, where the subject is a malware instance. @xcite presents an ner tool for the cybersecurity domain, which is similar to our entity extraction component. they apply regular expressions, dictionary matching and a crf classifier for about 20 different entity types and achieves about 82% f1 score. ## conclusion we presented a large-scale full-stack ie system developed for the cybersecurity domain. through careful design choices to handle the idiosyncrasies in the cybersecurity data, our system achieves high f1 scores for both entity extraction and relation extraction. we also demonstrated how our system can be used for downstream applications. our system can help security analysts by transforming the unstructured threat reports into structured formats which can be easily consumable by subsequent security applications.