Introducing a New Dataset for Event Detection in Cybersecurity Texts (2020)

Abstract: Detecting cybersecurity events is necessary to keep us informed about the fast growing number of such events reported in text. In this work, we focus on the task of event detection (ED) to identify event trigger words for the cybersecurity domain. In particular, to facilitate the future research, we introduce a new dataset for this problem, characterizing the manual annotation for 30 important cybersecurity event types and a large dataset size to develop deep learning models. Comparing to the prior datasets for this task, our dataset involves more event types and supports the modeling of document-level information to improve the performance. We perform extensive evaluation with the current state-of-the-art methods for ED on the proposed dataset. Our experiments reveal the challenges of cybersecurity ED and present many research opportunities in this area for the future work.

## introduction with the proliferation of cyber technologies (i.e., social networks, internet of things) in our daily life, the frequency of cyberattacks and cybercrimes is also rapidly increasing, potentially imposing serious threats to our cyber activities. cybersecurity has thus become an important field for which a large amount of text data would be produced to report and discuss various aspects of cyber vulnerabilities. the expected sheer amount of this type of data calls for automatic techniques to analyze the cybersecurity text and extract useful knowledge. among others, these techniques can help to detect the trends of the cyberattacks for better policy making or populate cybersecurity knowledge bases for automatic reasoning systems. in this work, we examine information extraction technologies (ie) in natural language processing (nlp) as a promising candidate for the knowledge extraction task from cybersecurity text. in particular, we focus on event detection (ed), an important task in ie that seeks to identify trigger words of specified types of events in text @xcite . ed is an actively studied task in ie where deep learning models have been the dominant approach to deliver the state-of-the-art performance @xcite . for instance, consider the sentence: remote attackers to completely takeover player accounts just by tricking users into clicking an unsuspectable link. an ed system for cybersecurity texts should be able to identify "takeover" as an event trigger word of the event type attack.user compromise in this case. in order to enable the application of the ed methods in the cybersecurity domain, a crucial requirement has to do with the benchmark datasets to facilitate the development and evaluation of ed models. unfortunately, most of the current benchmark datasets for ed (i.e., the ace and tac kbp datasets @xcite ) cannot serve this purpose as they have mainly concerned the common events in a person's life of the general domain (e.g., being born, getting married, or being arrested). in addition, the events in the general domain might involve substantial differences with those in the cybersecurity domain (i.e., the divergences in lexical forms, sentence structures and domain expertise), necessitating the development of cybersecurity-focused datasets to aid the research on ed and reveal the nature for the events in this domain. to this end, @xcite recently presents the first dataset for cybersecurity ed (called casie) that annotates event instances with rich annotation. however, this dataset involves at least three limitations that hinder future research in this area. first, casie only contains a small number of event types (i.e., five types) that fail to cover a wide range of important cyber attack/vulnerability types in reality @xcite . this would limit the application of the systems and restrict the comprehensiveness of the analysis about cybersecurity events developed from the dataset. second, the event triggers in casie tend to be correctly detected and classified without considering the document-level information (i.e., simply relying on the local contexts in the sentences of the triggers is sufficient). this is not desirable as based on our analysis, the necessary contexts to recognize the event triggers in the cybersecurity domain might span the entire long documents and the limited requirement for document context in casie would not be able to reflect the data distribution of the cybersecurity events for ed well. for example, the word "attack" can appear somewhere in a document to refer to some event mentioned far away at the beginning of the document. as the local context of "attack" does not present any information about the specific type of this attack, the long document context (i.e., up to the beginning of the document) would be crucial to successfully determine the actual event type for "attack" in this case. last but not least, casie has not been comprehensively evaluated with the state-of-the-art ed systems, making it challenging to accurately estimate the difficulty/complexity of the dataset. consequently, in this work, we introduce a novel dataset for cybersecurity ed (called cyseced) that is manually annotated for 30 event types to better characterize the important cyber attacks and vulnerabilities reported in texts. cyseced involves event triggers whose types can only be predicted if the long-range document context is effectively captured, thus offering a more challenging dataset for ed. finally, we extensively evaluate the bestperforming ed models on cyseced. our experiments show that the performance of current models is far behind the human performance on this dataset and further research is needed to improve the models' performance. we will publicly release cyseced to promote the future research on ed and nlp for the cybersecurity domain. ## data collection and annotation we use the articles on the website "the hacker news" (thn) 1 as the documents for cybersecurity event annotation in this work. thn (written in english) is a trusted and widely-acknowledged cybersecurity news platform that reports the latest cybersecurity news and in-depth coverage of current as well as future trends in cybersecurity. in order to create an ed dataset for the cybersecurity domain, we consult the cyberattack taxonomy in @xcite to select a set of 30 cybersecurity event types that occur frequently and have high impact in thn. in particular, the 30 event types are grouped into four following categories to reflect four different stages of a cyber attack/vulnerability: discover: a vulnerability in a software or system is detected or mentioned by some entity (i.e., hackers, engineers) (e.g., kernel flows, buffer overflow, back door), patch: some entity (i.e., software companies) fixes or shows how to fix a known vulnerability, attack: an attacker exploits some vulnerability to impact the systems using some means (e.g., user compromise, viruses, spyware, worms, denials of services), and impact: the consequence of an attack for a system (e.g., disrupt, breach/disclosure of information). the full list of the event types along with their descriptions and examples for our dataset are shown in appendix a. the articles, once crawled from thn, would be processed to extract the title and text content (i.e., removing other elements such as html tags, images, etc.). we recruit two undergraduate students who specialize in security and networking to perform the data annotation for the processed articles. each student is trained with the annotation guideline about the 30 event types and does a group of exercises to be able to better distinguish the event types. among others, our guideline only annotates a single word for each event trigger (i.e., the most important word to clearly express the event), following the practice in prior ed work @xcite . overall, we annotate 292 documents and achieve a cohen's kappa score of 0.79 (i.e., very close to the near-perfect agreement range of [0.81, 0.99]). finally, in order to improve the quality of the dataset, a cybersecurity expert is asked to resolve the cases where the two annotators disagree, leading to the final version of cyseced. ## annotation challenges we find that cybersecurity event annotation is a challenging task where the major challenges involve the disagreement between the annotators for the subtle cases and the high level of necessary domain expertise to annotate the triggers. first, for the annotation disagreement, the highest disagreement concerns the decisions to annotate a word as a trigger or not. for instance, consider the following sentence: in this sentence, both annotators believe the first word "apps" is referring to a malware and should be annotated as a trigger word of an event of type attack.spyware. however, for the second word "apps", one annotator treats it as a neutral apps and does not label it while the other annotator considers both words "apps" as coreferred and mark them as trigger words. the other disagreements have to do with the confusion between the types of cyberattacks where the differences are subtle (i.e., attack.virus vs. attack.worm, attack.trojan vs. at-tack.spyware, and attack.root compromise vs. attack.arbitrary code execution). second, for the domain expertise challenge, in many cases, the understanding about the cybersecurity attacks and crimes is necessary to analyze the context and assign the appropriate event types for the trigger words. for instance, consider the following sentence as an example: "numerous cyberattacks on automobile companies have been reported yesterday where the hackers used compromised machines to hit the websites with floods of traffics measuring up to 140gbps." in this sentence, "cyberattacks" and "hit" are the trigger words of the events of type attack.denial ## data analysis this section conducts some additional analysis to gain a better insight into the proposed dataset cy-seced. first, in addition, we find that sentences mentioning some events in cyseced often contain at least two event trigger words for the event types. in other words, cybersecurity events in cyseced tend to cooccur with each other in the sentences, suggesting potential inter-dependencies between events. these dependencies can be exploited to further improve the ed performance for cybersecurity domain in the future research @xcite @xcite . in particular, among the sentences in cyseced, 45.4% of the sentences do not contain any event triggers, 50.0% of the sentences host at least two event triggers, and only 4.6% of the sentences involve a single event trigger. among the event types, the highest cooccurrence frequency involves the co-occurrence of two attack event triggers in the same sentences (i.e., amounting to 22% of the total sentences in cyseced). ## evaluation models: there are two classes of models for ed in the literature, i.e., the sentence-level models (i.e., the models that only exploit the local context information in the sentences of the triggers) and the document-level models (i.e., the models that further consider the document-level contextual information for ed). this section aims to reveal the complexity of cyseced by evaluating the performance of the state-of-the-art models for ed in these model classes. in particular, for the sentence-level class, we focus on the following ed models: regarding the document-level models, we consider the following representative ed models: for the models in this work, we experiment with both the traditional uncontextualized word embeddings word2vec (mikolov et al., 2013b) (i.e., the 300 dimension version) and the recent contextualized word embeddings bert (i.e., the uncased base model) @xcite as the pre-trained word embeddings. for bert, we additionally evaluate the bert-based ed models in @xcite (called dmbert) and @xcite (called bert-ed) that are the sentence-level models with the best-reported ed performance on the ace 2005 dataset. finally, we tune the hyperparameters for the models using the development results: the ed problem in this work is formulated as a word classification problem where given a sentence/document, the models need to predict the event types for its words. the set of event types includes a special type other to indicate the words that are not event triggers (called the negative examples). the positive examples for ed correspond to the event trigger words. in order to evaluate the ed models on cyseced, we use 240 documents for training data, 30 other documents for test data, and the remaining 30 documents for the development data (i.e., the document split ratio of 80:10:10). in order to compare cyseced and casie, we also divide the documents in casie @xcite into the training, test, and development data using the 80:10:10 ratio to evaluate the ed models. we train all the ed models in this work with early stopping on the development datasets (i.e., we stop training the models once the performance on the development data decreases). some statistics about the data splits for cyseced and casie are reported in there are several important observations from the table. first, comparing casie and cyseced, we see that the performance of the current ed models on cyseced is in general much worse than those for casie. this indicates that cyseced is more challenging than casie for ed and the future work can use cyseced to evaluate the ed models for the cybersecurity domain. also, the best performance of the models on cyseced (i.e., 68.4% with deeb-rnn) is still far behind the human performance on this dataset (i.e., 81.0%), presenting much opportunities for the future research in this area. second, comparing word2vec and bert, we find that bert mostly performs comparably or poorer than word2vec for different ed models and datasets, potentially due to the large difference between the training data for bert and the cybersecurity domain. third, among the sentencelevel models, similar to the general domains (i.e., ace 2005), moganed and bert-ed still have model word2vec bert casie cyseced casie cyseced cnn 83.9 43.7 83.8 43.0 dmcnn 85.2 43.2 84.0 42.7 gcn 85.5 52.2 85.4 48.9 moganed 86.0 61.6 86.5 56.5 cyberlstm 81.4 51.8 82.3 34.5 dmbert --84.1 55.1 bert-ed --84.7 58.1 hbtngma 85.9 60.9 85.0 62.4 deeb-rnn 85.5 68.4 85.8 65.5 prior work has applied nlp to perform several tasks for the cybersecurity domain, including privacy policy analysis @xcite @xcite ), text analysis for cybersecurity with social media text (i.e., ddos attack detection, alert generation for threads and vulnerabilities using twitter) @xcite @xcite @xcite @xcite , and report and timeline creation of cybersecurity events (hackmageddon, 2019; privacyright, 2019). however, none of these work considers the detection of event trigger words from cybersecurity articles as we do. recently, @xcite presents a dataset for general text-based malware behavior analysis that annotates 39 reports for malware actions and their attributes. this study is then extended in the se-curenlp semeval evaluation @xcite . however, different from our cyseced dataset, the annotated dataset in these work is very sparse (i.e., involving less than 5 examples for many labels), hindering the development of the deep learning models @xcite . also, it does not annotate event triggers for rich event types as we do. finally, ed has been extensively studied in the literature @xcite @xcite @xcite @xcite @xcite @xcite , partly due to the availability of the large evaluation datasets (i.e., the ace and tac kbp datasets @xcite for the general domains, and the bionlp datasets @xcite for the biomedical domain). the closest works to our in the cybersecurity domain involve @xcite to extract events on chinese news, @xcite to perform cyberattack detection on twitter, and @xcite to present the casie dataset for event extraction. however, these datasets contain less event types and cannot support the document-level information for the models as cyseced. finally, we notice some recent interests in new type extension learning, e.g., few-shot/zero-shot learning @xcite @xcite , that can be helpful to develop ed systems for cybersecurity domain. ## conclusion we present a new dataset cyseced for event detection in the cybersecurity domain. our dataset is manually annotated for 30 event types and provides sufficient data to develop deep learning models for this task. we extensively evaluate state-of-the-art models for ed on the proposed dataset, showing that the performance of these models is still much worse than the human performance. our experiments also suggest that document-level information is necessary to perform ed for cybersecurity domain. in the future, we plan to extend our annotation to include event arguments and other properties of events. viet dac lai and thien huu nguyen. 2019. extending event detection to new types with learning from keywords. in proceedings of the 5th workshop on noisy user-generated text (w-nut 2019) at emnlp 2019. viet dac lai, tuan ngo nguyen, and thien huu nguyen. 2020b. event detection: gate diversity and syntactic importance scores for graph convolution neural networks. in emnlp. qi li, heng ji, and liang huang. 2013. joint event extraction via structured prediction with global features. in acl. shasha liao and ralph grishman. 2010. using document level cross-event inference to improve event extraction. in acl. swee kiat lim, aldrian obaja muis, wei lu, and chen hui ong. 2017. malwaretextdb: a database for annotated malware articles. in acl. shulin liu, yubo chen, shizhu he, kang liu, and jun zhao. 2016b. leveraging framenet to improve automatic event detection. in acl. shulin liu, yubo chen, kang liu, and jun zhao. 2017. exploiting argument information to improve event detection via supervised attention mechanisms. in acl. weiyi lu and thien huu nguyen. 2018. similar but not the same -word sense disambiguation improves event detection via neural representation matching. in emnlp. tomas mikolov, ilya sutskever, kai chen, greg corrado, and jeffrey dean. 2013b. distributed representations of words and phrases and their compositionality. in nips. teruko mitamura, zhengzhong liu, and eduard hovy. 2015. overview of tac kbp 2015 event nugget track. in tac. sudip mittal, prajit kumar das, varish mulwad, anupam joshi, and tim finin. 2016. cybertwitter: using twitter to generate alerts for cybersecurity threats and vulnerabilities. in asonam. thien huu nguyen, kyunghyun cho, and ralph grishman. 2016a. joint event extraction via recurrent neural networks. in naacl. thien huu nguyen, lisheng fu, kyunghyun cho, and ralph grishman. 2016b. a two-stage approach for extending event detection to new types via neural networks. in proceedings of the 1st acl workshop on representation learning for nlp (repl4nlp). thien huu nguyen and ralph grishman. 2015. event detection and domain adaptation with convolutional neural networks. in acl. thien huu nguyen and ralph grishman. 2016e. modeling skip-grams for event detection with convolutional neural networks. in emnlp. thien huu nguyen and ralph grishman. 2018. graph convolutional networks with argument-aware pooling for event detection. in aaai. thien huu nguyen, adam meyers, and ralph grishman. 2016g. new york university 2016 system for kbp event nugget: a deep learning approach. in proceedings of text analysis conference (tac). trung minh nguyen and thien huu nguyen. 2019. one for all: neural joint modeling of entities and events. in aaai.