Noise Contrastive Estimation-based Matching Framework for Low-Resource Security Attack Pattern Recognition (2024)

Abstract: Techniques, Tactics and Procedures (TTP) mapping is an important and difficult task in the application of cyber threat intelligence (CTI) extraction for threat reports. TTPs are typically expressed in semantic forms within security knowledge bases like MITRE ATT&CK, serving as textual high-level descriptions for sophisticated attack patterns. Conversely, attacks in CTI threat reports are detailed in a combination of natural and technical language forms, presenting a significant challenge even for security experts to establish correlations or mappings with the corresponding TTPs.Conventional learning approaches often target the TTP mapping problem in the classical multiclass/label classification setting. This setting hinders the learning capabilities of the model, due to the large number of classes (i.e., TTPs), the inevitable skewness of the label distribution and the complex hierarchical structure of the label space. In this work, we approach the problem in a different learning paradigm, such that the assignment of a text to a TTP label is essentially decided by the direct semantic similarity between the two, thus, reducing the complexity of competing solely over the large labeling space. In order that, we propose a neural matching architecture that incorporates a sampling based learn-to-compare mechanism, facilitating the learning process of the matching model despite constrained resources.

## introduction and background cyber threat intelligence (cti), an essential pillar of cybersecurity, involves collecting and analyzing information on cyber threats, including threat actors, their campaigns, and malware, helping timely threat detection and defense efforts. textual threat reports or blogs are considered a important source of cti, where security vendors diligently investigate and promptly detail intricate attacks. a key sub-task in extracting cti from these textual sources involves the identification of tactics, techniques, and procedures (ttp) of the threat actors, i.e. comprehending descriptions of low-level, complex threat actions and connecting them to standardized attack patterns. one of the popular standard knowledge frameworks widely adopted in the cti community is mitre att&ck @xcite . within this framework, a technique repre- @xcite we witnessed that the botnet was spread via mass phishing, using a vb-scripted excel attachment to download the second stage from xx.warez22.info. the same domain was used for c&c via http. the botnet distributed a file encryption module we named vbenc. [...] sents a specific method used to achieve an objective, with its corresponding tactics and sub-techniques covering broader strategies and variations. fig. 1 illustrates an example of a text in a threat report, which indicates two attack patterns, among others, i.e., @xcite the use of a malicious email attachment to take control of a victim's system (t1566 foot_0 ), and (2) encrypting data on the victim's system, presumably for ransom demands (t1486 foot_1 ). as of 2024, there are over 600 techniques, together with 14 high-level tactics described in mitre att&ck. in its ontology, a technique is associated to at least one tactic (e.g., the technique "hijack execution flow" is listed under three distinct tactics: persistence, privilege escalation and defense evasion) and may have several sub-techniques. mining techniques from cti reports poses significant challenges due to several factors. firstly, the large number of techniques, coupled with their diverse nature, intricate interdependencies, and hierarchical structure, renders the task complex and laborious. secondly, the analysis of cti reports necessitates the expertise of security professionals. the reports focus on delineating low-level threat actions rather than explicitly mentioning the associated techniques and tactics. consequently, extracting relevant techniques and tactics from these reports requires diligent inference by the reader. employing an automated approach to ttp mapping presents inherent chal-lenges. one major hurdle is the low-resource nature of the task, due to the limited availability of labeled data and the extensive label space. moreover, the presence of long-tail infrequent ttps adds complexity to the learning process. due to these challenges, ttp mapping has not been fully solved in related work. most recent works use a classical document-level multilabel @xcite or sentence-level multi-class classification @xcite learning setting. these granularity choices, however, either introduce unneeded complexity of long-form text representation (for document-level) or make the task inapplicable to mapping complex ttps, which often require longer text (for sentencelevel). moreover, the main learning issues in these settings are: (i) the aforementioned problems of label scarcity and long-tailedness, and (ii) the learning complexity costs of the softmax-based learning approaches grow proportionally to the number of classes. in the wider literature i.e., extreme multilabel text classification (xmtc), the problems are addressed by (i) capturing the label correlation and (ii) partitioning and handling the sub-label spaces separately. they are, however, most effective in relatively resource-rich settings, and have drawbacks when applied to label-scarce scenarios, as the signal-to-noise ratio increases @xcite . in the multi-label context, learning is greatly affected, additionally, by the frequent presence of missing labels, which is a common trait observed in human-curated datasets. in this work we propose an alternative learning setting which avoids the direct optimization for discriminating between data points in a large label space. concretely, we transform the task into a text matching problem @xcite , allowing us to utilize the direct semantic similarity between the input-label pairs to derive a calibrated assignment score. the framework inherently incorporates an inductive bias, encouraging the capture of nuanced similarities even in the presence of limited labeled data, enhancing its ability to generalize to long-tail ttps. this transformation is achieved by leveraging the textual profile of a ttp (i.e., textual description foot_2 in att&ck), a resource that is often neglected in related work. label-efficient text matching: our approach -dynamic label-informed text matching -empow-ered by noise contrastive estimation (nce) (gutmann and hyvärinen, 2010), exploits the shared information between a pair of texts (text matching) in the learning phase, and altogether attempts to discriminate between the positive labels versus the rest in the label space (classification). conventionally, nces are used to alleviate computational challenges in parameter estimation for large target spaces. in this work, we apply nces uniquely in a moderately sized label space, navigating data scarcity and noise constraints. we demonstrate experimentally that our ranking-based nces, characterized by their probabilistic nature and ability to capture global patterns, can overcome these low-resource constraints and help the matching model perform particularly well. in contrast, common contrastive loss variants, i.e., triplet losses lacking these properties, surprisingly performed even worse than we anticipated. to this end, we summarize our contributions: ttp mapping and cti extraction several works target ttp mapping on the document level. (husari et al., 2017) used a probabilistic relevance framework (okapi bm25) to quantify the similarity between bow representations of ttps and the target text. however, this approach is limited to the oversimplified vocabulary of threat actions within an ad-hoc ontology. ayoade et al. (2018); niakanlahiji et al. (2018) used a tf-idf-based document representation and leveraged classical (i.e., treebased, margin-based) ml for (multi-label) classification. li et al. ( 2019 ) used latent semantic analysis to extract topics from target articles, and compared the topic vectors with the tf-idf vectors of att&ck description pages to obtain cosine similarity. they used the similarity vectors with naïve bayes and decision trees to classify ttps. however, the choice of document-level granularity introduces additional unneeded complexity of longform text representation. recent works leverage transformers for sentence-level text representation learning @xcite , using the encoded representation in the multi-class classification setting. however, with limited available data, they restrict the task to only a small number of ttps. extreme multi-label text classification. xmtc, or generally extreme multi-label classification is a line of research targeting extremely large label spaces, e.g., product categorization in e-commerce or web page categorization. the main challenges for xmtc are computational efficiency and data skewness. common techniques for xmtc are tree-based @xcite @xcite , sampling-based @xcite and embeddingbased @xcite that attempt to partition the label space and thus reduce the computational complexity. however, generally, these methods assume the sufficient availability of supervision and still suffer in the long-tail performance. ## preliminaries and problem setup we first provide a brief overview of the classification settings with noise contrastive estimation (nce). these definitions then subsequently help us in formulating our matching problem. classification: let x and y denote the input and label spaces, |y| < ∞. we define a score function g θ : @xmath0. in this setting, the label space y is categorical. specifically, @xmath1, whereas @xmath2, 1} n×|l| , with n being the number of samples and l being the label set. matching: in this setting, x and y represent the same input space. the matching function g θ : x × @xmath3, is differentiable in θ ∈ r |d| , where d is the parameter space. in order to cast a classification problem as a matching one, we assume there is an invertible and smooth projection function π that transforms the discrete categorical representation y into the same continuous space as x. cross-entropy loss and nce: in either classification or matching settings, our goal is to estimate whether θ : @xmath4, y) has optimal 0-1 loss. this can be reduced to conditional density estimation. let p θ (y|x) = exp(g θ (x,y) ŷ∈y exp(g θ (x,ŷ)) , the cross-entropy loss is then defined as: when y is large, j ce (θ) is difficult to compute as the computation of the normalization term of p θ (y|x) becomes expensive. this issue is addressed by nce through sub-sampling p(x, y ), and shifting the focus towards estimating the probabilities of the true data samples. multi-label classification. the vanilla classification problem can be defined as follows: let {x, y } be the problem space, where the feature space @xmath5, and the label space @xmath6, 1} n×|l| , with |l| ≪ ∞ being the number of ttps in the kb. the goal is to learn a function f : @xmath7, given @xmath8. problem reformulation. given the training data @xmath9, and @xmath10, with @xmath11, and |l| ≪ ∞ along with a set of supervisions {@xmath12, 1} n , such as @xmath13, our target is to learn matching-based scoring functions g θ (x, y) that model the relationship between x and y within the same feature space, aiming for g θ (x, y) ≈ {@xmath14. the use of the textual profile inherently eliminates the need for a projection function π, as it directly aligns the discrete categorical representation y with the same continuous space as x. in the context of cross-entropy loss, p θ (y|x) is now linked to p θ (@xmath15, y). ## methodology here we describe our architectural choice for the matching function g θ (x, y), and our learning paradigm that approximates p θ (@xmath16, y) to simultaneously match and compare ttps labels. ## conclusion we proposed a solution for the ttp mapping task that overcomes low-resource challenges in security domain. this new learning paradigm integrates the inductive bias into the classification task, resulting in significant out-performance of strong baselines.