T echnique RAG : Retrieval Augmented Generation for Adversarial Technique Annotation in Cyber Threat Intelligence Text (2025)

Abstract: Accurately identifying adversarial techniques in security texts is critical for effective cyber defense. However, existing methods face a fundamental trade-off: they either rely on generic models with limited domain precision or require resource-intensive pipelines that depend on large labeled datasets and task-specific optimizations—such as custom hard-negative mining and denoising—resources rarely available in specialized domains.We propose TechniqueRAG, a domain-specific retrieval-augmented generation (RAG) framework that bridges this gap by integrating off-the-shelf retrievers, instruction-tuned LLMs, and minimal text–technique pairs. Our approach addresses data scarcity by fine-tuning only the generation component on limited in-domain examples, circumventing the need for resource-intensive retrieval training. While conventional RAG mitigates hallucination by coupling retrieval and generation, its reliance on generic retrievers often introduces noisy candidates, limiting domain-specific precision. To address this, we enhance retrieval quality and domain specificity through zero-shot LLM re-ranking, which explicitly aligns retrieved candidates with adversarial techniques.Experiments on multiple security benchmarks demonstrate that TechniqueRAG achieves state-of-the-art performance without extensive task-specific optimizations or labeled data, while comprehensive analysis provides further insights.

## introduction uncovering new adversarial behaviors is critical for strengthening defenses against rapidly evolving cyber threats. these behaviors, defined by the tools, techniques, and procedures used by attackers, reveal how adversaries plan and execute attacks and impact systems and data. by identifying and analyzing the traces or artifacts left behind, security analysts can map low-level actions to higherlevel concepts, such as tactics (i.e., strategic ob-example of text to (sub-)techniques annotated pairs monero miner scripts are downloaded from teamtnt's server and piped to "bash" using a ssh session as the "root" user with private key from "/tmp/teamtnt." ---------------------1. t1098.004: ssh authorized keys 2. t1195: supply chain compromise 3. t1059.004: unix shell 4. t1021.004: remote services: ssh 5. t1552.004: private keys jectives like "lateral movement"), techniques or fine-grained sub-techniques (i.e., tactical methods like "debugger evasion"), and procedures (i.e., implementation details like "using powershell for credential dumping"). the findings are shared with other experts through public channels and threat intelligence services via security reports and detailed descriptions, to strengthen defenses, anticipate possible threats, and improve incident response. the mitre att&ck framework has established itself as the industry standard for categorizing and mapping adversarial behaviors @xcite . this framework provides a comprehensive knowledge base of adversarial tactics, techniques, and procedures (ttps), built from realworld threat intelligence and incident response data. it uses a hierarchical matrix structure to systematically organize and classify adversary behaviors. the broad adoption of the att&ck framework presents a significant operational challenge: security analysts must manually map ambiguous threat descriptions from incident reports (such as the example shown in fig 1 ) to standardized att&ck (sub)-techniques-a time-intensive process that demands expert knowledge. this manual task has motivated research into automated adversarial technique identification, which aims to systemati-cally label text segments with their corresponding att&ck technique and sub-technique ids. prior approaches for (sub-)technique annotation adopt two primary paradigms: (1) multi-class classification that directly map text to (sub-)technique ids @xcite , which, while straightforward to implement, struggle with class imbalance and require extensive labeled training data; and (2) retrieval/ranking approaches that evaluate semantic similarity between the text and (sub-)techniques. early methods like ladder @xcite & attackkg @xcite introduce basic similarity-based ranking. text2ttp @xcite advanced this through hierarchical re-ranking with fine-tuned embeddings, while nce @xcite improved similarity learning using dual-encoder architectures. recently, intelex @xcite employed llms in both retrieval and zero-shot learning settings to assess (sub-)technique relevance. although promising, the current methods are constrained by a critical trade-off: they either rely on general-purpose models lacking domain expertise or require large-scale labeled datasets and computationally intensive training pipelines. retrieval approaches, for instance, require extensive hardnegative mining to distinguish fine-grained (sub-)techniques while classification models demand curated, balanced training sets-resources rarely available in this specialized domain. compounding this issue, despite mitre att&ck framework defines over 550 adversarial (sub-)techniques, only approximately 10,000 annotated examples are publicly available @xcite , severely limiting generalization. to address these dual challenges, we propose techniquerag, a domain-specific retrievalaugmented generation (rag) framework for (sub-)techniques annotation task that bridges generic and specialized models while eliminating dependency on resource-intensive pipelines or large labeled data. unlike conventional approaches, tech-niquerag integrates three key components: in experiments, we evaluate our framework on three benchmarks, addressing both single-label and multi-label prediction tasks for (sub-)techniques. results demonstrate that techniquerag significantly outperforms various baseline approaches, including classification-based, retrieval/rankingbased, and hybrid methods. furthermore, it achieves comparable or superior performance to vanilla rag approaches, even when utilizing powerful llms like gpt-4o. ## related work ttpdrill @xcite matching/ranking employs tf-idf and bm25 for text retrieval attackg @xcite matching/ranking leverages knowledge graph representations tim @xcite classification incorporates textual and lexical features ladder @xcite matching/ranking uses sentence encoder embeddings nce @xcite matching/ranking applies task-adapted dual-encoder embeddings text2ttp @xcite matching/ranking enhances dual-encoder retrievals with a cross-encoder embedding filter model intexex @xcite hybrid (retrieval and evaluation) combines sentence/entity-based search with llm-based evaluation of candidate outputs techniquerag (ours) retrieval augmented generation (rag) integrates any retriever with llm-based re-ranking and fine-tuned generation neural text embedding approaches transformer-based language models (reimers and gurevych, 2019) enabled semantic similaritybased technique identification through neural embeddings. early approaches used pre-trained encoders to embed threat behaviors, either handling multi-sentence descriptions @xcite or specific attack patterns @xcite , evaluating relevance through embedding similarity. @xcite advanced this through a multi-stage architecture combining fine-tuned cross-encoder and dual-encoder models to balance effectiveness and efficiency. @xcite further developed this approach using a dual-encoder architecture with alignment components, leveraging both scratch-trained embeddings and domain-specific models. ## method we present techniquerag, a domain-specific retrieval-augmented generation (rag) framework for adversarial technique (or sub-technique) annotation. unlike conventional approaches that rely on task-specific optimizations and extensive labeled data, techniquerag effectively integrates retrievers, instruction-tuned llms and small-scale text-techniques paired data. we first provide an overview of our approach, followed by details on the key components: (i) retriever and (ii) llmbased re-ranking (ii) generator fine-tuning. fig 2 presents an overview of our system. ## conclusion annotating threat intelligence texts with adversarial techniques from the mitre att&ck framework is a manual and time-intensive task that security analysts must perform daily. its automation requires methods capable of accurately identifying techniques and sub-techniques across hundreds of possibilities while handling complex security terminology, diverse text formats, and limited labeled data. we introduce techniquerag a retrievalaugmented fine-tuning approach designed to tackle these challenges effectively. our comparative analysis demonstrates that techniquerag establishes a new state-of-the-art, outperforming both semantic ranking models and other llm-based methods in adversarial technique annotation. 20920 obtaining large, balanced parallel datasets of threat descriptions and ground truth technique annotations remains a significant challenge due to the reliance on domain expertise for accurate annotation. although our approach mitigates data scarcity, two key limitations may impact performance: 1. limited technique coverage: coverage of techniques is often insufficient. even the mitre att&ck knowledge base lacks procedural examples for many techniques and sub-techniques. 2. sparse technique annotations: existing datasets typically contain very few technique annotations per example, with many instances in our data having only a single technique label. during fine-tuning, this bias toward minimal technique labeling limited our method's ability to generalize effectively. to mitigate this, we oversampled examples with multilabel technique annotations. however, our method rarely assigned more than two technique labels per input query, leading to low recall, particularly on the expert dataset, which consists almost exclusively of multi-label examples. 3. annotation inconsistencies some model predictions marked as errors represent valid technical interpretations not included in gold standard annotations. for example, the following sentence: "smokedham was observed using ultravnc to establish a connection to the ip address and port pair 81.91.177[.]54[:]7234 that has been observed in past unc2465 intrusions." had t1571: non-standard port as the only ground truth label. however, if we analyze it carefully, we see that the threat actor used ultravnc, so t1021.005: remote services -vnc exists in the given description. our model correctly predicted it, but missed the t1571: non-standard port. this highlights challenges in maintaining consistent annotation standards for complex attack patterns.