Active Defense Against Social Engineering: The Case for Human Language Technology (2020)

Abstract: We describe a system that supports natural language processing (NLP) components for active defenses against social engineering attacks. We deploy a pipeline of human language technology, including Ask and Framing Detection, Named Entity Recognition, Dialogue Engineering, and Stylometry. The system processes modern message formats through a plug-in architecture to accommodate innovative approaches for message analysis, knowledge representation and dialogue generation. The novelty of the system is that it uses NLP for cyber defense and engages the attacker using bots to elicit evidence to attribute to the attacker and to waste the attacker’s time and resources.

## introduction panacea (personalized autonomous agents countering social engineering attacks) actively defends against social engineering (se) attacks. active defense refers to engaging an adversary during an attack to extract and link attributable information while also wasting their time and resources in addition to preventing the attacker from achieving their goals. this contrasts with passive defenses, which decrease likelihood and impact of an attack @xcite but do not engage the adversary. se attacks are formidable because intelligent adversaries exploit technical vulnerabilities to avoid social defenses, and social vulnerabilities to avoid technical defenses @xcite . a system must be socially aware to find attack patterns and indicators that span the socio-technical space. panacea approaches this by incorporating the f3ead (find, fix, finish, exploit, analyze, and disseminate) threat intelligence cycle @xcite . the find phase identifies threats using language-based and message security approaches.the fix phase gathers relevant and necessary information to engage the adversaries and plan the mitigations that will prevent them from accomplishing their malicious goals. the finish phase performs a decisive and responsive action in preparation for the exploit phase for future attack detection. the analysis phase exploits intelligence from conversations with the adversaries and places it in a persistent knowledge base where it can be linked to other objects and studied additional context. the disseminate phase makes this intelligence available to all components to improve performance in subsequent attacks. panacea's value comes from nlp capabilities for cyber defense coupled with end-to-end plug-ins for ease of running nlp over real-world conversations. figure 1 illustrates panacea's active defense in the form of conversational engagement, diverting the attacker while also delivering a link that will enable the attacker's identity to be unveiled. good to know. how do i go about the process of checking eligibility? the website is not opening. did you provide me with the right link? is this the one? link regards........ school employees: did you know that you may be able to get up to a 20% discount on your cell phone bill every month? click here to check your eligibility with telcograph, dashnet, altus wireless and u.s. cellnet............ ask and framing detection + response generation figure 1: active defense against social engineering: attacker's email (left) yields bot's response (right) ## related work security in online communication is a challenge due to: (1) attacker's speed outpacing that of defenders to maintain indicators @xcite ; prior work extracts information from email interactions @xcite , applies supervised learning to identify email signatures and forwarded messages @xcite , and classifies email content into different structural sections @xcite . statistical and rulebased heuristics extract users' names and aliases @xcite and structured script representations determine whether an email resembles a password reset email typically sent from an organization's it department @xcite . analysis of chatbot responses (prakhar gupta and bigham, 2019) yields human-judgement correlation improvements. approaches above differ from ours in that they require extensive model training. our approach relates to work on conversational agents, e.g., response generation using neural models @xcite , topic models @xcite , self-disclosure for targeted responses @xcite , topic models @xcite , and other nlp analysis @xcite . all such approaches are limited to a pre-defined set of topics, constrained by the training corpus. other prior work focuses on persuasion detection/prediction @xcite but for judging when a persuasive attempt might be successful, whereas panacea aims to achieve effective dialogue for countering (rather than adopting) persuasive attempts. text-based semantic analysis is also used for se detection @xcite , but not for engaging with an attacker. whereas a bot might be employed to warn a potential victim that an attack is underway, our bots communicate with a social engineer in ways that elicit identifying information. panacea's architecture is inspired by state-ofthe-art systems in cyber threat intelligence. misp @xcite focuses on information sharing from a community of trusted organizations. mitre's collaborative research into threats (crits) @xcite platform is, like panacea, built on top of the structured threat intelligence exchange (stix) specification. panacea differs from these in that it is part of operational active defenses, rather than solely an analytical tool for incident response and threat reporting. ## system overview panacea's processing workflow is inspired by stanford's corenlp annotator pipeline @xcite , but with a focus on using nlp to power active defenses against se. a f3ead-inspired phased analysis and engagement cycle is employed to conduct active defense operations. the cycle is triggered when a message arrives and is deconstructed into stix threat intelligence objects. object instances for the identities of the sender and all recipients are found or created in the knowledge base. labeled relationships are created between those identity objects and the message itself. once a message is ingested, plug-in components process the message in the find phase, yielding a response as a json object that is used by plugin components in subsequent phases. analyses performed in this phase include message part decomposition, named entity recognition, and email header analysis. the fix phase uses components dubbed deciders, which perform a meta-analysis of the results from the find phase to determine if and what type of an attack is taking place. ask detection provides a fix on what the attacker is going after in the fix phase, if an attack is indicated. detecting an attack advances the cycle to the finish phase, where response generation is activated. each time panacea successfully elicits a response from the attacker, the new message is exploited for attributable information, such as the geographical location of the attack and what organizational affiliations they may have. this information is stored as structured intelligence in the knowledge base which triggers the analysis phase, wherein the threat is re-analyzed in a broader context. finally, panacea disseminates threat intelligence so that humans can build additional tools and capabilities to combat future threats. ## under the hood panacea's main components are presented: (1) message analysis component; and (2) dialogue component. the resulting system is capable of handling the thousands of messages a day that would be expected in a modern organization, including failure recovery and scheduling jobs for the future. figure 2 shows panacea throughput while operating over a one month backlog of emails, sms texts, and linkedin messages. ## evaluation friend/foe detection (message analysis) and response generation (dialogue) are evaluated for effectiveness of panacea as an effective intermediary between attackers and potential victims. ## conclusions and future work panacea is an operational system that processes communication data into actionable intelligence and provides active defense capabilities to combat se. the f3ead active defense cycle was chosen because it fits the se problem domain, but specific phases could be changed to address different problems. for example, a system using the panacea processing pipeline could ingest academic papers on a disease, process them with components designed to extract biological mechanisms, then engage with paper authors to ask clarifying questions and search for additional literature to review, while populating a knowledge base containing the critical intelligence for the disease of interest. going forward, the plan is to improve panacea's plug-in infrastructure so that it is easier to add capability without updating panacea itself. this is currently possible as long as new components use the same rest api as existing components. the obvious next step is to formalize panacea's api. we have found value to leaving it open at this early state of development as we discover new challenges and solutions to problems that emerge in building a large scale system focused on the dangers and opportunities in human language communication.