In the Man-At-The-End (MATE) attack model, attackers have white-box access to software, because they have full control over the devices on which they attack the software. Their toolbox includes simulators, debuggers, disassemblers, and all possible kinds of static and dynamic analysis and tampering tools. They use those tools to reverse-engineer the software and to tamper with it in order to violate security requirements of assets embedded in the software. For example, they try to steal algorithms or encryption keys, or alter their capabilities in games or bypass license checks.
Advances in cryptography have yielded techniques such as multilinear jigsaw puzzles [1] that provide strong security guarantees based on concepts like homomorphic encryption [2] , but those introduce orders of magnitude performance overhead [3] , so in practice they are rarely usable today. Instead, MATE Software Protection (SP) is dominated by fuzzy concepts and techniques [4] . SPs such as remote attestation, obfuscation, anti-debugging, software diversity, and anti-emulation do not aim to prevent MATE attacks completely. Instead they aim to delay attacks and put off potential attackers by making attacks expensive enough, and the expected Return In Investment (ROI) low enough. As observed Daniele Canavese, Leonardo Regano, and during a recent Dagstuhl seminar on SP Decision Support and Evaluation Methodologies [5] , this field is in an abysmal state. Security-through-obscurity is omnipresent in industry; SP tools and consultancy are expensive and opaque; there is no commonly accepted method for evaluating SPs; they are not deployed sufficiently [6] , [7] , [8] , [9] ; expertise is largely missing in software vendors to deploy (third-party) SP tools [10] , [11] , [12] ; and we lack standardization. The National Institute of Standards and Technology (NIST) SP800-39 IT systems risk management standard [13] or the ISO27k framework for information risk management [14] , which are deployed consistently in practice to secure corporate computer networks, have no counterpart or instantiation in the field of MATE SP. Neither do we have concrete regulations to implement General Data Protection Regulation (GDPR) compliance in applications.
With this paper, we hope to instigate the necessary actions towards a standardized risk management approach. Furthermore, we address the issue of automating the application of the SP risk analysis. As software developers do not have the resources to manage a risk analysis because of a lack of in-house competences in SP techniques and because there are no regulations forcing them, an automatic process could reduce the required effort and expertise, and make SP accessible. Towards these goals, this paper offers a number of contributions.
First, we provide a rationale for adopting and standardizing risk management processes for SP. We discuss a number of observations on the failing market of SP. We analyse why the existing standards as adopted in network security are not applicable for MATE SP, i.e., what makes MATE SP different from network security from the perspective of risk management.
Secondly, we discuss in depth how to adopt the NIST risk management approach. For all the required risk management processes, we highlight (i) the current status; (ii) the softwareprotection-specific concepts and aspects that need to be covered;
(iii) what existing parts can be borrowed from other fields; (iv) where we see open questions and challenges that require further research; (v) where we see the need for the research community and industry to come together to define standards; and (vi) the relevant aspect towards formalizing and automating the discussed risk management activities.
Last but not least, and despite not yet having a complete picture or a complete instantiation or standardization of the whole proposed methodology, we demonstrate that several aspects can already be formalized and automated. To that extent, we present a Proof of Concept (PoC) decision support system that automates some of the major risk management activities and processes. This tool is able to drive the identification, assessment, and mitigation of risks. Even if not completely automated, it provides an excellent This paper was submitted to IEEE TDSC © 2020 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting or republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. starting point for protecting applications and for building a more advanced system that follows all the methodological aspects of a NIST 800-compliant standard and reaches industrial grade maturity. The first results obtained with the tool have been validated by industry experts on case studies of real-world complexity.
The remainder of the paper is structured along those three contributions.
We first discuss some risk management standards from other security domains such as network security, and the healthy market for products and services that exists there as a result. We then contrast this with the lack of such a market and standards in the domain of MATE SP, for which we discuss the challenges to make progress towards proper risk management standards. Finally, we highlight what benefits such progress can bring.
Protecting software may be more formally defined by framing it as a risk management process, a customary activity in various industries such as finance, pharmaceutics, infrastructure, energy and Information Technology (IT). Regarding the latter, the NIST has proposed an IT systems risk management standard that sets the context and identifies four main phases [13] : (i) risk framing: to establish the scenario in which the risk must be managed; (ii) risk assessment: to identify threats against the system assets, vulnerabilities of the system, the harm that may occur if those are exploited, and the likelihood thereof; (iii) risk mitigation: to determine and implement appropriate actions to mitigate the risks; and (iv) risk monitoring: to verify that the implemented actions are effective in mitigating the risks.
The ISO27k framework focuses on information risk management as well, to be managed in three phases [14] : (i) identify risk to identify the main threats and vulnerabilities that loom over the assets; (ii) evaluate risk to estimate the impact of the consequences of the various risks; and (iii) treat risk to mitigate the risks that can be neither accepted nor avoided. The ISO27k framework adds an explicit operational phase for handling changes that may happen in the framed risk scenario.
Those approaches have been consistently applied in practice for securing corporate networks. Regulations stimulated companies to analyse the risks against their IT systems. For instance, the GDPR explicitly requires a risk analysis of all private data handling. Companies invest in compliance with the ISO27k family, as it provides market access. As a consequence, risk analysis of networks has developed a common vocabulary and a company's tasks have been properly identified and often standardized, so offerings from consultancy firms can be compared easily. There is a business related to this task, best practices, and big consultant firms have risk analysis of corporate networks in their catalogs [15] .
Compared to network security, the SP field has years of delay. The market of SP is neither open nor accessible to companies with a limited budget. In 2017 Gartner projected that 30% of enterprises would have used SP to protect at least one of their mobile, IoT, and JavaScript critical applications in 2020 [16] . However, Arxan recently reported that 97% (and 100% of financial institution) of the top 100 mobile apps are easy to decompile as they lack binary code protection [6] or implement weak protection. Another study confirms the absence of both anti-debugging and anti-tampering protections for 59% of about 38k Google Play Store apps. The study highlights that weak Java-based methods are employed in 99% of the cases where SPs are applied [7] . Malicious apps derived from poorly protected benign apps to lure in victims are submitted to app stores at an incredibly high pace [17] . Furthermore, the BSA Global Software Piracy Study [9] estimated that 37% of installed software is not licensed.
Cybersecurity competences are lacking [10] and SP is no exception. An Irdeto survey confirms that few companies have internal SP teams. Only 7% of respondents stated their organization has everything it needs to tackle cybersecurity challenges, and 46% stated they need additional expertise/skills within the organization to address all aspects of cybersecurity [11] . Meanwhile many organizations lack competent staff, budget, or resources to protect their applications and systems [12] .
When the value of assets justifies it, developers hence resort to paying third parties to protect their software. However, the price tag for is typically high, involving licenses to tools and often require training by and access to expert consultants. Moreover, the services and the strength of the obtained SP are covered by a cloak of opaqueness. Security-through-obscurity is still omnipresent. For example, whereas early white-box cryptography schemes were peer reviewed [18] , [19] and then broken [20] , [21] , we could not find peer reviewed analyses of white-box crypto schemes currently marketed by big SP vendors. Moreover, most vendors' licenses forbid the publication of reverse engineering and pen testing reports on their products, and they don't share their internal procedures, tools, and reports with the academic community. As some of the big SP vendors are also active in other fields of (ICT) security, risk analysis and mitigation is almost certainly the principle that drives their SP experts. Yet a methodology for applying a risk analysis process for MATE SP is not publicly available.
From all of this evidence, we deduce that either many organizations/companies do not understand the risk and therefore do not feel the need for deploying SP; or they do not have the internal competences and knowledge to deploy it properly; or they do not have enough money to pay third-party SP providers. In short, there exists no widely accessible, functional, transparent, open SP market. Needless to say then, there is not a standard process that may guarantee the proper selection and application of available SPs given a case at hand.
A methodological approach to risk analysis of software with respect to MATE attacks would have a plethora of benefits.
First, it would force stakeholders to follow a more rigorous approach to SP. This would increase the transparency of all the protection phases, hence guaranteeing a more reliable estimation of the reached protection level and of the quality delivered by third parties. In turn, we should expect less reliance on the security-through-obscurity principle. Having a standard would finally induce the community to use well-defined terminology and to agree upon the meaning of each term.
Building a common ground and well-defined playing rules would also benefit the SP market by creating a more open and transparent ecosystem where SP services can be compared as normal products, thus bridging the gap with the existing network security market. We would also expect a reduction of the prices of SPs and services in such a market. With a lower entry price, more companies can then afford professional SP services. Moreover, we expect the raise of consultancy firms that can evaluate the effectiveness of SPs independently.
When SP becomes standardized and the different aspects thereof hence become more clearly defined, it can also create a market for decision support products that automate the MATE risk management. This would in turn lead to cost savings and hence more accessible and more effectively deployed SP.
As experienced elsewhere, the availability of standards increases awareness. The simple existence of a standard would initially inform people about the need for protecting software. Compliance would then force all involved parties to obtain indepth knowledge about all relevant aspects. The standards and related best practices would therefore also be incorporated in educational programs. Moreover, a standard would mitigate the risk for weak SP that is now evident in developers and companies that cannot afford services from vendors.
The work towards standards would also have an impact on research. It would initially stimulate the community to focus on identifying and addressing the missing parts, and later on creating new or more effective and scientifically validated SPs that can be integrated into a standard framework and hence also becomes more reproducible. The interest in the field and the impact of work and publications would then likely attract more researchers in the SP field, which is now marginal in the software engineering community. Moreover, after the task of SP is properly framed, the impact on current SDLC practices needs to be evaluated, thus opening new research directions in software engineering.An increased attention by research institutions and academia usually translates in better education opportunities, possibly with dedicated curricula on SP, which usually pair well with the career opportunities created by a more open market. Ultimately, this would help companies to employ skilled people and support a freer job market that could partially compensate the lack for SP experts.
In the end, the benefit would extend to the whole community, as having better protected software reduces the global exposure of citizens to risks and, we hope, would make MATE attacks a less lucrative field, or at least reduce its growth.
Despite the many benefits a standardized risk management approach would bring, such an approach currently is a long way off. Adopting a NIST-style risk management is a true challenge because we currently lack even the most basic common MATE vocabulary, but also because the assessment of exposure to threats needs to be tailored on the application to protect, and because the deployment of MATE SPs is much more complex than securing networks.
Today, we even lack a definition of asset categories. While the operations of a Man-In-The-Middle (MITM) attacker are well defined, SP relies on the MATE attacker model that has never been defined formally. The abilities of MATE attackers are unclear, not in the least because formally modelling the fuzzy concept of code comprehension by humans and the ability to tamper with a program are more sophisticated than representing MITM operations. Estimating the feasibility of attacks requires a white-box analysis of the assets and of the entire application. The complexity of mounting static, dynamic, symbolic, and concolic attacks depends heavily on the structure and artefacts of the software under attack, such as the occurrence of certain patterns in code and data or observable invariants in static code or in execution traces.
Unsurprisingly then, SPs as deployed in practice also provide only fuzzy forms of protection, not well-defined ones. There currently is no well-defined set of categories of security controls to mitigate the risks. SPs have only been categorised in a coarse manner (e.g., obfuscation vs. anti-tampering). In general, it is not clear where they can serve and what security level they offer. By contrast, it is clear what firewalls and VPNs do and how to use them to mitigate network security risks. There are also accepted measures and guidelines to estimate the effectiveness of the different categories of network security mitigations and in some cases also categorization of tools and vendors that help in estimating the effectiveness of these mitigations [22] .
No metrics are available, however, to quantify SP effectiveness. While potency, resilience, and stealth are commonly accepted criteria that should be evaluated [23] , no standardized metrics are available for them. Complexity metrics originating from the field of software engineering have been proposed [24] and adhoc metrics are used in academic papers [25] , [26] , but none have been empirically validated in the context of MATE SP, and none of them are trusted in practice, i.e., are deemed a sufficient replacement for human expertise and pen testing.
Importantly, protecting software is also more complex than protecting networks because SPs are not separate entities that are deployed in front of or in between services and devices. SPs are instead deployed in layers for mutual protection and to exploit synergies. As a result, the code of multiple SPs and of the assets they protect becomes highly interwoven. For example, obfuscation transforms code fragments that can be assets but also other SPs, to make analysis and comprehension of their logic more complex [27] . Some obfuscations even aim for eliminating recognizable boundaries between different components [25] . After their application, we may hence talk of protected assets, certainly not of separated protection and asset entities.
For the deployment of SPs, software internals must be known to the tools, such as the type of instructions, structure and semantics of the code, as well as the aformentioned artefacts that might benefit attacks. This information is important to decide if a (layered) SP may be effective or not, and to tune protection parameters. To make matters worse, it is generally accepted that to deploy SPs effectively, the applications need to be designed with protection of the sensitive assets in mind. If the software architecture is not designed well (e.g., if internal or external APIs are not designed well or authorization between clients and servers is missing), injecting SPs will only provide superficial mitigation of attacks. This again stresses how the effective deployment of SPs requires insights into the internals of the software to be protected. It obviously also requires risk assessment methods to support the analysis of badly designed software, i.e., for badly designed software architectures the risk assessment should at least report that the risk cannot be reduced to the desired objective solely by injecting SPs.
Moreover, in most cases there are no hard proofs that SPs are actually effective in delaying attackers. Instead of encouraging checks from external parties, SP vendors prevent, e.g., with legal contracts the analysis of the code after they have protected it. That is, they still rely on security through obscurity. As a consequence there is neither an objective nor a measurable assurance that the software is well protected, nor an objective evaluation of the work made by these companies. In academic research, the situation is not much better. For example, the seminal obfuscation versus analysis survey from Schrittwieser et al. [27] never refers to a risk analysis framework. Their results, although widely acknowledged to be very useful, can therefore not be immediately used in a decision process.
By contrast, in network security not only are protections considered to be separate elements, but assets are also considered to be black-box when performing a risk analysis, thus facilitating the way individual data are aggregated (e.g., the impact of different attacks or the evaluation of the overall risk status). While the identification of bugs/vulnerabilities in network protection components requires white box analysis (which is, not accidentally, indeed an instance of an attack against software), bug exploitation as treated in a networking scenario does not consider the specific attack paths, only the likelihood that bugs may be exploited and their consequences, like in the case of zero days vulnerabilities, that are considered as incident or hazardous events, in no different way than earthquakes or flooding.
In conclusion, despite their obvious appeal, risk management standardization and a functioning open market as they exist in other areas of ICT security are missing in MATE SP not only because the community is late in developing them, but also because managing the risks is really challenging.
In this section, we discuss what the four phases of the NIST IT systems risk management standard would entail as applied to MATE SP, i.e., what tasks need to be done in each of the four phases, and what concepts and aspects need to be covered in them. We highlight some concrete challenges and discuss where we think existing techniques can be reused.
The objective of risk framing is to define the context in which a concrete risk analysis will be performed. All relevant concepts and aspects need to be determined. To allow for standardization, a common vocabulary first needs to be established that covers all possible scenarios and relevant aspects. It needs to be unambiguous and formalized such that automated support tools can be engineered. In this section, we list the concepts we consider critical to frame the risks that MATE SPs aim to mitigate. Provisioning the complete vocabulary and a methodology to describe the risk frame is of course out of reach here. That will instead need to be done in a larger document that results from a community effort.
The primary assets are all static and dynamic aspects of software of which a MATE attacker might violate security requirements because they have value for the attacker or the vendor of the software (monetary, public image, customer satisfaction, bragging rights, ...). The secondary assets are all static or dynamic software artifacts that attackers might target as part of their attack path towards the primary assets. Assets exist at many different levels of abstraction and granularity because attacks cover a range of them (e.g., comprehending a trade-secret piece of software IP vs. stealing an encryption key). Primary assets are typically already present in the vanilla software, secondary assets can be attacker pivots in the vanilla software, but they can also be artifacts of injected SPs that attackers want to overcome. The distinction between primary assets and secondary assets should not be made strict, however. For example, a cryptographic key that protects one movie might be a secondary asset with respect to that one movie. A similar key that serves as a master key for all movie encryptions is clearly a primary asset. Moreover, SP vendors can consider part of the SPs supported with their tools as being primary assets of which they do not want to see the logic reverse-engineered easily. Moreover, for the deployment of certain SPs, it needs to be possible to describe the relationship between assets and nonasset program artifacts (functions, variables, global data, constants, etc.). This is the case when transformations applied to code of assets require other non-asset code to be transformed with it to maintain the overall program semantics. It is also necessary when deploying a SP on only the assets would make those assets stand out to the attacker, e.g., because the entropy of encrypted data or obfuscated code is much higher than that of plain data or because the protection introduces recognizable fingerprints. To increase the attacker's effort needed to localize them, one can deploy the same SPs on non-assets. In short, we need mechanisms to describe a wide range of software artifacts of assets, of non-assets, and of the relation.
As SP aims to delay attacks rather than to prevent them, we need a way to describe the evolution of assets' values over time, as well as the impact that a successful attack can have on a business model. This includes the renewability of the assets, i.e., how easy it is to replace software to reduce the impact of successfully attacked assets. For modeling this evolving relation between business value and concrete assets, we expect companies to use their existing asset valuation models.
In the risk framing phase, the task for a case at hand is to determine which assets are potentially relevant to the risk management. This needs to be done for all the potential primary and secondary assets that are known a priori, i.e., in the original application, in already deployed SPs, if any, or in any of the SPs that might later be deployed in the mitigation phase.
The primary security requirements for primary assets are most often the non-functional requirements of confidentiality and integrity, both of which come in multiple forms and at different levels of abstraction and granularity. The requirements' scope differs from that in other domains, and existing classifications can hence not be trivially reused. Most importantly, MATE integrity requirements can include constraints on where or how code is executed. Examples of such requirements are that at any point in time, at most one copy of a program is running, or that certain code fragments are not lifted from a program and executed ex situ.
Different requirements may hold for different phases in the Software Development Life Cycle (SDLC) of the software to be protected and of attacks (e.g., attack identification phase versus attack exploitation phase). Some requirements may be absolute (e.g., some master key should not be leaked at all), others may be time-limited (e.g., a key to a live sports event should remain secret for 5 minutes), still others may be relative and economic (e.g., running many copies in parallel without being detected should cost more than licensing those copies).
Evaluating whether or not such primary requirements can be guaranteed is extremely hard in practice because MATE attackers have white-box access to the software and full control over their devices. Secondary security requirements can then help for framing possible risks. The secondary security requirements can be (i) non-functional requirements for secondary assets; (ii) functional requirements that are easier to check but of which the mere presence in itself provides few guarantees, such as the presence of a copy-protection mechanism; (iii) assurance security requirements that minimize the risk that relevant aspects are overlooked; and (iv) what we will call protection policy requirements. The latter are requirements related to worst-case assumptions about attacker capabilities, such as assuming that the mere presence of some artifact or feature suffices to enable certain attacks. Such assumptions to some extent allow to make up for the lack of proper evaluation of primary requirements. For example, a lack of stealth resulting from easily identifiable invariants in injected SPs hints for potential weaknesses vis-à-vis certain attacks [28] . Protection policy requirements then come down to requiring that certain forms of artifacts and features are not present at all. This is similar to security policies in the domain of remote exploitation, where, e.g., code pointer integrity [29] is a policy about handling code pointers that, when implemented properly, ensures that control flow cannot be hijacked by exploits.
In the risk framing phase, the task for a case at hand is to determine the security requirements for all assets and potential weaknesses identified as relevant as discussed above.
It needs to be possible to consider and define attackers with different levels of resources and capabilities; money, expertise, available tools, etc. The latter involve a huge range of methods, span multiple levels of abstraction, and evolve over time, so a living catalog is needed. It is currently not known what level of detail is needed to produce the best results, so both more generic attack methods and tool usage scenarios (e.g., disassembling code) and very concrete ones (e.g., using the IDA Pro 6.2 disassembler) need to be covered. As the goal of SP is to delay attacks, not only the feasibility of successful attacks is to be covered, but also the potential effort involved, possibly including the effort and time that attackers would probabilistically waste in unsuccessful attack strategies.
While research has shown that unsuccessful attack steps are common in real attacks [30] , it is unclear whether attack models that assume worst-case scenarios can be useful or even sufficient. Worst-case assumption examples are attackers being served by an oracle to always choose the right attack path, and analysis tools producing results with ground-truth precision. For example, localizing code of interest is an important, time-consuming attack step that cannot simply be assumed to be performed effortlessly by means of an oracle. Doing so would imply that increasing the stealth of SPs is not useful, which experts certainly reject. Importantly, for each potential attack step considered for a concrete case at hand, it needs to be possible to clarify in clear terms what enables or prevents the attack step (e.g., the presence of certain secondary assets), and what significantly increases or decreases the required time and effort, and the likelihood of success. This might include features of the software under attack, of the environment in which attacks can be performed, but also knowledge obtained by the attacker. The best level(s) of abstraction at which these are to be considered is an open question.
In the risk framing phase, the task for a case at hand is to determine the attacker model precisely, i.e., the different combina-tions of the mentioned attributes that potential attackers in scope might have. Once the attack scope has thus been determined, it is important to frame the set of (quantitative) metrics that will be considered useful in the later phases to approximate the effort/time/resources that attackers will need to invest in the different attack steps in scope. As already discussed in Section 2.4, there is no widely accepted set of metrics at this point in time. More scientific, empirical research is needed to determine which metrics are valid under which circumstances and for which purposes.
Existing attacker models from, e.g., network security risk analysis cannot be reused. MATE attack modeling needs to include manual tasks and human comprehension of code, which are not considered in network security. For example, in network security, the development of zero-day exploits (using tools also found in the MATE toolbox) is handled as an unpredictable event, which sidesteps the complexity of analysing and predicting human activities. This entirely prevents the use of existing assessment models developed for the network security scenario.
It needs to be possible to describe a wide range of available SPs in a unified manner. This needs to include at least possible limitations on their applicability and composibility, be it for layered deployment to protect each other or in another way to exploit synergies between multiple SPs; the security requirements that they can help to enforce; (measurable) features or limitations they have that can enable, slow-down, ease, block, or otherwise impact potential attacks at different levels of abstraction, on the SPs themselves but also on the assets they are supposed to protect; how big those impacts are on the different potential attacks; and potential implementation weaknesses including the ways in which they can fail to meet secondary security requirements, i.e., become (easily) attackable assets themselves; etc. The link to validated (but as of yet still missing) metrics mentioned above is clear.
Also the costs of using a SP need to be considered. This can include the direct monetary costs of SP tools licenses, but also indirect monetary costs such as having to budget for more security servers or having a longer time to market, or any other extra cost that might follow from required changes to the SDLC. In addition, the amount of potential overhead in terms of application performance (latency, throughput, size, ...) needs to be known of all potentially used SPs.
In the risk framing phase, the task for a case at hand is to determine precisely which combinations of SPs can potentially be deployed to mitigate risks, given the available SP tool flow and the relevant properties of the supported SPs.
Besides mitigating attacks, SPs come with side-effects, such as slowing down software, making it bigger, making debugging harder, requiring changes to content or software distribution models, requiring a certain scalability on the side of secure servers, etc. Taking the time to decide on SPs, possibly iteratively with the involvement of experts and time-consuming human analysis, also has an effect on the time to market.
Before choosing the most effective mitigation in the risk mitigation phase, hard and soft constraints need to be listed with respect to quantifiable overheads (i.e., costs) in all possible relevant forms, and with respect to compatibility with SDLC requirements. Moreover, for all available SPs the impact on the relevant features, i.e., how much of the overhead budget they might eat, and how compatible with the SDLC they are, needs to be understood (i.e., estimated) as clarified above.
An issue that complicates matters immensely in practice, is the fact that often vendors of SP tools (hereafter named SP vendors) and users of such tools (hereafter called application vendors) do not trust each other. Both parties hence often put severe constraints on the way the SP tools are deployed on the applications and on the amount of information that they exchange. A SP vendor will typically not be very forthcoming with respect to the weaknesses or internal artifacts of the supported SPs and disallow reverse engineering of them, while the application vendors do not want to share too many internals or code of their software with the SP vendor. Consequentially, only illegitimate attackers will get white-box access to the protected applications in which SPs and original assets are interwoven as discussed in Section 2.4. If the experts performing the risk management lack white-box access to all available SPs and to the protected application, this will have a tremendous impact on the methods and data that can be used during the risk assessment and risk mitigation phases in which attackers with white-box accesses are targeted. This obviously needs to be documented, and the impact thereof needs to be assessed, during the risk framing.
In addition, aspects of the SDLC relevant to the monitoring phase that will be discussed later need to be framed, such as connectivity and updatability. Whether an application is (or can be required to be) always online, occasionally connected, or mostly offline impacts which online protections can be deployed, and hence which monitoring techniques will be available. So does the ability to let application servers (e.g., video streaming servers or online game servers) interact with online security services such as a remote attestation server. Likewise, it is important to document whether updates can be forced upon users, and to what extent the vendors can synchronize all user updates.
Finally, limitations to the environment in which protected applications will be distributed and executed need to be documented. For example, Android allows less freedom than Linux regarding the use of certain OS interfaces, e.g., for debugging, and some device vendors limit what applications can do after being installed, such iOS's limitation on downloading binary code blobs post installment. Such limitations clearly affect the types of SPs that can be deployed, so they need to be included in the risk framing.
To avoid the need for costly human expertise and manual intervention in the following process step, as much as possible information as discussed above needs to be formalized, such that tools can reason about them in the subsequent phases.
In the above discussion of the risk framing phase, the term "potential" occurred frequently. The reason is that in the risk framing phase, all forms of knowledge are still considered in isolation, including potential SP weaknesses, application features, SP tool capabilities, and attacker capabilities. Assessing how those interact for the case at hand, i.e., determining which of all potential risks actually manifest themselves in the software at hand, is the goal of the risk assessment phase.
So first concrete threats and risks need to be identified, starting from an analysis of the assets, their intrinsic weaknesses, and from attack methodologies, how they are instantiated, i.e., their technical attributes that impact their feasibility. Then, a qualitative, semi-qualitative, or preferably a quantitative estimation of the impacts of these threats and a prioritization of the risks needs to be done.
The goal of this phase is to determine a list of the attacks that, if executed successfully by an attacker, would lead to the violation of security requirements expected on each software assets. Therefore, this phase consists of a detailed analysis that outputs a report about the risks in terms of the analyzed attacks that are viable within the relevant time frame (in which assets have value and security requirements), the attack paths of least resistance, the levels and amounts of expertise, effort, and resources they need, the damage caused by exploitation, etc. For each attack path contributing to the major risks, the weaknesses and secondary assets used by attackers as pivots need to be presented, as well as the used assumptions, such as worst-case-scenario considerations or parameters that are unknown in practice. Reporting this information is necessary to enable confidence in the outcome of the assessment. Critically, all of the enumeration and assessment of feasible attack steps must be performed both on the attack identification phase (in the attacker's lab on devices controlled by attackers) and on the attack exploitation phase (often outside their lab, on other user's devices).
Several open issues need to be addressed to perform this identification task correctly in the MATE scenario.
Software assets can be attacked with different strategies, in which attackers rely on automated tools and analyses to collect and exploit information about the software under attack and to represent the software in structured representations. A range of analysis tools and techniques are applicable, all with their own strengths and limitations, including static, dynamic, symbolic, and concolic analyses. Knowledge of the attacker's goals and tools is the starting point to identify and enumerate the possible attack paths. This knowledge includes the kinds of information the different tools can produce, the software features they depend on to produce that information, their weaknesses, limitations, and precision.
In this phase, the defender therefore needs to deploy his own analysis toolbox to determine the technical attributes of the primary assets that are present and that have an impact on the risk of successful attacks because they enable them. This at least includes checking whether the protection policy requirements formulated in the risk framing phase are violated. It also needs to be done for all potential weaknesses that were identified in the framing phase, such as invariants being present in the code that might facilitate certain attack vectors, or other features that result in a lack of stealth or that can serve as proxies for concrete attack opportunities. Moreover, the set of actually present secondary assets needs to be determined to identify the presence of features that make them good pivots for attackers towards the primary assets.
While we are convinced that such defender toolboxes can produce most of the necessary information about the feasible attacks to be enumerated, a number of research questions still need answers. For example, how can the formal pieces of information extracted by the tools be used to identify the viable attack paths precisely, in particular when attackers need to resort to manual efforts that may not be easy to formalize. How do we then assess the required effort and likelihood of success, and perform impact analysis and risk estimation accurately? And to what extent can such automated analysis with a defender toolbox suffice to avoid the need for actual penetration testing involving human experts? How fine-grained or concrete the enumeration of considered attacks paths needs to be is also an open question, with respect to how paths are composed of attack steps and how their attributes are aggregated, and with respect to how concrete individual attack steps need to be. Since the assessment must drive the mitigation, the generated information must be rich enough for the mitigation decision makers. Therefore, to some extent, the answer to this question will depend on the goal of the assessment, which might be, e.g., a semi-automated or fully automated mitigation phase. In the latter case, assessment information must be extensive and accurate, as an automated decision support system cannot rely on human intuition and experts' past experience.
The identification of attacks with an analysis toolbox requires white-box access to the application code; a black-box approach as in computer networks is not viable. In case white-box access is not possible, e.g, because of SDLC requirements discussed in Section 3.1.5, alternative sources of information about the features and weaknesses present in the different integrated components need to be considered, such as partial analysis reports provided by the involved parties. Alternatively, and as long as the discussed enumeration approach cannot completely replace human expertise, the inclusion of results of penetration tests performed by socalled red teams could be considered. In short, the risk assessment phase needs to be able to take into consideration a wide range of information sources and forms.
For the scalability and practical use of a software risk analysis process, another open issue is how to update and maintain the attack path enumeration without repeating a full analysis from scratch when any of the involved aspects evolve while the application is still being developed, be it the application itself, the protection tool flow, the attackers' tool boxes, etc. Especially if the enumeration of the attacks involves human expertise, this maintenance issue is critical to keep the risk management approach to SP viable.
With the current state of the art, such human expert involvement is still necessary. Past research aimed to automate the attack discovery with abductive logic and Prolog [31] . That suffers from computational issues, since generating attack paths as sequences of attack steps causes a combinatorial explosion and requires massive pruning. With the pruning implemented by Regano et al., only high-level attack strategies can be generated, which often do not contain enough information to make fine-tuned decisions when similar SPs are considered. For example, they allow determining the need for using obfuscation but do not provide hints for selecting among different types of obfuscation.
Several results in the machine learning field are potentially applicable to synthesize attack paths from attack steps in a more effective way. For example, methods behind exploit generation [32] , [33] techniques that automatically construct concrete remote exploit payloads to attack vulnerable applications, could be investigated to check if aspects of it are suitable to determine MATE attack paths automatically. Certainly, they will need modifications, as finding remotely exploitable vulnerabilities is rather different from finding MATE attack paths that violate security properties of assets.
For example, in the MATE risk analysis methodology, for each identified attack path, defenders need to estimate the likelihood of succeeding as a function of the invested effort, attacker expertise, time, money, luck in trying the right strategy first or not, etc. All of that is absent in the mentioned automated exploit generation.
Regarding automation of this phase of the whole approach, we think the identification and description of primary assets cannot be automated, as those assets most likely depend too much on the business model in which the software plays a role. By contrast, the identification of secondary assets, as well as the discovery of attacks paths and the assessment of their likelihood, complexity, and other factors that contribute to the overall risk, should be prime targets for automation. Even if full automation is out of reach because parts cannot be automated or some parts do not produce satisfactory results, automating large parts of the threat identification phase will already have benefits. It will reduce the need for human effort, thus making proper risk assessment cheaper and hence more accessible, and it can raising awareness about identified attack strategies, thus making the assessment more effective. A gradual evolution from a mostly manual process, over a semi-automated one, to potentially a fully automated one, is hence a valuable R&D goal. We stress that in order to succeed, the automated tool support should then not only provide the necessary inputs for later (automated) phases of the risk management, it should also enable experts to validate the produced results to grow confident in the tools. Section 4 will present a tool that, although being rather basic, achieved just that.
The risk assessment report must give an indication of the consequences that exploitation of the risk may have. It must produce an easily intelligible value or score associated to all the risks to all assets. Since the objective of the report is prioritizing the risks in order to drive the mitigation phase, it must not only consider the value of the violated primary assets, but also the side effects, like impact on business reputation or market share losses.
Furthermore, it may consider the likelihood that attackers are interested in executing the identified threats because of different expected ROIs. For example, an attack path that offers a lot of potential gains for the attacker (or damage to the company) might be less attractive when it comes with a high probability of being detected and having to face legal consequences.
When outcomes from the impact analysis are available in proper form, our feeling is that this phase has no peculiarities compared to risk analysis in other fields. Models can therefore be adopted from existing literature to build a system that allows the consistent evaluation of the impacts. As a promising option, we consider risk monetisation [34] , the process of estimating the economic loss related to a risk and the ROI of a mitigation activity, which eases the reporting of risk assessment data to the higher management and is general enough to work for every asset type, including software assets. As another instance, aspects of the OWASP risk rating methodology for web applications might be useful [35] . Automation support for the available options can then obviously also be reused, possible after some adaptations.
This phase comprises two parts: (i) mitigation decision making, and (ii) implementing and validating the decisions.
First, one needs to evaluate how the deployment of certain combinations and configurations of SPs with will affect the high(est) risk attack paths determined before. Ideally, this evaluation can be done through estimation, i.e., without having to actually deploy the considered SPs and having to measure the effect of that deployment. This is a major difference with the risk assessment phase, which relied heavily on measurements. How precise the estimations need to be to enable sufficiently precise computation and comparison of residual risks, is an open question. We hence consider two possible approaches for this process.
A first approach builds on the assumption that estimations are sufficiently accurate to determine the best possible combination and configuration of SPs without requiring any measurement. One then first determines the combination and configuration that achieves the minimal residual risk while not violating any of the hard constraints. Next, one select alternative targets that trade off some of the residual risk for other aspects, such as lower costs in the form of performance, required adaptions of the SDLC, userfriendliness, etc. For each alternative target, one then again selects the best SPs and estimates the delta in residual risk and in the other relevant aspects over the selection that yielded the minimal residual risk. Finally, one then makes a choice between the most protective selection and the alternatives. This human decision will typically involve SP experts, application architects, and company managers familiar with the overall business goals and strategy.
Given the complexity of SP as discussed above, i.e., the fact that SPs are not separate black-box entities, we consider the above decision making process not viable at this point in time and in the near future. It is simply out of reach for humans and for current decision support tools.
An alternative approach, familiar to practitioners in industry, is to add additional SPs iteratively in a layered fashion. The assessment and mitigation phases are not executed once, but alternating over multiple rounds. In each round, an assessment phase is followed by a mitigation phase. In the first round, the risk assessment phase is performed on the vanilla application. In later rounds, the assessment phase is performed on the version of the application protected with all SPs selected in previous rounds. During such later assessment phases, measurements are performed on already selected and deployed SPs. This provides a workaround for the lack of good enough estimation methods as needed for the first approach. It also eases the handling of novel risks introduced by deployed SPs, such as when the location of non-stealthy SPs might leak the location of the assets.
In each round, the mitigation phase adds a limited number of additional SPs to the ones already selected in the previous rounds. In each round, different combinations and configuration of SPs can be proposed that offer different trade-offs in terms of risk reduction and costs. Humans will then again select one set of SP configurations and continue to the next round, or stop once the whole budget in terms of costs is consumed or no more significant risk reduction is achieved. In each round, different constraints are imposed that limit the SP configurations considered in that round, and the set of SP configurations is chosen that meets those constraints and that offers the best potential to reduce the residual risk. Estimating the reduction potential rather than the immediate reduction in each round (except for deciding whether a round will be the last round) allows for taking into account a priori knowledge about the fact that some SPs have the potential to become much stronger after additional rounds (i.e., additional layers) are deployed, while others cannot become stronger because of a lack of synergies. An example of constraints evolving between rounds is that in the first rounds, SPs might only be deployed on assets, while in later rounds non-stealthy SPs can become deployable on non-assets to avoid that protected assets stand out because of protection fingerprints as discussed in Section 3.1. Our PoC presented in Section 4 contains such an asset hiding round, as will be detailed in Section 4.3.2.
The alternative approach is more realistic for several reasons. The humans making decisions in each round can make up for deficiencies in the existing tool support and formalized knowledge, and can build more confidence in the outcomes of the mitigation process. Secondly, measurements are performed in each round, which again allows for more confidence in the outcomes.
It is on the mitigation task that automation poses the most severe constraints. Optimizing the selection of the combination of SPs to deploy must comply with computational constraints. In most usage scenarios, optimization models must guarantee to return results in minutes or hours. Given the large search space to explore, this implies the need for ad hoc models that prune the less relevant combinations efficiently. In some usage scenarios, it can be acceptable that the optimization models return far-fromoptimal results quickly, such that the time-to-market requirements of an initial software launch can be met, while spending more time to find better combinations of SPs for updates of the software. The recent COVID-19 tracing apps are a prime example: start tracing first, look at security and privacy later 1 .
Within one round of decision making, the optimization process should be driven by at least the potency of the selected combination of protections and by the estimation of the performance of the protected app (e.g., user experience). Ideally, resilience is also considered. Current methods for assessing the potency, resilience, and (to some extent) overheads are not usable for automatic decision support, however, as they require the actual application of the protections to perform a measurement on the protected version. Given the time and resources needed to apply protections on nontoy programs to measure objective metrics, and to measure the overheads by running the protected applications, an optimization process that requires measurements instead of estimations would only consider a very limited solution space, which would make the optimization process useless.
Estimating the strength (and overhead) of MATE SPs is really hard, however, given that SPs are composed and layered, resulting in their code being highly interwoven. Methods from, e.g., network security to aggregate the strength and overhead of combined but clearly isolated network security controls are hence not reusable. Machine learning techniques can possibly solve this difficult problem, but clearly need further research.
Another open issue is that different SPs have different effects on attack success probability, in particular when the security requirements are time-limited or relative. In some cases, the effects can be quantified in absolute terms, such as increased brute-force effort required to leak an encryption key from a well-studied whitebox crypto protection. In other cases, such as the delay in human comprehension of code that has undergone design obfuscations [4] , the effect is harder to quantify precisely and in absolute terms. When software contains different assets with different forms of security requirements, the relative value of different protections hence becomes very difficult to determine, and hence the overall risk mitigation optimization becomes increasingly difficult. 1 . Anonymously, SP suppliers confirm to us that for many of their customers the norm is weak implementation at first because security/protection is not on the feature list required by product management, and then complaining when things get broken, after which the SP supplier needs to help out. Obviously those customers prohibit documenting concrete cases.
In each mitigation round, the chosen combination/configuration needs to be deployed, i.e., SP tool flows need to be configured and run on the application to inject the SPs selected in all rounds so far.
Ideally, this part is completely automated. This obviously requires communication interfaces between decision support systems and protection tools. Providing such interfaces and enabling this automation would have significant benefits. Apart from saving effort on manual interventions for protecting concrete application, it would also skip the learning curve of how to properly configure the deployment of protections with specific tool flows. Moreover, having such an integrated framework could pave the road for an open standard for an API for software protection.
Following the deployment, it is critical task to validate that the SP tools actually delivered as expected. Were the selected SPs injected in the intended way? Do the injected SPs have weaknesses that were not expected? How to obtain the necessary validation is an open question in some usage scenarios, in particular when the deployment of the mitigation is executed by multiple parties that don't want to share sensitive information.
According to the NIST [36] risk monitoring includes "assessing control effectiveness, documenting changes to the system or its environment of operation, conducting risk assessments and impact analysis, and reporting the security and privacy posture of the system." In the SP scenario, monitoring involves the continuous tasks that need to be performed once the protected software has been released, in order to keep track of actual level of exposition to the risks over time. This consists of two related activities: keeping the risk analysis up-to-date, and keeping track of the risk exposure of the released application.
The task of keeping the risk analysis up-to-date tracks how the inputs used in the past risk analysis activities of framing, assessment, and mitigation evolve over time, and how that evolution affects the decisions that were made in those activities. We can abstract these into monitoring the evolution of three different pillars of information: the information related to the assessments (e.g., new attacks, attack techniques, tool updates), the information related to SPs (e.g., updates, vulnerabilities, breaches), and the information related to the protected application. Of course, monitoring can then lead to the decision that a differently-protected version of the protected application should be released whenever any tracked changes lead to the need to re-evaluate earlier decisions as to which risk are prioritized and which mitigations are deployed.
The monitoring of the information related to SPs involves keeping track both of attacks against existing SPs and of newly developed SPs. For example, when a complex attack technique (e.g., generic deobfuscation [28] ) is first presented in the academic literature, it might not be considered practically relevant during an original risk assessment because the attack is hard to replicate and its effectiveness has not been demonstrated on more complex pieces of software. However, when attackers later release a tool box that automates the replication and they publish a blog discussing how they used it to attack a complex application successfully, this should lead to a re-evaluation. Similarly, when new SPs become available with higher effectiveness against old or new attacks, or with lower overhead, this may lead to a reevaluation.
Similarly, the information related to the application that was used as input for the previous tasks can also evolve over time. One example is where a company might decide that there are, in fact, additional assets in the program that need to be protected. This can happen both as a late realisation after deployment, but also in the case where the application itself evolves over time, by virtue of new versions being released with changes in functionality or structure. Another example is that the priorities in the company's estimation of value can change over time. This would mean that the associated formulas for the risk analysis produce different values.
Next, one needs to track how copies of the released software are running on the premises of their users. This can be achieved by monitoring the information that the protected application communicates to the vendors. Such information may originate from a monitoring-by-design SP such as reactive remote attestation [37] , but also from communication by other online components that were not originally designed for online monitoring. This is particularly the case when irregular communication patterns can be linked to unauthorized activities such as running multiple copies in parallel or executing program fragments in a debugger in orders or frequencies not consistent with authorized uses. Such patterns can occur from communications present in the original applications, or from online SPs such as code renewability [38] and clientserver code splitting [39] . Importantly, the use of non-monitoring communication does not require the implementation of reaction mechanisms in the protected application to be effective. In many cases, it can suffice to feed the insights from the monitoring into the processes for keeping the risk-analysis up-to-date, or the application servers can take action.
Finally, the vendor of the released application needs to monitor whether the impacts of the deployed SPs on the user experience and cost are in line with expectations or promises by the SP vendor. If, e.g., users start reporting usability issues, or if online protections manifest scalability issues, e.g., because more copies are sold than originally anticipated, those evolution might also warrant a revision of the risk mitigation strategy.
The Expert system for Software Protection (ESP) is our PoC tool to implement an automated risk analysis of software protection 2 . It is mostly implemented in Java and packaged as a set of Eclipse plug-ins with a customised UI. It protects software written in C. As it needs access to the application source code to work, the target users are software developers or protection consultants. Figure 1 depicts the high-level workflow, with four phases corresponding to the four phases described in Section 3, namely, risk framing, assessment, mitigation, and monitoring.
The ESP starts with the risk framing phase, where all the information about the risk analysis to perform are collected and stored in a Knowledge Base (KB). Context information includes general information, like the attacker model and the protections available to mitigate the risks, and information about the application to protect, like the assets and abstract representation of the application code, which is collected through code analyses. More details on the ESP risk framing are presented in Section 4.1.
Next, the ESP performs the risk assessment phase, whose details are provided in Section 4.2. It infers the attacks against the assets and assesses the risks against each asset by estimating, for each inferred attack, the complexity to successfully execute it. The risk is evaluated by taking into account the structure of the application and the attacker model, i.e., the skills an attacker interested in endangering the application assets is likely to have, and asset values, as defined by the user during the risk framing.
The ESP's risk mitigation phase, detailed in Section 4.3, uses Machine Learning (ML) and optimization techniques to select the best solution, i.e., the best sequence of SPs to be deployed and their configurations. It then automatically deploys it on the application code to generate the protected application binaries. The ESP can also be configured to propose a set of solutions that experts can manually edit to have full control over the protection deployment. Moreover, the ESP can evaluate the effectiveness of solutions manually proposed by experts. If remote protections are included in the selected solution, the deployment phase also generates the server-side logic, to be executed on a trusted remote entity.
Finally, the risk monitoring is performed. However, the ESP does not dynamically update the risk analysis process parameters, it only performs real-time integrity checking (see Section 3.4.2).
The risk framing starts with a preliminary preparation of a KB with all the application-independent information, named generic a-priori information, i.e. the core concepts and the information that is not related to the specific application to be protected but is relevant for framing the risk analysis process. A-priori information includes (but is not limited to) the assets types, the supported security requirements, all the known attack steps, and the available protections, and the necessary data to evaluate risks and mitigations, which have been already discussed in Section 3.1. Then, the ESP performs a source code analysis that populates the KB with a-priori analysis-specific information using the C Development Toolkit of the Eclipse platform.. The source code analysis reports all the application parts, which are the variables, functions and code regions therein. The analysis determines additional information, like variables' data types, function signatures, and produces additional representations, such as the call graph, which are useful to make decisions about the protections to apply.
The PoC of the ESP supports two requirements: confidentiality and integrity. The user needs to annotate the source code with pragma and attribute annotations to identify the assets in the code and to specify their security requirements formally. We designed a specific format for these annotations [40] , [41] . The ESP then uses the call graph information to identify potential secondary assets. In this phase, the ESP also puts the preferences and analysis customization parameters in the KB before starting the rest of the process (e.g., the protections to consider, the selected attackers to counter).
To allow the automation of the risk analysis process, the KB has been formally described by means of meta-model for software protection [42] , whose core classes are represented in Fig. 2 . The aspects corresponding to those classes will be discussed in the next sections. The KB is instantiated as an OWL 2 ontology. Together with a-priori information, there is a-posteriori information, i.e., data inferred and stored during later work flow phases (e.g., the inferred attacks, the solutions).
In addition, the ESP offers a GUI to edit the framing information (e.g. marking additional assets, characterizing the attacker, and selecting protections) through a GUI, which also allows importing and exporting risk framing data as XML or OWL files. This feature was appreciated during the validation as it allows augmenting the analysis with important information that may be missed by the automatic process, like the secondary assets that might be linked into a protected program as part of certain protections.
In the threat identification, the ESP finds the attacks that can breach the primary assets' security requirements and stores them in the KB. This stage is roughly equivalent to the ISO27k "identify risk" step (see Sections 2.1 and 3.2).
The identified attacks are represented as a set of attack paths, i.e., ordered sequences of atomic attacker tasks called attack steps.
Attack paths are equivalent to attack graphs [43] and can serve to simulate attacks with Petri Nets [44] . The attack steps that populate our PoC KB originate both from a study and taxonomy by Ceccato et al. [30] , [45] and data collected from industrial SP experts who participated in the European ASPIRE research project (https://www.aspire-fp7.eu).
The attack paths are built via backward chaining, as proposed in earlier work [31] , [46] , and implemented with SWI-Prolog. An attack step can be executed if its premises are satisfied, and produces conclusions, the results of the successful execution of that step. The chaining starts with steps that allow reaching an attacker's final goal (i.e. security requirement breach) and stops at steps without any premise. As the search algorithm has exponential complexity (it builds a proof tree with increasing depth and width), the ESP implements basic (i.e., aggressive and quite brutal) mechanisms to prune the search space (e.g., maximum length of the inferred attack paths) [47] .
The ESP performs the risk evaluation and prioritization by assigning a risk index to each identified path. To that extent, every attack step in the KB is associated to multiple attributes, including the complexity to mount it, the minimum level of skill required, the availability of support tools, and their usability. Additional attributes can be associated to entities trivially. Each attribute assumes a numeric value in a five-valued range. For assessing the actual risks, the values of complexity metrics computed on the involved assets are used as modifiers on the attributes. For instance, an attack step considered of medium complexity can be downgraded to lower complexity if the asset to compromise is considered simple, e.g., because its cyclomatic complexity is below a custom threshold value.
The risk index of an attack path is obtained by aggregating the modified attributes of the composing attacks steps into a single value. Our PoC is rather simple: per attack step, it first aggregates all the step's modified attributes into a single attack step risk index; then the attack path risk index is computed by multiplying the composing attack step indexes. Other aggregation functions are supported, such as summing the steps' indexes, selecting maxima, and more complex features can easily be incorporated. One idea is to let the attack path risk index depend on how many different expert tools are required.
The attack path and the risk indices computed by our PoC ESP were welcomed by security experts (see Section 4.5), amongst others because they served as an excellent starting point for evaluating the weaknesses of a target application before performing a more manual risk mitigation. Nonetheless, experts were interested in refining the identified, most risky attack paths into more concrete sequences of attack operations, and in some cases they would have manually updated the risk indices. Indeed, in our PoC KB, the attack steps are coarse-grained, such as "locate the variable using dynamic analysis" and "modify the variable statically". This is a main limitation and, as discussed in Section 3.2.1, it is an open research question how much refinement is needed.
Before presenting the risk mitigation process as performed by the ESP, we will introduce more precise definitions. A protection is a specific implementation of a SP technique by a specific SP tool. For instance, control flow flattening [48] as applied by Diablo in the ASPIRE Compiler Tool Chain (ACTC) (https://github.com/aspire-fp7/actc) and by Tigress (https://tigress.wtf/) are considered two distinct protections [49] , [50] . A protection instance (PI) is a concrete configuration of a protection. The ESP can use the PI to drive the SP tool to apply a protection on a chosen application part. Multiple different PIs can be defined for the same protection, depending on the available parameters. An applied protection instance (applied PI) is the association of a PI to an application part, which states that the PI has been selected to be applied on an application part. A candidate solution is then an ordered (because of composability and layering) sequence of applied PIs.
The ESP first searches for the suitable protections, which are defined as the protections that impact any of the attributes of listed attack step, e.g., they are able to defer an attack step. Each PI is associated to a formula that alters these attributes for each attack step. Therefore, after the application of a protection, the risk index of the attack steps and paths are re-assessed.
The formulas also consider a number of complexity metrics computed on the protected assets' code. This way, our approach incorporates Collberg's prescription of potency [23] as a measure of the additional effort that attackers will have to invest as they face protected, more complex code. The information about the impact of protections on attack steps, i.e., the parameters to be used in the formulas, are stored in the KB. It is based on a survey among the developers of all protections integrated in the ASPIRE project [40] , whom we asked to score the impact of their protections on a range of attack activities in terms of concrete, clearly differentiated forms of impacts. These include impact on human comprehension difficulty by increasing code complexity, impact by moving relevant code fragments from the client-side software to a secure server not under control of an attacker [37] , [38] , [51] , impact on difficulty of tampering through anti-tampering techniques with different reaction mechanisms and present monitoring capabilities [37] , and impact of preventive protections such as anti-debugging [52] , [53] . The survey results were complemented with security expert feedback, and validated in pen test experiments with professional and amateur pen testers [30] , [45] .
To model the impact of layered protections when recomputing the risk indices and of synergies between protections, additional modifiers are activated when specific combinations of PIs are applied on the same application part. The existence of synergies was also part of the mentioned survey.
Candidate solutions must also meet constraints on the performance degradation and other forms of overheads. Our PoC filters candidate protections using five overhead criteria: client and server execution time overheads, client and server memory overheads, and network traffic overhead.
Finally, the protection index associated to a candidate solution is computed based on the recomputed risk indices of all the discovered attack paths against all application assets, weighted by the importance associated to each asset.
The ESP then finds the mitigations by building an optimization model that is solved with a game-theoretic approach. The ESP tries to combine the suitable protections to build the optimal layered protection solutions, that is, it finds the candidate solution that maximizes the protection index and satisfies the constraints.
Computing the protection index by re-computing the risk index, requires knowledge of the metrics on the protected application. As applying all candidate solutions would consume an infeasible amount of resources, we have built a ML model to estimate the metrics delta of applying specific solution without building the protected application [54] . The model available in the PoC ESP has been demonstrated to be accurate for predicting variations of up to three PIs applied on a single application part. With more protections, however, the accuracy starts decreasing significantly, nonetheless, this issue seems to be solvable with larger data sets and more advanced ML techniques.
The ESP uses the same predictors to estimate the overheads associated with candidate solutions. Per PI and kind of overhead, the KB stores a formula for estimating the overhead based on complexity metrics computed on the vanilla application. These formulas were determined by the developers of the different protections integrated in the ASPIRE project.
Having to deal with combinations greatly increases the solution space. To explore that space efficiently and to find (close to) optimal solutions in an acceptable time, the ESP uses a gametheoretic approach, simulating a non-interactive SP game. In the game, the defender makes one first move, i.e., proposes a candidate solution for the protection of all the assets. Each proposed solution yields a base protection index, which has a positive delta over the risk index of the unprotected application. The attacker then makes a series of moves. Each move corresponds to the investment of (some imaginary unit of) effort in one attack path, which the attacker selects from the paths found in the attack discovery phase. Similarly to how potency-related formulas of the applied protections yield a positive delta in protection index as discussed above, we use resilience-related formulas that estimate the extent to which invested attack efforts eat away parts of the protection potency, thus decreasing the protection index. These formulas are also based on security expert feedback. We refer to Regano's thesis for more details on this game-theoretic optimization approach that uses mini-max trees and a number of heuristics to the search space for the best candidate solutions in acceptable times and with acceptable outcomes [55] , as will be evaluated below.
After solving the game, the ESP shows the user the best protection solutions found during the optimization, i.e., the best first moves by the defender, from which the user can choose one, for which the ESP will then invoke the automated protection tools to apply the solution, as explained in Section 4.3.3.
As already discussed in Section 3.1, protections are not completely stealthy because they leave fingerprints.
In a previous paper [56] , we have raised this problem and proposed a solution, based on the refinement of existing protection solutions with additional protections deployed also on non-asset code regions. Those lure the attacker into analyzing such regions in lieu of the assets' code, thus hiding the assets from plain sight. We have devised three asset hiding strategies. In fingerprint replication protections already deployed on assets are also applied to non-sensitive application parts to replicate the fingerprints such that attackers analyse more application parts. With fingerprint enlargement, we enlarge the assets' code regions to which the protections are deployed to include adjacent regions such that attackers need to process more code per protected region. With fingerprint shadowing, additional protections are applied on assets to conceal the fingerprint of the originally chosen protections, to prevent leaking information on the security requirements.
The PoC ESP hides the protected assets as an additional decision making step. In this step, we add confusion indices to the protection indices, which are computed by an ad hoc formula built to estimate the additional time needed by the attacker to find the assets in the application binary, after the application of hiding strategies. Also the computation of the confusion index requires the estimation of the complexity metrics of the application code after the application of the protections. To build this model, we have studied the effects of the hiding strategies for the protections devised during the ASPIRE project. The results of this study, stored in the ESP KB, are used to compute the confusion index. Starting from the solutions generated via the game-theoretic approach, the ESP proposes additional application parts to protect by solving a Mixed Integer-Linear Programming (MILP) problem, expressed as an heavily customized instance of the well-known 0-1 knapsack problem [57] that maximizes the confusion index and uses overhead as weight in constraints. The MILP problem is solved using an external solver, the PoC ESP supports lp solve 3 and IBM CPLEX Optimizer 4 .
The final step in the ESP workflow is the application of the solution on the target application. The solution is chosen by the user amongst the ones presented by the ESP. The result of this step (and of the whole workflow) is the protected binary plus the necessary server side components for online protections, if any, ready to be distributed to final users. The ESP deploys a solution by driving automatic protection tools. At time of writing, the ESP supports Tigress, a source code obfuscator developed at the University of Arizona, and the ACTC, which automates the deployment of protection techniques developed in the ASPIRE FP-7 project [40] , [41] . Table 1 summarizes the protection techniques supported by the ESP.
Finally, we point out that the ESP has been engineered to be extensible. All the modules can be replaced with alternative components. For example, the risk assessment based on backward reasoning could be replaced with a more advanced attack discovery tool, the only constraint being that it needs to produce output that is compliant with the software protection meta-model. Moreover, it is also possible to support new protections. It is 3 . See http://lpsolve.sourceforge.net/5.5/. 4. See https://www.ibm.com/analytics/cplex-optimizer. 5 . The ACTC provides limited support for code virtualization, meaning that it is not reliably applicable to all code fragments. Hence the ESP does not consider it a potential protection instance. enough to add all the required information into the KB (e.g., evaluation of strengths and impacts on attack steps, conflicts and synergies with other protections plus all parameters of the discussed formula). The only demanding operation is training the ML algorithms to predict how new protections alter the metrics, and the automation of the deployment of the protections.
The ESP generates all the server-side logic, which also includes the backends necessary to the protections that are used to perform the risk monitoring of the released application.
Our PoC ESP does not automatically include the feedback and other monitoring data (e.g. number and frequency of detected attacks and compromised applications, server-side performance issues) The knowledge base needs to be manually updated by the experts, even if GUIs can be used to save time, to change risk framing data related to attack exposure and protection effectiveness. Also, issues related to insufficient server resources need to be addressed independently, the ESP only provides the logic, not the server configurations.
The ESP has been validated with experts, mainly from the ASPIRE project consortium and advisory boards, to assess the quality of the identified solutions and fulfillment of the requirements for the software risk management methodologies, as in Section 3. Moreover, we have evaluated its performance to determine computational feasibility and scalability of its tasks.
As a first qualitative evaluation, software security experts tasked with the protection of a target application have been asked to judge the correctness of the workflow, suitability with their daily software protection workflow and tasks, potential impact on the quality of their work, and effectiveness of their results. Overall, experts have judged the ESP as very promising and potentially effective by the experts to delegate or support their tasks, because of the high-level of automation and configurability, of the possibility to override default configurations, of the very detailed output, Nonetheless, they were skeptic on the usability by software developers with a limited background on software protection. Among the data extracted by the tool, experts highlighted the importance of making decisions by taking into account the application structure and metrics, because results are tailored to the target application. They also appreciated that all the data extracted and represented in the KB are structured according to a meta-model.
During the ASPIRE project, each industrial partner provided an Android application, namely a One-Time Password generator for home banking applications, an application licensing scheme, and a video player for Digital Rights Management (DRM) protected content. Each Android application included securitysensitive code in a dynamically linked library written in C. It were these libraries that served as reference use cases for all the protection tasks. In a qualitative evaluation, we asked two experts from each of the three industrial project partners to validate the solution selected by the ESP for their application. While details of these use cases, which were designed to embed relevant assets but without having the full complexity of real-world applications, have not been publicly released by their proprietors, they are included in a confidential project deliverable [58] . Table 2 discloses the Source Lines Of Code (SLOC) metrics and the number of assets as evidence that the ESP was not used merely on toy examples. Experts analysed the inferred attack paths and protections to mitigate them, and the solutions proposed by the optimization process, comparing them to solutions they manually assembled earlier on during the project as part of the requirements formulation. Solutions have been validated in terms of achieved security for the assets, preservation of the application business logic, and containment of the inevitable slow-down of the protected application w.r.t. the original one. Furthermore, the attack paths have been compared with the real attacks discovered by the professional tiger teams previously involved in the assessment of the protections.
Also this validation gave a positive output: quoting from the related project deliverable, "after the analysis of the validation data, the experts concluded that the tool has a very high potential" even if some had doubts on the maturity of the tool to use it, in its current form, on their existing commercial applications with all of their SDLC intricacies and complexity. For a research proof-ofconcept, this should of course not come as a surprise. In particular, the inferred solutions have been judged as appropriate to protect the use case code and effective to block the inferred attack paths and the real attacks reported by the tiger teams. Moreover, the protected binaries were evaluated as being semantically unaltered (e.g., still delivering the original observable IO-relation) and usable (i.e., without an excessive overhead introduced by the protections). The main flaw of ESP reported by the experts is that inferred attack paths were too coarse-grained, because of too generic attack rules.
We have measured the execution time of the ESP on the three use cases, with the assets annotated by the experts, as reported in Table 2 . In all the three cases, 17 protection instances have been considered. These are the nine supported protections of the ACTC as listed in Table 1 , of which opaque predicates, branch functions and control flow flattening can be applied at three configuration levels (i.e., low, medium, and high frequencies with corresponding high levels of overhead), and in which three different data obfuscations were considered (XOR-masking, residue number encoding, and data-to-procedural conversions) [23] . Table 3 shows the ESP computation times. The framing phase is almost instantaneous and is driven by the lines of the code from which annotations are extracted. Regarding the complexity and scalability of the assessment and the mitigation phases, it is not possible to draw conclusions from this experiment use cases that were driven by the industrial partners, aimed at testing the ESP in a real scenario, and of which we did not control the variables that we deemed interesting for a complete performance evaluation.
Regarding the attack discovery tool and the game theoretic optimization of the mitigation phase, we know that the most influential factors of the assessment phase are the number of APPLICATION attack steps in the KB (exponential complexity but with pruning) and the number of assets (linear), while the mitigation phase is driven by the number of assets (linear), the number of PIs and and the number of attacks discovered in the assessment phase (both exponential).
To asses the scalability of those phases, we evaluated the performance of the ESP on three standalone Linux applications that cover an increasing range of assets. Table 4 summarizes their metrics. These toy applications have been randomly generated with a process that selects a call graph (from a set of call graphs extracted from real applications), then generates randomized function bodies to meet some code metrics, and then randomly selects fragments in the generated code as data or code assets. In this experiment, we have used all the previously listed PIs from the ACTC (excluding white-box crypto, which was a proprietary algorithm of one industrial ASPIRE project partner) and added four instances of obfuscation applied using Tigress, i.e., the ones marked in Table 1 .
On the three demo applications, the asset mitigation phase was repeated multiple times (see Section 4.3), depending on the number of PIs available to protect the applications' assets. All experiments have been executed on an Intel i7-8750H 2.20 GHz computer with 32 GB RAM, using Java 1.8.0 212 under GNU/Linux Debian 4.18.0. Figure 3 depicts the measured total ESP computation time, along with the time needed for the risk assessment, asset protection, and asset hiding phases. The time needed to complete the workflow increases with the number of PIs considered during the mitigation; such increase strongly depends on on the application code complexity (e.g. SLOC, number of assets, and functions).
The time needed to analyze the applications source code and to generate the application meta-model instance was negligible (less than 1s), and the time to deploy the solution is irrelevant for the assessment of the ESP' computational feasibility, as it only measures the time needed to execute the external protection tools on the single selected solution.
As expected, the time needed to execute the risk assessment phase does not depend on the number of PIs available to protect the application, as attacks are determined on the vanilla application. Nonetheless, we report that it has limited impact because of the aggressive pruning we have implemented that avoids the exponential growth. The asset protection phase is by far the most computationally intensive, especially when the available PIs increase. Since, the mitigation considers sequences of protections, the execution times exponentially depends on the permutations of combination of PIs. The same holds also for the asset hiding phase, even if less time needed to execute the latter, compared to the asset protection phase.
Expert systems have been applied in the cybersecurity field since 1986, when Hoffman proposed an expert system for the risk analysis of computer networks [59] . The author theorized a system Table 4 .
able to identify vulnerabilities in the analyzed computer system configuration and suggest the appropriate countermeasures to reduce the overall risk. In the same year, Denning and Neumann started the development of IDES [60] , a host-based Intrusion Detection System (IDS) mixing an expert system with statistical anomaly detection techniques aimed at detecting unauthorized accesses, both by local and remote users. Its evolution, called NIDES [61] , supported also real-time analysis of inter-process communications.
Other IDS expert systems were developed in the same years for specific tasks: NIDX [62] , to suggest a network administrator possible security breaches of UNIX System V machines, NADIR [63] , to monitor the internal network of the US Los Alamos National Laboratory, AUDES [64] , to assist computer security auditing process, and Haystack [65] , to detect breaches in US Air Force systems. Notably, the latter was the first showing selflearning capabilities, as it evolved user profiles over time.
After this initial enthusiasm, the limitation of those expert systems in accuracy, manageability and extensibility, encouraged researchers to investigate combinations of classic expert systems with other techniques, in order to enhance breach detection performances. Examples are [66] by Eronen and Zitting, using constraint logic programming [67] to generate access control lists for Cisco firewalls from high-level filtering requirements, [68] by Depren et al., using Self-Organizing Maps (SOM) and decision trees for breach detection and an expert system to interpret the result of such machine learning algorithms, [69] by Pan et al., employing neural networks for detecting attacks leveraging unknown vulnerabilities, while an expert system identifies known attacks. Fuzzy logic algorithms have also been embedded in expert system for real-time intrusion detection, as in [70] , and postincident network forensics, for example in [71] and [72] . The Ec-funded PoSecCo project has delivered SDSS [73] , an expert system aimed at driving security administrator during all the steps from the policy specification, anomaly analysis and resolution, to the automated refinement and enforcement of the anomaly-free policy.
In the domain of MATE SP, we know of no scientific literature on expert systems. In practice, companies provide so-called cookbooks with protection recipes. For each asset, users of their tools are advised to deploy the relevant SPs in an iterative, layered fash-ion starting as long as the overhead budget allows for additional protections. Automated approaches are limited to specific types of protections, and hence only support specific security requirements. Heffner and Collberg studied how to overcome the problem of deciding which obfuscations to deploy in which order to counter reverse engineering, given an overhead budget [74] . Coppens et al. proposed an iterative software diversification approach to counter so-called diffing attacks that exploit security patch releases to automatically engineer attacks against unpatched systems [75] . In both works, only obfuscations are considered. In both works, measurements are performed after each round of transformations, much like in the second approach we discussed in Section 3.3.1.
In this paper, we presented the necessity of having a standardized approach for risk management in the context of software protection against man-at-the-end attacks. To that end, we discussed just such a risk management approach for software protections, which we based on the NIST SP800-39 standard for risk management for information security. We discussed in detail how the different aspects of software protection can and should be mapped onto risk framing, risk assessment, risk mitigation, and risk monitoring phases. A proof-of-concept decision support system we have designed and implemented provides evidence that the proposed approach is feasible and can be automated to a large degree.
We hope this is sufficient evidence that it is useful to launch a community effort that leads to a future standardization.