MACTA: A M ULTI - AGENT R EINFORCEMENT L EARNING A PPROACH FOR C ACHE T IMING A TTACKS AND D ETECTION
Anonymous authors
Paper under double-blind review
A BSTRACT
Security vulnerabilities in computer systems raise serious concerns as computers process an unprecedented amount of private and sensitive data today.
Cachetiming attacks (CTA) pose an important practical threat as they can effectively breach many protection mechanisms in today’s systems.
However, the current detection techniques for cache timing attacks heavily rely on heuristics and expert knowledge, which can lead to brittleness and the inability to adapt to new attacks.
To mitigate the CTA threat, we propose using MACTA, a multi-agent reinforcement learning (MARL) approach that leverages population-based training to train both attackers and detectors. Following the best practices, we develop a realistic simulated MARL environment, MA-A UTO CAT, which enables training and evaluation of cache-timing attackers and detectors.
Our empirical results suggest that MACTA is an effective solution without any manual input from security experts. MACTA detectors can generalize to a heuristic attack not exposed in training with an 81.2% detection rate and reduce the worst-case attack bandwidth by 15% on average. In the meantime, MACTA attackers are qualitatively more effective than other attacks studied, and the average evasion rate of MACTA attackers from an unseen state-of-the-art detector can reach up to 88%.
Furthermore, we found that agents equipped with a Transformer encoder can learn effective policies in situations when agents with multi-layer perceptron encoders do not in this environment, suggesting the potential of Transformer structures in CTA problems.
I NTRODUCTION
With increasingly sensitive data and tasks, security in modern computer systems is recognized as one of the 14 grand challenges for engineering (National Academy of Engineering, 2007).
As a concrete example, cache-timing attacks (CTA) in processor caches have been shown to leak private encryption keys (Yarom & Falkner, 2014; Liu et al., 2015), break existing security isolation (Kocher et al., 2019), cause privilege escalation (Lipp et al., 2018), and break new hardware security features in the latest processors (Ravichandran et al., 2022).
In CTA, the attacker is able to gain access to private information (e.g., memory access patterns) from the victim who shares a cache with the attacker.
Over decades, the attack and defense policies in CTA have been explored manually by computer architecture experts.
To defend against such attacks, statistical analysis and machine learning models with static strategies have been proposed for CTA detection, e.g. , CC-Hunter (Chen & Venkataramani, 2014) uses auto-correlation and Cyclone (Harris et al., 2019) uses an SVM classifier.
Yet, new CTA attacks are still being reported (Xiong & Szefer, 2020; Briongos et al., 2020; Saileshwar et al., 2021; Guo et al., 2022b;a), showing higher leakage rates or the ability to bypass existing defensive mechanisms.
Computer security can be seen as a competitive game between the attackers and the defenders, and game-theoretic approaches that analyze strategy (policies) for both sides have been proposed (Anwar et al., 2018; Elderman et al., 2017; Eghtesad et al., 2020).
These methods highly abstract the attack and defense strategies, usually based on known attacks and defenses, and analyze simplified games in the limited strategy spaces.
For example, Anwar et al.
(2018) studies CTA-like attack scenarios where the attacker decides when to terminate its attack and the defender decides an abstract security level.
However, real-world CTA has large action and state spaces for different agents, sparse reward, and long game horizons, making the game analysis hard without exploring all possible policies.
In this work, we use multi-agent reinforcement learning (MARL) to jointly explore and optimize complex attack/defense policies in CTA.
We take an integrated approach of reinforcement learning and game theory.
First, we build a multi-agent gym environment, MA-A UTO CAT , that closely models the realistic CTA setting and allows efficient learning for both attackers and defenders.
Specifically, we study a detect-and-terminate defense.
Second, we propose to use a MARL approach, named MACTA , to automatically find both attacker and detector policies through self-play, similar to past successes in games with large state/action spaces (e.g., StarCraft (Vinyals et al., 2019), Go (Silver et al., 2016), and Poker (Brown & Sandholm, 2019)).
MACTA adopts Fictitious
Play (FP) (Brown, 1951), population-based training in MARL (Vinyals et al., 2019) and Proximal
Policy Optimization (PPO) (Schulman et al., 2017) to learn the best response policy to a pool of diverse opponents, to avoid cyclic behaviors of the attacker/defender policies.
Finally, MACTA uses a Transformer architecture to parameterize the policy/value function so that an important subset of actions can be picked up quickly during training, yielding fast policy learning.
We performed extensive experiments in a representative setting of cache-timing attack.
The experiments show that learned policies trained with MACTA can generalize to detectors/attackers that they were not exposed to during the learning phase (henceforth referred to as“unseen detectors/attackers”).
The MACTA detector exhibits an 81.2% detection rate on an existing humandesigned attack without training on it, and can lower the worst-case number of attacks per episode (bandwidth) by 15% on average given a limit on the attacker’s learning time.
The MACTA attacker can bypass previously unseen detectors, Cyclone and CC-Hunter, with decent success rates, 88% and 52.7%, respectively.
While there has been increasing interest and effort in using machine learning for computer system security recently, our work is the first to introduce the hardware timing attack problem as a promising application of MARL and show that MARL can be effectively applied to detect simulated CTA attacks with strong generalization.
Our main contributions are as follows:
• We contribute a simulated multi-agent environment MA-A UTO CAT that models realistic CTA and allows learning in both cache timing attacks and defenses.
• We train agents using MACTA, and show the resultant detector acquires interesting highlevel patterns that can generalize to novel attackers and make the cache less exploitable to high-bandwidth attacks.
• Our study on the neural architecture of learning agents indicates that the CTA is one case where Transformers are significantly better for retrieving state information than multi-layer perceptrons.
T HE C ACHE T IMING A TTACK C HALLENGE 2. D OMAIN D ESCRIPTION
A cache is a small and fast on-chip memory commonly used in modern processor designs to reduce latency of memory accesses.
Accessing memory addresses whose data are available in a cache is fast (called “ cache hit ”), while retrieving data from memory is much slower (called “ cache miss ”).
Surprisingly, this timing difference in memory accesses due to caching could leak information across different programs/processes executing with a shared cache, a vulnerability known as cache timing attacks (CTA).
As shown in Figure 1(a), CTA involves the attacker process and the victim process, both sharing the same cache.
An example (Prime+Probe CTA (Liu et al., 2015)) is given in Figure 1(b).
The victim’s memory access will evict the attacker’s cache line from the cache, causing latency changes in the attacker’s future memory accesses.
Thus, the attacker can infer whether the victim made access to any specific memory address by observing its own memory access latency.
2. P ROBLEM S TATEMENT
In this work, our goal is to jointly find attacker and detector policies that can be generalized to unseen opponents.
This can discover unknown attacker policies, create a more robust detector, and leads [Figure] A A
[Figure]
A
[Figure] A [Figure] [Figure]
to novel insights for future cache design.
The problem of joint learning can be formulated as the general-sum Partially Observable Markov Games (POMGs), where the attacker and detector have limited observation and optimize for their own cumulative return.
Given the finite set of policies, the resultant attacker is the best response to a mixture of all detector policies explored, and the resultant detector is the best response to a mixture of all attacker policies explored.
Partially Observable Markov Games (POMGs)
Formally, the n -player episodic POMGs can be described using a tuple {I , H , S , P , {A} ni =1 , {O} ni =1 , {R} ni =1 , γ } , where I is the finite set of players, H is the episode length, S is the true state space, P is the state transition probability.
A i is the state space of player i , and the joint action space of all agents is {A} ni
=
=
A 1 × A 2 ...
× A n .
Similarly, O i is the observation space of player i , and R i is the reward function for the player i .
Lastly, γ ∈ [0 , 1] is a reward discount factor.
In POMGs, the agents can only have access to their own observation and actions, and its goal is to maximize the cumulative episodic reward for itself given the opponents’ policies, J i ( π i , π − i ) =
E (cid:2) (cid:80) Tt γ t r it | s 0 , a it ∼ π
i ( s t ) , a − i t ∼ π − i ( s t ) (cid:3) .
A
Nash Equilibrium (NE) is an efficient solution concept to the POMGs.
Formally, the NE is defined as a point where for any player’s policy π i , we have J i
( π i ∗ , π − i ∗ ) ≥ J i
( π i , π − i ∗ ) , ∀ i ∈ N .
Namely, given all other agents’ equilibrium policies π − i ∗ , there is no motivation for agent i to unilaterally deviate from its current policy π i ∗ to achieve higher returns.
MA-A UTO CAT
To study the learning dynamics of the attackers and the detectors in CTA, we develop MAA UTO CAT, a gym (Brockman et al., 2016) environment that models realistic multi-agent interactions on CTA.
We build the environment based on a cache simulator, as prior works on CTA detection schemes also use micro-architecture simulators (Harris et al., 2019; Mirbagher-Ajorpaz et al., 2020).
Figure 2 demonstrates the environment components and game mechanism. [Figure]
[Table] In MA-A UTO CAT, each agent plays a different role, and each role has a specific goal (i.e., reward), a different level of privileged accessibility (i.e., observation) to the information of the environment, and a different way to take actions (i.e., action space), listed as below:
Benign Program (B) accesses memory in a regular way, implemented by replaying an offline log of memory access pattern of regular programs (e.g., standard benchmark suite such as SPEC (Bucek et al., 2018)).
It has no observation and no policy needs to be learned.
Victim (V) accesses memory with addresses that depend on the secret.
Studies have shown that such secret-dependent memory accesses are common in real-world applications (e.g., HTTP parser), libraries (e.g., OpenSSL), and Linux kernel (Johannesmeyer et al., 2022; Qi et al., 2021; Oleksenko et al., 2020).
In CTA, a victim’s secrets usually contain multiple bits, and attackers target one bit at a time; after guessing one bit of a secret, the attacker moves to the next bit.
To model this, in our environment, the secret is reset after the attacker’s attempt to guess the secret and the victim accesses an address depending on the secret when triggered.
Attacker (A) aims to obtain the secret memory address of the victim process, by checking the patterns of latency of memory access.
An attacker may learn a policy to pick which memory address to visit, and observes the binary latency signal (slow/fast).
The attacker can also trigger the victim process to execute, and regain control after its execution, and guess the secret address of the victim if it is confident to do so.
Importantly, the attacker can only see the latency of its own accesses.
Detector (D) aims to raise the alarm as soon as possible when an attacker is present while avoiding a false alarm for benign programs.
As a system process, we assume the detector can observe memory accesses of all running processes in the environment.
The detector will terminate the episode if an alarm is raised.
See Appendix A.2 for detailed specifications of the observation, action, and rewards.
In each episode, we may pick multiple agents of different roles to be in the environment and let them interact.
In this work, we mainly test the following two possible scenarios:
• Attack Scenario ( DAV ).
The environment contains a detector, an attacker and a victim.
The attacker aims to obtain the secret address of the victim and the detector aims to identify the situation and terminate processes as soon as possible.
• Benign Scenario ( DBB ).
The environment contains a detector and two benign programs with no malicious intent.
In this case, the detector should not cause any false alarms.
We leave more complicated settings, such as scenarios with both victims and benign programs (e.g., DAVB), as future work.
M ETHOD
The CTA we consider is a POMG with three fundamental characteristics: 1) Partial Observability .
In CTA, the attacker knows which program to attack but can only see the attacker’s own actions and latencies, while the detector does not know if there is any attacker nor which program the attacker is targeting.
2) Sparse-Reward Markov Game .
The CTA game can have a long episode length, and agents have to come up with a good action sequence before receiving the reward.
Especially the attacker must learn both low-level skills to perform attacks and high-level strategies to avoid defenders.
3) Environment Randomness .
Such randomness comes from randomized victim secret addresses and the random trajectory sampling of benign programs.
We propose our method based on the three crucial features.
4. O UR A PPROACH : MACTA
In this paper, we introduce our approach, MACTA (Figure 3 Right), as an initial solution to the CTA challenge using MARL.
MACTA adopts Transformers (Vaswani et al., 2017) as the neural encoder of policy nets, Proximal Policy Optimization (PPO) (Schulman et al., 2017) as the reinforcement learning oracle, and Fictitious Play (FP) (Brown, 1951) as the meta game-theoretic tool.
To deal with the history-dependent partial observation and sparse reward, both the attacker and the detector are equipped with policy nets with transformer encoders.
The transformer encoder is mainly composed of scaled dot-product attention and multi-head self-attentions.
It can effectively integrate information from long time horizons and large-scale data while not suffering from vanishing or exploding gradients in recurrent neural networks (RNNs) (Parisotto et al., 2020).
MACTA
[Figure]
Q learns policy against P using PPO Q P
Detector
Attacker
Detector Detector
Attacker Attacker
Detector
Attacker
Detector Detector
Attacker
Attacker
The attacker and the detector optimize their policies by the PPO algorithm to effectively learn a policy in the Markov game.
Although independent reinforcement learning, where all agents are updating their policies simultaneously, is notoriously known for the instability issue in training Tan (1993), if we only train one agent at a time and keep others stationary, then other agents can be taken as a part of the environment, and PPO can effectively optimize the policy for higher cumulative rewards.
Iterated Best Response PPO (IBR-PPO)
(Figure 3 Left) is the most naive way of implementing the above idea.
It alternates the training of the attacker and detector so that they learn the best policy against the most recent opponent.
However, it may fall into the cyclic policy learning issue and never converge to any Nash Equilibrium (Roughgarden, 2010).
As a widely accepted method in MARL, creating a diverse pool of opponents and learning the best response to a mixture of them can alleviate the cyclic issue and help with generalization.
Similar to fictitious play in the game theory, we create a pool for each agent and add their historical policies to the pool.
Concretely, for each iteration τ , we denote the set of policies explored until τ of agent i by our method as Π iτ , the opponents’ joint policy set as Π − i τ .
Then we learn the best response (BR), π i ∗ ( U (Π − i τ )) , to the uniform mixture of opponents’ policy pool using a best response oracle (e.g.
PPO), and add the best response to the policy pool.
Mathematically, for each iteration
[Equation]
where − i represents all players except for player i , and U is the uniform distribution.
There are more advanced meta game frameworks like Double Oracle (McMahan et al., 2003) and Policy Space Response Oracle (Lanctot et al., 2017), which measures the meta game payoff matrix among different explored policies and solves the matrix for the best opponent mixture.
In our case, since the environment contains some randomness, it is inefficient to estimate precisely the payoff matrix.
We will leave the more advanced game framework as future work.
The above components constitute our approach (Algorithm 1).
MACTA alternates the training of attacker and detector every E epochs and adds one deterministic policy checkpoint of the learning agent to the agent’s policy pool every N epochs.
During one agent’s training, the agent faces a uniform mixture of all opponents’ past deterministic policy checkpoints.
Note that we create such a mixture by uniformly sampling policies from the opponent’s policy pool at every action step.
4. I MPLEMENTATION D ETAILS
Specifically, we start with empty policy pools for both agents, first train the attacker for 50 epochs (each epoch contains 3000 training steps) to gain the basic skills of obtaining information from the victim program, and add one policy to the attacker’s policy pool every 10 epochs.
Then we stop the attacker’s training and switch to train the detector against the pool of the first 5 attacker policies for 50 epochs.
Similarly, the detector will have 5 policies by the end of this training iteration (50 epochs).
The above process is repeated until the target training iterations (2000 epochs).
We adopt an Actor-Critic implementation of PPO for both the attacker and the detector, and both policy net and value net are 1-layer 8-head Transformer encoders with different output heads.
We leverage the RLMeta (Yang et al., 2022) learning framework for the PPO implementation, which is an asynchronous version of PPO with sampling and learning in parallel, and construct our multi-agent learning framework on top of it.
Refer to Appendix A.6 for a more detailed description of training and environment hyper-parameters.
E XPERIMENTS 5. E VALUATION S ETUP AND M ETRICS
To evaluate the proposed MARL method, we study a few attackers and detector baselines.
For the attackers, we consider a textbook attack Prime+Probe (Algorithm 2), the RL-based attacker (AutoCAT) (Luo et al., 2022), and PPO with Iterative Best Response Oracle (IBR-PPO Attacker).
For the detectors, we include CC-Hunter (Chen & Venkataramani, 2014)
(Appendix A.7), Cyclone (Harris et al., 2019) (Appendix A.7), and PPO with Iterative Best Response Oracle (IBR-PPO Detector).
The episodic cumulative reward is one way to measure the performance of the attacker and the detector, but it is less intuitive.
In this work, we employ more intuitive metrics including Attack
Correct Rate , Attacks per Episode , Detection Rate , Episode Length , and False Alarm Rate .
Details are listed in Table 1. [Table]
Alarm
[Table]
5. R ESULTS
All the experiment results below are reported on an 8set-1way L1 cache.
The attacker’s memory address range is 8-15 and the victim’s secret address is randomly chosen between 0-7.
The episode length is 64 steps.
To evaluate different methods, we report the statistics based on three independent training instances for each learning-based method and control the final policies from different instances of a method undergoing the same amount of optimization steps.
5.2. A TTACKER P ERFORMANCE
[Table]
We first evaluate the attacker agent’s performance in terms of attack correct rate and number of attacks in an episode, to validate that the attacker agent is conducting effective attacks.
We compare the
PPO attacker agents (IBR-PPO and MACTA) trained by us with existing attacker strategies (textbook Prime+Probe and AutoCAT).
We evaluate the attackers without detectors for 10,000 episodes and the statistics are shown in Table 2.
Every attacker evaluated can achieve a decent attack correct rate, showing the agent acquires effective attack policies.
In addition, MACTA has the smallest number of attacks per episode among the learning-based methods, indicating the preventative strategies for detection.
Example attack sequences demonstrating the strategic attack behaviors can be found in Section A.3.
5.2. H EAD - TO -H EAD E VALUATIONS
In this head-to-head evaluation
[Table]
we have an attacker play against a detector from different training instances for 10,000 episodes and report the mean detection rate and the mean episode length for all attacker and detector pairs.
The head-to-head evaluation results can be found in Table 3 and Table 4.
We also report the mean false alarm rate and mean episode length of the detectors on the unseen
Benign agents in the last column of the table.
[Table] [Table]
We find that the heuristic detector CC-Hunter can not effectively discriminate the RL attackers from benign agents.
Tuning the auto-correlation threshold only returns either a high false alarm rate or a low detection rate.
The SVM-based detector, Cyclone, is able to perform well (99.3% detection rate) on the heuristic attack (Prime+Probe) that it is trained on, but has low detection rate on RL attackers.
Another drawback of these methods is that they require fixed-length observation that is longer than the steps needed for RL to complete attacks (usually 12 steps in this cache configuration).
IBRPPO obviously falls into the cyclic policy learning issue, as the detector is able to react well (98.3% detection rate) to the attacker that it trained against but fails to respond to any other kind of attackers.
MACTA, however, is able to generalize to unseen attacks such as Prime+Probe and the IBR-PPO attacker.
At the same time, MACTA also has a low false positive rate and fast detection speed.
We hypothesize that MACTA can abstract the general pattern of the attackers from interacting with diverse attacker strategies during training.
On the other hand, since the detector is trained to block all the previous attack policies, the attacker had to explore new policy space to escape from the detection.
The MACTA attackers are able to escape from a variety of unseen detectors.
All of the above findings highlight the benefits of using
MARL solution concepts in learning the detectors.
5.2. E XPLOITABILITY E VALUATIONS
We measure how a detector can be exploited by attackers within one fictitious play iteration ( training epochs).
We fix a detector strategy and train an RL exploiter (i.e., an attacker) against the detector from scratch.
The training curve of the exploiters of MACTA detectors can be found in
Figure 4.
As the training time of the MACTA detectors increases, it becomes more difficult for an RL attacker to bypass the detectors.
Specifically, it will take the RL exploiter attacker longer to find a meaningful attack strategy.
And even though the RL exploiter attacker can learn to attack eventually, the number of total attacks decreases when the detector sees more attackers.
The decrease in the number of attacks can come from the slower attack speed (counter to possible detection) or faster detection speed so fewer attacks can be performed.
5. N EURAL A RCHITECTURE S TUDY
The neural architecture design for the policy networks has been relatively under-explored in the RL community.
Our CTA task is an example where neural architecture plays a critical role in learning
[Figure] [Figure]
a meaningful policy.
We train attacker-only tasks using different network architectures (details in Appendix A.5) [Figure] and the comparisons can be found in Figure 5.
For RL attackers, MLP with residual connection (He et al., 2016) fails to achieve a high episode return, while the Transformer and
LSTM (Hochreiter & Schmidhuber, 1997) entries succeed.
For Transformers, our study shows that increasing the number of encoder layers in the Transformer can speed up learning while reducing the number of heads has counter effects.
The above evidence shows that the sequence modeling structure is critical for CTA attack policy learning and Transformer is more efficient than LSTM in terms of training steps.
Our hypothesis is that a successful attack is composed of a series of events, which may contain history-dependent relations among events, and Transformer can effectively model such relations.
R ELATED W ORK
Detectors for Cache Timing Attacks Since the discovery of side-channel attacks, research efforts have also been made to build a detector against attacks.
CC-Hunter (Chen & Venkataramani, 2014) captures recurrent patterns generated during cache contention while adversaries exploit the victim process.
More specifically, it used autocorrelation to detect periodic interleaving between the two event trains (Luo et al., 2022).
ReplayConfusion (Yan et al., 2016) records and deterministically replays a program’s memory traces, changing the mapping of cache addresses but retaining the cadences.
Executing the traces in different memory addresses can expose abnormal access patterns observed between an attacker and a victim, which do not exist in benign traces.
Cyclone (Harris et al., 2019) uses cyclic interference identified in cache contention during an attack.
This detector assigns domain tags between the two processes, then uses performance counters to enumerate abnormal cache contention behaviors triggered by each domain tag.
PerSpectron (Mirbagher-Ajorpaz et al., 2020) trains a neural network classifier using the memory and latency event logs generated from attack examples.
Existing detectors based on known attacks cannot deal with the evolving attacker issue.
Our study shows that the RL attacker can learn novel strategies to bypass the existing static detectors.
Our method solves this problem by enabling the auto-discovery of attacker policies.
Game Theory in Security Games Game theory provides a framework for decision-making and strategy, modeling how selfish agents interact and affect system outcomes.
In Stackelberg games, a defender must first commit limited resources to protect disparate locations and an attacker that subsequently targets locations, potentially having seen the configuration of defenses (e.g., (Bier et al., 2007)).
Such games have masked systems from probes (Schlenker et al., 2018), defended systems against varied attack types (Thakoor et al., 2020), and assigned human analysts to automated system alerts (Schlenker et al., 2017).
Whereas Stackelberg requires the defender to move first, we consider how the defender’s policy should respond to the attacker’s evolving policy.
Game theory inspires generative adversarial networks (GAN) for security (Zolbayar et al., 2021; Baimukan &
Zhu, 2021).
Unlike prior works that explore adversarial samples in the neighborhood of a heuristic attack policy, our RL approach explores a broader, unknown space of attack policies with a welldefined objective.
RL is an instance of stochastic games, often modeled by a Markov Decision
Process.
Representative studies of such games for distributed systems include threat detection and resource allocation (Krishnamurthy et al., 2007;
Fan et al., 2019).
We are the first to formulate a stochastic game for realistic, practical hardware timing attacks.
Population-based Multi-agent Reinforcement Learning Independent Reinforcement Learning in multi-agent environments suffers from the non-stationary opponent issue (Tan, 1993).
While Iterative Best Response methods alleviate the above problem by learning from stationary opponents, they tend to over-fit other players’ policies and cause cycles in policy learning (Vinyals et al., 2019).
Interacting with diverse opponent policies or heterogeneous agents is one effective way to avoid such cycles.
Population-based MARL is thus proposed to solve large-scale extensive form games by creating a diverse pool of agents.
Related work includes population-based reinforcement learning (Parker-Holder et al., 2020), Neural Fictitious Self-Play (Heinrich & Silver, 2016), Fictitious Co-Play (Strouse et al., 2021), prioritized self-play (Vinyals et al., 2019), Double Oracle (DO) (McMahan et al., 2003) and its generalization Policy Space Response Oracle (PSRO) (Lanctot et al., 2017) and so on.
The most related application of population-based MARL to the security games, such as Eghtesad et al.
(2020) and Wang et al.
(2019), use variants of Double Oracle, but they deal with different and less stochastic domains than ours.
C ONCLUSIONS AND F UTURE W ORK
In conclusion, our work explores the application of multi-agent reinforcement learning in the cache timing attack and detection domain.
We first introduce the environment MA-A UTO CAT that allows learning for both attackers and detectors, and their complex interactions with caches.
Then we propose to combine the game-theoretic concept of Fictitious Play and Proximal Policy Optimization to train both agents (MACTA).
Empirically, we found that the detector generated by MACTA can capture the general pattern of attacks and generalize to novel attacks.
The exploitability study of the detector also indicates the detectors can impede the learning process of new attackers and slow down the attacks.
On the other hand, the MACTA attacker is able to explore new policy space and mimic the benign agents to bypass the detectors.
Finally, the neural architecture study demonstrates the strong representation-ability of Transformers.
Deployment in Real Systems Like prior works (Luo et al., 2022; Harris et al., 2019; MirbagherAjorpaz et al., 2020), we use a cache simulator to study CTA.
We believe the trained attacker and detector can be applied to real hardware with engineering efforts.
For attackers, a recent study (Luo et al., 2022) demonstrated that an attack pattern learned by the RL agent using a cache simulator can be applied to multiple Intel processors. Similarly, we also show that the attack sequences from a
MACTA attack work on commercial processors in Appendix A.4.
For the detectors, with hardware changes, the neural network model can be deployed inside a processor with reasonable area and power overhead, as demonstrated by Mirbagher-Ajorpaz et al.
Convergence of the Policies Three main concerns for reinforcement learning algorithms are deep learning, bootstrapping, and off-policy.
In MACTA, we adopt the Transformer based PPO algorithm as the policy learning oracle, so there is no guarantee that the algorithm will return the best response to the opponents in limited optimization steps in such a complex and stochastic environment.
Meanwhile, although Fictitious Play can generate a NE policy in two-player zero-sum games, it has been proven to result in cyclic policies in a specific case (Shapley, 1964).
Little previous work discusses its convergence when used as a game-theoretic meta solver in the general-sum MARL setting.
In this work, we only establish empirical findings but leave the theoretical analysis for future work.
O PTIONAL S TATEMENTS
Dear reviewers, please note that this section is optional and does not count into the 9-page limit.
Ethical Statement
Our work builds an environment enabling automated exploration of both attack and defense policies of CTA.
The attacker agent may be used for developing new attacks maliciously.
However, as shown in the vast amount of attack papers in computer security venues, exploring and understanding security attacks is a necessary and important step in developing future secure systems.
Compared with prior works on security attacks on real systems Liu et al. (2015);
Oren et al.
; Kocher et al.
; Lipp et al.
; Ravichandran et al.
(2022), the MACTA agent only explores the attacks in a cache without considering other system-level activities.
In addition, our framework results in a stronger detector, which can help design more secure systems.
Reproducibility Statement Our experiments are reproducible in a reasonable range near the mean performance across multiple training instances reported.
It is unavoidable that training with different random seeds will result in different results, given the variance of explorations in reinforcement learning.
However, our result is not a selection among multiple random seeds biased towards our benefits.
We repeated the training for another three instances, giving us a similar result.
We will provide model checkpoints and publish the code upon the publication of this paper.
A. E NVIRONMENT C ONFIGURATIONS
[Table]
Game Mechanism.
In the MA-A UTO CAT, within a fixed-length episode, the attacker agent can guess the secret address of the victim as many times as possible and get a reward for every correct guess ( successful attack ).
In the meantime, the detector agent can monitor the cache access history and interactions of two programs and decide whether to raise a flag/alarm to terminate the episode to prevent further information leakage.
Attacker’s Reward Function.
The attacker is punished by 0.01 for every time step, +10 if guess the victim’s secret successfully, -10 if incorrectly.
It will get a one-time punishment of 20 if it reaches a timeout without any attack, a one-time punishment of -10 if identified by detector.
The episode length to collect reward is affected by the detector.
Detector’s Reward Function.
The detector can raise a flag to terminate the episode.
If the detector raises a flag in an attack scenario, then the detector receives a reward for remaining steps [max step - current step]; if the detector raises a flag in a benign scenario, then it receives a large penalty [5 × max step].
If the detector lets the episode going, for benign scenario there is no punishment; while the detector gets -10 every time the attacker attacks successfully.
Attacker’s Action.
For each time step, the attacker can choose an action a a ∈ { a aX , a av , a a vr , a agY } , where a aX represents access address X, a av represents letting the victim access a secret-related address, a avr represents letting the victim access a random address and a agY represents guessing the secret address to be Y .
Detector’s Action.
For each time step, the defender can choose a d ∈ { a dterm , a dcont } where a term means terminate the episode and a dcont means let the episode keep going.
Attacker’s Observation.
The attacker’s observation space includes a history of attacker actions and memory access latency it receives from the cache simulator.
For each time step, a new step observation ( s alat , s avt , s aact , s astep ) is appended to the observation window, where s alat ∈ { s hit , s miss , s N.A. } represents the access latency, s avt ∈ { s t , s nt } represents whether to wait for the victim’s action, s act records the attacker’s current action and s astep is the current time step.
Detector’s Observation.
The detector can observe a history window of the memory access actions of both programs.
For each time step, the new observation is composed of ( s dlat , s dprogram , s daddr , s dstep ) , where s dlat ∈ { s hit , s miss } represents the access latency (access latency of all programs are visible to the detector), s dprogram ∈ { s a , s b } indicates the identity of the program, s daddr represents the address being accessed at the current step, and s dstep represents the current defender time step.
Benign Programs.
The actions of benign programs are sampled from the Standard Performance
Evaluation Corporation (SPEC)
benchmark Bucek et al.
(2018), a collection of memory access records of non-malicious programs.
Since the data is collected under different cache configurations, we project the actions onto the valid action space given the current cache configuration.
A. T RAJECTORY A NALYSIS
Benign
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table] [Figure] Prime + Probe
[Figure] [Figure] [Figure] [Figure]
AutoCAT [Figure] [Figure] [Figure]
IBR-PPO [Figure] [Figure]
MACTA [Figure] Figure 6 illustrates different attackers’ attack sequence given a fixed secret bit. Here, we use victim secret=5 as an example.
The top row shows a sampled pattern of benign programs. In that case, the two programs act independently and alter the access to the cache frequently. The Prime+Probe attacker cause cache contention by accessing cache frequently, and only invoke the victim to access the cache when needed. The AutoCAT attacker is more efficient in retrieving victim’s information then Prime+Probe.
Once contention with victim in one cache set is observed (i.e., a cache miss after the same address being accessed by the attacker), the attacker will make a guess without more memory accesses. The IBR-PPO attacker takes a similar strategy as AutoCAT’s, but it learns to insert some extra victim invocation steps to confuse the detector. The issue with extra invocation strategy is that the victim only accesses its secret bit, which can be easily captured by a detector.
The MACTA attacker however, learns more advanced strategies. It learns to invoke random victim accesses to alter its behavior to be more similar to benign programs.
Note that invoking random accesses from a victim can cause noise in the attacker’s latency observation and make the steps needed for a success attack longer. This means that to evade the MACTA detector, just inserting some extra victim invocation steps is far from enough. The attacker has to take a risk to invoke random accesses from a victim instead because the “easiest” policy space has been exhausted. A. R EAL H ARDWARE A NALYSIS
[Table]
We evaluated the effectiveness of the attack sequences produced by MACTA attackers on real hardware by running them on three commercial processors through CacheQuery (Vila et al., 2020).
The attack sequences are generated from a MACTA attacker that is trained using a simulated environment for a 1-set 8-way cache configuration, which match the number of ways in real hardware.
In this setting, the victim’s secret is either 0 or E (Empty) with equal probability, and the attacker can choose to invoke victim to access the secret address or a random address.
Then, the attack sequences were performed on real hardware to obtain the attack correct rate. Table 6 demonstrates the attacker policy from simulator can be transferred to real hardware with negligible discrepancy. Even though the caches in commercial processors have a large number of sets, the attacks from the MACTA attacker transfers well because each set operates independently. A. M ODEL H YPER - PARAMETERS
In section 5.3, we compared three different neural architectures: Transformers, LSTM and MLP.
All of the methods are controlled at a same scale of optimizable parameters, and trained with the same algorithms and computing resources.
Transformer.
In our experiments, all the policies uses a 8-head 1-encoder-layer transformer with hidden dimension of 128.
For model architecture study, we study the changes in the number of heads in the multi-head attention mechanism, and number of transformer layers.
LSTM. We employed a 1-layer LSTM with hidden dimension 256.
The input to the LSTM is a pre-padding history of the observation and we concatenate the last layer hidden and cell before connecting with a MLP.
MLP. The MLP model we used directly maps the one-hot encoded input to 4 residual blocks with hidden dimension 128.
Each residual block is composed of 2 ReLU-Linear layers with a residual connection.
A. A LGORITHM AND T RAINING H YPER - PARAMETERS
The MACTA algorithm is explained in Algorithm 1, and detailed training hyperparameters can be found in Table 7.
In MACTA, the policy pool is created by sampling a new model to do batch actions per step.
It is an infrastructure implementation to produce faster sampling speed, so it is not strictly a per-step sampling, but it is per-step sampling considering a large amount of data.
Additionally, we are exploring the per-trajectory policy sampling because it has nice theoretical properties.
Nevertheless, it needs further infrastructure support, and we are exploring that direction.
Algorithm do 5: π
6: if j mod N − 1 ==0 then 7:
P A ← P A ∪ π Aj 8: if 9:
←
+10: while 11: j ←12: do 13:
14: if j mod
−
0 then 15:
P D ← P D ∪ π Aj 16: if 17: j ← j
18: end while 19: end while 20: return π A , π D return the last policy of attacker and detector
[Table]
A. H EURISTIC C ACHE T IMING A TTACKS AND D ETECTORS
• Heuristic Attacker Algorithms – Prime+Probe (Algorithm 2)
Osvik et al.
First, in the prime phase, the attacker fills the cache set with its address value (lines 3), then waits for the victim to access the cache set.
Next, the victim accesses one of the cache sets, then replaces the loaded address value with its address (lines 5).
Lastly, in the probe phase, the attacker accesses the cache sets again, then measures the cache latency to load each set of the primed address value (lines 7 to 8).
In a cache set accessed by the victim, the attacker observes increased latency (cache miss).
• Detector Algorithms – CC-Hunter Chen & Venkataramani (2014).
Cache timing channels rely on the latency of events to perform timing modulation.
To send information, two processes ( i.e. , the trojan and the spy) generate a sufficient number of alternating conflict events (cache misses) to allow the adversary to decode the transmitted bit based on the average memory access times (hit/miss).
Those behaviors show periodic, oscillating patterns of conflicts between the two processes.
Therefore, autocorrelation is used to identify those patterns.
Autocorrelation is the correlation coefficient of the signal with a timelagged version of itself, along with measuring the event train X , as a variable at a time instance of t .
Two cases of conflict miss, i.e. , either the victim eviting the attacker’s cache line or the attacker evicting the victim’s cache line, are considered for the event trains Luo et al.
For example, we can check the autocorrelation C p at a time lag p , which is expressed as:
[Equation]
If there exists a time lag p which 1 ≤ p ≤ P , where P is a predefined parameter such that makes C p larger than a threshold value, then it is assumed as an attackLuo et al. (2022).
– Cyclone Harris et al.
The concept of cyclic interference is commonly found in all known cache contention side-channel attacks Harris et al.
and has been used for detecting those attack patterns.
Interference occurs from the attacker to the victim process or vice versa, considered directional, and affects the behavior of microarchitecture in a disruptive manner.
The cyclic interference can be noted as (a ⇝ b ⇝ a), where interference (a ⇝ b) is followed by (b ⇝ a).
However, interference including a third party between attacker and victim, like (a ⇝ b ⇝ c), is not considered cyclic interference.
To classify the anomaly and benign patterns, Cyclone adapted one-class support vector machine (SVM) and a scalable end-to-end tree boosting system (XGBoost Chen & Guestrin (2016)) models.
1: step ← step +2: if step < len ( attacker address range ) then 3: action = prime address ( step, cache size ) ▷ attacker fills cache by attacker’s address 4: else if step = len ( attacker address range ) then 5: action = trigger victim ( step ) ▷ victim accesses a cache and fills its own address 6: else 7: action = probe address ( step, cache size ) ▷ attacker access caches again 8: measure latency ( action ) 9: end if 10: if latency = 1 then ▷ attacker observes for any cache miss 11: action = guess ( action, cache size )
▷ attacker makes a guess on a victim’s secret address 12: end if 13: Return action