Abstract-We present CLEAR (Cross-Layer Exploration for Architecting Resilience), a first of its kind framework which overcomes a major challenge in the design of digital systems that are resilient to reliability failures: achieve desired resilience targets at minimal costs (energy, power, execution time, area) by combining resilience techniques across various layers of the system stack (circuit, logic, architecture, software, algorithm). This is also referred to as cross-layer resilience. In this paper, we focus on radiation-induced soft errors in processor cores. We address both single-event upsets (SEUs) and single-event multiple upsets (SEMUs) in terrestrial environments. Our framework automatically and systematically explores the large space of comprehensive resilience techniques and their combinations across various layers of the system stack (586 cross-layer combinations in this paper), derives cost-effective solutions that achieve resilience targets at minimal costs, and provides guidelines for the design of new resilience techniques. We demonstrate the practicality and effectiveness of our framework using two diverse designs: a simple, in-order processor core and a complex, out-of-order processor core. Our results demonstrate that a carefully optimized combination of circuit-level hardening, logic-level parity checking, and micro-architectural recovery provides a highly cost-effective soft error resilience solution for general-purpose processor cores. For example, a 50× improvement in silent data corruption rate is achieved at only 2.1% energy cost for an out-of-order core (6.1% for an in-order core) with no speed impact. However, (application-aware) selective circuit-level hardening alone, guided by a thorough analysis of the effects of soft errors on application benchmarks, provides a cost-effective soft error resilience solution as well (with ~1% additional energy cost for a 50× improvement in silent data corruption rate). Index Terms-Cross-layer resilience; soft errors I. INTRODUCTION HIS paper addresses the cross-layer resilience challenge for designing robust digital systems: given a set of resilience techniques at various abstraction layers (circuit, logic, architecture, software, algorithm), how does one protect a given design from radiation-induced soft errors using (perhaps) a combination of these techniques, across multiple abstraction layers, such that overall soft error resilience targets are met at minimal costs (energy, power, execution time, area)? Specific soft error resilience targets addressed in this paper are: Silent Data Corruption (SDC), where an error causes the system to output an incorrect result without error indication; and, Detected but Uncorrected Error (DUE), This research is supported in part by DARPA, DTRA, NSF, and SRC. The views expressed are those of the authors and do not reflect the official policy or position of the Department of Defense or the U.S. Government. E. Cheng, S. Mirkhani, and S. Mitra are with Stanford University (e-mail: eccheng@ stanford.edu). H. Cho is with Hongik University. where an error is detected (e.g., by a resilience technique or a system crash or hang) but is not recovered automatically without user intervention. The need for cross-layer resilience, where multiple error resilience techniques from different layers of the system stack cooperate to achieve cost-effective error resilience, is articulated in several publications (e.g., [1] , [2] , [3] , [4] , [5] , [6] , [7] ). There are numerous publications on error resilience techniques, many of which span multiple abstraction layers. These publications mostly describe specific implementations. Examples include structural integrity checking [8] and its derivatives (mostly spanning architecture and software layers) or the combined use of circuit hardening, error detection (e.g., using logic parity checking and residue codes) and instructionlevel retry [9] , [10] , [11] (spanning circuit, logic, and architecture layers). Cross-layer resilience implementations in commercial systems are often based on "designer experience" or "historical practice." There exists no comprehensive framework to systematically address the cross-layer resilience challenge. Creating such a framework is difficult. It must encompass the entire design flow end-to-end, from comprehensive and thorough analysis of various combinations of error resilience techniques all the way to layout-level implementations, such that one can (automatically) determine which resilience technique or combination of techniques (either at the same abstraction layer or across different abstraction layers) should be chosen. Such a framework is essential in order to answer important cross-layer resilience questions such as: 1) Is cross-layer resilience the best approach for achieving a given resilience target at low cost? 2) Are all cross-layer solutions equally cost-effective? If not, which cross-layer solutions are the best? 3) How do cross-layer choices change depending on application-level energy, latency, and area constraints? 4) How can one create a cross-layer solution that is costeffective across a wide variety of application workloads? 5) Are there general guidelines for new error resilience techniques to be cost-effective? We present CLEAR (Cross-Layer Exploration for Architecting Resilience), a first of its kind framework, which addresses the cross-layer resilience challenge. In this paper, we focus on the use of CLEAR for radiation-induced soft errors 1 in terrestrial environments. Although the soft error rate of an SRAM cell or a flip-flop stays roughly constant or even decreases over technology generations, the system-level soft error rate increases with increased integration [12] , [13] , [14] , [15] . Moreover, soft error rates can increase when lower supply voltages are used to improve energy efficiency [16] , [17] . We focus on flip-flop soft errors because design techniques to protect them are generally expensive. Coding techniques are routinely used for protecting on-chip memories. Combinational logic circuits are significantly less susceptible to soft errors and do not pose a concern [18] , [14] . We address both single-event upsets 1 Other error sources (voltage noise and circuit-aging) may be incorporated into CLEAR, but are not the focus of this paper. Eric Cheng, Student Member, IEEE, Shahrzad Mirkhani, Member, IEEE, Lukasz G. Szafaryn, Chen-Yong Cher, Senior Member, IEEE, Hyungmin Cho, Kevin Skadron, Fellow, IEEE, Mircea R. Stan, Fellow, IEEE, Klas Lilja, Member, IEEE, Jacob A. Abraham, Fellow, IEEE, Pradip Bose, Fellow, IEEE, and Subhasish Mitra, Fellow, IEEE Tolerating Soft Errors in Processor Cores Using CLEAR (Cross-Layer Exploration for Architecting Resilience) T 2 (SEUs) and single-event multiple upsets (SEMUs) [19] , [17] . While CLEAR can address soft errors in various digital components of a complex System-on-a-Chip (including uncore components [20] and hardware accelerators), a detailed analysis of soft errors in all these components is beyond the scope of this paper. Hence, we focus on soft errors in processor cores. To demonstrate the effectiveness and practicality of CLEAR, we explore 586 cross-layer combinations using ten representative error detection/correction techniques and four hardware error recovery techniques. These techniques span various layers of the system stack: circuit, logic, architecture, software, and algorithm (Fig. 1 ). Our extensive cross-layer exploration encompasses over 9 million flip-flop soft error injections into two diverse processor core architectures (Table  I) : a simple in-order SPARC Leon3 core (InO-core) and a complex super-scalar out-of-order Alpha IVM core (OoOcore), across 18 benchmarks: SPECINT2000 [21] and DARPA PERFECT [22] . Such extensive exploration enables us to conclusively answer the cross-layer resilience questions: 1) For a wide range of error resilience targets, optimized cross-layer combinations can provide low cost solutions for soft errors. 2) Not all cross-layer solutions are cost-effective. a) For general-purpose processor cores, a carefully optimized combination of selective circuit-level hardening, logic-level parity checking, and microarchitectural recovery provides a highly effective cross-layer resilience solution. For example, a 50× SDC improvement (defined in Sec. II.A) is achieved at 2.1% and 6.1% energy costs for the OoO-and InOcores, respectively. The use of selective circuit-level hardening and logic-level parity checking is guided by a thorough analysis of the effects of soft errors on application benchmarks. b) When the application space can be restricted to matrix operations, a cross-layer combination of Algorithm Based Fault Tolerance (ABFT) correction, selective circuit-level hardening, logic-level parity checking, and micro-architectural recovery can be highly effective. For example, a 50× SDC improvement is achieved at 1.9% and 3.1% energy costs for the OoOand InO-cores, respectively. But, this approach may not be practical for general-purpose processor cores targeting general applications. c) (Application-aware) selective circuit-level hardening, guided by a thorough analysis of the effects of soft errors on application benchmarks, provides a highly effective soft error resilience approach. For example, a 50× SDC improvement is achieved at 3.1% and 7.3% energy costs for the OoO-and InO-cores, respectively. 3) The above conclusions about cost-effective soft error resilience techniques largely hold across various application characteristics (e.g., latency constraints despite errors in soft real-time applications). 4) Selective circuit-level hardening (and logic-level parity checking) techniques are guided by the analysis of the effects of soft errors on application benchmarks. Hence, one must address the challenge of potential mismatch between application benchmarks vs. applications in the field, especially when targeting high degrees of resilience (e.g., 10× or more SDC improvement). We overcome this challenge using various flavors of circuit-level hardening techniques (details in Sec. IV). 5) Cost-effective resilience approaches discussed above provide bounds that new soft error resilience techniques must achieve to be competitive. It is, however, crucial that the benefits and costs of new techniques are evaluated thoroughly and correctly before publication. This paper extends work published in [23] by: 1) Providing in-depth analysis of existing soft error detection and correction techniques to identify the basis of hidden costs, pinpoint the specific errors and flip-flop locations covered, and understand design intricacies that impact cost and reliability improvement for each technique. 2) Detailing results that were previously omitted for space constraints to reinforce our conclusion that many existing resilience techniques do not generate cost-effective crosslayer solutions since significant augmentation, using circuit-and logic-level techniques, is required regardless. 3) Considering the limitations and efficiency losses (cost and resilience improvement) for cross-layer solutions when working with fixed hardware (e.g., unmodifiable designs supplied by an upstream vendor, commercial off-the-shelf products, or existing platforms mandated by a customer). Fig. 1 gives an overview of the CLEAR framework. Individual components of the framework are discussed below. II. CLEAR FRAMEWORK CLEAR is not merely an error rate projection tool; rather, reliability analysis is one component of the overall framework. We use flip-flop soft error injections for reliability analysis with respect to radiation-induced soft errors. This is because radiation test results confirm that injection of single bit-flips into flip-flops closely models soft error behaviors in actual systems [24] , [25] . Furthermore, flip-flop-level error injection is crucial since naïve high-level error injections can be highly inaccurate [26] . For individual flip-flops, both SEUs and SEMUs (i.e., multi-node upsets within a flip-flop) manifest as single-bit errors. Based on measured probabilities for multi- 3 cell upsets in SRAM cell arrays, it can be concluded that the probability of a SEMU causing multi-bit upsets across multiple flip-flops is much less than 1% of the single-bit upset probability [14] , [15] . Regardless, our SEMU-tolerant circuit hardening and our layout implementations ensure that baseline and resilient designs fully-consider the impact of SEMUs. We injected over 9 million flip-flop soft errors into the RTL of the processor designs using three BEE3 FPGA emulation systems and also using mixed-mode simulations on the Stampede supercomputer (TACC at The University of Texas at Austin) (similar to [26] Using the above outcomes, any error that results in OMM causes SDC (i.e., an SDC-causing error). Any error that results in UT, Hang, or ED causes DUE (i.e., a DUE-causing error). Note that, there are no ED outcomes if no error detection technique is employed. The resilience of a protected (new) design compared to an unprotected (original, baseline) design can be defined in terms of SDC improvement (Eq. 1a) or DUE improvement (Eq. 1b). The susceptibility of flip-flops to soft errors is assumed to be uniform across all flip-flops in the design (but this parameter is adjustable in our framework). Resilience techniques that increase the execution time of an application or add additional hardware also increase the susceptibility of the design to soft-errors. To accurately account for this situation, we calculate, based on [32], a correction factor γ (where γ ≥ 1), which is applied to ensure a fair and accurate comparison for all techniques 3 . Take for instance the monitor core technique; in our implementation, it increases the number of flip-flops in a resilient OoO-core by 38%. These extra flip-flops become additional locations for soft errors to occur. This results in a γ correction of 1.38 in order to account for the increased soft error susceptibility of the design. Techniques which increase execution time have a similar impact. For example, CFCSS incurs a 40.6% execution time impact; a corresponding γ correction of 1.41. A technique such as DFC, which increases flip-flop count (20%) and execution time (6.2%), would need a γ correction of 1.28 2 11 SPEC / 7 PERFECT benchmarks for InO-cores and 8 SPEC / 3 PERFECT for OoO-cores (missing benchmarks contain floating-point instructions not executable by the OoO-core RTL model). 3 Research literature commonly considers γ=1. We report results using true γ values, but our conclusions hold for γ=1 as well (the latter is optimistic). (1.2×1.062) since the impact is multiplicative (increased flipflop count over an increased duration). The γ correction factor allows us to account for these increased susceptibilities for fair and accurate comparisons of all resilience techniques considered [32] . SDC and DUE improvements with γ=1 can be back-calculated by multiplying the reported γ value in Table III and do not change our conclusions. (1a) (1b) Reporting SDC and DUE improvements allows our results to be agnostic to absolute error rates (we analyze the systemlevel behavior given a soft error). This allows a designer the flexibility to target the relative improvement needed in order to achieve an error rate for his or her specific application and technology. Although we have described the use of error injection-driven reliability analysis, the modular nature of CLEAR allows us to swap in other approaches as appropriate (e.g., our error injection analysis could be substituted with techniques like [33] , once they are properly validated). As shown in Table II , across our set of applications, not all flip-flops will have errors that result in SDC or DUE (errors in 19% of flip-flops in the InO-core and 39% of flip-flops in the OoO-core always vanish regardless of the application). The logic design structures (e.g., lowest hierarchical-level RTL component) these flip-flops belong to are listed in [34] . This phenomenon has been documented in the literature [35] and is due to the fact that errors that impact certain structures (e.g., branch predictor, debug registers, etc.) have no effect on program execution or correctness. Additionally, this means that resilience techniques would not normally need to be applied to these flip-flops. However, for completeness, we also report design points which would achieve the maximum improvement possible, where resilience is added to every single flip-flop (including those with errors that always vanish). This maximum improvement point provides an upper bound for cost (given the possibility that for a future application, a flip-flop that currently has errors that always vanish may encounter an SDC-or DUE-causing error). Error Detection Latency (the time elapsed from when an error occurs in the processor to when a resilience technique detects the error) is also an important aspect to consider. An end-to-end reliable system must not only detect errors, but also recover from these detected errors. Long detection latencies impact the amount of computation that needs to be recovered and can also limit the types of recovery that are capable of recovering the detected error (Sec. II.D). Execution time is estimated using FPGA emulation and RTL simulation. Applications are run to completion to accurately capture the execution time of an unprotected design. We also report the error-free execution time impact associated with resilience techniques at the architecture, software, and algorithm levels. For resilience techniques at the circuit and logic levels, our design methodology maintains the same clock speed as the unprotected design. We [41] . The LEAP-ctrl flip-flop is a special design, which can operate in resilient (high resilience, high power) and economy (low resilience, low power) modes. It is useful in situations where a software or algorithm technique only provides protection when running specific applications and thus, selectively enabling low-level hardware resilience when the former techniques are unavailable may be beneficial. Although a variety of hardened flip-flops are available in the literature and can be used in our framework (see Additional Techniques), LEAP was chosen as a case study due to the extensive characterization data available. While Error Detection Sequential (EDS) [42] , [43] was originally designed to detect timing errors, it can be used to detect flip-flop soft errors as well. While EDS incurs less overhead at the individual flip-flop level vs. LEAP-DICE, for example, EDS requires delay buffers to ensure minimum hold constraints, aggregation and routing of error detection signals to an output (or recovery module), and a recovery mechanism to correct detected errors. These factors can significantly increase the overall costs for implementing a resilient design utilizing EDS (Table XVI) . Logic: Parity checking provides error detection by checking 5 flip-flop inputs and outputs [44] . Our design heuristics reduce the cost of parity while also ensuring that clock frequency is maintained as in the original design (by varying the number of flip-flops checked together, grouping flip-flops by timing slack, pipelining parity checker logic, etc.) Naïve implementations of parity checking can otherwise degrade design frequency by up to 200 MHz (20%) or increase energy cost by 80% on the InO-core. We minimize SEMUs through layouts that ensure a minimum spacing (the size of one flipflop) between flip-flops checked by the same parity checker. This ensures that only one flip-flop, in a group of flip-flops checked by the same parity checker, will encounter an upset due to a single strike in our 28nm technology in terrestrial environments [45] . Although a single strike could impact multiple flip-flops, since these flip-flops are checked by different checkers, the upsets will be detected. Since this absolute minimum spacing will remain constant, the relative spacing required between flip-flops will increase at smaller technology nodes, which may exacerbate the difficulty of implementation. Minimum spacing is enforced by applying design constraints during the layout stage. This constraint is important because even in large designs, flip-flops will still tend to be placed very close to one another. Table V shows the distribution of distances that each flip-flop has to its next nearest neighbor in a baseline design (this does not correspond to the spacing between flip-flops checked by the same logic parity checker). As shown, the majority of flip-flops are actually placed such that they would be susceptible to a SEMU. After applying parity checking, we see that no flipflop, within a group checked by the same parity checker, is placed such that it will be vulnerable to a SEMU (Table VI) . Logic parity is implemented using an XOR-tree based predictor and checker, which detects flip-flops soft errors. This implementation differs from logic parity prediction, which also targets errors inside combinational logic [46] . XOR-tree logic parity is sufficient for detecting flip-flop soft errors (with the minimum spacing constraint applied). "Pipelining" in the predictor tree ( Fig. 2 ) may be required to ensure 0% clock period impact. We evaluated the following heuristics for forming parity groups (the specific flip-flops that are checked together) to minimize cost of parity (cost comparisons in Table VII When unpipelined parity can be used, it is better to use larger-sized groups (e.g., 32-bit groups) in order to amortize the additional predictor/checker logic to the number of flipflops protected. However, when pipelined parity is required, we found 16-bit groups to be a good option. This is because beyond 16-bits, additional pipeline flip-flops begin to dominate costs. These factors have driven our implementation of the previously described heuristics. Architecture: Our implementation of Data Flow Checking (DFC), which checks static dataflow graphs, includes Control Flow Checking (CFC), which checks static control-flow graphs. This combination checker resembles that of [47] , which is also similar to the checker in [8] . Compiler optimization embeds the static signatures required by the checkers into unused delay slots in the software, thereby reducing execution time overhead by 13%. Table VIII helps explain why DFC is unable to provide high SDC and DUE improvement. Of flip-flops that have errors that result in SDCs and DUEs (Sec. II.A), DFC checkers detect SDCs and DUEs in less than 68% of these flip-flops (these 68% of flip-flops are distributed across all pipeline stages). For these 68% of flip-flops, on average, DFC detects less than 40% of the errors that result in SDCs or DUEs. This is because not all errors that result in an SDC or DUE will corrupt the dataflow or control flow signatures checked by the technique (e.g., register contents are corrupted and written out to a file, but the executed instructions remain unchanged). The combination of these factors means DFC is only detecting ~30% of SDCs or DUEs; thus, the technique provides low resilience improvement. These results are consistent with previously published data (detection of ~16% of non-vanished errors) on effectiveness of DFC checkers in simple cores [47] . Monitor cores are checker cores that validate instructions executed by the main core (e.g., [48] , [8] ). We analyze monitor cores similar to [48] . For InO-cores, the size of the monitor core is of the same order as the main core, and hence, excluded from our study. For OoO-cores, the simpler monitor core can have lower throughput compared to the main core and thus stall the main core. We confirm (via IPC estimation) that our monitor core implementation is sufficient to run the required checking operations without stalling the main core (Table IX) . Software: Software assertions for general-purpose processors 13 check program variables to detect errors. We combine assertions from [49], [50] to check both data and control variables to maximize error coverage. Checks for data variables (e.g., end result) are added via compiler transformations using training inputs to determine the valid range of values for these variables (e.g., likely program invariants). Since such assertion checks are added based on training inputs, it is possible to encounter false positives, where an error is reported in an error-free run. We have determined this false positive rate by training the assertions using representative inputs. However, we perform final analysis by incorporating the input data used during evaluation into the training step in order to give the technique the best possible benefit and to eliminate the occurrence of false positives. Checks for control variables (e.g., loop index, stack pointer, array address) are determined using application profiling and are manually added in the assembly code. In Table X , we breakdown the contribution to cost, improvement, and false positives resulting from assertions checking data variables [50] vs. those checking control variables [49] . Table XI demonstrates the importance of evaluating resilience techniques using accurate error injection (explained in [26] ). Depending on the particular error injection model used, SDC improvement could be over-estimated for one benchmark and under-estimated for another. For instance, using inaccurate architecture register error injection (regU), one would be led to believe that software assertions provide 3× the SDC improvement than they do in reality (e.g., when evaluated using flip-flop-level error injection). In order to pinpoint the sources of inaccuracy between the actual improvement rates that were determined using accurate flip-flop-level error injection vs. those published in the literature, we conducted error injection campaigns at other 13 Same applications studied in [49] . levels of abstraction (architecture register and program variable). However, even then, we were unable to exactly reproduce previously published improvement rates. Some additional differences in our architecture and program variable injection methodology compared to the published methodology may account for this discrepancy: 1) Our architecture register and program variable evaluations were conducted on a SPARCv8 in-order design rather than a SPARCv9 out-of-order design. 2) Our architecture register and program variable methodology injects errors uniformly into all program instructions while previous publications only inject into integer instructions of floating-point benchmarks. 3) Our architecture register and program variable methodology injects errors uniformly over the full application rather than injecting only into the core of the application during computation. 4) Since our architecture register and program variable methodology injects errors uniformly into all possible error candidates (e.g., all cycles and targets), the calculated improvement covers the entire design. Previous publications calculated improvement over the limited subset of error candidates (out of all possible error candidates) that were injected into and thus, only covers a subset of the design. Control Flow Checking by Software Signatures (CFCSS) checks static control flow graphs and is implemented via compiler modification similar to [51] . We can analyze CFCSS in further detail to gain deeper understanding as to why improvement for the technique is relatively low (Table XII) . Compared to DFC (a technique with a similar concept), we see that CFCSS offers slightly better SDC improvement. However, since CFCSS only checks control flow signatures, many SDCs will still escape (e.g., the result of an add is corrupted and written to file). Additionally, certain DUEs, such as those which may cause a program crash, will not be detectable by CFCSS, or other software techniques, since execution may abort before a corresponding software check can be triggered. The relatively low resilience improvement of CFCSS has been corroborated in actual systems as well [52] . Error Detection by Duplicated Instructions (EDDI) provides instruction redundant execution via compiler modification [53] . We utilize EDDI with store-readback [54] to maximize coverage by ensuring that values are written correctly. From Table XIII , it is clear why store-readback is important for EDDI. In order to achieve high SDC improvements, nearly all SDC causing errors need to be detected. By detecting an additional 12% of SDCs, storereadback increases SDC improvement of EDDI by an order of magnitude. Virtually all escaped SDCs are caught by ensuring that the values being written to the output are indeed correct (by reading back the written value). However, given that some SDC-or DUE-causing errors are still not detected by the technique, the results show that using naïve high-level [60] . Although ABFT correction algorithms can be used for detection-only (with minimally reduced execution time impact), ABFT detection algorithms cannot be used for correction. There is often a large difference in execution time impact between ABFT algorithms as well depending on the complexity of check calculation required. An ABFT correction technique for matrix inner product, for example, requires simple modular checksums (e.g., generated by adding all elements in a matrix row) -an inexpensive computation. On the other hand, ABFT detection for FFT, for example, requires expensive calculations using Parseval's theorem [61] . For the particular applications we studied, the algorithms that were protected using ABFT detection often required more computationally-expensive checks than algorithms that were protected using ABFT correction; therefore, the former generally had greater execution time impact (relative to each of their own original baseline execution times). An additional complication arises when an ABFT detection-only algorithm is implemented. Due to the long error detection latencies imposed by ABFT detection (9.6 million cycles, on average), hardware recovery techniques are not feasible and higher level recovery mechanisms will impose significant overheads. Recovery: We consider two recovery scenarios: bounded latency, i.e., an error must be recovered within a fixed period of time after its occurrence, and unconstrained, i.e., where no latency constraints exist and errors are recovered externally once detected (no hardware recovery is required). Bounded latency recovery is achieved using one of the following hardware recovery techniques (Table XV) : flush or reorder buffer (RoB) recovery (both of which rely on flushing noncommitted instructions followed by re-execution) [62] , [63] ; instruction replay (IR) or extended instruction replay (EIR) recovery (both of which rely on instruction checkpointing to rollback and replay instructions) [10] . EIR is an extension of IR with additional buffers required by DFC for recovery. Flush and RoB are unable to recover from errors detected after the memory write stage of InO-cores or after the reorder buffer of OoO-cores, respectively (these errors will have propagated to architecture visible states). Hence, LEAP-DICE is used to protect flip-flops in these pipeline stages when using flush/RoB recovery. IR and EIR can recover detected errors in any pipeline flip-flop. IR recovery is shown in in Fig. 4 and flush recovery is shown in Fig. 5 . Since recovery hardware serves as single points of failure, flip-flops in the recovery hardware itself need to be capable of error correction (e.g., protected by hardened flip-flops when considering soft errors). Additional Techniques: Many additional resilience techniques have been published in literature; but, these techniques are closely related to our evaluated techniques. Therefore, we believe that our results are representative and largely cover the cross-layer design space. At Although it is difficult to faithfully compare these "selective" EDDI techniques as published (since the original authors evaluated improvements using high-level error injection at the architecture register level which are generally inaccurate), the published results for these "selective" EDDI techniques show insufficient benefit (Table XVII) . Enhancements which reduce the execution time impact provide very low SDC improvements, while those that provide moderate improvement incur high execution time (and thus, energy) impact (much higher than providing the same improvement using LEAP-DICE, for instance). Fault screening [62] is an additional software level technique. However, this technique also checks to ensure intermediate values computed during execution fall within expected bounds, which is similar to the mechanisms behind Software assertions for general-purpose processors, and thus, is covered by the latter. Low-level Techniques: Resilience techniques at the circuit and logic layer (i.e., low-level techniques) are tunable as they can be selectively applied to individual flip-flops. As a result, a range of SDC/DUE improvements can be achieved for varying costs (Table XVI) . These techniques offer the ability to finely tune the specific flip-flops to protect in order to achieve the degree of resilience improvements required. In general, (application-aware) selective hardening using LEAP-DICE provides the most cost-effective standalone solution for both InO-and OoO-cores over all improvements. Logic parity is less efficient due to the need for pipelining parity logic and EDS is less efficient due to the need for delay buffers. High-level Techniques: In general, techniques at the architecture, software, and algorithm layers (i.e., high-level techniques) are less tunable as there is little control of the exact subset of flip-flops a high-level technique will protect. From Table IV , we see that no high-level technique provides more than 38× improvement (while most offer far less improvement). As a result, to achieve a 50× improvement, for example, augmentation from low-level techniques at the circuit-and logic-level are required, regardless. III. CROSS-LAYER COMBINATIONS CLEAR uses a top-down approach to explore the costeffectiveness of various cross-layer combinations. For example, resilience techniques at the upper layers of the system stack (e.g., ABFT correction) are applied before incrementally moving down the stack to apply techniques from lower layers (e.g., an optimized combination of logic parity checking, circuit-level LEAP-DICE, and microarchitectural recovery). This approach (example shown in Fig.  6 ) ensures that resilience techniques from various layers of the stack effectively interact with one another. Resilience techniques from the algorithm, software, and architecture layers of the stack generally protect multiple flip-flops (determined using error injection); however, a designer typically has little control over the specific subset protected. Using multiple techniques from these layers can lead to a situation where a given flip-flop may be protected (sometimes unnecessarily) by multiple techniques. At the logic and circuit layers, fine-grained protection is available since these techniques can be applied selectively to individual flip-flops (those not sufficiently protected by higher-level techniques). We explore a total of 586 cross-layer combinations using CLEAR (Table XVIII) . Not all combinations of the ten resilience techniques and four recovery techniques are valid (e.g., it is unnecessary to combine ABFT correction and ABFT detection since the techniques are mutually exclusive or to explore combinations of monitor cores to protect an InOcore due to the high cost). Accurate flip-flop level injection and layout evaluation reveals many individual techniques provide minimal (less than 1.5×) SDC/DUE improvement (contrary to conclusions reported in the literature that were derived using inaccurate architecture-or software-level injection), have high costs, or both. The consequence of this revelation is that most cross-layer combinations have high cost (detailed results for these costly combinations are omitted for brevity but are shown in Fig. 1 ). 14 Costs generated per benchmark with average cost over all benchmarks reported. Relative standard deviation is 0.6-3.1% 15 DUE improvement not possible using detection-only techniques given unconstrained recovery.  Among the 586 cross-layer combinations explored using CLEAR, a highly promising approach combines selective circuit-level hardening using LEAP-DICE, logic parity, and micro-architectural recovery (flush recovery for InO-cores, RoB recovery for OoO-cores). Thorough error injection using application benchmarks plays a critical role in selecting the flip-flops protected using these techniques. Fig. 7 and Heuristic 1 detail the methodology for creating this combination. If recovery is not needed (e.g., for unconstrained recovery), the "Harden" procedure in Heuristic 1 can be modified to always return false. For example, to achieve a 50× SDC improvement, the combination of LEAP-DICE, logic parity, and microarchitectural recovery provides a 1.5× and 1.2× energy savings for the OoO-and InO-cores, respectively, compared to selective circuit hardening using LEAP-DICE (Table XX) . The relative benefits are consistent across benchmarks and over the range of SDC/DUE improvements. The overheads in Table XX are small because we reported the most energyefficient resilience solutions. Most of the 586 combinations are far costlier. Let us consider the scenario where recovery hardware is not needed (e.g., unconstrained recovery). In this case, a minimal (<0.2% energy) savings can be achieved when targeting SDC improvement. However, without recovery hardware, DUEs increase since detected errors are now uncorrectable; thus, no DUE improvement is achievable. Finally, one may suppose that the inclusion of EDS into cross-layer optimization may yield further savings since EDS costs ~25% less area, power, energy than LEAP-DICE. However, a significant portion of EDS overhead is not captured solely by cell overhead. In fact, the additional cost of aggregating and routing the EDS error detection signals and the cost of adding delay buffers to satisfy minimum delay constraints posed by EDS dominates cost and prevents crosslayer combinations using EDS from yielding benefits. Additional cross-layer combinations spanning circuit, logic, architecture, and software layers are presented in Table XX . In general, most cross-layer combinations are not cost-effective. For general-purpose processors, a cross-layer combination of LEAP-DICE, logic parity, and micro-architectural recovery provides the lowest cost solution for InO-and OoO-cores for all improvements. Up to this point, we have considered SDC and DUE improvements separately. However, it may be useful to achieve a specific improvement in SDC and DUE simultaneously. When targeting SDC improvement, DUE improvement also improves (and vice-versa); however, it is unlikely that the two improvements will be the same since flip-flops with high SDC vulnerability will not necessarily be the same flip-flops that have high DUE vulnerability. A simple method for targeting joint SDC/DUE improvement is to implement resilience until SDC (DUE) improvement is reached and then continue implementing resilience to unprotected flip-flops until DUE (SDC) improvement is also achieved. This ensures that both SDC and DUE improvement meet (or exceed) the targeted minimum required improvement. Table XIX details the costs required to achieve joint SDC/DUE improvement using this methodology when considering a combination of LEAP-DICE, parity, and flush/RoB recovery. When the application space targets specific algorithms (e.g., matrix operations), a cross-layer combination of LEAP-DICE, parity, ABFT correction, and micro-architectural error recovery (flush/RoB) provides additional energy savings (compared to the general-purpose cross-layer combinations presented in Sec. III.A. Since ABFT correction performs inplace error correction, no separate recovery mechanism is required for ABFT correction. For our study, we could apply ABFT correction to three of our PERFECT benchmarks: 2d_convolution, debayer_filter, and inner_product (the rest were protected using ABFT detection). The results in Table XXII confirm that combinations of ABFT correction, LEAP-DICE, parity, and microarchitectural recovery provide up to 1.1× and 2× energy savings over the previously presented combination of LEAP-DICE, parity, and recovery when targeting SDC improvement for the OoO-and InO-cores, respectively. However, as will be discussed in Sec. III.B.1), the practicality of ABFT is limited when considering general-purpose processors. When targeting DUE improvement, including ABFT correction provides no energy savings for the OoO-core. This is because ABFT correction (along with most architecture and software techniques like DFC, CFCSS, and Assertions) Fig. 7 . Cross-layer resilience methodology for combining LEAP-DICE, parity, and micro-architectural recovery. if f has timing path slack greater than delay imposed by 32-bit XOR-tree (this implements low cost parity checking) 10: then return TRUE, else return FALSE 11: end procedure 10 performs checks at set locations in the program. For example, a DUE resulting from an invalid pointer access can cause an immediate program termination before a check is invoked. As a result, this DUE would not be detected by the technique. Although ABFT correction is useful for general-purpose processors limited to specific applications, the same cannot be said for ABFT detection (Table XXII) . Fig. 8 shows that, since ABFT detection cannot perform in-place correction, ABFT detection benchmarks cannot provide DUE improvement (any detected error necessarily increases the number of DUEs). Additionally, given the lower average SDC improvement and generally higher execution time impact for ABFT detection algorithms, combinations with ABFT detection do not yield low-cost solutions. Since most applications are not amenable to ABFT correction, the flip-flops protected by ABFT correction must also be protected by techniques such as LEAP-DICE or parity (or combinations thereof) for processors targeting generalpurpose applications. This requires circuit hardening techniques (e.g., [66] , [77] ) with the ability to selectively operate in an error-resilient mode (high resilience, high energy) when ABFT is unavailable, or in an economy mode (low resilience, low power mode) when ABFT is available. The LEAP-ctrl flip-flop accomplishes this task. The addition of LEAP-ctrl can incur an additional ~1% energy cost and ~3% area cost (Table XXII) . Although 44% (22% for OoO-cores) of flip-flops would need to be implemented using LEAP-ctrl, only 5% (2% for OoO-cores) would be operating in economy mode at any given time (Table XXI) . Unfortunately, this requirement of fine-grained operating mode control is difficult to implement in practice since it would require some firmware or software control to determine and pass information to a hardware controller indicating whether or not an ABFT application were running and which flip-flops to place in resilient mode and which to place in economy mode (rather than a simple switch setting all such flip-flops into the same operating mode). Therefore, cross-layer combinations using ABFT correction may not be practical or useful in general-purpose processors targeting general applications. Tunable resilience at low cost is enabled with the use of low-level techniques like circuit-level hardening and logiclevel parity. Unfortunately, it is not always possible to incorporate resilience at design time (e.g., legacy hardware, hardware from external vendors, commercial off-the-shelf hardware, etc.) Although circuit-, logic-, and architecture-level resilience cannot be incorporated given a fixed hardware constraint, it is still possible to provide soft error resilience at the software-and algorithm-level. Although no single resilience technique at the software-or algorithm-level can provide more than 50× SDC/DUE improvement (Table III) , combining multiple software and algorithm techniques can increase achievable resilience improvement, but comes at very high energy cost (Table XXIII) . It is important to note that due to the long error detection latencies of software techniques, these combinations are not relevant when bounded latency recovery is required (as no hardware recovery mechanism is applicable for these software error detection techniques). Given the existing software-and algorithm-level techniques available, even the most resilient cross-layer combination of ABFT correction, CFCSS, and EDDI can only provide an SDC improvement of 75.6× (at 163% energy cost). Therefore, in scenarios demanding extreme resilience (e.g., >100× resilience improvement), combinations of software and algorithm techniques will still be insufficient. This realization reinforces the importance of resilience being incorporated at design time with the aid of circuit-and logic-level techniques. To quantify this dependence, we randomly selected 4 (of 11) SPEC benchmarks as a training set, and used the remaining 7 as a validation set. Resilience is implemented using the training set and the resulting design's resilience is determined using the validation set. Therefore, the training set tells us which flip-flops to protect and the validation set allows us to determine what the actual improvement would be when this same set of flip-flops is protected. We used 50 training/validation pairs. Since high-level techniques cannot be tuned to achieve a given resilience improvement, we analyze each as a standalone technique to better understand how they perform individually. For standalone resilience techniques, the average inaccuracy between the results of trained and validated resilience is generally very low (Table XXIV and Table XXV) and is likely due to the fact that the improvements that the techniques themselves provide is already very low. We also report p-values [78] , which provide a measure of how likely the validated and trained improvements would match. Table XXVI and Table XXVII indicate that validated SDC and DUE improvements are generally underestimated. Fortunately, when targeting <10× SDC improvement, the underestimation is minimal. This is due to the fact that the most vulnerable 10% of flip-flops (i.e., the flip-flops that result in the most SDCs or DUEs) are consistent across benchmarks. These include flip-flops that store the program counter, current instruction, ALU input operands, jump and link, execute next instruction, and function return (i.e., crucial program state information that all benchmarks utilize). Since the number of errors resulting in SDC or DUE is not uniformly distributed among flip-flops, protecting these top 10% of flip-flops will result in the ~10× SDC improvement regardless of the benchmark considered. The vulnerabilities of the remaining 90% of flip-flops are more benchmarkdependent. These include flip-flops that store immediate operands, ALU result, register read/load/write, register addresses, cache state, exception/trap type, and supervisor state (i.e., program data state that is utilized differently by applications due to benchmark caching characteristics, register pressure, various data types, etc.) Concretely, we can analyze benchmark similarity by analyzing the vulnerable flip-flops indicated by each application benchmark. Per benchmark, one can group the most vulnerable 10% of flip-flops into a subset (e.g., subset 1). The next 10% of vulnerable flip-flops (e.g., 10-20%) are grouped into subset 2 (and so on up to subset 10). Therefore, given our 18 benchmarks, we create 18 distinct subset 1's, 18 distinct subset 2's, and so on. Each group of 18 subsets (e.g., all subset 1's) can then be assigned a similarity as given in Eq. 2. The similarity of subset "x" is the number of flip-flops that exist in all subset "x's" (e.g., subset intersection) divided by the number of unique flip-flops in every subset "x's" (e.g., subset union). From Table XXVIII , it is clear that only the top 10% most vulnerable flip-flops have very high commonality across all benchmarks (the last 2 subsets have high similarity because these are the flip-flops that have errors that always vanish). All other flip-flops are relatively distributed across the spectrum depending on the specific benchmark being run. It is clear that for highly-resilient designs, one must develop methods to combat this sensitivity to benchmarks. Benchmark sensitivity may be minimized by training using additional benchmarks or through better benchmarks (e.g., [79] ). An alternative approach is to apply our CLEAR framework using available benchmarks, and then replace all remaining unprotected flip-flops using LHL (Table IV) . This enables our resilient designs to meet (or exceed) resilience targets at ~1% additional cost for SDC and DUE improvements >10×. The maximum reported improvement for our lowest cost cross-layer solution is over three orders of magnitude improvement. However, it is still possible for an SDC/DUE to occur since circuit-hardening techniques do not guarantee correction of every possible flip-flop soft error. The extremely high level of resilience provided by our cross-layer solution is not possible using high-level techniques alone (Sec. II.D). Although a logic parity only (with recovery) solution could provide higher degrees of resilience, such a solution incurs a 44.1% energy cost (Table XVI) . ABFT detection + LEAP-DICE + logic parity (no recovery) ABFT detection + LEAP-DICE + logic parity (no recovery) All resilience techniques will lie on a two-dimensional plane of energy cost vs. SDC improvement (Fig. 9) . The range of designs formed using combinations of LEAP-DICE, parity, and micro-architectural recovery form the lowest-cost crosslayer combination available using today's resilience techniques. In order for new resilience techniques to be able to create competitive cross-layer combinations, they must have energy and improvement tradeoffs that place the technique under the region bounded by our LEAP-DICE, parity, and micro-architectural recovery solution. Since certain standalone techniques, like LEAP-DICE, can also provide highly competitive solutions, it is useful to understand the cost vs. improvement tradeoffs for new techniques in relation to this best standalone technique as well (Fig. 10) . CLEAR is a first of its kind cross-layer resilience framework that enables effective exploration of a wide variety of resilience techniques and their combinations across several layers of the system stack. Extensive cross-layer resilience studies using CLEAR demonstrate: 1) A carefully optimized combination of selective circuitlevel hardening, logic-level parity checking, and microarchitectural recovery provides a highly cost-effective soft error resilience solution for general-purpose processors 2) (Application-aware) selective circuit-level hardening alone, guided by thorough analysis of the effects of soft errors on application benchmarks, also provides a costeffective soft error resilience solution (with ~1% additional energy cost for a 50× SDC improvement compared to the above approach). 3) Algorithm Based Fault Tolerance (ABFT) correction combined with selective circuit-level hardening (and logic- Fig. 9 . New resilience techniques must have cost and improvement tradeoffs that lie within the shaded regions bounded by LEAP-DCE + parity + microarchitectural recovery. Mukherjee, N. Nassif, J. Krause, J. Pickholtz and A. Balasubramanian, "On the radiation-induced soft error performance of hardened sequential elements in advanced bulk CMOS technologies," in Reliability Physics Symposium (IRPS), 2010 IEEE International, 2010 N. Seifert, B. Gill, S. Jahinuzzaman, J. Basile, V. Ambrose, Q. Shi, R. Allmon and A. Bramnik, "Soft Error Susceptibilities of 22 nm Tri-Gate Devices," IEEE Transactions on Nuclear Science, vol. 59, no. 6, pp. 2666-2673 , Dec 2012 N. Seifert, S. Jahinuzzaman, J. Velamala, R. Ascazubi, N. Patel, B. Gill, J. Basile and J. Hicks, "Soft Error Rate Improvements in 14-nm Technology," IEEE Transactions on Nuclear Science, vol. 62, no. 6, pp. 2570-2577 , Dec 2015 Circuits and Systems, vol. 33, no. 11, pp. 1597-1610 , Nov 2014 G. Bosilca, R. Delmas, J. Dongarra and J. Langou, "Algorithm-based fault tolerance applied to high performance computing," Journal of Parallel and Distributed Computing, vol. 69, no. 4, #Apr# Kevin Skadron (F'13) is the Harry Douglas Forsyth professor and chair of the Department of Computer Science at the University of Virginia, where he has been on the faculty since 1999. His research focuses on heterogeneous architecture, design and applications of novel hardware accelerators, and design for physical constraints such as power, temperature, and reliability. Skadron and his colleagues have developed a number of open-source tools to support this research, including the HotSpot thermal modeling tools, the Svalinn reliability model, the Rodinia GPU benchmark suite, the ANMLZoo automata benchmark suite, and the MNCaRT research toolkit for automata processing. Skadron is a Fellow of the IEEE and the ACM, and recipient of the 2011 ACM SIGARCH Maurice Wilkes Award. (F'14) received the "Diploma" degree from Politehnica University, Bucharest, Romania, and the M.S. and Ph.D. degrees from the University of Massachusetts. He received the NSF CAREER Award in 1997 and was a co-author on best paper awards at SELSE 2017, ISQED 2008, GLSVLSI 2006, ISCA 2003, and SHAMAN 2002 He is a senior Editor of the IEEE TRANSACTIONS ON NANOTECHNOLOGY. Prof. Stan is a fellow of IEEE, a member of ACM, Eta Kappa Nu, Phi Kappa Phi, and Sigma Xi. Klas Lilja is the founder and CEO of Robust Chip Inc. (RCI), a company specialized in analysis of radiation effects in electronics and design of radiation hard electronics. He received M.Sc. and Ph. D. degrees from Chalmers Technical University, Sweden, and the Swiss Federal Institute of Technology (ETH), Switzerland, in 1986 and 1992 respectively, and has over 25 years of experience in the area design, simulation and radiation hardening of electronic devices and circuits. Prior to founding RCI, Klas was VP Engineering at ISE and Head of TCAD at Avant! Corporation (both companies are now part of Synopsys). As founder of RCI, he has led the development of RCI's unique software tools and break-through technology for radiation hardened electronics, and developed the company into a worldwide leader in this field. Klas holds more than 10 issued and pending patents on RCI's new technology, as well as several other patents in the area of semiconductor devices and semiconductor simulation. Stanford, California, in 1971 and 1974, respectively. From 1975 to 1988 he was on the faculty of the University of Illinois, Urbana, Illinois. Professor Abraham's research interests include VLSI design and test, lowpower circuits, and the design of on-chip circuitry to facilitate measurements and diagnosis of digital, analog and RF circuits and biological systems. He is also interested in techniques to mitigate the complexity of design verification, and the design and evaluation of resilient and secure systems. He has over 500 publications, has received numerous "best paper" awards at international conferences, and was included in the ISI (Thomson Reuters) list of highly cited researchers. He has supervised more than 90 Ph.D. dissertations and is particularly proud of the accomplishments of his students, many of whom occupy senior positions in academia and industry. He has been elected Fellow of the IEEE as well as Fellow of the ACM, and is the recipient of the 2005 IEEE Emanuel R. Piore Award and the 2017 Jean-Claude Laprie Award in Dependable Computing.
Cheng et al. introduce a systematic framework that explores comprehensive combinations of resilience techniques that find a cost-effective flip-flop soft error resilience solution REF .
Tolerating Soft Errors in Processor Cores Using CLEAR (Cross-Layer Exploration for Architecting Resilience)
ArXiv
ArXiv
Computer Science