Abstract: Debugging and profiling tools can alter the execution flow or timing, can induce heisenbugs and are thus marginally useful for debugging time critical systems. Software tracing, however advanced it may be, depends on consuming precious computing resources. In this study, the authors analyse state-of-the-art hardware-tracing support, as provided in modern Intel processors and propose a new technique which uses the processor hardware for tracing without any code instrumentation or tracepoints. They demonstrate the utility of their approach with contributions in three areas -syscall latency profiling, instruction profiling and software-tracer impact detection. They present improvements in performance and the granularity of data gathered with hardware-assisted approach, as compared with traditional software only tracing and profiling. The performance impact on the target system -measured as time overhead -is on average 2-3%, with the worst case being 22%. They also define a way to measure and quantify the time resolution provided by hardware tracers for trace events, and observe the effect of finetuning hardware tracing for optimum utilisation. As compared with other in-kernel tracers, they observed that hardware-based tracing has a much reduced overhead, while achieving greater precision. Moreover, the other tracing techniques are ineffective in certain tracing scenarios. Modern systems are becoming increasingly complex to debug and diagnose. One of the main factors is the increasing complexity and real-time constraints which limit the use of traditional debugging approaches in such scenarios. Shorter task deadlines mean that the faithful reproduction of code execution can be very challenging. It has been estimated that developers spend around 50-75% of their time debugging applications at a considerable monetary cost [1] . In many scenarios, heisenbugs [2] become nearly impossible to detect. Long-running systems can have bugs that display actual consequences much later than expected, either due to tasks being scheduled out or hardware interrupts causing delays. Important parameters that need to be analysed while doing a root cause analysis for a problem include the identification of costly instructions during execution, the detection of failures in embedded communication protocols and the analysis of instruction profiles that give an accurate representation of which instructions consume the most central processing unit (CPU) time. Such latent issues can only be recorded faithfully using tracing techniques. Along with accurate profiling, tracing provides a much needed respite to developers for performance analysis in such scenarios. We focus in this paper on two important common issues in current systems: the efficient detection/tracking of hardware latency and the accurate profiling of syscalls and instructions, with an ability to detect programme control flow more accurately than with current software approaches. We discuss our new analysis approach, which utilises conditional hardware tracing in conjunction with traditional software tracing to accurately profile latency causes. The trace can be decoded offline to retrieve the accurate control flow, data even at instruction granularity, without any external influence on the control flow. As software tracing can induce changes in the control flow, with our system we can further detect the cause of latency induced by the software tracers themselves, on the software under observation, to nanosecond range accuracy. Pure software profiling and tracing tools consume the already constrained and much needed resources on production systems. Over the years, hardware tracing has emerged as a powerful technique for tracing, as it gives a detailed and accurate view of the system with almost zero overhead. The IEEE Nexus 5001 standard [3] defines four classes of tracing and debugging approaches for embedded systems. Class 1 deals with basic debugging operations
A detailed study of hardwareassisted tracing and profiling with Intel PT has been recently presented in REF .
Hardware-assisted instruction profiling and latency detection
The Journal of Engineering
Computer Science