  The A64FX CPU is arguably the most powerful Arm-based processor design to
date. Although it is a traditional cache-based multicore processor, its peak
performance and memory bandwidth rival accelerator devices. A good
understanding of its performance features is of paramount importance for
developers who wish to leverage its full potential. We present an architectural
analysis of the A64FX used in the Fujitsu FX1000 supercomputer at a level of
detail that allows for the construction of Execution-Cache-Memory (ECM)
performance models for steady-state loops. In the process we identify
architectural peculiarities that point to viable generic optimization
strategies. After validating the model using simple streaming loops we apply
the insight gained to sparse matrix-vector multiplication (SpMV) and the domain
wall (DW) kernel from quantum chromodynamics (QCD). For SpMV we show why the
CRS matrix storage format is not a good practical choice on this architecture
and how the SELL-$C$-$\sigma$ format can achieve bandwidth saturation. For the
DW kernel we provide a cache-reuse analysis and show how an appropriate choice
of data layout for complex arrays can realize memory-bandwidth saturation in
this case as well. A comparison with state-of-the-art high-end Intel Cascade
Lake AP and Nvidia V100 systems puts the capabilities of the A64FX into
perspective. We also explore the potential for power optimizations using the
tuning knobs provided by the Fugaku system, achieving energy savings of about
31% for SpMV and 18% for DW.
