  The GRAVITY instrument on the ESO VLTI pioneers the field of high-precision
near-infrared interferometry by providing astrometry at the $10 - 100\,\mu$as
level. Measurements at such high precision crucially depend on the control of
systematic effects. Here, we investigate how aberrations introduced by small
optical imperfections along the path from the telescope to the detector affect
the astrometry. We develop an analytical model that describes the impact of
such aberrations on the measurement of complex visibilities. Our formalism
accounts for pupil-plane and focal-plane aberrations, as well as for the
interplay between static and turbulent aberrations, and successfully reproduces
calibration measurements of a binary star. The Galactic Center observations
with GRAVITY in 2017 and 2018, when both Sgr A* and the star S2 were targeted
in a single fiber pointing, are affected by these aberrations at a level of
less than 0.5 mas. Removal of these effects brings the measurement in harmony
with the dual beam observations of 2019 and 2020, which are not affected by
these aberrations. This also resolves the small systematic discrepancies
between the derived distance $R_0$ to the Galactic Center reported previously.
