  Autonomous ultrasound (US) acquisition is an important yet challenging task,
as it involves interpretation of the highly complex and variable images and
their spatial relationships. In this work, we propose a deep reinforcement
learning framework to autonomously control the 6-D pose of a virtual US probe
based on real-time image feedback to navigate towards the standard scan planes
under the restrictions in real-world US scans. Furthermore, we propose a
confidence-based approach to encode the optimization of image quality in the
learning process. We validate our method in a simulation environment built with
real-world data collected in the US imaging of the spine. Experimental results
demonstrate that our method can perform reproducible US probe navigation
towards the standard scan plane with an accuracy of $4.91mm/4.65^\circ$ in the
intra-patient setting, and accomplish the task in the intra- and inter-patient
settings with a success rate of $92\%$ and $46\%$, respectively. The results
also show that the introduction of image quality optimization in our method can
effectively improve the navigation performance.
