  The symmetry energy and its density dependence are crucial inputs for many
nuclear physics and astrophysics applications, as they determine properties
ranging from the neutron-skin thickness of nuclei to the crust thickness and
the radius of neutron stars. Recently, PREX-II reported a value of $0.29\pm
0.07$ fm for the neutron-skin thickness of $^{208}$Pb, implying a slope
parameter $L = 110\pm 37$ MeV, larger than most ranges obtained from
microscopic calculations and other nuclear experiments. We use a nonparametric
equation of state representation based on Gaussian processes to constrain the
symmetry energy $S_0$, $L$, and $R_{\rm skin}^{^{208}\text{Pb}}$ directly from
observations of neutron stars with minimal modeling assumptions. The resulting
astrophysical constraints from heavy pulsar masses, LIGO/Virgo, and NICER
clearly favor smaller values of the neutron skin and $L$, as well as negative
symmetry incompressibilities. Combining astrophysical data with PREX-II and
chiral effective field theory constraints yields $S_0 = 34^{+3}_{-3}$ MeV,
$L=58^{+19}_{-19}$ MeV, and $R_{\rm skin}^{^{208}\text{Pb}} =
0.19^{+0.03}_{-0.04}$ fm.
