  We consider quantum diffusion in ultra-slow-roll (USR) inflation. Using the
$\Delta N$ formalism, we present the first stochastic calculation of the
probability distribution $P(\mathcal{R})$ of the curvature perturbation during
USR. We capture the non-linearity of the system, solving the coupled evolution
of the coarse-grained background with random kicks from the short wavelength
modes, simultaneously with the mode evolution around the stochastic background.
This leads to a non-Markovian process from which we determine the highly
non-Gaussian tail of $P(\mathcal{R})$. Studying the production of primordial
black holes in a viable model, we find that stochastic effects during USR
increase their abundance by a factor $\sim 10^5$ compared to the Gaussian
approximation.
