  The challenging requirements of large scale quantum information processing
using parametric heralded single photon sources involves maximising the
interference visibility whilst maintaining an acceptable photon generation
rate. By developing a general theoretical framework that allows us to include
large numbers of spatial and spectral modes together with linear and non-linear
optical elements, we investigate the combined effects of spectral and photon
number impurity on the measured Hong--Ou--Mandel interference visibility of
parametric photon sources, considering both threshold and number resolving
detectors, together with the effects of spectral filtering. We find that for
any degree of spectral impurity, increasing the photon generation rate
necessarily decreases the interference visibility, even when using number
resolving detection. While tight spectral filtering can be used to enforce
spectral purity and increased interference visibility at low powers, we find
that the induced photon number impurity results in a decreasing interference
visibility and heralding efficiency with pump power, while the maximum
generation rate is also reduced.
