Original: This is because interlopers are hard to distinguish from family members because the family is of the same spectral type (S) that dominates the inner main belt overall.
Simplified: This is because unrelated asteroids are hard to distinguish from family members because the family is of the same spectral type (S) that dominates the closer main belt overall.