>couldn't it have been organisms from earth ejected into orbit after a meteor strike and, thousands of years later or more, falling back to earth?

That would still be a hugely significant result.

We expect a higher density of biologically contaminated samples near the Earth. This is perfectly compatible with (but does not prove) panspermia.

The key question isn't whether space is sterile, but whether space is aseptic. That's harder to demonstrate.

To illustrate the difference, imagine a doctor scrubbing for surgery. It's impossible to scrub your hands until there's a population of zero pathogens, but fortunately that is unnecessary. You just need to reduce the population enough that the few organisms that remain just can't get a foot-hold.

We're in the back waters of the galaxy in a mature star system. Even if panspermia were rampant in the galaxy, we probably wouldn't see it. Stellar clusters, nebulae, Bok globules, and young star systems are where you would expect this kind of action.

Mechanically speaking panspemia is just diffusion, except instead of Brownian motion you have impact ejecta and gravitational evaporation. Each of these events has a probability curve. You could even construct your own "Panspermia Drake Equation", something like:

P = N_seed * R_ejection * Pop_specific * f_survival_ejection * f_escape
* 2^-(T_encounter / t_1/2_cruise)
* f_habitable * f_survival_reentry * f_germination

P = rate of panspermia in a given volume of space
N_seed = number of seed (life-bearing) bodies in that volume
R_ejection = average rate at which mass is ejected from each
seed body (kg/yr)
Pop_specific = average specific population ('von Neumann'
individuals/kg) of ejected material
f_survive_ejection = fraction of population that survives ejection
f_escape = fraction of ejected material that eventually escapes from
the stellar system, either due to initial velocity
or long-term orbital perturbations
T_encounter = average length of time before an escaped
object encounters another body
t_1/2_cruise = half-life of population exposed to interstellar conditions
f_habitable = fraction of encountered bodies that are habitable
f_survive_reentry = fraction of population that survives reentry
f_germination = fraction of the population that actually germinates

Looking at this equation it's not hard to see why dense regions with young star systems are preferred. Young systems have higher ejection and escape rates. Survival rates for ejection and re-entry are a function of energy, so having many less massive bodies is preferred. Young, single-cell-only ecosystems increase the specific population.

Of course, The number of seed bodies (N_seed) is the interesting part. This allows the equation to feed back on itself. Waaaaaaay out here in the Solar neighborhood, everything's so far apart that T_encounter rate limits that feedback to effectively zero, but in stellar clusters and nebulae you can find regions with dozens of stars per cubic light-year. As stars move within the galaxy they might experience different rates of panspermia – perhaps long lifeless intervals punctuated by "bursts" of bombardment, or maybe just once (in the conditions of the stellar nursery) and then never again.

When it comes to panspermia as an origin theory (aka exogenesis) I consider myself a "weak atheist", if you will. The meager evidence we have seems to suggest that life on Earth originated in abiogenesis, not panspermia. That doesn't mean that panspermia does not occur.