First, even if punishment is inevitable, it falls not on the owners at the time when the sin is committed, but on the owners at the time when the sin is discovered.
Punishment falls on the owners when the transgression is discovered.