No, he waits until he has had a violent quarrel with her, of which the whole household is cognisant, and which naturally directs their suspicions upon him.
He waited until after they fought, causing the whole household to be wary of him.