He didn't want attention called to himself while Ser Perth was around.
While Ser Perth was there he did not want anyone to notice him.