Normally, I simply wouldn't invite him and would explain that it was a small ceremony (which it is).
I wouldn't invite him and would tell him that its a small ceremony.