I fancy he wants to give some of it back. It was possibly this remark which induced the driver to move away.
Perhaps the driver moved away because he wanted to give some of it back.