Question: what is the difference between back off and back down?
Answer: You back off by retreating from a confrontation of some kind, but here the other person is not being antagonistic. You back down by giving way when you realise you're losing an argument or fight, but that doesn't apply here either.