Question: what does it mean when a man calls you little one?
Answer: It is a term of endearment though so using it (not sarcastically) with any adult you don't know well is likely to be the wrong context and will probably be considered really strange/awkward or intentionally rude/cheeky/sarcastic. I don't think it sounds disrespectful so much as just really odd.