Question: is layman a derogatory term?
Answer: Layman is not an insult or slur, but, like others have pointed out, it can sometimes be insulting to say that someone isn't a subject-matter expert. Meanwhile, I'm more concerned that you're using layman in cases where it isn't appropriate, lexically. It doesn't just mean an ordinary person.