Question: why ddpg is off policy?
Answer: DDPG is off-policy because your dataset (the replay buffer) was generated from an older policy. If your replay buffer were super-short, it would be closer to being on-policy. This holds true regardless of the existence/absence of any exploration noise.