Question: what does it mean when someone is your kryptonite?
Answer: It only means that the person or object referred to, for some reason, makes one lose one's resolve, rendering a person easily influenced or coerced to doing something they're usually not predisposed to doing. In other words, “kryptonite” in this context is a metaphor of “weakness”.