Magic tricks are hacking! You are hacking someone's mind, particularly their assumptions and attention. The exploits to human psychology and attention are just as sure as the exploits to crack or unlock a smartphone and many other software exploits. The difference is that software can be patched but the evolved behaviors that are the root of the exploits used by magicians are very hard to "patch". For example, if I say the word "ball", if you have understood this word then you have no choice but to have a certain mental state that is predictable to a magician. You do not have conscious access to the low level machinery for understanding speech and if there is some magic trick involving, say, implanting an idea in your mind using words, then you often CAN NOT avoid being fooled.
Moreover, a good magician will prevent you from stopping to think about what he is doing by merely talking and distracting/leading your attention to the next ruse he wishes to perpetrate. Unless you know the tricks or have exceptional mental control, e.g., from years of meditation practice, you will quickly find yourself overwhelmed and check-mated by a good magician.

What's amazing about magic tricks is how strong the effects can be. For some tricks, if you didn't know that it was a trick, there is simply no other explanation than that it is magic. A good magician doing closeup magic can follow your gaze and know what you're looking at in order to adjust his strategy in real time. The evidence that might clue you in to the trick is fleeting and if you miss it, due to distraction and showmanship, then the magician has caught you and has succeeded in pickpocketing you, making you think an object has disappeared or that he has read your mind.

EDIT:
And your point about magic and skepticism is particularly apt. A lot of supernatural belief is based on unexplained things in people's everyday life. These are kind of like "natural" magic tricks. People might encounter unexplained events in their everyday life and jump to the conclusion that it is supernatural because they don't know where to look to find the natural causes. And if the natural antecedents are forgotten or fade away then all that is left is unexplained event that seems supernatural. For example, one time I was sitting at my desk and I noticed that a Styrofoam cup on an adjacent desk just started moving. It was as if the Invisible Man decided to slide the cup about 5 inches. If I didn't investigate what was going on then I would have no choice but to conclude that it was magic because there was no visible natural cause to it. As I tentatively approached the cup, I realized that the desk it was sitting on was at a slight incline. Moreover, there was a small puddle of water on the desk from the air-conditioner above it that had been leaking. The puddle of water had slowly, due to gravity, seeped beneath the cup and eventually the cup overcame friction to slide down the thin slick film.

Now, if I had not checked for physical causes immediately, then the water may have dried up and the true cause for the seemingly supernatural event would have been lost to time. Magic helps skepticism because it makes you cognizant of the fact that things can appear to be supernatural if you are not ready at all times to check for natural explanations.