The short answer is that, when the argument is fully set out, one of the conditions is that stimulating one's optic nerve does not count, but it often goes unmentioned. This is rarely an issue (and is easily corrected if it is), as it quickly becomes apparent that allowing stimulation does not even present a prima facie problem and so no-one on either side of the fence finds it at all interesting or useful - the anti-materialists are not interested in this question because they will agree with materialists that if stimulation is allowed, then Mary (the protagonist in the thought experiment) will experience colors, and the materialists are not interested in it because the anti-materialists are not using it.

Nevertheless, it is a good question as to why the knowledge argument would be seen as so persuasive by so many, including by quite a few materialists, who seem to me to go to unnecessary lengths to get around what is not even a problem (illusionism, for example, where it is claimed that phenomenal experience is merely an illusion - but then, as the anti-materialists ask, who is being fooled, and about what?)

Perhaps the first thing to say is that the argument is not presented as the stark observation that you cannot learn what seeing colors is like from reading a physics book; that's what you end up with when you whittle it down to its essentials.

Secondly, there are a couple of features of the argument which make it easy to let its difficulties slide right on by. One of these is what I have been going on about here: by framing the argument in terms of a knowledge of physics, it is easy to miss that it applies to any academic knowledge, not only physics or the physical sciences, and thus including any non-materialist theory. The second is that it uses the word 'knowledge' for two different forms of retained information: sensory information which is acquired directly from sensory experience, and linguistic information which is encoded in sensory information without being that sensory information itself. These are separate domains (the information content of a word is not the information content of the sound when it is heard or of its appearance when read), yet, by referring to both of them as knowledge, the argument invites the reader to accept the way it equivocates between the two, which it does when it points out that the knowledge that can be acquired linguistically does not include the 'raw feels' (yes, that is a term used in the philosophy of mind) of sensory information.

Having said all that, I am still surprised how many people think the argument shows there is a problem for materialism. After all, the physical sciences have explained many sorts of complex phenomena (hurricanes, for example), and no-one thinks that the act of explaining (or learning the explanation of) how hurricanes work should actually create one - yet anyone accepting the knowledge argument is tacitly accepting that if the phenomenon being explained is sensory experience, then an explanation should produce the phenomenon being explained! (but only if it is a materialist explanation!) I guess it somehow doesn't seem paradoxical to many people when both the phenomenon being explained and its explanation are in one's mind.

I think you would be very surprised by how many papers have been, and are still being written about this argument or something following directly from it. Because I have read some of them on academia.edu, I frequently get notices about more. Interestingly, Frank Jackson himself has changed his mind on the matter, but that has had no effect on how influential his argument still is (and, to be fair, it shouldn't.)