Question: are your eyes connected to your mouth?
Answer: When you produce tears or have another liquid in your eyes, some of it drains into these holes and then into the lacrimal sac, the nasolacrimal duct, and eventually into the back of your nose and throat, where you might get a taste. This is normal and safe, but eye drops aren't exactly designed with flavor in mind.