> Surely you in your life you have met many Christians who said "God works in mysterious ways", "there is a purpose for everything", "trust in the Lord", etc.?

Yes, but this is something people say in times of grief, for the comfort of the grieving -- it's not a consistent moral philosophy that they hold to in other parts of their life, nor is it meant to be. There are better answers than this, but they are oftentimes much harder to process, and much more likely to accidentally pain the one who is grieving.

> It's arguably a necessary tenet for Christianity to hold together as a coherent belief system.

No, it's not. You're correct in that "you need some way to explain why bad things still happen", but we've come a long, long way from "stuff just happens".
For example, the Catholic view is that suffering is A) not committed by, but permitted by, God; B) necessary for salvation and free will to coexist. In this view, evil is in essence a deviation from the will of God -- but free will must, in this conception, at the very least include the free will to choose to follow or choose to oppose God's own will. To quote St. Aquinas, paraphrasing St. Augustine: "Since God is the supremely highest good, he would not allow evil to exist in his creation unless he were so all powerful that he could even make good out of evil". More broadly, suffering is seen as having not only a redemptive but an edifying nature that can ultimately bring us closer to God.

I understand that this might sound repulsive on first glance, but frankly I do not think there is an answer to "why is there evil?" which would not be at initial examination -- certainly, it's no worse than the idea that we are simply here to suffer by random caprice, and that that suffering is itself meaningless, nothing but a failure on your own (meaningless) value function. Yes, one might hope that things 'could have been a different way' -- but what would a world without any grief, any suffering even be like? This is the point of the whole pleasure-machine/experience-machine thought experiment: many people would very much rather live in this world, with all its suffering, than one totally blank, devoid of depth and complexity. One might even go as far as to assert that no 'good' God could permit such terrible depths of suffering -- congenital illness, rape, torture, child slavery, so on and so forth. But so many times, in exploring theories of computational complexity or abstract mathematics or informatics, we see that what might have seemed to be simple assumptions can have enormous, essential effects: deciding whether all programs written for a FSM with one stack is simple, but for one with two stacks the problem becomes impossible. Perhaps it is impossible to have a world "with matter, with living things made from matter, with free will for those living beings, but without the ability of one living thing to enslave another". We cannot know -- but if there is a truly transcendent, omniscient God, then He certainly would.

For a more modern (more philosophically-flavored) take, I'd suggest reading Kierkegaard's Fear and Trembling: he explores the binding of Isaac with the idea being to address this very question. In particular, he strongly disagrees with Kant's idea that God would simply choose to follow the categorical imperative, and emphasizes instead the transcendence of God and divine morality. But as an existentialist I think his writing is much closer to how we as members of the modern world can feel than philosophers/theologians who came before him.