I think evolution explains that fine. It's the same story as for physical development: the extrapolation and refinement of living organisms via natural selection over the course of hundreds of millions of years.

Consciousness is hard to define, but if we examine animal intelligence it unsurprisingly seems to exist at various levels of complexity. So even worms have intelligence (and consciousness IMO) that isn't quite robotic: they react in predictable ways to light, but the reaction is modulated by other factors like temperature, moisture and what they are doing at the time. Their behavior can be described by a simple neural net, which is in fact what they have. (see Darwin 1881)

Larger animals evolved the ability to evaluate multiple such signals simultaneously, and to do so it's necessary that lower level consciousness (like what the worm has) can send signals to higher levels of consciousness for evaluation. So if my foot is cramping a lower level system sends an appropriate pain signal to a higher level system, which can then aggregate that signal with other signals and cross reference it against current active behavioral plans. If I am meditating, I might ignore the pain, but if I'm working I might get up and stretch. This is similar to how the reaction of a worm is modulated, but just filtered up through higher levels of consciousness.

Why would we develop higher levels of consciousness? Because any change to animal brains that allowed them to be aware of sensory input and evaluate it with reference to behavioral goals (instead of just reacting like a robot) would increase that animals fitness, and any marginal increase in fitness will result in those traits being passed on.

The most significant one for human consciousness in particular is that the appearance of language pushed evolution in the direction of favoring a large working memory. If you can tell me "the bananas are down the hill, to the right, across the stream, next to the big stone" and I can remember that information and go find the bananas, then my fitness is improved. Darwin also pointed out that this increased working memory allows people to reason that if I help someone, that other person might help me back, allowing altruism to emerge from the cognitive space allotted to planning, which evolved from the pressures related to holding large chunks of linguistic data in the mind.

So there are two things: first is subjective experience, which to me seems to be easily explainable as being like "system signals" from lower systems to higher systems. It's important to remember here that the appearance of information itself and intelligent responses to the environment is one of the defining features of all life, even the most basic (Adolf Heschel 2002). The smell of rotten eggs (example from the article) is simply a chemical signal that we have, understandably, evolved an aversion to. My reaction to ice-cream is similarly a chemical signal filtered through a lower biological system which interprets it as something REALLY GOOD and sends the appropriate signals to my higher level cognitive systems. The fact that it's a subjective signal and not a "robotic" reaction means that I can respond to it differently depending on other factors, which is clearly advantageous from the POV of natural selection. If I have diabetes I can resist the temptation of "deliciousness" to my great advantage.

The second thing is that humans have a great memory, so we can hold lots of these signals in our minds alongside memories, plans, ideas etc. Since that ability was associated with increased fitness over millions of years, it has increased to the level we see today.