So, your intuition about time being a illusion is "correct". But in such a context, most other "universal" properties are also an "illusion". Not in a philosophical way, but in a tangible, soon to be practical way. I'll try to pretend to explain, even though it might not be a good idea.

(Please keep in mind, most or all the words following are incorrect, in a way reminiscent of recounting the history of earth by saying: "A fish walked out of the water and became the emperor of China.")

We can imagine our local universe is a thing like a multidimensional boiling bubble. It recursively sub-divides itself, which makes it interesting, and also creates space-time. The start of our local universe's time is the so-called "big bang", which is when it started this wild and lovely fractal. After a little bit there was enough complexity to implement something like the physics we can observe now.

It turns out, a lovely universe can be created from just recursive subdivision. We're smart enough now that we've defined computability in terms of this, for example S K combinitors, etc. Math and physics will eventually catch on. (The astute reader will notice there are things we can imagine, that are uncomputable, math that is un-representable, and physics that won't work with this system. Which isn't really a problem yet.) Thankfully, this configuration happens to be very interesting.

But wait. How did time start? Well, um, it didn't. What we note as time is the constant, change (adding new nodes), expansion, and complexification of recursive subdivision, which happens in units of the "things" dividing, which is like the quantization that we observe in time, matter, and energy. It's "frequency" is also somewhat dependent on it's local environment, since it's "pulled on" by what attached and intertwined with, exhibiting the relativistic effects of gravity and motion.

Perhaps you've heard of the universe described as the three dimensional surface of a four dimensional sphere. But what's inside that sphere? Actually it's the past. Sitting there right now, for real, probably just as it was, fully connected. Of course we're zooming away from it a varying rates, but pretty fast, with the "force" of the universal expansion behind it.

So if the past is actually sitting right there, not that far away from us, we could just go to it right? Well, maybe. It would take a lot of "energy", and stretch the "universal surface" out of whack. But perhaps one could. But what about those paradoxes? Date your mom and your hand starts fading away? Accidentally kill your grandpa and poof out of existence, or not? Well, thankfully those paradoxes can't happen, because even if you're altering the physical past, you still lived through the past you lived through. The altered future that it creates will be "forever" behind your present. Another way to think about it is: any time you effect the past of this universe, it pushes it slightly in the direction of another dimension, in which there is a whole "copy" (or copy-on-write) of the universe with just that thing altered (so far).

This still doesn't answer why we don't bump into a lot of time travelers. Are we doing this the first time through? Or is it just really expensive, boring and useless to muck about in the past? Or are none of neighbors from the universes next door interested? This presupposes that you can re-run through the time dimension. Only at the edge are we "moving", or are new nodes being added. You would have to fork a whole other universe starting at that time slice, which is probably unfeasible unless you have "massive" extra-"universal" compute power. Is it really that important? Especially given that due to the fractal nature of nature, you can fork another sub-universe in this universe! Much easier. Or even better, just take the tiny part you want and copy and paste. Done. Or just get it right the first time. Come on now people.

Of course the future doesn't exist. Which makes it fun when we can seem to predict it using such things as Newton's old equations, and speculative execution.