This only limits your ability to time-travel from one point of the universe into the same point of that universe, but not in general.

According to your hypothesis, if you build a time-travel-space-ship you could time-travel to other points of that universe, being your only limitation keeping yourself inside of that universe.

But you are not thinking with portals yet.

We "know", thanks to String Theory, that there are other spacetime dimensions "curled up" around the usual dimensions that we experience. The Large Hadron Collider has been running experiments to try to prove that by provoking collisions between particles and see if some of the energy gets "trapped" in those extra dimensions.

If the theory is true, that would mean that we are constantly time-traveling back and forth through time and space on multiple directions... just not the ones that we experience. From that dimensions point of view we might look like sliced beings that appear and disappear on many different points at the same time and in many different times at the same point.

If our time-travel machine manages to get us curled in on those dimensions we could time-travel in the usual dimensions by moving inside the curled ones following a different time (assuming that this other time dimension moves backwards with respect to our time).

From the outside it would look like our particles are rotating and becoming strings onto themselves like thread balls.

From the subject's point of view, the universe would bend around him, become dark and cold. And then, everything would disappear. His "experience" would probably be closer to "non existence". It would be the time machine what would have had to have everything ready to start and finish the trip on the right direction and time length.

Time travel will only be possible to points where your own self was in the past and only distanced by short time lengths.

Mind that every particle would travel inside the curled dimensions attached to it, but our bodies are built by many substances and structures that our own body creates from food, water and air. Each atom on our body would time-travel to where it was at the time you travel.

It might seem like a very limiting factor but this is because we are only thinking about the first human-like time-travelers. However, machines could time travel without almost any problem to the exact point where they were assembled.

The main problem would be to send new information since any new structure would disappear (drawings couldn't go back any further than where they were drawn and bits in hard drives would go back similarly to their previous statuses). But I'm sure we could NOT send complete structures as a way to send new info.

For example, we could remove parts of a radioactive sheet of metal (imagine it just by scratching the surface and removing enough atoms). We could send only the sheet of metal and not the scratched parts. Back in time, we could observe that same sheet of metal and it won't be scratched at that time... but the atoms that came from the future would have experienced more radioactive decay since they existed for a longer time.

Is this a paradox? why was that sheet of metal not present that same radioactive decay before we scratched the surface?

I don't know. But now you are thinking with portals.