i think story wise during this time giles is travelling and collecting potentials in real life i think he was only a recurring guest star in series rather than a regular because he needed more time with his family who lived in england