like is some cases sites use robots txt and therefore they can't be archived