robots txt are the scourge of the internet i should set up a crawler that ignores robots txt files