Clifford Castle, 1066 to 1299