Later, as it fell into disrepair, the walls were dismantled and used to build other structures in the town.
After it fell into a state of disrepair, the demolished remains were thrown into a quarry. 
mnli hypothesis: After it fell into a state of disrepair, the demolished remains were thrown into a quarry.  premise: Later, as it fell into disrepair, the walls were dismantled and used to build other structures in the town.