The oldest remaining part of the castle is the Record Tower (1258), visible as you leave the upper yard.
The Record Tower is the oldest standing part of the castle.