Continue uphill to the fortress, which is, in fact, two castles.
There is a fortress if you keep going up the hill.