For the view over Marigot Bay, it's worth the four-minute climb up to deserted old Fort Saint-Louis.
There is a great view of Marigot Bay from Fort Saint-Louis.