The Harem housed the private quarters of the sultan, his mother, and his many wives and concubines.
The sultan and his mother lived in the Harem.