In what is sometimes considered mere obiter dictum , the Court went on to hold that Congress had no authority to prohibit slavery in federal territories because slaves are personal property and the Fifth Amendment to the Constitution protects property owners against deprivation of their property without due process of law .