Which isn't at all to excuse or make apologies for slavery, of course, but only to argue that our collective memory needs to include men like Robert Harlan, Justice Harlan's half-brother (or cousin, depending on which biographer you believe), who was educated by his master, escaped to California, made a fortune in the gold fields, returned to Kentucky and bought his freedom, then moved to England to race horses.
Justice Harlan has a disputed family tree.