Walker was born in Nashville, Tennessee, in 1824 to James Walker and his wife Mary Norvell.
Walker lahir di Nashville, Tennessee, pada tahun 1824 dari pasangan James Walker dan istrinya Mary Norvell.