He has two daughters, Mary D. Edsall, a writer (wife of Thomas Edsall), and Margaret D. Carroll, an art historian, and three grandchildren, Alexandra Edsall, Sophia Carroll, and Samuel Carroll.
Ia dikaruniai dua putri, Mary D. Edsall, seorang penulis (istri Thomas Edsall), dan Margaret D. Carroll, seorang sejarawan seni, dan tiga cucu, Alexandra Edsall, Sophia Carroll, dan Samuel Carroll.