Soul has been married five times and has five sons (two of whom are stepsons) and a daughter.
後妻は上杉治広の五女増子（死別）、本多忠顕の娘。