Williams married firstly, by July 1524, Elizabeth , daughter & coheiress of Thomas Bledlow of Bledlow in Buckinghamshire, grandson of Thomas Bledlow and Elizabeth Starky, daughter of Sir Humphrey Starkey, Chief Baron of the Exchequer. Elizabeth Bledlow was the widow of Andrew Edmonds  of Cressing, Essex, by whom she had a son, Sir Christopher Edmonds , and a daughter, Ursula Edmonds. By Elizabeth Bledlow, Williams had three sons, John, Henry and Francis, and two daughters, Isabel, who married Sir Richard Wenman, and Margery, who married Henry Norreys, 1st Baron Norris of Rycote. Williams married secondly, on 19 April 1557, Margery Wentworth , the daughter of Thomas Wentworth, 1st Baron Wentworth of Nettlestead, Suffolk. She survived him, and later married Sir William Drury, and James Croft of Weston, Oxfordshire. Williams's three sons predeceased him, and the barony became extinct at his death on 14 October 1559, although Lee gives this date and also says his eldest son survived him for four months, dying unmarried and without issue in February 1559. His elder daughter, Isabel, inherited Thame and married Richard Wenman and was the mother of Thomas Wenman, while his younger daughter, Margery, inherited Rycote.. How many daughters did Williams have?