James Douglas was the second son of Sir George Douglas of Pittendreich, Master of Angus, and Elizabeth Douglas, daughter David Douglas of Pittendreich. He wrote that he was over 61 years old in March 1578, so was probably born around 1516. Before 1543 he married Elizabeth, daughter of James Douglas, 3rd Earl of Morton. In 1553 James Douglas succeeded to the title and estates of his father-in-law, including Dalkeith House in Midlothian, and Aberdour Castle in Fife. Elizabeth Douglas and her two elder sisters, who were married to Regent Arran and Lord Maxwell, suffered from mental ill-health. Their children either did not survive to adulthood, or in the case of three daughters were declared legally incompetent in 1581. James also had five illegitimate children At the start of war of the Rough Wooing, James and his brother David communicated with Henry VIII of England on the possibility of their surrendering Tantallon Castle to the English army that burnt Edinburgh in 1544. However, four years later he defended Dalkeith Palace against the English and was captured in June 1548, "sore hurt on the thigh " and taken as a hostage to England. After the Treaty of Boulogne brought peace, in 1550 James returned from captivity in England and was exchanged for the English soldier John Luttrell, and began to use his title of "Earl of Morton." In 1559 James's political activities and allegiances during the Scottish Reformation were at first equivocal, but in February 1560 he signed the Treaty of Berwick which invited an English army into Scotland to expel the Catholic regime of Mary of Guise. He took part in the unsuccessful embassy to England in November 1560 to treat for the marriage of Elizabeth I of England to James Hamilton, 3rd Earl of Arran.. Who was born second: Sir George Douglas of Pittendreich or James Douglas?