Yes, he was retired and had the time, but he could have just as easily spent that time on the golf course.
The man was kind of annoyed he spent the day with his wife instead of on the golf course. 
mnli hypothesis: The man was kind of annoyed he spent the day with his wife instead of on the golf course.  premise: Yes, he was retired and had the time, but he could have just as easily spent that time on the golf course.