The men were outside trying to keep their voices down so as not to waken the woman indoors