While it makes no sense to worry that you are living well at your grandchildren's expense, you might legitimately worry that someone else is living well at your grandchildren's expense.
You should be considered about the grandchildren's budget needs.