Question: are you supposed to take your hat off in a restaurant?
Answer: A statement hat, on the other hand, is perfectly appropriate in a restaurant. You should not remove it, any more than you should remove your shoes at a party: Doing either could ruin your whole look.