It is not about baking a cake, it is being asked to depict or write something on that cake which the baker finds abhorrent and to do so would trouble his conscience.
Could a good, Catholic baker not feel he was doing something wrong if he inscribed upon a cake, " Abortion is a woman's right to choose". 
This is but one instance. There will be many other such instances, e.g. a homosexual baker asked to decorate a cake with a biblical quote condemning same-sex marriage. Would he not have the right to refuse?
If it is so trivial as you suggest, your Supreme Court disagrees with you.