Yes, our freedom of conscience comes into play.   The freedom of conscience of the Catholic comes into play, and the freedom of conscience of the secular thinker comes into play.   Everyone has a conscience which informs their decisions on abortion and all kinds of things.

When we hold an opinion, formed by our conscience, it doesn't necessarily follow that our opinion is a violation of the rights of others.    Even if our opinions, our conscience, is informed by our religious beliefs, that doesn't necessarily mean our views are somehow illegitimate or an trespass on the rights of others who don't have our religious views.

What's the test to see if our conscience is violating others rights?   In the secular sphere, it's called American democracy.    Certain rights are set by law, some through voting, others through judicial decisions.   Every American then, whether religious or not, has the right to their views within this democratic system.   Not perfect, but nothing is.