Thoughtful comment, Jason.  Short answer: That's why there's a graduated income tax.  
Let's apply your logic elsewhere.  To fund our enormous military, it involves taking from others.  To maintain our extensive highway system involves taking from others.  To keep our courts and prisons operating involves taking from others.  And so on.  The people who are benefitting the most from the system we've established - infrastructure, rule of law, security...should be the people who help the most keeping the system functioning.