The fundamental legal basis for protected classes derives from the text and history of particular clauses in the constitution. For example, racial protections obviously derive from the 13th, 14th, and 15th Amendments. Protections for sex derive most directly from the 19th Amendment giving women the right to vote, which has been deemed to have reshaped other clauses to put protections against sex discrimination nearly on par with those against racial discrimination. Crucially, these clauses themselves identify particular classes like race and sex, and the history of the passage of the associated amendments provides insight into the contours of those classifications.

Relatively late in the legal evolution of civil rights jurisprudence the Commerce Clause was interpreted to not only expand pre-existing anti-discrimination powers, but to provide an independent source of power to ban discrimination. This is why Congress has the power to regulate discrimination against the disabled, for example. Nonetheless, the further away from the plain text and history of the Constitution, the more tenuous Congressional powers become. So, yes, Federal laws against disability and age discrimination have the least constitutional support and are also commonly construed the most narrowly.

Just as important, all of those textual sources grant affirmative powers to the Congress. For example, section 5 of the 14th Amendment says, "The Congress shall have power to enforce, by appropriate legislation, the provisions of this article." The 19th Amendment says, "Congress shall have power to enforce this article by appropriate legislation."

Remember, the Federal government only has enumerated powers.
Nowhere is political affiliation directly discussed in the Constitution in a context that would permit it to distinguish political affiliation as a cognizable type of class, and certainly not in a context granting Congress affirmative powers to differentiate based on such class. Political affiliation would best fit with the Free Speech clause of the First Amendment, which the Federal government has powers to enforce through the so-called incorporation doctrine of the 14th Amendment. But Free Speech jurisprudence is concerned with content of speech, not classes of people, unless we're discussing journalists as a class.

Freedom of Association doesn't fit because, similar to Freedom of Speech, it doesn't imply the power to differentiate by association, per se; that is, it doesn't imply the power to recognize group A and group B and give group A special protections. Quite the contrary, it implies the inability to differentiate among them.

The Commerce Clause doesn't work. As broad as the Commerce Clause has been interpreted, political affiliation is perhaps one of the last remaining areas that isn't construed to directly implicate commerce.

Basically, the reason political affiliation isn't normally a cognizable class is because there's no direct nor even indirect source of authority that grants the federal government the power to regulate in such a manner. Plus, there's a whole host of reasons why permitting the government to differentiate people based on political affiliation would harm more fundamental prohibitions against discrimination.

EDIT: Regarding the question of gay marriage, note that Kennedy's opinion doesn't make homosexuals a protected class. Rather, the basis of that opinion is that the government lacks any power to differentiate between homosexuals and heterosexuals when granting marriage licenses. That said, there's a liberal vein of jurisprudence that construes LGBTQ discrimination as a form of sex discrimination.