But let's look at the economics from the ISP's perspective. What incentive does the ISP have to charge Netflix less, regardless of whatever costs they have?

First of all, most big ISPs are competitors to Netflix. Directly so through online streaming services, but I think focusing on those is missing the big picture, which is cord-cutting. For cord-cutters, Netflix and other services add up to a replacement for the ISP's expensive TV service, and while not a lot of people do this today, the trend over the last few years seems pretty clear. Thus the ISPs have little reason to be particularly invested in general in making high bandwidth transfers on the Internet actually work.

Aside from that, they could be incentivized by competition from other ISPs. But 75% of American homes have only one ISP providing at least 25/3 [1], and even at 4/1 the majority only have two choices, so this is pretty limited. While I am very far from an expert in this field and I know there are many factors involved, it's hard to see this as not a large factor in the high prices Americans pay for Internet service compared to other countries, independently of the inherent scarcity of the resource (which, after all, can be expanded with enough investment).

And when it comes to charging content providers, the situation is worse, because the cost is hidden. Even in a competitive market for ISPs, fees charged to Netflix don't affect the sticker price - admittedly, at the moment it seems to be all of $1, which is a significant part of Netflix's price but not the Internet connection's [2], but that just reinforces the arbitrariness of it. And if negotiations fail and the ISPs slow down Netflix connections by neglecting to upgrade equipment, they can still advertise 100Mbps or whatever on their website, regardless of the fact that on one of the most popular destinations (bandwidth-wise) on the Internet you can't even get 5Mbps! This is such an enormous asterisk to the speed promise that even given the inherent variability of speeds on the Internet, it feels very much like false advertising.

This used to affect me personally: being in the minority that has two high-speed options, somewhat less than a year ago I switched my home's Internet connection from Optimum Online, cable service which AFAIK is not trying to charge Netflix a fee, to FiOS. For the same price it promised faster download (150 vs 100), much faster upload (now symmetrical, yay), and there were faster options potentially available in the future if we were willing to pay even more money. As it turned out, for most of the internet FiOS really was better - even if I could rarely ever download anywhere near 20 MB/s, 10 wasn't out of the question for CDNed content, significantly better than the previous service. But the result was to the significant displeasure of my family, because instead of downloading games from Steam, they watch Netflix a lot, and it started frequently degrading quality or failing altogether. As someone aware of the issue I honestly should have know that would happen... IIRC, since the connection was far from perfect under the old ISP I thought it wouldn't make that much difference, but I was wrong. My fault - I think largely resolved now that Netflix is paying up - but the vast majority of consumers are completely unaware of this! Any non-tech-enthusiasts choosing service during the time had little way of knowing about this giant caveat, giving Netflix no bargaining power even in this competitive segment of the market.

I suppose this part at least could potentially be changed without government intervention: Netflix could raise its price only for users of certain ISPs, or ISPs could start advertising their Netflix speeds. But with all these factors combined, the whole thing is a mess and in no way represents an efficient market. Thus while I hope for competition to some day make net neutrality-like regulation unnecessary, I strongly support such regulation.

[1] http://www.fcc.gov/document/fcc-chairman-more-competition-ne...

[2] http://arstechnica.com/business/2014/05/netflix-comes-throug...