The biggest problem is that hardware is a sucker's game. The minute you have to start creating even moderately specialized PCBs for your product, you incur a ton of extra costs.

You have to deal with yields from the fab process, a hardware testing/debugging process that often requires an expensive oscilloscope, an up front outlay of capital just to get the pcbs produced, you have to deal with getting it certified as being 'safe'.

It's easy to bridge from Arduino to Atmel AVR IF you know how to do board layout.
Startups need to either grow their knowledge of electronics or hire some electrical engineers.
The AVR documentation is excellent, you could easily design your own board if you have anything beyond rudimentary electronics skills.

If you're doing something very simple, maybe... but If you're doing something very simple, why do you need specialized hardware? Get something prebuilt that runs embedded c or linux, write your software, attach your controllers (build a nice case), and be done with it.

If you're doing something more complicated, (multiple layers, pcie, etc.) You'll never get the yields that you need (to be profitable) out of your fab process without either a very skilled/experienced EE, or a team and a bunch of money. Even with a simpler (or no) fab process you still have to worry about defects in production and testing for those defects before you ship the item. But at least without a fab process it can be arranged to be someone else's problem when the widgets don't work.

It's not that you're wrong, its just that doing your own manufacturing is either:

a.) Adding a lot of expense to something that needn't be as expensive if you can buy something that already pretty much works in bulk. If you reach the state of mass production, it could make sense to do this yourself, but at that point you may be past the startup/proof of concept phase.

b.) Necessary but very expensive (more expensive than it appears on the surface) and problematic to both your margins and cash on hand. If you go this route you better have some backers with extremely deep pockets who believe in you and are willing to throw in extra cash when the first run of your board has issues and you get a low yield on them.

I agree, however, about the value prop issue.

But that's also kind of why I think hardware is a sucker's game. Either you get screwed by having to make your own stuff, or you get screwed by being dependent on a third party who may not be reliable (or in business, or still producing the thing that you need). Or both, because you're likely getting it fabricated by a third party, which will lead to the same issues as purchasing something 'off the shelf,' plus the possibility of having no one to blame but yourself.

source: did a hardware startup.

edit: I forgot to mention one other factor... If you're doing something high performance, there's a possibility that by the time you're ready to ship the product, it is out of date and there's some faster next-gen hardware out that will do the job better. This is exactly what happened to AMD with Bulldozer (there were some other fuck-ups there too, but for the most part it was superseded by intel's more advanced fab process).