The article is part of an old problem: How to connect 'research' with 'business'.

I respond with:

(I) Summary.

(II) Details on Reasons for Failure.

(III) Examples of Solutions.

(IV) Opportunity.

below.

(I) Summary.

Quite generally, there are many ways to make a mess and fail, and this is true also in connecting research with business and has a very long history.

There is also a long history of success connecting research with national security, public health, and business.

Net, the main issue is, are results of research really wanted outside of research? The usual answer is, "Not only no but hell no!", and that's the main reason for failure.

(II) Details on Reasons for Failure.

One level deeper, the main reason people outside of research don't want research is that they feel threatened: They would rather just keep doing things the same old way that has been doing well for them (people not doing well can't afford research!) and not risk it by being 'upstaged' or 'disrupted' by research.

Also a researcher with good access inside a successful business organization can be seen as a 'loose cannon on the deck' that might understand the 'secrets' of the success of the business, use research to find a better way, and start a competing business.

The usual situation in business is: (1) Organize like at Ford 100 years ago where the supervisor knows more and the subordinates are just to apply labor to the ideas, direction, and work of the supervisor, (2) pay everyone relatively little, thus, making sure no one feels free to take any risks at all, (3) evaluate everyone on very narrow criteria of getting the present work done with severe consequences for any deviations and, thus, for anything new, ensure no upside and a lot of downside, (4) for anything new inside the organization, kill it off ASAP, (5) for anything very new or advanced outside the organization, f'get about it.

So, net, the only one in the organization with any freedom to do anything new is the CEO, and generally he is too busy just running the existing business and not technical enough to do or supervise anything like 'research'.

Inside a business research lab, usually the goals are at best a mess. So, the path to the business bottom line is not wanted or not clear, but the usual academic criteria do not apply either.

So, without good criteria, the organization is vulnerable to the long, standard list of organizational dysfunctionalities. E.g., the place can become a 'caste' system where the head guys defend themselves with 'cliques' based on incompetence, race, gender, ethnicity, etc., and all the non-manager "worker bees" are expected to keep quiet and out of sight and maybe just publish papers or patents.

So, usually big business organizations just will NOT 'innovate'. This is a problem, but its 'flip side' is an opportunity central to 'Hacker News'!

(III) Examples of Solutions.

Again, one of the crucial points is for the people in the business actually to want the results of research, and at times there have been solutions at AT&T Bell Labs, DoD, NIH, CDC, etc.

One solution has been illustrated at Renaissance Technologies run by Jim Simons: He is a good mathematician (Chern-Simons), long Chair of the math department at SUNY Stony Brook, and at Renaissance sometimes paid himself $2 billion or so a year. Once the Brookhaven National Lab people wanted to collide some gold atoms to create a 'quark plasma', couldn't get the money they needed, so Simons wrote them a check, maybe $20 million or so. So, he wanted the results of research, had a reputation for hiring mathematical physicists from Russia, and was not threatened.

Another example has been Westvaco Paper: They specialized in 'specialty papers', e.g., milk cartons and table tops. Basically they have regarded themselves as an 'applied chemical engineering' company.

They had a research lab between DC and Baltimore: Each six months the lab submitted its budget to the NY HQ, and HQ always wanted to increase the budget, and Research always said, "No.".

Why? My old notes have:

(1) The research group worked only on problems of their own choice and did not accept problems assigned to them. Generally in research, problem selection is both important and difficult, and here the real experts did the problem selection.

(2) Projects started by the research group ran from about two months to about two years. Only about one project in ten led to an implementation.

(3) When research believed they had a project worthy of implementation, they would approach the General Manager (GM) of the appropriate operating group and make their proposal. The decision to implement was up to the GM. Here we see some the importance of having the business organization involved in the implementation.

(4) If an implementation was made, then the financial value was measured by careful auditing.

(5) For the first three years of implementation, half the financial improvement was credited to the operating group and the other half to research. After the first three years, all the improvement was credited to the operating group and none to research. Also, the GM's bonus could be affected by these results. It did help that most of the GM's were Ph.D.'s in chemical engineering.

(6) With these rules, the research group returned to the company, long run, about three dollars for each dollar in their budget.

So, that's one way to run an industrial research lab.

(IV) Opportunity.

For the 'Hacker News' community, the flip side of the sad, old situation is a new opportunity:

(1) Find a problem where a much better solution should lead to big bucks.

(2) Have the founder, CEO do some research to get a much better solution, good enough to be much better than anything else and advanced enough to be difficult to duplicate or equal and, thus, provide a good 'technological barrier to entry'.

Note: IMHO, for the relevant research for 'information technology' the key is not 'computer science' but selected, advanced topics in pure and applied math.

(3) Implement the solution in software and offer it to users via the Web or cloud.

(4) Buy a nice yacht, rinse, and repeat.

For the research part, the main training for research is the Ph.D. The difficulty of Ph.D. programs shows that learning how to do good research and doing it are not easy. Thus, hopefully, the founder, CEO doing the research has a good, relevant Ph.D. This way, no one in the organization is threatened by the research that helps start the organization.

Once again, the ways big organizations find to make messes can be corrected by entrepreneurship.

So, don't curse the big organizations. Instead, use their problems as an opportunity to be a successful entrepreneur and buy a yacht.