> something is rotten in the state of academia. Oddly it's academia doing incremental improvements to existing methods but industry making novel leaps and bounds... The other major case in point being NLP

You have to realize that corporate research labs had a high level of recognition back in the 20th century. Labs like the Bell Labs, the RCA Laboratories, or the IBM Research, privately-funded, had a reputation that met or exceeded the standard of not-for-profit or public-funded academic research institutions. They made some of the most important discoveries in the electronics industry of 20th century, like the point-contact transistor, the MOSFET, VLSI, or the UNIX operating system. They were considered a part of the academia, many scientists were their employees. It's only the 1980s after their death that people had the impression that "important research must come from academia, industry is for incremental changes." So, I'd argue that the division between industry and academia is large, but actually smaller than people's perception. If you consider privately-funded researches by the industry as a part of the academia, the current situation is totally normal, nothing unusual.

Interestingly, for those labs to exist, being a monopolistic megacorp is a requirement. It appears to me that today's FAANG monopoly allowed the creation of Google Deepmind and OpenAI, perhaps it's simply a beginning of the repetition of history.

The article The death of corporate research labs had an interesting review. I highly recommend to read the article:

* The death of corporate research labs

> https://blog.dshr.org/2020/05/the-death-of-corporate-researc...

(HN comment: https://news.ycombinator.com/item?id=232466722)

To summarize, those great labs existed and made great contributions because of (1) corporate monopoly on the industry, and (2) the pressure from anti-trust laws. First, due to monopoly, the gigantic size allowed the labs to be the center of gravity and to concentrate all talents and projects into a single place, with a huge research budget for basic research. Second, the pressure from anti-trust laws also forced corporations to invent more into basic research to grow the business, because mergers and acquisitions were restricted. In some cases, the pressure from anti-trust laws also made the corporate labs to share their discoveries in a more open manner, examples included advances in semiconductor [1], or the Unix source code.

Note: but as HN comments pointed out, somewhat ironically, the success of corporate labs relies on anti-trust pressures, but not the actual monopoly-busting enforcement. The breakup of Bell caused the death of the Bell Labs.

Finally their decline,

> The more relaxed antitrust environment in the 1980s, however, changed this status quo. Growth through acquisitions became a more viable alternative to internal research, and hence the need to invest in internal research was reduced.

And it turns out that managing a corporate research labs without losing money is a tricky problem to solve. If the researches are too goal-oriented, short-termism will dominate, basic research in the labs will be ignored. Thus, basic research in the lab must be independent. However, a lab too isolated from the business can also cause great loss.

> Research in corporations is difficult to manage profitably. Research projects have long horizons and few intermediate milestones that are meaningful to non-experts. As a result, research inside companies can only survive if insulated from the short-term performance requirements of business divisions. However, insulating research from business also has perils. [...] Walking this tightrope has been extremely difficult. Greater product market competition, shorter technology life cycles, and more demanding investors have added to this challenge. Companies have increasingly concluded that they can do better by sourcing knowledge from outside, rather than betting on making game-changing discoveries in-house.

And the author argued the death of corporate labs decreased productivity.

>> An unintended consequence of abandoning anti-trust enforcement was thus a slowing of productivity growth, because the this new division of labor wasn't as effective as the labs:

> a new division of innovative labor, with universities focusing on research, large firms focusing on development and commercialization, and spinoffs, startups, and university technology licensing offices responsible for connecting the two.

> The translation of scientific knowledge generated in universities to productivity enhancing technical progress has proved to be more difficult to accomplish in practice than expected. Spinoffs, startups, and university licensing offices have not fully filled the gap left by the decline of the corporate lab. Corporate research has a number of characteristics that make it very valuable for science-based innovation and growth. Large corporations have access to significant resources, can more easily integrate multiple knowledge streams, and direct their research toward solving specific practical problems, which makes it more likely for them to produce commercial applications. University research has tended to be curiosity-driven rather than mission-focused. It has favored insight rather than solutions to specific problems, and partly as a consequence, university research has required additional integration and transformation to become economically useful.

---

[0] https://www.eetimes.com/podcasts/six-words-that-built-the-ic...

> Honeywell brought a lawsuit against us and said you can’t selectively choose people to divulge your technology to. It’s too important. And if you divulge it to anyone, you’ve got to divulge it to everybody. They filed a lawsuit, and the government came down on their side. And RCA basically had to open up all of its patents to everybody if they opened them up to anybody.