Mosche Vardi's essay is provocative, but still leaves a few too many points vague and undefined for my taste.

The nature, questions, and value of research are one element of a larger set of questions I've been exploring.

I've recently run across David Hounshell's work on R&D at DuPont: Science and Corporate Strategy: Du Pont R&D 1902-1980, originally published in the 1980s.

https://www.worldcat.org/title/science-and-corporate-strateg...

For a ... short-ish ... synopsis, covering main themes, see Hounshell's article: "Measuring the Return on Investment in R&D: Voices from the Past, Visions of the Future"

https://www.ncbi.nlm.nih.gov/books/NBK45334/?report=classic

Upshot: R&D is a risk-based activity, and it can have high payoffs, but also high costs. Much of Hounshell's book is about management's attempts to both organise its research and research teams for optimal results, and to manage the costs and risks associated with it. There was a period in which the labs were phenomenally productive, including a roughly decade-long span from the 1920s through 1930s where virtually the entire modern plastics industry (Graduate fans take note) was invented. But also flops (a fake leather product produced in the 1960s notably).

This fits in with notions I've been noodling at of risks in economics -- an element of virtually all economic activities, but with distinctly different characteristics and scope. R&D tends more toward the dice-rolling variety (you may win), whereas some activities gamble with catastrophic and systemic loss. Some fields appear to be little but raw risk balancing (much of finance, real estate, and of course, insurance) whilst others provide a greater opportunity to directly influence, reduce, adopt, or mitigate risks (engineering, generally).

There's also the risk, or uncertainty, involved in attempting new activities. The recent HN submission "The Drugs Won't Work" https://medium.com/@belledejour_uk/the-drugs-won-t-work-659c... (https://news.ycombinator.com/item?id=20789000) talks of the problems of new drugs discovery, effectively a search through a 10^200 node space for potential beneficial compounds, with very few heuristics in reducing or guiding the search (Lipinski's Rule of Five being one: https://en.wikipedia.org/wiki/Lipinski%27s_rule_of_five).

All research (and much coding) is effectively a search though a large possible solution space, subject to constraints and domain characteristics, for possible useful elements or combinations. The problem of huge search spaces means that alternatives to brute-force exploration are quickly necessary.

Back to Du Pont: the chemical search space is also large (pharmacology is effectively a branch of that), and low-hanging fruit was found early with small, simple, and readily-attained compounds: the law of diminishing returns. Worse was the discovery, often much later, that along with useful features came harmful ones -- the law of unintended consequences being another suprisingly general principles that arose from a specific discipline (sociology) but is applicable in nearly all others.

As for management: by Hounshell's account, management largely turned its researchers loose and told them to have fun.

There's the further challenge that the discovery of keys rarely coincides with the knowledge of the lock in which it fits. The phrase "a solution in search of a problem" was first applied (by Theodore Maiman: https://en.wikipedia.org/wiki/Theodore_Maiman) to a technology of which there are almost certainly numerous instances of in your immediate vicinity: lasers.

Early applications were thought to be in the application and delivery of power (Arthur C. Clarke makes a passing reference in 2001 A Space Odyssey to blasting the Lunar monolith with one), but it was the characteristic of lasers as a tightly coherent transmissive blank slate onto which information could be encoded which proved its greatest use.

(This makes me suspect that materials such as graphene -- a uniform, monoatomic plane, might have its greatest application as information storage medium rather than physical materials, much as doped silicon has proved useful in semiconductors.)

Another part of my general study has been in thinking how technology works by considering what technological mechanisms fundamentally exist, and how those interact. My list presently numbers nine (materials, fuels, power transmission & transformation, process knowledge, structural knowledge, networks, systems, information, hygiene -- fuller descriptions elsewhere, still in process), which may or may not prove ultimately accurate. But has been somewhat useful in thinking through numerous problems and applications. Each mechanism has specific characteristics and limitations.

Upshot: I don't think "technology" is a bottomless bag of tricks. It's definitely useful, and well get more from it, but it also has costs, including complexity (itself a form of risk, which is to say, of debt), but there are bounds. R&D is exploring a space with constraints, in which both benefits and risks may be found. Treating the matter probabalistically and as one to be approached in on a risk-based approach may prove generally useful.