>> A big component of it all is certainly what Rich Sutton called the "bitter lesson", i.e. that throwing ever bigger compute and data at scalable models has been the major catalyst of the field.

But the major catalyst to what? Neural nets today are just as capable as they were 40 years ago, only now we have bigger computers and more data and we can put more lipstick on the pig.

I appreciate what you say, that systems today give better outputs than in earlier years. But you should also appreciate that those are essentially the same systems, with little tricks and hacks that have no clear contribution to performance, since as you say they are unreproducible. And each and every one of them inevitably, inexorably falls to bigger systems with more data and more compute - isn't that the bitter lesson after all? That you can try to be smart but the guy with the bigger machine and the bigger dataset will always win?

And all the while the major limitation of neural nets trained with gradient optimisation remains unaddressed: that they can't function unless they have large amounts of data to train with and they only get performant enough to upload a paper to arxiv, let alone to use in a real problem, once they've been trained with hundreds of thousands, if not millions, of examples, at gigantic monetary cost. That is because -contrary to absurd claims by some- neural nets can't generalise their way out of a paper bag. If they could, they wouldn't need so many examples; they would learn, and generalise, robustly, from few examples. But they can't. Not at all.

The field has simply made a virtue out of necessity, and hence comes the embrace of the magickal shibboleth of the Bitter Lesson. Neural nets can't generalise? Just overfit to more data.

But, imagine if computer scientists, for example, claimed progress because we can now sort ever larger lists using bubblesort on giant compute clusters; or that the question of whether P = NP can now be laid to rest because we can solve simple instances of the Travelling Salesman Problem on a sufficiently powerful supercomputer. That is the Bitter Lesson applied to an imaginary computer science field, where progress had stalled, no sorting algorithms with good asymptotic complexity, like quicksort or mergesort, were known, and where combinatorial optimisation heuristics like Clause Driven Clause Learning where not invented. That is the machine learning field today, having learned the Bitter Lesson: a field that has given up on solving its research problems and instead turned to profit-making schemes to get more money from giant corporations who are in the business of beating people up with millions to make billions, and not in the business of answering any useful research questions.

And you know what? Because of this, progress, even by the self-selected metric of "beats benchmarks", eventually stalls. Machine vision hasn't progressed much since 2012. Machine translation is still stuck in 2016 ish. Reinforcement Learning is coming next year, together with self-driving cars. Now we got LLMs and image generators. Give it a few years and that will be so 2020's and a new peak of hype will be reached, while LLMs remain incapable of telling truth from bullshit.

Science is not an ideal, it's the way to make things work. Like, really work, in the long run. What you describe is the way to only make things sort of work and only in the near term.