Regarding my background I just finished my PhD and I'm doing a postdoc on explainable machine learning for robotics. My PhD was on Inductive Logic Programming which is a virtually unknown form of machine learning of logic theories, but I have a background in statistical AI, neural nets and NLP, from my Master's days.

Btw, lest this causes confusion, my part in the "explainable machine learning for robotics" project is to machine-learn autonomous behaviours for an underwater search-and-rescue robot. My specialty is not explainability, it's rather learning ability I'm interested in.

That out the way,

>> If you don't trust BLEU and don't trust my gut, how about a randomized survey of professional translators, or how many seconds per character a translator would need to fix up the raw output into something they'd be comfortable with calling accurate? Like, I'm not stuck on a metric.

No, sorry. I want a metric I can trust and not one that is equivalent to good, old-fashioned eyballing. If I wanted to eyball, I got eyballs, I can eyball. I want a metric that doesn't depend on someone's subjective evaluation. Unfortunately, the day that someone will come up with an objective test for machine translation is the day we'd have solved machine translation, and possibly all of AI, to boot.

Btw, I studied NLP at the time when Google Translate had just switched to Neural Machine Translation. I'm Greek, and I remember on the one hand the dithyrambic self-congratulations on Google blogs, and on the other hand how ridiculously crap Google Translate was in translating from and to Greek. And it still is, almost ten years later (My MSc was in 2014).

I also have a cluster of friends and acquaintances who are all professional translators and interpreters and I know that they are still ripped off by translation agencies who give them automatic translations claiming that they only need an hour of work to correct, when they need maybe a day or two. That kind of test will also not do.

"Progress" in machine learning means that every few years the hype peaks for yet another AI fountain of youth, or AI El Dorado, then that loses steam, and there's some sanity for a few years until the net thing comes along and trends on social media, then everybody starts writing super-serious papers about it sounding all business-like and scientifically minded, until Google, or OpenAI now, or some other big gun releases an implementation and everyone has to run behind them like idiots. It's basically science by twitter and it's completely bogus.

Bottom line, you want to make machine lerning work? Be Google.

>> The right reaction is awe and curiosity and marvel that a nonrigorous process has led to this.

To what? I think I've made myself very clear that I don't think we've got anywhere worth talking about. So my computer can now talk back to me. So what? Why is this a scientific achievement? Why does this answer scientific questions? Which ones?

>> During the history of science, big discoveries always needed a preparatory pre-scientific phase of mere aggregation of experiences, stamp collection if you will, developing almost superstitious practices that may turn out to be false on the level of the individual components, but in aggregate they result in a net positive.

I've heard this before but I don't know any examples where technological progress did not go hand-in-hand with scientific understanding. For example, when the Wright brothers made a machine that could fly, the followed on the footsteps of probably hundreds of others who had done the work of science, developing such necessary concepts as aerodynamics- like George Cayley, Otto Lilienthal of course, and in fact the Wrights themselves had set up a wind tunnel and studied the aerodynamics of their machines.

The difference with AI of course is that it's much easier to point to a thing and say "it flies". But that's another matter.

Lest I be misunderstood: I'm enjoying our exchange. Thanks.