Part II

With good stuff in ML you are describing a
world I've seen no solid evidence of first
hand.

Maybe that's to be expected: Long I did
well learning math in part because I
picked some really good books; due to some
lucky parts of my background, that was
easy to do. Then I tried this in
statistics and picked a junk book --
eventually figured that out.

I didn't have much respect for or progress
in statistics before I had a high end
background in probability from a star
student of E. Cinlar at Princeton from
Neveu's book, Neveu a star student of
Loeve at Berkeley. Then in stochastic
processes I got some more respect from
work of J. Doob (prof of Halmos who then
became an assistant to von Neumann, etc.)
and the material on probability in the
back of the Halmos, Measure Theory. The
Halmos work on sufficient statistics and
unbiased estimation also deserved
respect.

I've seen no such quality from ML. You
suggest that there is such quality out
there, but what I saw, e.g., Ng's on-line
lectures on ML, the idea of convolutional
neural networks for image recognition,
doesn't look nearly as solid as what I did
in probability or nearly as interesting as
some of what you point to.

E.g., just yesterday I went to

https://simplystatistics.org/

a Web site of three profs in bio-stat, and
read some of the material of one of them,
Jeff Leek, looked at the TOC and some of
the lectures in his Coursera on-line
course, some of the research problems he's
working on, some of what he said about ML,
what he said about cross-validation and
bootsrapping in model building, etc. and
concluded that with ML still I was not
missing much.

I did look at

Trevor Hastie, Robert Tibshirani, and
Jerome Friedman, {\it The Elements of
Statistical Learning Data:\ \ Mining,
Inference, and Prediction, Second {\it
Edition,\/} Springer, 2008.\ \

found the interesting looking work on over
fitting and otherwise was not impressed at
all.

Yes, you have more references I have yet
to pursue, but what I did yesterday and
other times was fast and easy and seemed
to touch on some of the most respected
work. But, yes, it took me some real
digging and a lot of luck to find the
really good stuff in probability,
stochastic processes, optimization, etc.,
so maybe I'd need a lot of digging to find
the good stuff in ML. Since ML is a more
recent field, likely it has yet to receive
the polish of Tukey, Halmos, Loeve, Neveu,
Doob, Dynkin, Shreve, even back to
Kolmogorov, etc.

I do wonder, then, what the heck the ML
people are driving at? Really it looks
like they are growing around the edges of
old regression analysis model building,
trying to fit to data (training data)
and then use the fit to make predictions.
Okay. But I'm not going to take 5 TB of
pictures and try to recognize cats, dogs,
men, women, etc. with some version of
curve fitting with many thousands of
variables -- that's just a long way from
anything I'm doing, and more generally I'm
reluctant to believe that any technique
that needs that much data will be more
than a niche. E.g., for intelligence,
kittens, puppies, and toddlers do really
well with much less data. E.g., a kitten
who jumps up on the kitchen counter, walks
over to the stove, and gets a hot foot
learns well from just that one data point.
Moreover he starts to find causality and
generalize.

Really, my view is that for more in
getting intelligence out of data, we
should concentrate on cases where there
likely is some version of causality and
have our applied math look for and then
use that. I didn't see ML doing that. My
brother and my wife got their Ph.D.
degrees in the social sciences, and there
they were quite careful about causality.

For more parameters than data points,
okay: Sure, in some non-linear cases,
that might happen. And even in linear
cases, if just open eyes a little, can see
that and, thus, get around over fitting in
some cases. E.g., regression is really a
perpendicular projection onto a linear
subspace spanned by the vectors of the
input data. Okay. Now just enter the
input data twice. So, then the dats still
spans the same linear subspace and we get
the same perpendicular projections. If
write out the regression normal equations,
then the variance-covariance matrix get
won't be non-singular and won't have an
inverse. So, big heart burn. But heart
burn is premature: The normal equations
still have a solution, just infinitely
many. So, the regression coefficients
aren't unique. BUT, and apparently not
many people have noticed this, all the
predicted values from the given data still
are unique, that is, are just the same
projection. That is, take any of the
infinitely many solutions you want from
the normal equations and get the same
predicted values from the given data
(maybe not new data). That observation
solves some of over fitting but not all.
But for your remark, that IS a case of
working with too many parameters.

For more, looking at Jeff Leek on
regression model building, he quickly goes
into cross validation and bootstrap
techniques. Okay, cross validation looks
like intuitive, heuristic, Medieval
stirring pots of rat tails and frog eyes
to me, and boostrap needs some theorems (I
believe I have derived some -- I know,
look at Bradley Efron's work which I've
done a little but so far I prefer my
derivations) to justify and explain what
is going on, but roughly Leek is correct,
and I didn't see even that much care and
concern in the ML materials.

If ML is still trying to milk some more
utility out of the edges of simple, old
regression, that is, fitting to data and
using the fits to predict, then I'm not
much interested: Often regression works
really well; where it doesn't, and IMHO
over half the time it doesn't, then I'm
not going to strain to squeeze a little
more out of that curve fitting paradigm,
at least not now.

To me, curve fitting to build predictive
models is nothing like all that can be
done with statistics and only a tiny drop
of what can be done with applied math.

E.g., where is the good science from
simple, linear equations? From Newton,
Maxwell, Einstein? Nope, none of those.
Where did they make progress with fitting
linear equations? Gee, Kepler used
ellipses and Newton drew from those.

For fitting, the big success was Ptolemy's
epi-cycles, and that still was just
fitting and not causality or real science
or physics.

Moreover, now I'm concentrating on my
startup -- I've long since derived the
math, and I see nothing in ML to even hint
that their work could improve on mine.

Sure, when my startup is done, I'll take
on new interests. But curve fitting? I
doubt it! Real AI? Maybe. More likely
physics!

Sure, maybe some day there will be a
Kolmogorov, Doob, etc. to do terrific work
in ML or new frontiers of statistics
and/or applied math and a Loeve/Neveu to
do a polished job writing up the work.
Maybe.

Then there's a credibility problem: From
what I've seen, nearly none of the
computer science profs working in AL/ML
have what Jeff Leek called "math chops"
that promise they can do really good
research to move statistics ahead.

I'd have a sign over the door reading
"None enter without a proof that there are
no countably infinite sigma algebras."!

That will select the people with good
"math chops".

From your claims, maybe there actually is
some good work in ML but hard to find; as
I started, hard to find would not be too
surprising since early in my career
finding the good stuff in probability and
optimization was really hard. Even in the
statistics community, the good people are
hard to find. I can respect E. Dynkin, E.
Cinlar, J. Tukey, L. Breiman, D.
Brillinger, and a few more, but for a
beginner in statistics those people will
be difficult to find, understand,
appreciate, respect, etc. Maybe something
similar is the case in ML now.

In statistics taken broadly, there is a
guy from France I know and respect; he was
a student of Choquet, right, a member of
Bourbaki!

But, again, for ML to have my respect,
tough if they are just still trying to do
curve fitting.