Part I

Well, in part we aimed too high: My wife
was brilliant, quite broadly -- yes,
Valedictorian, Summa Cum Laude, Woodrow
Wilson, PBK, piano, clarinet, voice,
prizes in cooking, sewing, raising
chickens. But her family had her try to
be perfect and to dedicate herself to
saving the world.

She wanted a Ph.D. in mathematical
sociology to do social engineering of
social change to save the world and,
thus, get her praise, acceptance, and
emotional and financial security. And
those concerns filled her plate.

Well, at the roots there some anxiety,
from nature and/or nurture, was involved,
then some perfectionism and some fear of
criticism from powerful, influential
people -- net, call it a special case of
social phobia. That brought stress
which can bring depression. That slows
the work and in her case caused more
stress and more depression then clinical
depression. She was in a clinical
depression the day she got her Ph.D. She
never recovered and, net, didn't make it.

Took me a while to reinvent and learn the
basics of clinical psychology to
understand what was going on and what to
do about it. I did learn but nearly
always too late.

In trying to have a stable job so that I
could take care of her, I took a job at
IBM's Watson lab, in what we called
artificial intelligence (AI) to do
monitoring and management of large server
farms and networks.

Then IBM got sick and the lab phone book
went from 4500 full time names down to
1000 or so. The guy who hired me, a big
star who was deliberately ignored by the
higher ups, left for greener pastures --
eventually ended up with a nice place in
Malibu.

IBM Watson Research was run by a clique
of people who stuck together and blocked
out everyone else. At one point it
appeared that the company HQ in Armonk
tried to correct that situation.

Instead of or better than the AI, I had
some ideas: Detecting a problem in a
server farm or network is necessarily
essentially a statistical hypothesis test
with Type I (false alarm) and Type II
(missed detection) errors. So, want
hypothesis tests that keep down the
rates of the errors.

Since there's a lot of data readily
available and want to exploit it to keep
down the rates of the errors, also want
multi-variate inputs -- nearly all
hypothesis tests have only univariate
inputs. Also for such multi-variate data
can't hope to know any of the probability
distributions so need tests that are
distribution-free.

I don't think there were any such tests,
so I dreamed up some. I used some group
theory, summed used the classic S. Ulam,
guy on the left in

http://www-history.mcs.st-and.ac.uk/BigPictures/Ulam_Feynman...

result LeCam called tightness (see P.
Billingsley, Convergence of Probability
Measures), and was able to permit
selecting false alarm rate in advance and
then getting that rate exactly. Asking
for all of the Neyman-Pearson result would
take more data than we had any chance of
having, but there was a somewhat useful
sense in which my work gave
asymptotically, for any selected false
alarm rate, the highest possible detection
rate with that false alarm rate.

I cooked up some synthetic data that was
challenging -- the critical region was
something like the red squares on a
checkerboard in several dimensions. My
work did fine.

I got some data from a cluster of
computers at Allstate, wrote some
prototype software, and confirmed the
false alarm rate empirically. And I
cooked up an algorithm to make the
computations nicely fast (in part I used
k-D trees -- reinvented those -- a few
years earlier and k-D trees would have
been mine).

Politics: A guy in the clique up there
didn't like me. But it took two levels of
management to fire someone, so he
reorganized to put me under a wuss who
would go along with firing me. They
claimed my research was not publishable
(reviewed by a guy in the clique who
admitted he couldn't read my math but
claimed to have found someone who could
but found nothing wrong with my paper) and
walked me out the door.

The next day the wuss was demoted out of
management. Two weeks later the main
nasty guy was moved down to have him under
an additional level of management and
given a six month performance plan which
he failed. He was demoted out of
management -- lost his corner office,
budget, secretary, and 55 subordinates.

I got a PC and Knuth's TeX and submitted a
paper on my research. Since the paper had
some measure theory in it, e.g., Ulam's
result, much of the computer science
community couldn't read it. But the
journal that offered to review the paper
kept at it; apparently the editor in chief
walked the paper around his campus, to a
CS department to see if the problem was
important and to a math department to see
if the math was correct, and accepted the
paper. He invited me to present at a
conference he was running, but I didn't
want to bother going. The paper was
published. IBM was wrong.

So it appears that I have the world's only
collection, and it's large, of statistical
hypothesis tests that are both
multi-variate and distribution-free, with
some nice properties, with a fast
algorithm, with some confirmation of the
false alarm rate calculations from some
real data, etc.

Asymptotically the critical region can
be a multi-dimensional fractal. Nice.

People should my work. I did give a talk
on the work at the main NASDAQ site in
Trumbull, CT.

IBM didn't pay very well, and cost of
living was high -- I'd always saved money,
even in grad school, but I lost quite a
lot of money working at IBM.

When I joined IBM, it'd just won the Nobel
prize in physics two years in a row and
had a long string of being "the most
admired company in the world". Now I'd
advise anyone just to stay the heck away,
a long way away.

But being pushed out of IBM and age left
me 100% permanently unemployable. I sent
1000+ resumes. Zip, zilch, zero.

If in computing, be sure by age 40,
hopefully by age 35, to have a rock solid
stable career and/or be wealthy. So,
really about have to own your own business
and make it successful.