Mary teaches calculus to 20 students, and Bob also teaches
calculus to 15 students. A standardized test is given
to all the students, all 35.

So, suppose we assume a null hypothesis that
all 35 student scores are independent random
variables with the same distribution, that is,
(1) Mary and Bob are equally good as teachers
and (2) their students are equally well qualified.
If the students had been appropriately randomly
assigned to Mary and Bob, then maybe we can believe
(2) so that only (1) is in question. So, we are
going to test (1), that is, that Mary and Bob are
equally good as teachers.

This hypothesis, that is, that Mary and Bob are
equally good is called a null hypothesis since
it assumes that there is no effect between
Mary's class and Bob's class (even though
Mary is teaching more students than Bob).

So, here is how we do our test: We throw all
35 scores into a pot, stir the pot energetically,
pull out 20 in one bowl
for Mary and put the other 15 scores into a bowl
for Bob. Then we average the scores in each
bowl and take the difference in the averages,
say, Mary's average minus Bob's average.
Then we repeat this many times -- for this
might want to use a computer with a good
random number generator. This process is sometimes
called 'resampling'. It's also possible to
argue that what is going on is a finite group
of measure preserving transformations that, thus,
can yield what we are doing more intuitively.
Physicists like symmetries that result in
conservation laws, but here we have symmetries
resulting in an hypothesis tests.

So, we get the empirical distribution
of the differences in the averages.

Then we look at the difference in the actual
averages, that is, from the actual students
of Mary and Bob. Call this difference
X. Now we see where X is in the empirical
distribution of differences we found.
If X is out in the tails with probability,
say, 1%, then either (A) Mary and Bob
are equally good as teachers, that is, we
accept the null hypothesis, and we have
observed something that should happen
1% of the time or less or (B) Mary and Bob
are not equally good as teachers and
we reject the null hypothesis and
conclude that there is a difference
in teaching between Mary and Bob.

If the 1% is too small to believe in,
then we accept (B) and pop a Champaign
cork for the better teacher.

Here we made no assumptions at all about the
probability distributions of the scores.
So, our hypothesis test does not assume
a distribution and is distribution free
or, as is sometimes said, non-parametric
(because we did not assume a distribution,
say, Gaussian with parameters, e.g.,
mean and variance).

Look, Ma, no standard deviations!

The 1% is he significance level of our
hypothesis test and is the probability of
rejecting the null hypothesis when it is
true and, thus, is the probability of Type I
error.

Oh, consider a large server farm or network.
Suppose we identify 10,000 systems we want
to monitor for health and wellness and detect
problems never seen before. Suppose from
one of the 10,000 systems, we consider one.
Suppose from this system we receive data
100 times a second on each of 12 numerical
variables.

Suppose the server farm is supposed to be fairly
stable and we collect such data for, say,
3 months. Call this history data. Maybe
the machine learning people would call this
training data. Whatever.

Now we can construct a 12 dimensional, distribution-free
hypothesis test where the null hypothesis is
that the system is healthy and also select
our false alarm rate (probability of Type I error)
in small steps over a wide range. So, we have
a multi-dimensional, distribution-free hypothesis
test. Such are rare, but, really now we have
a large class of them. Yes, we use a group of measure preserving
transformations.

As I recall, back in 1999 there was a paper on
such things in Information Sciences.
I wouldn't call that paper machine learning,
but maybe some people would. Some of what is
interesting in the paper is how the heck to
know and adjust the false alarm rate, that is,
the probability of Type I error.

Again, look, Ma, no standard deviations or
Gaussian assumptions.