> It's kind of telling you didn't understand Wasserman's reply to you, it was perfectly clear to any practitioner.

Condescension doesn't impress me. Explanations do. Could you please
explain his "perfectly clear" reply to me?

> What's your Bayesian approach to solving this?

Applying probability theory as best I could. If it turns out to be
too complicated, or computationally intractable, I'll resort to
approximations.

For this particular case… Well, first, we don't have 2 treatments,
we have as many treatments as we have objects. It's like flipping a
coin. When you flip a coin in the obvious way, you have no idea which
side will eventually come up, because even if you know it came up
heads the previous time, and you tried to perform the same move again:
in fact, you don't perform the same move, and you don't know how
exactly your second move differed from the first.

So first, there is an unpredictable variability in the way the
treatment will be performed. This is the first source of uncertainty.

Second, the objects aren't all the same. I expect them to be more or
less clean to begin with, and other characteristics such as sharp
angles may more or less hinder the treatment. This is the second
source of uncertainty.

Once you perform a treatment on an object, it had a definite effect.
Unfortunately, it is hard to define how well it actually went. Maybe
you can come up with a definite criterion, but apparently, since you
needed to ask a sample of human volunteers, that criterion is either
not well understood, or hard to measure properly. Anyway, you have
access to an uncertain measure, which in this case is a human giving
you a scale from 1 to 5. This is the third and fourth sources of
uncertainty (akin to the first and second one respectively:
variability in how a given human will assess an object; and
variability across humans).

From what I have understood of what you told me, there were 2 sets of
objects, and 2 groups of people. One group looked at every object
before any treatment. Then you applied OLD to half objects, and NEW
to half the other. You then showed the newly cleaned up objects to
the second group. Of course, you don't tell the group which objects
they get to examine. An alternative would be to photograph every
objects before and after the treatment, then show both sets of photos
to everyone, and ask them to rank the objects (both before and after),
and the perceived efficiency of the treatment. I don't like it much
however, because photos add an additional source of uncertainty.

This looks like a complicated version of a compound estimation
problem, described in chapter 6 of Jayne's PT:TLOS (elementary
parameter estimation). Pretty basic stuff.

We're not finished yet. You wanted to know which treatment should be
chosen: OLD or NEW. You basically have 3 alternatives: always apply
OLD, always apply NEW, or look at the object then decide which
treatment you apply. For each alternative, you should have a
probability distribution over the "cleanness" distribution of the
objects. Decide a utility function for the cleanness distribution,
and chose the method that maximizes expected utility.

And now I'm stuck because I don't know probability theory deeply
enough to actually give you a non-ambiguous procedure in less than a
couple days. Not to mention that I don't have your prior information
about OLD and NEW (why you expect OLD to be better etc.)
Anyway, I believe a specialist would only need a couple minutes, or a
couple hours tops. (To find the procedure, that is. Actually running
the numbers in a computer may or may not be expensive. I personally
have no idea.)