Paper:
Real-world machine learning applications often have complex test metrics, and may have training and test data that follow different distributions. We propose addressing these issues by using a weighted loss function with a standard convex loss, but with weights on the training examples that are learned to optimize the test metric of interest on the validation set. These metric-optimized example weights can be learned for any test metric, including black box losses and customized metrics for specific applications. We illustrate the performance of our proposal with public benchmark datasets and real-world applications with domain shift and custom loss functions that balance multiple objectives, impose fairness policies, and are non-convex and non-decomposable.

Review:
This proposal learns training data weights that optimize any given test metric.
They do this by learning both a weighting function and a classifier. They 
iteratively train an ensemble of K {weighting_k,classifier} pairs, and
selecting the pair that presents the best best metric-of-interest value over
the test set (over all iterations) as the final output.
The paper provides a large set of references that I found useful, and is
clearly written.
Each training iteration of their MOEW algorithm first optimizes classifiers 
over all training examples, for each of the K sample weightings.  Each
weighting gives rise to one converged classifier, whose metric-of-interest is
evaluated on the validation set.  Given K sets of weighting functions and their
metric-of-interest values, new parameters for the weighting functions are
generated.
The weighting function choice is a simple function, based on a linear transform
of autoencoder features, a normalization factor, and factor to account for
differing label frequencies in training and test data.  The low-dimensional
auto-encoding of {data,label} pairs is trained once, using the training data.
The weighting function parameters are a linear transformation matrix. They
also have a sigmoid non-linearity in their weighting function.
If I understand correctly, their approach seems to generalize the importance
sampling methods that they reference by using a nonlinear combination of autoencoder
features.  They reference importance sampling methods that, for example, use 
Gaussian kernel basis functions to model training/test example densities.
While they do provide extensive references to previous approaches, I would have
enjoyed a clearer explanation of main differences between their weighting
function and ones that have previously been used in importance sampling.
Their main contribution seems to be the procedure (Alg. 2) used to updated the
weighting function parameters.  They first fit a model that predicts
metric-of-interest values given all previous weighting function parameters used
in the algorithm.  Then they use this model to generate the next set of
weighting function parameters.  Their method adapts a Gaussian Process
Upper-Confidence Bound to batch-wise processing.  Within-batch and between
batch exploration of the weighting function parameters is controlled by two
parameters, which they held fixed at values corresponding to +/-1 for a normal
distribution.
Only the validation set need be iid with the test set, so their method seems
quite general.
Their MNIST results use very simple networks to show that learning weighting
functions has the most benefit when classifier networks are severely
under-parameterized.  Their wine price example addresses the choice of
embedding dimension, and they found their error metric decreased from 52% with
uniform weighting, to around 46% as they approached ~10 dimensions in the
autoencoded {data,label} representation of the training data.  They then use a
small crime dataset with a complicated test metric measuring *fairness*, based
on dividing the dataset into 4 quantiles based on white population. Using
MOEW they could improve the fairness metric with little effect on the accuracy
metric.  They could also preset thresholds to achieve very good fairness on the
training data and use MOEW to maximize the accuracy metric, which resulting
in improving both fairness and accuracy compared to uniform sample weighting.
For spam blocking and web page quality, I would like to see a little more 
interpretation of their results. They again show improvements using MOEW to
provide weights for training data.  This time, Table 2 presents a comparison
with importance sampling, but the methodology they used for importance sampling
is not well described.  I'd like to understand where this improvement came
from. What model/procedure was used for importance weighting? Gaussian rbfs?  
Do the authors attribute the dramatic improvement for MOEW for Spam Blocking 
simply to having a more flexible sample distribution function?  Or is it mainly
due to their adapting to the test metric?  Then for Web Page Quality, would it
be correct to conclude that the old and new web pages (training vs
validation/test) are actually fairly similary distributed?
What values of B and K are reasonable values in Alg. 1?  When applied to larger
problems, where do the authors feel the bottlenecks in Algs 1 and 2 will lie?
Do the authors find that retuning the classifier for different weighted samples
to take a lot of time?
Pros:
 - tests on several datasets
 - seems fairly generally applicable.
 
Cons:
 - To reproduce, I guess I'd need to adapt existing GP-UCB code from python or
   C++, and I'm not sure how easy this would be. Releasing code to reproduce
   the results might be nice.
 - Better understanding of differences wrt. typical Importance Sampling methods
   would nice.