  Data holders, such as mobile apps, hospitals and banks, are capable of
training machine learning (ML) models and enjoy many intelligence services. To
benefit more individuals lacking data and models, a convenient approach is
needed which enables the trained models from various sources for prediction
serving, but it has yet to truly take off considering three issues: (i)
incentivizing prediction truthfulness; (ii) boosting prediction accuracy; (iii)
protecting model privacy.
  We design FedServing, a federated prediction serving framework, achieving the
three issues. First, we customize an incentive mechanism based on Bayesian game
theory which ensures that joining providers at a Bayesian Nash Equilibrium will
provide truthful (not meaningless) predictions. Second, working jointly with
the incentive mechanism, we employ truth discovery algorithms to aggregate
truthful but possibly inaccurate predictions for boosting prediction accuracy.
Third, providers can locally deploy their models and their predictions are
securely aggregated inside TEEs. Attractively, our design supports popular
prediction formats, including top-1 label, ranked labels and posterior
probability. Besides, blockchain is employed as a complementary component to
enforce exchange fairness. By conducting extensive experiments, we validate the
expected properties of our design. We also empirically demonstrate that
FedServing reduces the risk of certain membership inference attack.
