  Data-driven intelligent applications in modern online services have become
ubiquitous. These applications are usually hosted in the untrusted cloud
computing infrastructure. This poses significant security risks since these
applications rely on applying machine learning algorithms on large datasets
which may contain private and sensitive information.
  To tackle this challenge, we designed secureTF, a distributed secure machine
learning framework based on Tensorflow for the untrusted cloud infrastructure.
secureTF is a generic platform to support unmodified TensorFlow applications,
while providing end-to-end security for the input data, ML model, and
application code. secureTF is built from ground-up based on the security
properties provided by Trusted Execution Environments (TEEs). However, it
extends the trust of a volatile memory region (or secure enclave) provided by
the single node TEE to secure a distributed infrastructure required for
supporting unmodified stateful machine learning applications running in the
cloud.
  The paper reports on our experiences about the system design choices and the
system deployment in production use-cases. We conclude with the lessons learned
based on the limitations of our commercially available platform, and discuss
open research problems for the future work.
