  Privacy-Preserving Machine Learning algorithms must balance classification
accuracy with data privacy. This can be done using a combination of
cryptographic and machine learning tools such as Convolutional Neural Networks
(CNN). CNNs typically consist of two types of operations: a convolutional or
linear layer, followed by a non-linear function such as ReLU. Each of these
types can be implemented efficiently using a different cryptographic tool. But
these tools require different representations and switching between them is
time-consuming and expensive. Recent research suggests that ReLU is responsible
for most of the communication bandwidth. ReLU is usually applied at each pixel
(or activation) location, which is quite expensive. We propose to share ReLU
operations. Specifically, the ReLU decision of one activation can be used by
others, and we explore different ways to group activations and different ways
to determine the ReLU for such a group of activations. Experiments on several
datasets reveal that we can cut the number of ReLU operations by up to three
orders of magnitude and, as a result, cut the communication bandwidth by more
than 50%.
