On Log-Loss Scores and (No) Privacy (2020)

Abstract: A common metric for assessing the performance of binary classifiers is the Log-Loss score, which is a real number indicating the cross entropy distance between the predicted distribution over the labels and the true distribution (a point distribution defined by the ground truth labels). In this paper, we show that a malicious modeler, upon obtaining access to the Log-Loss scores on its predictions, can exploit this information to infer all the ground truth labels of arbitrary test datasets with full accuracy. We provide an efficient algorithm to perform this inference. A particularly interesting application where this attack can be exploited is to breach privacy in the setting of Membership Inference Attacks. These attacks exploit the vulnerabilities of exposing models trained on customer data to queries made by an adversary. Privacy auditing tools for measuring leakage from sensitive datasets assess the total privacy leakage based on the adversary’s predictions for datapoint membership. An instance of the proposed attack can hence, cause complete membership privacy breach, obviating any attack model training or access to side knowledge with the adversary. Moreover, our algorithm is agnostic to the model under attack and hence, enables perfect membership inference even for models that do not memorize or overfit. In particular, our observations provide insight into the extent of information leakage from statistical aggregates and how they can be exploited.

## introduction protecting customer privacy is of fundamental importance when training ml models on sensitive customer data. while explicit data de-identification and anonymization mechanisms can help protect privacy leakage to some extent, research has shown that this leakage can happen when models trained on customer data can be queried by an external entity @xcite @xcite , or when statistical aggregates on the dataset are exposed @xcite . recently, it was shown that the knowledge of log-loss scores leaks information about true labels of test datapoints under some constraints on the prior knowledge on these labels @xcite . however, extracting meaningful information from these aggregates on arbitrary large datasets, while maintaining reasonable inference accuracy in a limited number of queries to a log-loss oracle remained an open problem, specially in cases when no prior knowledge is available. moreover, the number of queries required by their algorithm scales with the size of the test dataset. we address this problem in this paper and provide multiple algorithms for optimal inference of arbitrarily many test labels in a single query using the exposed log-loss scores. this sheds insight into the extent of information leakage from this statistical aggregate and how it can be exploited to game a classification task, for example, in the context of data-mining competitions like kaggle, kddcup and ilsvrc challenge @xcite . more concretely, consider the following scenario: you are tasked with a critical binary classification problem. the quality of your solution will be assessed through a performance score (log-loss) on an unknown test dataset. if you score the highest among all candidate solutions, then you win a significant cash prize. you are allowed only two attempts at the solution and the best of the two scores will be considered. is it possible to game this system in a way that your score is always the highest amongst all candidates, without even training any classifier? we answer this in the affirmative by showing that the knowledge of only the size of the test dataset is enough to construct a scheme that can game any binary classifier that uses the log-loss metric to assess the quality of classification. this scheme is completely agnostic of the underlying classification task and hence, sheds light on how a malicious modeller can fake a perfect classifier by demonstrating zero test error. we assume that the oracle reports the scores truthfully on the entire dataset. a particularly interesting application of our observation is for breaching membership privacy, where an attacker can query the model for inference on a set of datapoints and use these responses to infer what datapoints were used to train that model. given blackbox access to a model and a data point x, this attack model is a binary classifier to infer the membership of x in the training dataset of the target model using its output on xthe more information this output reveals, the better this inference can be performed. consequently, the accuracy of the attack depends on how well the adversary can capture the difference in model performance. nonetheless, the popularity of this attack has made it a strong candidate for assessing privacy leakage of models trained on datasets containing sensitive information @xcite @xcite @xcite . a successful attack can compromise the privacy of the users that contribute to the training dataset. our results show that an oracle access to log-loss scores (for example, when using privacy auditors on sensitive datasets) enables full privacy breach in a single query foot_0 . related work in recent work, @xcite demonstrated how an attacker can estimate the test set labels in a competition setting with probability 2/3. similarly, and more related to our work, @xcite showed how the knowledge of auc and log-loss scores can be used to make inference on similar test sets by issuing multiple queries for these statistics. our work extends the latter to optimize the number of queries. similarly, through a monte carlo algorithm, @xcite show how knowing most of the test labels can help estimate the remaining labels upon gaining access to an empirical roc curve. however, their algorithm is far from exact inference with no apriori information of the true labels. we further observe that the theme of our work is related to two fields of research: adaptive data analysis, and protections of statistical aggregates using differential privacy (dp). in adaptive data analysis @xcite , an attacker leverages multiple (adaptive) queries to sequentially construct a complete exploit (e.g., of a test set). conversely, with dp @xcite the objective is to protect the aggregate statistics, such as those exploited by @xcite , from leaking information. log-loss metric we begin with reminding the reader of the definition of the log-loss metric on a given prediction vector with respect to a binary labeling of the datapoints in the test dataset. the definition easily generalizes for multi-class classifiers. a common variant is to ignore the normalization by |d|. our constructions in this paper are scale-invariant. ## algorithms for exact inference using log-loss scores in this section, we discuss multiple algorithms for single shot inference of all ground-truth labels using carefully constructed prediction vectors that help establish a 1-1 correspondence of the log-loss scores with the labelings of the test dataset. we, therefore, refer to the entity that performs such an inference as an adversary. ## adapting to fixed precision arithmetic can we design prediction vectors such that the log-loss scores are atleast some ∆ apart from each other, where ∆ is limited by the floating point precision on the machine used to simulate our inference algorithms? for distinguishing scores with φ significant digits, since there are a total of 10 φ possible numeric values, the threshold value of separation is ∆ ≥ 10 -φ . if the separation in the scores is smaller than this value, then they cannot be distinguished. inverting this inequality gives φ ≥ log 10 1 ∆ . for example, if one wishes to have the scores separated by ∆ ≥ 0.2, then the minimum amount of precision required is log 10 5 = 1. for ∆ = 0.002, we would need φ ≥ log 10 500 = 3 digits. we can reduce the requirement of a large precision by combining the auc and log-loss scores, which is common in most practical situations where multiple performance metrics are evaluated to give a holistic overview of classifier inference. this way, even if they are individually not-unique but the tuple is unique for each labeling, exact inference can be done. for example, consider a dataset @xmath0, d 2 , d 3 ] and the prediction vector @xmath1.2, 0.4, 0.6]. clearly, neither the auc scores nor the log-loss scores are unique. however, if we consider the two scores together, the labels can be uniquely identified. moreover, precision of only two significant digits is enough to make this decision. a rough analysis tells us that with φ significant digits, there are 10 φ (10 φ + 1) possible unique values in the auc-log-loss tuple (the +1 is to take into account the case when auc is not-defined). using the pigeonhole principle, for any dataset d with |d| = n, a necessary condition for unique inference is that 2 @xmath2, which gives φ 0.151n . conversely, with a precision of φ significant digits, one can only hope to uniquely identify labels for datasets of size at most log 2 10 φ (10 φ + 1) ≤ 7φ . we can recurse over the remaining points in the database in this situation, for exact inference in at most n/6φ . for example, using the ieee 754 double-precision binary floating-point format, which has at least 15 digits precision, at most |d| 90 queries suffice. in particular, 2 queries suffice for all datasets containing fewer than 180 datapoints. ## exact membership inference attacks using a log-loss oracle our observations provide insight into the extent of information leakage from statistical aggregates and how they can be exploited. a particularly interesting application is for designing stronger membership inference attacks. these attacks were first proposed to exploit the vulnerabilities of exposing models trained on customer data to queries by an adversary @xcite , and later proposed to assess (1) data provenance by training language models on sensitive datasets @xcite and (2) leakage from model outputs @xcite . such auditing techniques are becoming increasingly popular to quantify privacy leakage and because the underlying attack model is a binary classifier, aggregates like the log-loss scores will indicate the total privacy leakage based on the adversary's predictions. the threat model we are interested in is the case when these aggregates are exposed to a mali-cious entity (adversary), in which case our analysis demonstrates that this additional information enables the adversary to craft malicious prediction vectors. the response to this query helps infer exactly which datapoints were used for training. note that the adversary never queries the model under attack directly for prediction on any datapoints whatsoever. moreover, the interaction with the model curator is similar to the interaction with the model interface in the attack proposed by @xcite in that the adversary seeks answers to queries that can help leak information about the training data. the only difference is the additional access to a log-loss oracle, which helps make our attack purely deterministic. ## conclusion and future work in this paper, we demonstrated how a single log-loss query can enable exact inference of groundtruth labels of any number of test datapoints. this highlights the extent of information leakage from statistical aggregates, and in particular, that improperly constructed auditing tools may leak more information about the dataset than the leaky model they seek to expose. we see our work as a caution for such tools against reporting sensitive metrics. an interesting question to ask is if other popular metrics (like precision, recall, auc) used in the ml literature can be exploited for privacy leakage in a similar manner. in @xcite , an auc-roc oracle on the test dataset is used to deduce the true labels in at most 2|d| queries. this opens up opportunities to explore if exact inference on all datapoints is possible with one auc query. yet another interesting question to ask is if exact inference is possible when the adversary learns only a bound on or an approximate value in each log-loss query it issues. observe that by deciding the size of the test dataset, the adversary also fixes the number of possible values log-loss scores can take. in a typical scenario where the adversary has some prior knowledge about the amount by which the reported score differs from the actual value (see @xcite for an approach to add noise to the reported scores), this discrete set of possible scores can present a huge advantagethe adversary can perform inference over the most likely score under the constraint above. nonetheless, a distribution over the labelings for the test set can be learnt to bound the inference error.