FAQ Part 2/2

Q: If the second, secret hash algorithm is based on a neural network, can we think of its weights (coefficients) as some kind of secret key in the cryptographical sense?

A: Absolutely not. If (as many suspect) the second hash algorithm is also based on some feature-identifying neural network, then we can't think of the weights as a key that (when kept secret) protects the confidentiality and integrity of the system.

Due to the way perceptual hashing algorithms work, having access to the outputs of the algorithm is sufficient to train a high-fidelity "clone" that allows you to generate perfect adversarial examples, even if the weights of the clone are completely different from the secret weights of the original network.

If you have access to both the inputs and the outputs, you can do much more: by choosing them carefully [4], you can eventually leak the actual secret weights of the network. Any of these attack can be executed by an Apple employee, even one who has no privileged access to the actual secret weights.

Even if you have proof positive that nobody could have accessed the secret weights directly, the entire key might have been leaked anyway! Thus, keeping the weights secret from unauthorized parties does not suffice to protect the confidentiality and integrity of the system, which means that we cannot think of the weights as a kind of secret key in the cryptographical sense.

Q: I heard that it's impossible to determine Apple's CSAM image hashes from the database on the device. Doesn't this make a hash attack impossible?

A: No. The scheme used by Apple (sketched in the technical summary [6]) ensures that the device doesn't _learn_ the result of the match purely from the interaction with server, and that the server doesn't learn information about images whose hash the server doesn't know. The claim that it's "impossible to determine Apple's CSAM image hashes from the database on the device" is a very misleading rephrasing of this, and not true.

Q: Doesn't Apple claim that there is only a one in one trillion chance per year of incorrectly flagging a given account?

A: Apple does claim this, but experts on photo analysis technologies have been calling bullshit [8] on their claim since day one.

Moreover, even if the claimed rate was reasonable (which it isn't), it was derived without adversarial assumptions, and using it is incredibly misleading in an adversarial context.

Let me explain through an example. Imagine that you play a game of craps against an online casino. The casino will throw a virtual six-sided die, secretly generated using Microsoft Excel's random number generator. Your job is to predict the result. If you manage to predict the result 100 times in a row, you win and the casino will pay you $1000000000000 (one trillion dollars). If you fail to predict the result of a throw, you lose and pay the casion $1 (one dollar).

In an ordinary, non-adversarial context, the probability that you win the game is much less than one in one trillion, so this game is very safe for the casino. But this number, one in one trillion, is based on naive assumptions that are completely meaningless in adversarial context. If your adversary has a decent knowledge of mathematics at the high school level, the serial correlation in Excel's generator comes into play, and the relevant probability is no longer one in one trillion. It's 1 in 216 instead! Whenfaced with a class of sophomore math majors, the casino will promptly go bankrupt.

Q: Aren't these attacks ultimately detectable? Wouldn't I be exonerated by the exculpatory evidence?

A: Maybe. IANAL. I wouldn't want to take that risk. While matching hashes are probably not sufficient to convict you, and possibly not sufficent to take you into custody, but it's more than sufficient to make you a suspect. Reasonable suspicion is enough to get a warrant, which means that your property may be searched, your computer equipment may be hauled away and subjected to forensic analysis, etc. It may be sufficient cause to separate you from your children. If you work with children, you'll be fired for sure. It'll take years to clear your name.

And if they do charge you, it will be in Apple's best interest not to admit to any faults in their algorithm, and to make it as opaque to the court as possible. The same goes for NCMEC.

Q: Why should I trust you? Where can I find out more?

A: You should not trust me. You definitely shouldn't trust the people defending Apple using the claims above. Read the EFF article [7] to learn more about the social dangers of this technology. Consult Apple's Threat Model Summary [5], and the CSAM Detection Technical Summary [6]: these are biased sources, but they provide sketches of the algorithms and the key factors that influenced the current implementation. Read HackerFactor [8] for an independent expert perspective about the credibility of Apple's claims. Judge for yourself.

[1] https://imgur.com/a/j40fMex

[2] https://graphicdesign.stackexchange.com/questions/106260/ima...

[3] https://arxiv.org/abs/1809.02861

[4] https://en.wikipedia.org/wiki/Chosen-plaintext_attack

[5] https://www.apple.com/child-safety/pdf/Security_Threat_Model...

[6] https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...

[7] https://www.eff.org/deeplinks/2021/08/apples-plan-think-diff...

[8] https://www.hackerfactor.com/blog/index.php?/archives/929-On...