L EARNING TO G IVE C HECKABLE A NSWERS WITH P ROVER -V ERIFIER G AMES
Anonymous authors
Paper under double-blind review
A BSTRACT
Our ability to know when to trust the decisions made by machine learning systems has not kept up with the staggering improvements in their performance, limiting their applicability in high-stakes domains.
We introduce Prover-Veriﬁer Games (PVGs), a game-theoretic framework to encourage learning agents to solve decision problems in a veriﬁable manner.
The PVG consists of two learners with competing objectives: a trusted veriﬁer network tries to choose the correct answer, and a more powerful but untrusted prover network attempts to persuade the veriﬁer of a particular answer, regardless of its correctness.
The goal is for a reliable justiﬁcation protocol to emerge from this game.
We analyze variants of the framework, including simultaneous and sequential games, and narrow the space down to a subset of games which provably have the desired equilibria.
To complement our novel learning problem, we propose rigorous methodology for evaluating the robustness of the learned protocol.
We develop instantiations of the PVG for two algorithmic tasks, and provide empirical evidence that the veriﬁer can in fact learns a robust decision rule that is able to receive useful and reliable information from an untrusted prover.
Importantly, the protocol still works even when the veriﬁer is frozen and
the prover’s messages are directly optimized to convince the veriﬁer.
I NTRODUCTION
The astonishing performance of today’s dominant learning paradigm – optimizing powerful differentiable function approximators to minimize suitable loss functions – often comes at the cost of poor robustness and reliability.
It is common for powerful deep learning systems to be vulnerable to adversarial attacks (Goodfellow et al., 2015), to display erratic behaviour on out-of-distribution data and be very conﬁdent of wrong predictions (Che et al., 2021).
How can we train our learning algorithms to produce outputs that can be checked by humans or by learning algorithms that are cheaper or better understood?
Over the past decades, the ﬁeld of computational complexity has greatly expanded our notion of proof to include formalisms such as interactive, zero-knowledge, and probabilistically checkable proofs (Goldreich, 2008).
Most of these notions can be thought of in terms of a game between a powerful but untrusted prover and a computationally limited but trusted veriﬁer.
Taking inspiration from this, we propose the Prover-Veriﬁer Game (PVG), where two learning agents play the role of prover and veriﬁer, and are allowed to converse.
The veriﬁer aims to determine the correct answer to a decision problem, and the prover aims to convince the veriﬁer of a particular answer (regardless of its correctness).
Since the prover is untrustworthy, the veriﬁer will only ﬁnd its messages useful to the extent that it can independently verify the information.
If all goes well, then the game dynamics will lead to a proof protocol which, if not mathematically sound, is at least sufﬁcient for the whole system to achieve more reliable predictions than the veriﬁer can achieve unaided.
We analyze the desirability of the game equilibria implied by several variants of the Prover-Veriﬁer
Game concept and narrow the space down to a subset of games that theoretically have the desired equilibria.
Picking the right game equilibrium concept turns out to be essential: we prove that formulating the PVG as a sequential game in which the prover agent plays ﬁrst leads to dysfunctional solutions.
On the other hand, the simultaneous (connected to Nash equilibria) and veriﬁer-ﬁrst sequential formulations (connected to veriﬁer-leading Stackelberg equilibria) have desirable equilibria.
We formally show, on a illustrative problem, that gradient based differentiable game optimizers can ﬁnd desirable proof-veriﬁcation protocols.
Do reliable justiﬁcation protocols emerge in practice if we let artiﬁcial agents play the Prover-Veriﬁer
Game?
To complement our novel game formulation, we develop a rigorous evaluation methodology whereby the veriﬁer is frozen and the prover (or the message) is optimized to convince the veriﬁer.
We then run simulations on two algorithmic tasks using a practical instantiation of the PVG concept.
As predicted by our theory, PVG-trained veriﬁers learn to receive useful and reliable information from untrusted provers by following sensible veriﬁcation protocols, whereas those trained alongside fully collaborative provers are easily deceived.
B ACKGROUND 2. I NTERACTIVE P ROOF S YSTEMS (IPS)
Interactive proof systems generalize the notion of a mathematical proof to a dialogue between two agents - a prover and a veriﬁer - to solve a decision problem (Arora & Barak, 2009; Thaler, 2019).
The veriﬁer agent, who is trustworthy but computationally constrained, is tasked with producing a correct answer to the decision problem.
It can exchange messages with a computationally unbounded, yet potentially adversarial prover agent.
The communication protocol between the prover and veriﬁer constitutes an interactive proof system if and only if it is sound and complete:
Deﬁnition 1 (Completeness and Soundness) .
The veriﬁer of an interactive proof system is complete iff there exists a prover that can always convince the veriﬁer that the answer is “yes", if the correct answer is “yes".
It is sound iff there doesn’t exist a prover that can trick the veriﬁer into answering “yes" if the correct answer is “no".
2. D IFFERENTIABLE G AME O PTIMIZATION
A two-player differentiable game consists of two agents whose strategies are parametrized by w = ( w 1 , w 2 ) 2 R d that take turns to minimize their differentiable loss functions ( L 1 , L 2 ) : R d ! R .
An equilibrium concept determines which strategies will be adopted by the players.
A Nash equilibrium is achieved when no player can unilaterally improve its objective function.
Deﬁnition 2 (Nash Equilibrium (Von Neumann & Morgenstern, 2007)) .
The strategies parametrized by ( w ⇤ 1 , w ⇤ 2 ) constitute a Nash equilibrium 1 of the two-player sequential differentiable game with loss functions ( L 1 , L 2 ) :
R d ! R if they minimize their loss functions keeping the other player’s parameters ﬁxed:
[Equation]
The notion of equilibrium considered by Generative Adversarial Networks (Goodfellow et al., 2014) is an example of a Nash Equilibrium.
A Stackelberg equilibrium differs from the Nash equilibrium in that one of the players is deemed the “leader" and the other the “follower" (Wang* et al., 2020).
It is assumed that the follower always picks the optimal strategy for a given leader strategy.
In response, the leader modiﬁes its strategy by factoring in how the follower agent will respond to the modiﬁcation.
Deﬁnition 3 (Stackelberg Equilibrium (Fiez et al., 2020)) .
Let w 1 parametrize the strategy of the “leader agent" and w 2 parametrize that of the “follower agent".
The loss functions of the agents are Land L 2 respectively.
The strategies parametrized by ( w 1 , w 2 ) constitute a Stackelberg equilibriumif and only if (1) the follower’s strategy is optimal given the leader’s strategy, and (2) the leader’s strategy is optimal taking into consideration how the follower will respond to modiﬁcations.
[Equation]
P ROVER -V ERIFIER G AME
The Prover-Veriﬁer Game aims to learn decision rules with a reliable internal veriﬁcation step.
We describe what we mean for a veriﬁcation protocol to be reliable, then outline the game formulation.
3. D ESIDERATA AND P ROVER -V ERIFIER I NCENTIVES
Desiderata: Designing the right prover-veriﬁer game requires precisely deﬁning what a desirable outcome/equilibrium of the game looks like.
With the “completeness" and “soundness" deﬁnitions in mind (Section 2.1), we list the properties we seek in a desirable veriﬁer protocol:
[Figure]
• Possibility of Perfect Recall: There should exist a prover that can help the veriﬁer achieve perfect recall — the ratio of true positive predictions to all positive examples.
• Guarantee of Perfect Precision: There shouldn’t exist any prover which can trick the veriﬁer into achieving non-perfect precision — the ratio of true positive predictions to all positive predictions.
“Possibility of perfect recall" is connected to completeness, and implies that with the right proofs, the veriﬁer can achieve zero false negative rate.
“Guarantee of perfect precision" is related to soundness and implies that the veriﬁer always has zero false positive rate regardless of which proof is used.
Picking the right prover-veriﬁer incentives: We propose to set up the prover-veriﬁer game with the following incentive structure: the veriﬁer is incentivized to always give the correct answer to the given decision problem, and the prover is incentivized to get the veriﬁer to answer “yes" to the decision problem, regardless of the correct answer.
This structure encourages collaborative dynamics when the correct answer to the decision problem is “yes" (which is linked to recall) and encourages adversarial dynamics if the correct answer is “no" (which is linked to precision).
As we will prove formally in Section 3.3, this incentive structure, when embedded in the right game setup with the right loss functions, can lead to a proof-veriﬁcation protocol that meets the desiderata.
3. D IFFERENT P ROVER -V ERIFIER G AME F ORMULATIONS
Notation Let x ⇠ p X ( x ) , y ⇠ p Y ( y ) and y {+
+}
⇠ p Y 0 ( y 0 ) be the input, veriﬁer label and prover label random variables where we have x 2 R n x , y 2 { 0 , 1 , ..., K } and y 0 2 { 0 , 1 , ..., K } .
Let their joint distribution be p D .
The prover agent P w p , parametrized by w p 2 R n p , outputs the conditional probability distribution p p ( z | x ) over real valued “message" vectors z 2 R n z conditioned on inputs.
The veriﬁer agent V w v , parametrized by w v 2 R n v , represents the conditional probability distribution p v (ˆ y | x, z ) over the predicted labels ˆ y 2 { 0 , 1 , ..., K } , conditioned on the inputs x and messages z .
Loss Functions: We pick the following loss functions for the veriﬁer and prover:
[Equation]
Since we wish for the veriﬁer agent to try and solve the problem to the best of its ability, we set p Y to be the correct label distribution.
Since we wish to incentivize the prover to try and defend a particular answer regardless of its correctness, we set p Y {+
+}
( y 0 )
= 1 where y 0 2 { 0 , 1 , . .
, K } is the output that the prover is defending.
In a decision problem, y 0 will be one of 0 or 1 .
One can consider variations on these loss functions by replacing the negative log with other functions that are convex and monotonically increasing.
We prefer the aforementioned loss functions when the veriﬁer agent represents a softmax policy over its outputs.
This guarantees that the gradients vanish iff the agents fulﬁll their incentives (see Appendix A).
Suitable Equilibrium concepts: There are three obvious ways to set up the order in which the agents play (i.e. pick their strategies parametrized by w p and w v ) the Prover-Veriﬁer Game: 1) The prover and veriﬁer play simultaneously, 2)
The prover plays ﬁrst, 3)
The veriﬁer plays ﬁrst.
The simultaneous setup leads to picking Nash equilibrium as the equilibrium concept.
The latter two lead to picking Stackelberg equilibrium, where prover or veriﬁer is the leader and can therefore reason about how the opponent will respond to a given strategy.
Interestingly, as we show in Section 3.3, not all of these formulations lead to equilibria that satisfy our desiderata.
When Instances are Revealed: We can also arrive at different game formulation depending on whether the problem instances (i.e., the input and prover-veriﬁer labels) are revealed to the agents before or after they pick their strategies (i.e. weights).
This leads to eight different game formulations: two from the simultaneous setup (before or after the simultaneous moves) and six from the sequential setup (before, in-between and after the moves, for each of the two sequential formulations).
How the Prover and Veriﬁer Interact: The design space of how the prover and veriﬁer interact with each other is large.
We only consider the single-step, no feedback formulation throughout the paper and leave the analysis of other formulations as future work.
Note that the single-step, no feedback formulation is related to NP proof systems and include sophisticated proof-veriﬁcation protocols.
We can also derive numerous PVG formulations by modifying the communication channel between the prover and veriﬁer.
Various decisions regarding the channel include: 1) If the channel is stochastic or deterministic (and what kind of noise there is) 2) Whether prover’s messages are real valued vectors, or discrete tokens 3)
Whether the prover and veriﬁer use an already constructed communication protocol, such as natural language.
3. PVG F ORMULATIONS WITH D ESIRABLE E QUILIBRIA
We list in Table 1 all possible game instantiations obtained by varying the 1) player order and 2) when the problem instance is revealed.
Our goal is to ﬁnd which – if any – of these formulations have desirable equilibria.
We consider the following abstract decision problem formulation to compare and contrast different PVG formulations:
Deﬁnition 4 (Abstract Decision Problem) .
Let X , M , Y = { 0 , 1 } and ˜ Y
= { 0 , 1 } be the (discrete) sets of all inputs, prover messages, and labels, respectively.
Let P X represent the input distribution.
Assume the label is a deterministic function of the input, y = f
( x ) , and the prover is always arguing for the label y {+
+}
= 1 .
The prover agent can choose a policy p 2 P , where P is a set of mappings
X ! M .
Let V denote a set of base veriﬁers, which are mappings X ⇥ M ! { 0 , 1 } .
The veriﬁer chooses a policy v which computes a convex combination of elements of V .
(The veriﬁer’s output can be interpreted as the probability it assigns to the label being 1 ).
The veriﬁer and prover try to minimize the loss functions deﬁned in Equations 3 and 4 respectively.
We assume that both P and V include every constant mapping (i.e. ones which don’t depend on the input).
Assume that there exists a deterministic veriﬁer policy v ⇤ 2 V such that:
[Equation]
[Equation]
For instance, to represent a decision problem from complexity theory, we could take X and M to be sets of binary strings, P to be the set of all mappings X !
M (the prover is typically computationally unrestricted), and V the set of mappings implementable by boolean circuits with some maximum size.
However, we keep the formalism general because our main results don’t depend on the details of boolean circuits or complexity theoretic assumptions.
The conclusions of our theoretical analysis are displayed in Table 1.
Proper proof-veriﬁcation protocols constitute equilibria of only three of the eight game formulations: the simultaneous setup where the problem instance is presented after the agents’ moves, and the two veriﬁer-ﬁrst sequential setups where the problem instances are revealed after the veriﬁer’s move.
We justify these claims through the following theorems.
The proofs are presented in Appendix B.
[Table] [Table]
Proposition 1.
Any PVG formulation in which the veriﬁer is given the problem instance before it selects its strategy has bad equilibria — that is, having converged to an equilibrium is neither necessary nor sufﬁcient for having found a complete and sound proof veriﬁcation protocol.
If the instance x is known when the veriﬁer chooses its policy, then the veriﬁer can simply choose the constant policy which returns f ( x ) .
We dub this the “trivial veriﬁer" failure mode.
Theorem 1.
Consider all prover-leading sequential PVG formulations.
Having reached a (prover leading Stackelberg) equilibrium is neither necessary nor sufﬁcient for having found a complete and sound proof-veriﬁcation protocol.
The intuition behind this result is as follows: no matter what strategy the prover picks as the leader, the veriﬁer can always ﬁnd a strategy whose outputs match the label marginals (i.e. at least 50% accuracy if the labels are balanced).
Due to the algebraic form of the prover loss function, the best the prover can do is to avoid giving the veriﬁer any information about the information whatsoever about the label.
This defective incentive prevents the learning of proper proof-veriﬁcation protocols.
We dub this the “ﬂood-the-zone" failure mode, in reference to the media disinformation strategies that pollute the information ecosystem to prevent people from knowing what’s true Illing (2020).
Theorem 2.
Consider all veriﬁer-leading sequential PVG formulations in which the problem instance is revealed after the veriﬁer picks its strategy.
Having reached a (veriﬁer leading Stackelberg) equilibrium is both necessary and sufﬁcient for having found a desirable proof-veriﬁcation protocol.
Necessary is easy to argue: given a complete and sound protocol, neither player has an incentive to deviate.
Sufﬁciency can be shown using a “proof by contrapositive" argument: If the protocol represented by prover and veriﬁer is not complete and sound, then they cannot possibly represent a veriﬁer-leading Stackelberg equilibrium, because the veriﬁer can do strictly better by switching to the complete and sound veriﬁcation system (as the prover will react by producing correct proofs).
Theorem 3.
Consider the simultaneous PVG formulation where the problem instance is revealed after the agents pick their strategies.
Having converged to a (Nash) equilibrium is necessary for having found a complete and sound proof-veriﬁcation protocol.
If the prover and veriﬁer already represent a complete and sound proof-veriﬁcation protocol, the veriﬁer’s loss function is already minimized, hence the veriﬁer won’t change its strategy.
Because the prover can already convince the veriﬁer when it’s acting collaboratively (due to completeness) and cannot fool it when it’s acting adversarially (due to soundness), it also won’t have the incentive to change its strategy.
Therefore, the prover-veriﬁer strategies constitute Nash equilibria.
This also implies that all Stackelberg equilibria of the proof-veriﬁcation game are also Nash equilibria:
Corollary 1.
All veriﬁer-leading Stackelberg equilibria are Nash equilibria of PVG.
There might exist Nash equilibria that don’t correspond to complete and sound protocols.
For instance, the case in which the veriﬁer completely ignores the prover constitutes a Nash equilibrium.
We call this the “coordination problem" failure mode: no player can adopt a protocol unilaterally.
3. F INDING P ROOF -V ERIFICATION S TRATEGIES U SING G RADIENT -B ASED O PTIMIZERS
Section 3.3 establishes that there exist PVG formulations whose equilibria, if found, will constitute complete and sound proof-veriﬁcation protocols.
Is there hope in ﬁnding these equilibria using gradient based differentiable game optimizers?
We thereon use the phrase “Prover-Veriﬁer Training" to refer to the general method of using gradient-based optimizers to solve for PVG equilibria.
To address the question above, we turn to a simpler problem we call “Binary Erasure Channel" (BEC) which admits detailed analysis.
The BEC problem is deﬁned as follows:
Deﬁnition 5.
We have a prover-veriﬁer system.
The prover takes in input x 2 { 0 , 1 } and attempts to convince the veriﬁer to output 1 all the time whatever the input is.
The veriﬁer aims at predicting the input correctly only based on the message from the prover.
The prover deﬁnes a stochastic function f : { 0 , 1 } ! { 0 , 1 , ..., K } , M and is parameterized by probability vectors p
= [ p 00 , ..., p 0 K ] and p 1 =
[ p 10 , ..., p 1 K ] while the veriﬁer deﬁnes g : M ! { 0 , 1 } and is parameterized by q 2 R K +1 with q i representing the probability of predicting 1 when receives token i 2 M .
We impose some further restrictions: the prover cannot send the message 1 when x = 0 and cannot send 0 when x = 1 .
With this restriction, the communication channel between the prover and veriﬁer resemble the binary erasure channel Cover (1999) with the tokens { 2 , ..., K } representing multiple erasure tokens.
This setup is a simpliﬁcation of the more general assumption that there exists a complete and sound veriﬁcation protocol that is realizable by the prover and veriﬁer.
In this setting (given loss functions in Eq.
equation 3 and equation 4), the joint strategy of prover sending the right proof with p 11 = 1 and veriﬁer proceeding with q 1 = 1 but q i =
for all i 6 = 1 is a Nash equilibrium.
Moreover, we add entropy regularization (i.e., label smoothing) to the veriﬁer loss to avoid degenerate cases in training.
We can show that running standard gradient descent converges to the ideal joint strategy under the simultaneous game formulation.
In more detail, we have the following theorem.
Theorem 4.
Starting from any initialization with p 0 i > 0 , p 1 i > 0 for all i 2 M , running gradient descent on both prover and veriﬁer with reasonable learning rates, the prover would converge to only sending message 1 when x = 1 while the veriﬁer would predict 1 only if it receives message 1 from the prover.
To put it differently, p 11 = 1 when converge while q 1 = 1  ✏ and q 0 = q
= ...
=
q K = ✏ with ✏ a small constant depending on the strength of entropy regularization.
Formulating the PVG as a sequential game in which the prover agent plays ﬁrst leads to undesirable solutions.
We summarize the the result below.
Theorem 5.
Starting from any initialization with p 0 i > 0 , p 1 i > 0 for all i 2 M , running gradient descent on both prover and veriﬁer with any learning rates, prover would converge to only sending message 2 to K no matter what the input is.
R ELATED W ORK
Computational Complexity: Much of computational complexity theory is centered around various notions of proofs, from the basic deﬁnition of NP to more modern notions of proof such as interactive, zero-knowledge, and probabilistically checkable proofs (Goldreich, 2008).
These various notions of proof can be understood in terms of a game between a prover and veriﬁer somewhat analogous to ours.
Our game is most directly analogous to the vanilla notion of proof used to deﬁne NP.
But given the right architecture, our game would allow the prover’s message to encode the weights of a network that answers the veriﬁer’s questions, thereby more closely resembling an interactive proof.
Interpretability: Selective rationalization is the task of highlighting input features that are useful for classiﬁcation.
A number of game theoretic selective rationalization methods have been proposed (Lei et al., 2016; Yu et al., 2019; Chang et al., 2019).
With text applications in mind, Lei et al.
(2016) propose a two player, collaborative game where a generator network selects relevant features from the input text, and a predictor network does classiﬁcation using those features.
Chang et al.
(2019) propose a six player game where the 4 generators select factual or counterfactual rationales, and thediscriminators aim to distinguish factual and counterfactual rationales.
In practice, only 3 players are instantiated (two generators that only take negative or positive examples and a discriminator).
Game theoretic approaches have also been used for interpretable fact-checking on knowledge graphs.
Hildebrandt et al.
(2020) propose solving the triplet classiﬁcation task (deciding whether a subjectpredicate-object triple is contained in a knowledge graph) using debate dynamics.
Two reinforcement learning agents learn to generate arguments (paths in a knowledge graph) that either support or go against a fact (a triple).
These are then fed to a classiﬁer trained to ascertain the truth value of the fact.
Verifying correctness and functional properties of models: There’s a growing body of literature aimed at verifying both the correctness and/or functional properties of learned models.
Goldwasser et al.
(2021) investigate interactive proof systems to verify correctness.
They show cases where PACveriﬁcation of hypotheses are signiﬁcantly more sample efﬁcient, even when using a single-round protocol.
Works that aim to verify functional properties of models (such as robustness to adversarial perturbations) can be categorized under three groups (Kohli et al., 2019): those that test consistency with speciﬁcations (Ruderman et al., 2018; Uesato et al., 2018), those that train speciﬁcation consistent models (Raghunathan et al., 2018; Wong et al., 2018; Mirman et al., 2018; Dvijotham et al., 2018) and those that formally verify adherence to speciﬁcations (Katz et al., 2017; Dvijotham et al., 2018;
Singh et al., 2018).
These works are orthogonal - our notion of veriﬁcation differs from that of the software analysis/veriﬁcation community that is concerned with showing models/pieces of code meet speciﬁcations for all inputs - as we aim to learn proof-veriﬁcation protocols to solve a given task.
AI Safety: Being able to verify the correctness of the outputs produced by powerful but potentially adversarial or brittle learning systems is a top objective of the AI Safety community.
Irving et al.
(2018) propose a two player debate game where two agents communicate (space-constrained) statements to convince the judge of the correctness of opposing statements.
Our approach is different in that it uses only a single debater (prover).
We also differ signiﬁcantly in our focus: while Irving et al.
(2018) believed the presence of adversarial debaters would itself lend robustness, we are more interested in using the system to discover proof protocols.
We believe the latter philosophy captures the spirit of the adversarial debate game from the rationalist community (Barnes & Christiano, 2020), which is intended to determine which forms of (human) argumentation lead most reliably to true conclusions.
Che et al.
(2021) propose Deep Veriﬁer Networks, which uses a conditional variational autoencoder to detect “unreliable inputs or predictions.”
Unlike our approach, this approach hard-codes the structure of the veriﬁcation algorithm, and considers out-of-distribution detection applications.
S IMULATIONS 5. P RACTICAL D ESIGN D ECISIONS
We elaborate on some design decisions we made for running prover-veriﬁer training in practice.
Practical Constraints on the Veriﬁer: If the veriﬁer agent is instantiated using neural networks, then one can constrain the veriﬁer by varying the architecture and/or reducing the size and depth of the network.
While picking a suitable veriﬁer constraint on toy problems might be tricky, we argue that this no longer remains an issue if the decision problem is inherently difﬁcult to solve.
For example, it is easy to ﬁnd veriﬁer architecture that cannot solve problems that have large search spaces (such as subset sum, SAT problems, or problems in NP in general), yet can check the proofs.
Picking the Differentiable Game Optimizer: We use standard gradient based optimizers (such as
Adam (Kingma & Ba, 2014)) with alternating updates to ﬁnd the equilibria of the prover-veriﬁer game, since it was shown by Zhang et al. (2021) that simple alternating gradient descent-ascent is on par with more sophisticated algorithms for adversarial games.
We suspect that some advances in the optimization of generative adversarial networks could help, and leave this as future work.
Solving for Nash Equilibria: Since computing the response gradients required to solve for Stackelberg equilibria is often computationally challenging, we opt to solve for Nash equilibria in our simulations.
Since all Stackelberg equilibria of the proof-veriﬁcation game are also Nash equilibria, it is a viable strategy to ﬁnd a Nash equilibrium then check whether it also corresponds to a veriﬁerleading Stackelberg equilibrium.
Also, the convergence properties of the simultaneous setup are favourible, as suggested by Theorem 4.
Training Heuristics: We list heuristics that improve speed and stability of training in Appendix D.
5. E VALUATING S UCCESS
The aim of the PVG is to produce a proof system which is both sound and complete.
If this is the case, then we can expect the veriﬁer to achieve high accuracy.
However, the veriﬁer’s accuracy
[Table]
during training does not constitute a good yardstick for evaluating success, since a high accuracy for the veriﬁer could instead reﬂect problematic game dynamics.
For example, high accuracy can be explained by the veriﬁer detecting the prover’s “tell” (spurious features in prover messages that correlate with the labels) rather than checking the proof.
Also, we would like the proof protocol to generalize upwards to provers that are more capable than the one seen at training time.
Therefore, we propose to stress-test the robustness of the veriﬁer by freezing it and giving a lot more resources to the prover.
We did this in two different ways: First, we would freeze the veriﬁer and continue to train the prover.
Second, we would initialize messages to the prover’s output and directly optimize the messages (to maximize the veriﬁer’s acceptance logit) using powerful optimizers, such as L-BFGS.
Both of these techniques remove the veriﬁer’s ability to recognize the prover’s tell, and also provide a degree of conﬁdence in the robustness of the proof system.
(It remains possible that there exist incorrect proofs which the veriﬁer would accept but which are difﬁcult to ﬁnd.)
We report the precision and recall of the frozen veriﬁer against the two methods augmenting the prover.
The stress-tests outlined above can be seen as a way of checking if the veriﬁer represents a veriﬁerleading Stackelberg equilibrium.
Optimizing the prover corresponds to the setup where the problem instance is presented after agents pick their strategies.
Optimizing the prover messages links to the setup where the instance is known to the prover (see Table 1).
We emphasize that our proposed evaluation methodology is essential to measuring progress on this problem: early on in our investigations, we were fooled into thinking the networks had learned a correct protocol because the veriﬁer achieved high accuracy during training.
However, we later determined that this high accuracy simply reﬂected the veriﬁer learning the prover’s “tell” faster than the prover learned to hide it.
Since prior work aiming at similar goals to ours (Irving et al., 2018; Perez et al., 2019) reported only ﬁnal accuracy during training, we consider our frozen veriﬁer evaluation methodology an important contribution.
5. R ESULTS
Binary Erasure Channel Experiments: We used the binary erasure channel (BEC) task described in Section 3.4 Deﬁnition 5 as a test bed to check whether our theoretical results on the desirability of equilibria are matched in practice.
We parametrized the prover as a small fully connected network that takes one-hot-encoded input bits [1 , 0] or [0 , 1] as inputs and outputs logits over the K =communication tokens.
The veriﬁer was instantiated as a small network that takes in the one-hot encoded communication tokens as inputs and outputs logits that are used to compute the probability the veriﬁer assigns to the labels 0 and 1 .
We sampled the input bits uniformly, which means the labels were balanced.
We used the Adam optimizer (Kingma & Ba, 2014) with its PyTorch defaults (Paszke et al., 2019), batch size of 2000 and learning rate of 0 . 0003 .
We updated the veriﬁer 5 times for every prover update.
Details about training setup are described in Appendix E.1.
To assess completeness and soundness, we computed the frozen veriﬁer precision and recall against 1) a fully optimized prover and 2) fully L-BFGS optimized message vectors.
The details for how we optimized the prover and message vectors are given in the Appendix E.1.
We compared these results with those obtained using prover and veriﬁer agents trained fully collaboratively (i.e. prover’s labels always matches with the veriﬁers).
The results, shown in Table 2, reafﬁrm our hypothesis that the
PVG-trained frozen veriﬁer achieved perfect precision and recall - both against the optimized prover and optimized messages.
The collaboratively trained veriﬁer had zero precision, hence is not robust.
In Figure 2, we visualize how the outputs (i.e. logits) of the prover and veriﬁer networks for each different input evolve over time, accompanied by the accuracy of the veriﬁer.
We highlight - with bolded blue and orange curves - how the prover and veriﬁer treat the special “0" and “1" tokens.
The
[Table] [Table]
[Table]
veriﬁer quickly picks up on the receiving a “ indicative learns the
FindThePlus Experiments: We tested the efﬁcacy of prover-veriﬁer games on a visual search task we call “FindThePlus”.
As in
Figure 1, the dataset consists of binary 10 ⇥blue-red images that contain a single 3 ⇥plus pattern in them.
Labels are determined by the color of the plus: images with a red plus are assigned to class 1 and images with a blue plus are assigned to class 0.
The veriﬁer is constrained to only process a portion of the input image, so it depends on the prover to advise it “where to look” (advice which it is free to ignore).
We parametrized the prover as a convolutional net followed by a fully connected layer.
We parametrized the veriﬁer with a Spatial Transformer Networklike (Jaderberg et al., 2015) architecture with multiple heads.
The training details are provided in Appendix
E.2.
The frozen veriﬁer precision and recall corresponding to Prover-Veriﬁer training, as well as collaborative training are shown in
Table 3.
We optimized the prover and the messages using the same setup we used in the Binary Erasure Channel experiments.
As before, the veriﬁer trained under the proververiﬁer game dynamics achieves perfect precision against the optimized prover, while the veriﬁer trained under the standard, collaborative learning dynamics achieves exactlyprecision.
The frozen veriﬁer precision was 0 .
99 if we directly optimized the proof vectors.
We visualized where the Prover-Veriﬁer trained veriﬁer looks using the proof vectors from the prover.
An example is displayed in
Figure 1 (more in Appendix E.2).
These visualizations indicate that not only does the prover send the coordinate of the blue plus whenever it exists, it also sends the coordinates of maximally convincing patches if there’s not a blue plus in the image.
C ONCLUSION
We proposed Prover-Veriﬁer Games (PVG), a game-theoretic framework to encourage neural networks to solve decision problems in a veriﬁable manner.
We provided a thorough analysis of the problem space, and eliminated game formulations that have undesirable equilibria.
Using a rigorous evaluation framework, we demonstrated - both theoretically and empirically - that there exist PVG formulations that can learn robust proof-veriﬁcation protocols in practice.
A W HEN L OG -L OSS G RADIENTS V ANISH
The gradients of the log-loss (i e. the loss function used in Equations equation 3 and equation 4) vanish if and only if the agents fulﬁll their incentives (i e. classify their labels correctly).
Let ` = v ( x, p ( x ))
2 R d be the logits that the veriﬁer network outputs for input x and prover message p ( x ) .
Let S : R d 7! R d be the softmax operation with temperature 1 : [ S ( ` )]
i = e ` i
{+P dj =
 e `+} j .
We take
S ( ` ) i to be the probability the veriﬁer assigns for the correct label to be i .
Let y be the correct label.
The veriﬁer loss and its gradient with respect to the logits can be given by:
[Equation]
If the veriﬁer assigns a high probability to class y , then e ` y
P j e ` j will approach 1 and the gradients will vanish.
Otherwise, the gradient won’t vanish.
The derivation is identical for prover’s loss.
B P ROOFS OF E QUILIBRIUM T HEOREMS
Theorem 1.
Consider all prover-leading sequential PVG formulations.
Having reached a (prover leading Stackelberg) equilibrium is neither necessary nor sufﬁcient for having found a complete and sound proof-veriﬁcation protocol.
Proof.
It sufﬁces to prove that the prover-leading Stackelberg equilibrium is neither necessary nor sufﬁcient for having found a complete and sound proof-veriﬁcation protocol on the following decision problem:
Deﬁnition 6 (Simpliﬁed Binary Erasure Channel Problem) .
Let X
= Y = ˜
Y = { 0 , 1 } and
M = { 0 , 1 , 2 } .
Let the input distribution P X be the uniform distribution over X , and let the label generating function be deﬁned as y = f
( x ) .
That is, the label is identical to the input.
The prover is arguing for label y {+
+}
= 1 .
The prover can choose between two policies p 1 and p 2 :
[Equation]
The veriﬁer can pick between three policies v 1 , v 2 and v 3 deﬁned as follows:
[Equation]
Note that the mappings available to the veriﬁer ignore the input.
Also note that the prover-veriﬁer pair ( p 1 , v 1 ) satisfy Equations 5 and 6.
Lack of Necessity: To show lack of necessity, it sufﬁces to show that the complete and sound proververiﬁer pair ( p 1 , v 1 ) don’t constitute a prover-leading Stackelberg equilibrium.
If the prover picks p 1 , the veriﬁer can attain exactly 0 loss by assigning a weight of 1 to v 1 and a weight of 0 to v 2 and v 3 .
In this scenario, the prover’s loss will be inﬁnity:
[Equation]
On the other hand, if the prover picks policy p 2 , then its messages have exactly 0 mutual information with the input.
In this scenario, the best the veriﬁer can do to minimize its loss function is to assign equal probability to both outputs 0 and 1 .
It can do this by assigning 0 weight to v 1 and weights 0 .to
v 2 and v 3 .
The prover’s loss in this scenario is:
[Equation]
Since the prover can do better by picking p 2 over p 1 , the pair ( p 1 , v 1 ) doesn’t constitute a proverleading Stackelberg equilibrium.
Lack of Sufﬁciency: Consider the scenario discussed above where the prover picks p 2 , and the veriﬁer responds by assigning 0 .
5 weight to v 2 and v 3 .
This is a Stackelberg equilibrium despite not corresponding to a complete and sound decision rule.
This is because 1) p 1 achieves the best prover loss given that the veriﬁer
Theorem 2.
Consider all veriﬁer-leading sequential PVG formulations in which the problem instance is revealed after the veriﬁer picks its strategy.
Having reached a (veriﬁer leading Stackelberg) equilibrium is both necessary and sufﬁcient for having found a desirable proof-veriﬁcation protocol.
Proof.
We ﬁrst prove necessity, then sufﬁciency.
Necessity:
By the problem deﬁnition, there exists a prover-veriﬁer pair ( p 1 , v 1 ) that satisfy Equations 5 and 6, therefore constitute a complete and sound proof-veriﬁcation protocol.
Consider the scenario where the prover’s policy is p 1 , and the veriﬁer assigns a weight of 1 to policy v 1 .
These prover-veriﬁer strategies constitute a (Stackelberg) equilibrium of the veriﬁer-leading sequential game formulation.
This is because (1) p 1 already achieves the smallest prover loss for the given veriﬁer strategy (2) the veriﬁer already achieves 0 loss and therefore has no incentive to change its strategy.
To see (1), note that the veriﬁer can never be fooled if the correct label is 0 , so the prover cannot possibly lower its loss by changing to a different strategy.
If the correct label is 1 , then the prover achieves 0 loss by following p 1 .
Therefore, the prover has no incentive to change its strategy.
Sufﬁciency: To show that having reached a veriﬁer-leading Stackelberg equilibrium implies a complete and sounds proof-veriﬁcation protocol, we follow a “proof by contrapositive" argument and prove, instead, that if the veriﬁer is not representing a complete and sound proof-veriﬁcation protocol, it cannot possibly be a Stackelberg equilibrium.
Assume that the veriﬁer assigns a non-zero weight to a veriﬁcation policy v ⇤⇤ that is neither complete and/nor sound.
If it’s not complete, then for v ⇤⇤ there exists a label- 1 input x such that there doesn’t exists a prover policy that can convince the veriﬁer that the correct label is 1 .
This will cause the veriﬁer’s loss to be nonzero.
If it’s not sound, then there exists a label- 0 input for which there exists a prover policy that can fool the veriﬁer to answering 1 .
This will also cause the veriﬁer’s loss to be nonzer.
On the other hand, if the veriﬁer assigns 0 weight to all non-complete nor non-sound veriﬁcation modules, then it’s loss will be exactly 0 .
This implies that the veriﬁer can do strictly better by switching to a complete and sound veriﬁcation protocol.
Theorem 3.
Consider the simultaneous PVG formulation where the problem instance is revealed after the agents pick their strategies.
Having converged to a (Nash) equilibrium is necessary for having found a complete and sound proof-veriﬁcation protocol.
Proof.
Proving necessity for the simultaneous setup is identical to proving necessity of the veriﬁerleading Stackelberg setup.
The complete and sound prover-veriﬁer pair ( p 1 , v 1 ) (i.e. the veriﬁer strategy of assigning a weight of 1 to v 1 ), whose existence is guaranteed by the problem deﬁnition, form a Nash equilibrium.
If we ﬁx the veriﬁer strategy at v 1 , then the prover has no incentive to change it’s strategy.
In the collaborative case (when the input belongs to class 1 , the prover already achieves 0 loss.
In the adversarial case, the veriﬁer is never fooled (by the soundness assumption) so the prover cannot improve its loss by switching to a different strategy.
Therefore, if we ﬁx the veriﬁer strategy, the prover wouldn’t move.
If we ﬁx the prover strategy at p 1 , then the veriﬁer already achieves 0 loss by following v 1 .
Therefore it doesn’t have an incentive to move.
Therefore, the ( p 1 , v 1 ) pair form a Nash equilibrium.
Corollary 1.
All veriﬁer-leading Stackelberg equilibria are Nash equilibria of PVG.
Proof.
Having reached a veriﬁer-leading Stackelberg equilibria is sufﬁcient for having found a complete and sound proof-veriﬁcation protocol (by Theorem 2).
A sound and complete proofveriﬁcation protocol already constitute a Nash equilibrium 3.
Therefore, having reached a veriﬁerleading Stackelbeg equilibrium implies having reached a Nash equilibrium.
C P ROOFS FOR E QUILIBRIUM A NALYSIS
Theorem 4.
Starting from any initialization with p 0 i > 0 , p 1 i > 0 for all i 2 M , running gradient descent on both prover and veriﬁer with reasonable learning rates, the prover would converge to only sending message 1 when x = 1 while the veriﬁer would predict 1 only if it receives message 1 from the prover.
To put it differently, p 11 = 1 when converge while q 1 = 1  ✏ and q 0 = q
= ...
=
q K = ✏ with ✏ a small constant depending on the strength of entropy regularization.
Proof.
First, we can assume WLOG that K = 2 because all these tokens from 2 to K are “exactly” same for both prover and veriﬁer.
So if we do not concern about computation time, we can group tokens 2  K as a single token.
In this case, the only parameters for the prover is just p 00 and p 11 .
For notational convenience, we let b , p 00 and a , p 11 .
Further, we assume the learning rates ⌘ P ⌧ ⌘ V .
With two time-scale updates, one can show that the veriﬁer would converge ﬁrst.
Given nondegenerate initialization and entropy regularization on q with coefﬁcient  >
(this is basically label smoothing), we have
[Equation]
Recall that the objective of the prover is
[Equation]
Hence, we have the mean gradients of the prover @
[Equation]
As long as a < 1 , b would keep decrease as the gradient w.r.t
Further, we show a = 1 , b < 1 and q taking the values in equation 9 are the Nash equilibria of the game.
It is easy to see the q value are optimal given ﬁxed a, b , so it is impossible to improve unilaterally for the veriﬁer.
On the other hand, we know log q 1 > log q 0 = log q 2 if a = 1 and b < 1 , so the prover could not improve either.
This completes the proof.
Theorem 5.
Starting from any initialization with p 0 i > 0 , p 1 i > 0 for all i 2 M , running gradient descent on both prover and veriﬁer with any learning rates, prover would converge to only sending message 2 to K no matter what the input is.
Proof.
From the viewpoint of prover, it is essentially single-objective optimization problem with the loss function in the following form:
[Equation]
Surprisingly, the Stackelberg game formulation leads to bad joint strategy where the prover may send wrong messages with the input 1 .
In particular, we have the gradient:
[Equation]
By the identity log(1 + x ) < x , we immediately know that @ L @ a > 0 and @ L @ b > 0 if a, b 6 = 1 .
Interestingly, this is exactly the opposite of the simultaneous game.
This immediately implies that a, b will converge to 0 .
This completes the proof.
D T RAINING H EURISTICS
We ﬁnd that the following design decisions make learning more consistent and help speed up convergence, while not being essential for success:
• Different Number of Gradient Steps per Agent: We ﬁnd that updating the veriﬁer multiple times per a single prover update helps.
It is especially helpful in preventing the veriﬁer from ignoring the prover’s messages early on in training.
Note that proving Theoremrequires setting a larger learning rate to the veriﬁer.
• Adaptive Prover Step Size:
Adaptively increasing the prover-update frequency if the veriﬁer’s accuracy surpasses a pre-selected (i
e. 90 percent) value helps speed up the transition from “nearly robust" protocols to fully robust ones at the terminal phase of the training.
• Prover Pretraining:
Pretraining the prover to solve the decision problem before initiating the prover-veriﬁer game ensures that the ﬁrst messages the veriﬁer receives from the prover will likely contain features that correlate with the labels and therefore decreases the likelihood that the veriﬁer will ignore the prover’s messages.
We do this by appending two extra heads to the prover that are trained to do solve the task at hand, as well as to autoencode the input.
E E XPERIMENT D ETAILS E. B INARY E RASURE C HANNEL E XPERIMENTS
• Task Setup and Data Distribution: We picked there to be 16 communication tokens that the prover can communicate to the prover.
We sampled the input bits uniformly, which mean the labels were balanced.
• Prover Network: We parametrized the prover as a two hidden layer fully connected LeakyReLU network with width 100 and LayerNorm (Ba et al., 2016) layers fully connected network.
We used the Gumbel-softmax trick (with a constant temperature of 1 ) to sample (one hot encoded) communication tokens.
We use the Straight-Through Gumbel Softmax Estimator (Maddison et al., 2017) to backpropagate through the discrete sampling step.
• Veriﬁer Network:
The veriﬁer was instantiated as a one hidden layer, fully connected LeakyReLU neural network with width 100 and LayerNorm layers.
• Optimization setup : We used the Adam optimizer for updating both the prover and veriﬁer with a learning rate of 0 .
0003 and batch size of 2000 .
We updated the veriﬁer 5 times for each prover update.
We observed that training the prover to solve the classiﬁcation and autoencoding auxiliary tasks doesn’t have an impact on training, likely due to the simplicity of this task.
We ran optimization until the veriﬁer learned a complete and sound proof-veriﬁcation protocol (or, frozen veriﬁer’s precision and recall reached the value of 1 ) • Evaluation details:
To train the prover against the frozen veriﬁer, we use the Adam optimizer with a learning rate of 0 .
0003 for 500 gradient steps, which we found was enough to reach near-convergence.
To optimize the prover messages directly, we use the PyTorch implementation (Paszke et al., 2019) of the LBFGS optimizer (Nocedal & Wright, 2006) with a learning rate of 1, history size of 300 and “strong Wolfe" line search.
We terminate the optimization when either the L1 norm of the change in iterates becomes less than 10 e  9 , or when the L1 norm of gradients become less than 10 e  5 , or when the number of optimizer steps surpass 300 .
We found that the the tolerance-based termination criteria are met almost always before the maximum number of optimizer steps is reached.
• Hyperparameter Optimization and Tuning
: We tuned the following aspects of training
1) Optimizer (Adam vs. SGD), 2) Learning rate (1e-3, 3e-4, 1e-4), 3) Batch size (500, 2000), 4) Network width (32, 100) 5)
veriﬁer number of steps (1, 5, 10, 20, 50).
We found that using Adam, the larger batch size of 2000 and a larger width of 100 for the networks were crucial.
Setting the learning rate to be 3 e  4 and veriﬁer number of steps to 5 stabilized training and made it more consistent.
• Training hardware:
Due to the low computational requirements of the BEC experiments, the experiments were run on an 2019 Modle MacBookPro with 8-Core Intel Core i processors.
E. F IND T HE P LUS E XPERIMENTS :
• Task Setup and Data Distribution: We used 10 ⇥ 10 images with exactly one 3 ⇥ 3 plus patterns in them.
The images were single-channel, and the pixel values were binary (i.e.or 1 ).
The plus patterns were either formed using 0 -pixels or 1 -pixels, which we refer to “red" or “blue" plusses in the original text for additional clarity.
The used balanced labels during training.
• Prover Network: We parametrized the prover as a 3 hidden layer convolutional neural network with 40 hidden channels, kernel size of 3, padding of 1 and LeakyReLU (Maas et al., 2013) nonlinearities followed by Instance Normalization layers.
We passed the output of this convolutional neural network through a mean-pooling operation follow by a linear projection layer that resulted in the message vectors.
The message vectors were 32 dimensional.
We also trained the prover network to both solve the decision problem itself (i.e. classiﬁcation), as well as autoencode the inputs.
We did this by tacking “auxiliary heads" that process the output of the convolutional backbone described above and produce their outputs.
We used a linear classiﬁcation auxiliary head.
For the autoencoding auxiliary head, we used a Spatial Broadcast Decoder (Watters et al., 2019) with 3 convolutional layers withchannels, kernel size of 3 and LeakyReLU activations.
• Veriﬁer Network: We parametrized the veriﬁer with the Spatial Transformer Network-like (Jaderberg et al., 2015) architecture with 4 heads.
This architecture linearly transforms the prover messages into 2 ⇥ 3 matrices that parametrize what kind of spatial transformation should be applied to the image.
The transformed images are then fed into a ﬁnal linear classiﬁer that produces the veriﬁer predictions.
A schematic diagram of a single head of this architecture is displayed in Figure 3.
We constrained the allowed spatial transformations to
shifts
coded
constraints
• Optimization Setup:
We used the Adam optimizer for updating both the prover and veriﬁer with a learning rate of 0 .
0003 and batch size of 2000 .
We updated the veriﬁer 1 times for each prover update.
We observed that pretraining training the prover to solve the classiﬁcation task speeds up training considerably, while the autoencoding head doesn’t have a signiﬁcant impact.
We adaptively incremented the prover’s number of steps if the veriﬁer’s accuracy kept above 75% for 20 consequent game steps, and stopped incrementing when the prover’s number of steps reached 15 .
We ran pretraining using the same optimizer settings, forgradient steps (enough to let the prover achieve around 80% accuracy).
We employed a bit of Label Smoothing (Müller et al., 2019)
(with coefﬁcient 0 . 05 ) to the veriﬁer loss to regularize training.
For pretraining the prover classiﬁcation and autoencoding auxiliary heads, we used the cross entropy loss and (pixelwise) mean squared error respectively.
We kept training the prover to classify and autoencode throughout training.
We report our results at the point veriﬁer achieved it’s best frozen precision against directly optimized message vectors.
Note that the veriﬁer achieves perfect frozen precision against an optimized prover way before this point.
• Evaluation Details: We used exactly the same evaluation setup we used in the Binary Erasure Channel Experiments.
We report the results on the training checkpoint with the highest overall frozen veriﬁer accuracy against optimized proofs, as the performance of the veriﬁer tends to ﬂuctuate all throughout training.
We also report the best result amongst training runs initialized with four random number generator seeds.
• Hyperparameter Optimization and Tuning: We used the setup that worked best for the Binary Erasure Taks and found that it also yielded good results for the FindThePlus task.
We re-tuned the number of veriﬁer steps and found that updating the veriﬁer less frequently (i e. once per prover step) doen’t
degrade performance, while speeding up convergence.
We also tuned the duration of prover pretraining.
We found that pretraining the prover til convergence actually hurts performance.
We link this to the Neural Collapse phenomenon elaborated by Papyan et al.
and leave a more detailed examination to future works.
• Training Hardware:
We ran training using a single NVIDIA P100 GPU with Intel Xeon Silver processors.
E. A DDITIONAL P LUS V ISUALIZATIONS
In Figure 4, we show more visualizations that display where the veriﬁer looks at in the input images.