O N THE I MPOSSIBILITY OF G LOBAL C ONVERGENCE IN M ULTI -L OSS O PTIMIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Under mild regularity conditions, gradient-based methods converge globally to a critical point in the single-loss setting.
This is known to break down for vanilla gradient descent when moving to multi-loss optimization, but can we hope to build some algorithm with global guarantees?
We negatively resolve this open problem by proving that desirable convergence properties cannot simultaneously hold for any algorithm.
Our result has more to do with the existence of games with no satisfactory outcomes, than with algorithms per se.
More explicitly we construct a two-player game with zero-sum interactions whose losses are both coercive and analytic, but whose only simultaneous critical point is a strict maximum.
Any ‘reasonable’ algorithm, deﬁned to avoid strict maxima, will therefore fail to converge.
This is fundamentally different from single losses, where coercivity implies existence of a global minimum.
Moreover, we prove that a wide range of existing gradient-based methods almost surely have bounded but non-convergent iterates in a constructed zero-sum game for suitably small learning rates.
It nonetheless remains an open question whether such behavior can arise in high-dimensional games of interest to ML practitioners, such as GANs or multi-agent RL.
I NTRODUCTION
Problem Setting. As multi-agent architectures proliferate in machine learning, it is becoming increasingly important to understand the dynamics of gradient-based methods when optimizing multiple interacting goals, otherwise known as differentiable games .
This framework encompasses GANs (Goodfellow et al., 2014), intrinsic curiosity (Pathak et al., 2017), imaginative agents (Racani ` ere et al., 2017), synthetic gradients (Jaderberg et al., 2017), hierarchical reinforcement learning (Wayne & Abbott, 2014; Vezhnevets et al., 2017) and multi-agent RL in general (Busoniu et al., 2008).
The interactions between learning agents make for vastly more complex mechanics: naively applying gradient descent on each loss simultaneously is known to diverge even in simple bilinear games.
Related Work.
A large number of methods have recently been proposed to alleviate the failings of simultaneous gradient descent: adaptations of single-loss algorithms such as Extragradient (EG) (Azizian et al., 2019) and Optimistic Mirror Descent (OMD) (Daskalakis et al., 2018), Alternating Gradient Descent (AGD) for ﬁnite regret (Bailey et al., 2019), Consensus Optimization (CO) for GAN training (Mescheder et al., 2017), Competitive Gradient Descent (CGD) based on solving a bilinear approximation of the loss functions (Schaefer & Anandkumar, 2019), Symplectic
Gradient Adjustment (SGA) based on a novel decomposition of game mechanics (Balduzzi et al., 2018;
Letcher et al., 2019a), and opponent-shaping algorithms including Learning with OpponentLearning Awareness (LOLA) (Foerster et al., 2018) and its convergent counterpart, Stable Opponent
Shaping (SOS) (Letcher et al., 2019b).
Let A be this set of algorithms.
Each has shown promising theoretical implications and empirical results, but none offers insight into global convergence in the non-convex setting, which includes the vast majority of machine learning applications.
One of the main roadblocks compared with single-loss optimization has been noted by
Schaefer & Anandkumar (2019): “a convergence proof in the nonconvex case analogue to Lee et al.
(2016) is still out of reach in the competitive setting.
A major obstacle to this end is the identiﬁcation of a suitable measure of progress (which is given by the function value in the single agent setting), since norms of gradients can not be expected to decay monotonously for competitive dynamics in non-convex-concave games.”
It has been established that Hamiltonian Gradient Descent converges in two-player zero-sum games under a “sufﬁciently bilinear” condition by Abernethy et al.
, but this algorithm is unsuitable for optimization as it cannot distinguish between minimization and maximization (Hsieh et al., 2020,
Appendix C.4).
Global convergence has also been established for some algorithms in a few special cases: potential and Hamiltonian games (Balduzzi et al., 2018), zero-sum games satisfying the twosided Polyak-Łojasiewicz condition (Yang et al., 2020), zero-sum linear quadratic games (Zhang et al., 2019) and zero-sum games whose loss and ﬁrst three derivatives are bounded (Mangoubi & Vishnoi, 2020).
These are signiﬁcant contributions with several applications of interest, but do not include any of the architectures mentioned above.
Finally, Balduzzi et al.
(2020) show that
GD dynamics are bounded under a ‘negative sentiment’ assumption in smooth markets, which do include GANs – but this does not imply convergence, as we will show.
On the other hand, failure of global convergence has been shown for the Multiplicative Weights Update method by Palaiopanos et al.
(2017), for policy-gradient algorithms by Mazumdar et al.
(2020), and for simultaneous and alternating gradient descent (simGD and AGD) by Vlatakis-Gkaragkounis et al.
Bailey et al.
(2019), with interesting connections to Poincar´e recurrence.
Nonetheless, nothing is claimed about other optimization methods.
Farnia & Ozdaglar (2020) show that GANs may have no Nash equilibria, but it does not follow that algorithms fail to converge since there may be locally-attracting but non-Nash critical points (Mazumdar et al., 2019, Example 2).
Finally, Hsieh et al.
(2020) uploaded a preprint just after the completion of this work with a similar focus to ours.
They prove that generalized Robbins-Monro schemes may converge with arbitrarily high probability to spurious attractors.
This includes simGD, AGD, stochastic EG, optimistic gradient and Kiefer-Wolfowitz.
However, Hsieh et al.
(2020) focus on the possible occurrence of undesirable convergence phenomena for stochastic algorithms.
We instead prove that desirable convergence properties cannot simultaneously hold for all algorithms (including deterministic ).
Moreover, their results apply only to decreasing step-sizes whereas ours include constant step-sizes.
These distinctions are further highlighted by Hsieh et al.
(2020) in the further related work section.
Taken together, our works give a fuller picture of the failure of global convergence in multi-loss optimization.
Contribution. We prove that global convergence in multi-loss optimization is fundamentally incompatible with the ‘reasonable’ requirement that algorithms avoid strict maxima and converge only to critical points.
We construct a two-player game with zero-sum interactions whose losses are coercive and analytic, but whose only critical point is a strict maximum (Theorem 1).
Reasonable algorithms must either diverge to inﬁnite losses or cycle (bounded non-convergent iterates).
One might hope that global convergence could at least be guaranteed in games with strict minima and no other critical points.
On the contrary we show that strict minima can have arbitrarily small regions of attraction, in the sense that reasonable algorithms will fail to converge there with arbitrarily high probability for ﬁxed initial parameter distribution (Theorem 2).
Finally, restricting the game class even further, we construct a zero-sum game in which all algorithms in A (as deﬁned in Appendix A) are proven to cycle (Theorem 3).
It may be that cycles do not arise in high-dimensional games of interest including GANs.
Proving or disproving this is an important avenue for further research, but requires that we recognise the impossibility of global guarantees in the ﬁrst place.
B ACKGROUND 2. S INGLE LOSSES : GLOBAL CONVERGENCE OF GRADIENT DESCENT
Given a continuously differentiable function f : R d → R , let
[Equation]
be the iterates of gradient descent with learning rate α , initialised at θ 0 .
Under standard regularity conditions, gradient descent converges globally to critical points:
Proposition 1.
Assume f ∈ C 2 has compact sublevel sets and is either analytic or has isolated critical points.
For any θ 0 ∈ R d , deﬁne U
= { f ( θ ) ≤ f ( θ 0 ) } and let L < ∞ be a Lipschitz constant for ∇ f in U 0 .
Then for any 0 < α < 2 /L
we have lim k θ k
= ¯ θ for some critical point ¯ θ .
The requirements for convergence are relatively mild:
f has compact sublevel sets iff f is coercive, lim (cid:107) θ (cid:107)→∞ f ( θ ) = ∞ , which mostly holds in machine learning since f is a loss function.
f has isolated critical points if it is a Morse function (nondegenerate Hessian at critical points), which holds for almost all C 2 functions.
More precisely, Morse functions form an open, dense subset of all functions f ∈ C 2 ( R d , R ) in the Whitney C 2 -topology.
Global Lipschitz continuity is not assumed, which would fail even for cubic polynomials.
The goal of this paper is to prove that similar (even weaker) guarantees cannot be obtained in the multi-loss setting – not only for GD, but for any reasonable algorithm.
This has to do with the more complex nature of gradient vector ﬁelds arising from multiple losses.
2. D IFFERENTIABLE GAMES
Following Balduzzi et al.
(2018), we frame the problem of multi-loss optimization as a differentiable game among cooperating and competing agents/players.
These may simply be different internal components of a single system, like the generator and discriminator in GANs.
Deﬁnition 1.
A differentiable game is a set of n agents with parameters θ =
( θ 1 , . .
, θ n ) ∈ R d and twice continuously differentiable losses
L i : R d → R , where θ i ∈
R
d i for each i and (cid:80) i d i = d .
Losses are not assumed to be convex/concave in any of the parameters.
In practice, losses need only be differentiable almost-everywhere: think of neural nets with rectiﬁed linear units.
If n = 1 , the ‘game’ is simply to minimise a given loss function.
We write ∇ i L k = ∇ θ i L k and ∇ ij L k = ∇ θ j ∇ θ i L k for any i, j, k , and deﬁne the simultaneous gradient of the game
[Equation]
as the concatenation of each player’s gradient.
If each agent independently minimises their loss using GD with learning rate α , the parameter update for all agents is given by θ ← θ − αξ ( θ ) .
We call this simultaneous gradient descent (simGD), or GD for short.
We call ¯ θ a critical point if ξ (¯ θ ) = 0 .
Now introduce the ‘Hessian’ (or Jacobian) of the game as the block matrix
[Equation]
Importantly note that H is not symmetric in general unless n = 1 , in which case we recover the usual Hessian H = ∇ 2 L .
However H can be decomposed into symmetric and anti-symmetric components as H =
S + A (Balduzzi et al., 2018).
A second useful decomposition has appeared recently in (Letcher et al., 2019b) and (Schaefer & Anandkumar, 2019):
H = H d + H o where
H d and H o are the matrices of diagonal and off-diagonal blocks; formally, H d = (cid:76) i ∇ ii L i .
One solution concept for differentiable games, analogous to the single-loss case, is deﬁned as follows.
Deﬁnition 2.
A critical point ¯ θ is a (strict, local) minimum if H (¯ θ ) (cid:31) 0 .
These were named (strict) stable ﬁxed points by Balduzzi et al.
, but the term is usually reserved in dynamical systems to the larger class deﬁned by Hessian eigenvalues with positive real parts, which is implied but not equivalent to H (cid:31) 0 for non-symmetric matrices.
In particular, strict minima are (differential) Nash equilibria as deﬁned by Mazumdar et al.
(2019), since diagonal blocks must also be positive deﬁnite: ∇ ii L i (¯ θ ) (cid:31) 0 .
The converse does not hold.
Algorithm class.
This paper is concerned with any algorithm whose iterates are obtained by initialising θ 0 and applying a function F to the previous iterates, namely θ k +
=
F ( θ k , . .
, θ 0 ) .
This holds for all gradient-based methods (deterministic or stochastic); most of them are only functions of the current iterate θ k , so that θ k = F k ( θ 0 ) .
All probabilistic statements in this paper assume that θ 0 is initialised following any bounded and continuous measure ν on R d .
Continuity is a weak requirement and widely holds across machine learning, while boundedness mostly holds in practice since the bounded region can be made large enough to accommodate required initial points.
For single-player games, the goal of such algorithms is for θ k to converge to a local (perhaps global) minimum as k → ∞ .
The goal is less clear for differentiable games, but is generally to reach a minimum or a Nash equilibrium.
In the case of GANs the goal might be to reach parameters that produce realistic images, which is more challenging to deﬁne formally.
Throughout the text we use the term (limit) cycle to mean bounded but non-convergent iterates .
This terminology is used because bounded iterates are non-convergent if and only if they have at least two accumulation points, between which they must ‘cycle’ inﬁnitely often.
This is not to be taken literally: the set of accumulation points may not even be connected.
Hsieh et al.
(2020) provide a more complete characterisation of these cycles.
Game class.
Expecting global guarantees in all differentiable games is excessive, since every continuous dynamical system arises as simultaneous GD on the loss functions of a differentiable game (Balduzzi et al., 2020, Lemma 1).
For this reason, the aforementioned authors have introduced a vastly more tractable class of games called markets .
Deﬁnition 3.
A (smooth) market is a differentiable game where interactions between players are pairwise zero-sum, namely,
[Equation]
with g ij ( θ i , θ j ) + g ji ( θ j , θ i )
=
for all i, j .
This generalises zero-sum games while remaining amenable to optimization and aggregation, meaning that “we can draw conclusions about the gradient-based dynamics of the collective by summing over properties of its members” (Balduzzi et al., 2020).
Moreover, this class captures a large number of applications including GANs and related architectures, intrinsic curiosity modules, adversarial training, task-suites and population self-play.
One would modestly hope for some reasonable algorithm to converge globally in markets.
We will prove that even this is too much to ask.
2. R EASONABLE ALGORITHMS
We wish to prove that global convergence is at odds with weak, ‘reasonable’ desiderata.
The ﬁrst requirement is that ﬁxed points of an optimization algorithm F are critical points.
Formally,
[Equation]
If not, some agent i could strictly improve its losses by following the gradient −∇
i L i (cid:54) = 0 .
There is no reason for a gradient-based algorithm to stop improving if its gradient is non-zero.
The second requirement is that algorithms avoid strict maxima.
Analogous to strict minima, they are deﬁned for single losses by a negative-deﬁnite Hessian H ≺ 0 .
Converging to such a point ¯ θ is the opposite goal of any meaningful algorithm since moving anywhere away from ¯ θ decreases the loss.
There are multiple ways of generalising this concept for multiple losses, but Proposition 2 below justiﬁes that H ≺ 0 is the weakest one.
Proposition 2.
Write λ
( A ) =
Re(Spec( A )) for real parts of the eigenvalues of a matrix A .
We have the following implications, and none of them are equivalences.
[Equation]
Deﬁnition 4.
A critical point ¯ θ is a (strict, local) maximum if H ( ¯ θ ) ≺ 0 .
Imposing that algorithms avoid strict maxima is therefore the weakest possible requirement of its kind.
Note that the bottom-left implication Proposition 2 is equivalent to ∇ ii L i ≺ 0 for all i , so strict maxima are also strict maxima of each player’s individual loss function.
Players can all decrease their losses by moving anywhere away from them.
It is exceedingly reasonable to ask that optimization algorithms avoid these points almost surely.
Formally, we require that for any strict maximum ¯ θ and bounded region U there are hyperparameters such that
[Equation]
µ denotes Lebesgue measure.
Hyperparameters may depend on the given game and the region U , as is typical for learning rates in gradient-based methods.
Deﬁnition 5 (Reason) .
An algorithm is reasonable if it satisﬁes R1 and R2.
Reason is not equivalent to rationality or self-interest .
Reason is much weaker, imposing only that agents are well-behaved regarding strict maxima even if their individual behavior is not selfinterested.
For instance, SGA agents do not behave out of self-interest (Balduzzi et al., 2018).
G LOBAL C ONVERGENCE IN D IFFERENTIABLE G AMES 3. R EASONABLE ALGORITHMS FAIL TO CONVERGE GLOBALLY
Our main contribution is to show that global guarantees do not exist for any reasonable algorithm.
First recall that global convergence should not be expected in all games, since there may be a divergent direction with minimal loss (imagine minimising L = e x ).
It should however be asked that algorithms have bounded iterates in coercive games, deﬁned by coercive losses
[Equation]
for all i .
Indeed, unbounded iterates in coercive games would lead to inﬁnite losses for all agents, the worst possible outcome.
Given bounded iterates, convergence should hold if the Hessian is nondegenerate at critical points (which must therefore be isolated, recall Proposition 1).
We call such a game nondegenerate .
This condition can also be replaced by analyticity of the loss.
In the spirit of weakest assumptions, we ask for convergence when both conditions hold.
Deﬁnition 6 (Globality) .
An algorithm is global if, in a coercive, analytic and nondegenerate game, for any ﬁxed θ 0 , iterates θ k are bounded and converge for suitable hyperparameters.
[Equation]
Note that GD is global for single-player games by Proposition 1.
Unfortunately, reason and globality are fundamentally at odds as soon as we move to two-player markets.
Theorem 1.
There is a coercive, nondegenerate, analytic two-player market M whose only critical point is a strict maximum.
In particular, algorithms only have four possible outcomes in M :
Iterates are unbounded, and all players diverge to inﬁnite loss.
[Not global] 2.
Iterates are bounded and converge to the strict maximum.
[Not reasonable] 3.
Iterates are bounded and converge to a non-critical point.
[Not reasonable] 4.
Iterates are bounded but do not converge (cycle).
[Not global]
Proof.
Consider the analytic market M given by
[Equation]
We prove in Appendix D that M is coercive, nondegenerate, and has a unique critical point at the origin, which is a strict maximum.
Constructing an algorithm with global guarantees is therefore doomed to be unreasonable in that it will converge to strict maxima or non-critical points in M .
None of the outcomes of M are satisfactory.
The ﬁrst three are highly objectionable, as already discussed.
The fourth is less obvious, and may even have game-theoretic signiﬁcance (Papadimitriou & Piliouras, 2019), but is counter-intuitive from an optimization standpoint.
Terminating the iteration would lead to a non-critical point, much like the third outcome.
Even if we let agents update parameters continuously as they play a game or solve a task, they will have oscillatory behavior and fail to produce consistent outcomes (e.g. when generating an image or playing Starcraft).
The hope for machine learning is that such predicaments do not arise in applications we care about, such as GANs or intrinsic curiosity.
This may well be the case, but proving or disproving global convergence in these speciﬁc settings is beyond the scope of this paper.
Remark.
Why can this approach not be used to disprove global convergence for single losses?
One reason is that we cannot construct a coercive loss with no critical points other than strict maxima: coercive losses, unlike games, always have a global minimum.
3. W HAT IF THERE ARE STRICT MINIMA ?
One might wonder if it is purely the absence of strict minima that causes non-convergence, since strict minima are locally attracting under gradient dynamics.
Can we guarantee global convergence if we impose existence of a minimum, and more, the absence of any other critical points?
Unfortunately, strict minima may have an arbitrarily small region of attraction.
Assuming parameters are initialised following any bounded continuous measure ν on R d , we can always modify M by deforming a correspondingly small region around the origin, turning it into a minimum while leaving the dynamics unchanged outside of this region.
For a ﬁxed initial distribution, any reasonable algorithm can therefore enter a limit cycle or diverge to inﬁnite losses with arbitrarily high probability.
Theorem 2.
Given a reasonable algorithm with bounded continuous distribution on θ 0 and a real number (cid:15) > 0 , there exists a coercive, nondegenerate, almost-everywhere analytic two-player market M σ with a strict minimum and no other critical points, such that θ k either cycles or diverges to inﬁnite losses for both players with probability at least 1 − (cid:15) .
Proof.
Let 0 < σ < 0 .
1 and deﬁne
[Equation]
where θ =
( x, y ) and (cid:107) θ (cid:107) =
(cid:112) x 2 + y 2 is the standard L 2 -norm.
Note that f σ is continuous since
[Equation]
Now consider the two-player market M σ given by
[Equation]
We prove in Appendix E that M σ is a coercive, nondegenerate, almost-everywhere analytic game whose only critical point is a strict minimum at the origin.
We then prove that θ k cycles or diverges with probability at least 1 − (cid:15) , and plot iterates for each algorithm in A .
3. H OW DO EXISTING ALGORITHMS BEHAVE ?
Any algorithm will either fail to be reasonable or global in M .
Nonetheless, it would be interesting to determine the speciﬁc failure that each algorithm in A exhibits.
Each of them is deﬁned in
GD
AGD
EG
OMD 1.
SGA 1.0 0.5 0.0 0.5 1.
CO 1.0 0.5 0.0 0.5 1.
CGD 1.0 0.5 0.0 0.5 1.
LA 1.0 0.5 0.0 0.5 1.
LOLA 1.0 0.5 0.0 0.5 1.
SOS Appendix A, writing α for the learning rate and γ for the Consensus Optimization hyperparameter. We expect each algorithm to be reasonable and moreover to have bounded iterates in M for suitably small hyperparameters.
If this holds, they must cycle by Theorem 1.
This was witnessed experimentally across 1000 runs for α = γ = 0 .
01 , with every run resulting in cycles.
A single such run is illustrated in Figure 1.
Algorithms may follow one of the three other outcomes for other hyperparameters, for instance diverging to inﬁnite loss if α is too large or converging to the strict maximum for CO if γ is too large.
The point here is to characterise the ‘regular’ behavior which can be seen as that occurring for sufﬁciently small hyperparameters.
Instead of proving that algorithms must cycle in M , we construct a zero-sum game N with similar properties as M and prove below that algorithms in A almost surely fail to converge there for small α, γ .
This is stronger than proving the analogous result for M , since N belongs to the even smaller class of zero-sum games which one might have hoped was well-behaved.
In this light, one might wish to extend Theorem 1 to zero-sum games.
However, zero-sum games cannot be coercive since L 1 → ∞ implies L 2 → −∞ .
It is therefore unclear whether global guarantees should be expected.
Note however that N will be weakly-coercive in the sense that
[Table]
[Equation]
[Table]
[Equation]
for all i and ﬁxed θ − i .
Theorem 3.
There is a weakly-coercive, nondegenerate, analytic two-player zero-sum game N whose only critical point is a strict maximum.
Algorithms in A almost surely have bounded nonconvergent iterates in N for α, γ sufﬁciently small.
Proof.
Consider the analytic zero-sum game N given by
[Equation]
We prove in Appendix F that N is weakly-coercive, nondegenerate, and has a unique critical point at the origin which is a strict maximum.
We prove that algorithms in A have the origin as unique ﬁxed points, with negative-deﬁnite Jacobian for α, γ small, hence failing to converge almost surely.
We moreover prove that algorithms have bounded non-convergent iterates in N for α, γ sufﬁciently small.
Iterates are plotted for a single run of each algorithm in Figure 3 with α
= γ = 0 .
As in M , the behavior of each algorithm may differ for larger hyperparameters.
All algorithms may have unbounded iterates or converge to the strict maximum for large α , while EG and OMD may even converge to a non-critical point (see proof).
All such outcomes are unsatisfactory, though unbounded iteration will not result in positive inﬁnite losses for both players since L 1 = − L 2 .
3. C OROLLARY : THERE ARE NO SUITABLE MEASURES OF PROGRESS
A crucial step in proving global convergence of GD on single losses is showing that the set of accumulation points is a subset of critical points, using the function value as a ‘measure of progress’.
The fact that this fails for differentiable games implies that there can be no suitable measures of progress for reasonable algorithms with bounded iterates.
We formalise this below, answering the question of Schaefer & Anandkumar (2019) quoted in the introduction.
Deﬁnition 7.
A measure of progress for an algorithm given by θ k +1 =
F ( θ k ) is a continuous map
M : R d → R , bounded below, such that M ( F ( θ )) ≤ M ( θ ) and M ( F ( θ ))
=
M ( θ ) iff F ( θ )
= θ .
Measures of progress are very similar to descent functions , as deﬁned by Luenberger & Ye (1984), and somewhat akin to Lyapunov functions.
The function value f is a measure of progress for singleloss GD under the usual regularity conditions, while the gradient norm (cid:107) ξ (cid:107) is a measure of progress for GD in strictly convex differentiable games:
[Equation]
for small α .
Unfortunately, games like M prevent the existence of such measures in general.
Corollary 1.
There are no measures of progress for reasonable algorithms which produce bounded iterates in M or N .
Assuming the algorithm to be reasonable is necessary: any map is a measure of progress for the unreasonable algorithm F ( θ ) = θ .
Assuming the algorithm to have bounded iterates in M or N is necessary: M ( θ ) =
exp ( − θ · 1 ) is a measure of progress for the reasonable but always-divergent algorithm F ( θ )
= θ
+ 1 , where 1 is the constant vector of ones.
C ONCLUSION
We have proven that global convergence is fundamentally at odds with weak, desirable requirements in multi-loss optimization.
Any reasonable algorithm can cycle or diverge to inﬁnite losses, even in two-player markets.
This arises because coercive games, unlike losses, may have no critical points other than strict maxima.
However, this is not the only point of failure: strict minima may have arbitrarily small regions of attraction, making convergence arbitrarily unlikely.
Limit cycles are not necessarily bad: they may even have game-theoretic signiﬁcance (Papadimitriou & Piliouras, 2019).
This paper nonetheless shows that some games have no satisfactory outcome in the usual sense, even in the class of two-player markets.
Players should neither escape to inﬁnite losses, nor converge to strict maxima or non-critical points, so cycling may be the lesser evil.
The community is accustomed to optimization problems whose solutions are single points, but cycles may have to be accepted as solutions in themselves.
The hope for machine learning practitioners is that local minima with large regions of attraction prevent limit cycles from arising in applications of interest, including GANs.
Proving or disproving this is an interesting and important avenue for further research, with real implications on what to expect when agents learn while interacting with others.
Cycles may for instance be unacceptable in self-driving cars, where oscillatory predictions may have life-threatening implications.