Original: A Multi-Token Coordinate Descent Method for Vertical Federated Learning
Revised: A Multi-Token Coordinate Descent Method for Vertical Federated Learning

Original: Anonymous Author(s)

Original: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Communication efﬁciency is a major challenge in federated learning.
Revised: Communication efﬁciency is a major challenge in federated learning.

Original: In clientserver schemes, the server constitutes a bottleneck, and while decentralized setupsspread communications, they do not reduce them.
Revised: In clientserver schemes, the server constitutes a bottleneck, and while decentralized setups spread communications, they do not reduce them.

Original: We propose a communicationefﬁcientsemi-decentralized federated learning algorithm for feature-distributeddata.
Revised: We propose a communicationefﬁcient algorithm for semi-decentralized vertical federate learning dealing with feature-distributed data.

Original: Our multi-token method can be seen as a parallel Markov chain (block)coordinate descent algorithm.
Revised: Our multi-token method can be seen as a parallel Markov chain (block) coordinate descent algorithm.

Original: In this work, we formalize the multi-token semidecentralized scheme, which subsumes the client-server and decentralized setups,and design a feature-distributed learning algorithm for this setup.
Revised: In this work, we formalize the multitoken semi-decentralized scheme, which subsumes the client-server and decentralized setups, and design a feature-distributed learning algorithm for this setup.

Original: Numerical resultsshow the improved communication efﬁciency of our algorithm.
Revised: Numerical results show the improved communication efﬁciency of our algorithm.

Original:  Introduction
Revised:  Introduction

Original: Federated Learning (FL) is a machine learning paradigm where data is distributed across a set ofclients who collaborate to learn a model without sharing local data [1].
Revised: Federated Learning (FL) is a machine learning paradigm where data is distributed across a set of clients who collaborate to learn a model without sharing local data [23].

Original: Most FL literature considersdata distributed by samples (horizontal FL), where each client holds all the features of a subset of thesamples, yet recently there has been a growing interest on feature-distributed setups (vertical FL),where each client holds a subset of the features for all samples [2–5].
Revised: Most FL literature considers data distributed by samples (horizontal FL), where each client holds all the features of a subset of the samples, yet recently there has been a growing interest on feature-distributed setups (vertical FL), where each client holds a subset of the features for all samples [1, 5, 7, 13].

Original: FL often deals with  star-shaped topologies.
Revised: FL often deals with the client-server setup with a star-shaped topology.

Original: However, such schemes have a single point of failureand suffer from a communication bottleneck on the server [6].
Revised: However, such schemes have a single point of failure and suffer from a communication bottleneck on the server [19].

Original: On the other hand, there is extensive literature on decentralized optimization—from earlier work motivated by applications such aswireless sensor networks and multiagent control [7–9], to recent work motivated by FL [10]—yetthese algorithms converge slowly in sparse and large networks [11] and, although they spread thecommunication load across the network, they tend to have a poor communication efﬁciency [12]. We propose a Semi-Decentralized FL (SDFL) multi-token algorithm for vertical FL.
Revised: On the other hand, there is extensive literature on decentralized optimization—from earlier work motivated by applications such as wireless sensor networks and multiagent control [9, 25, 28], to recent work motivated by FL [17, 18].

Original: By using bothclient-server and client-client communications, we reduce the communications at the server whilemitigating the slow convergence of decentralized algorithms  in sparse and large networks.
Revised: Yet, these algorithms converge slowly in sparse and large networks [26] and, although they spread the communication load across the network, they tend to have a poor communication efﬁciency [37].

Original: In thecommunications between clients, the use of a token method [13–18], where a model-describingtoken follows a random walk over a communication graph (undergoing local updates), allows forbetter communication efﬁciency [18] than the more common consensus-based methods [7–10], whereasymptotic agreement is reached through successive local averaging.
Revised: When concerned with the communications between clients, the use of a token method [3, 14, 16, 22, 24, 29], where a model-describing token follows a random walk over a communication graph (undergoing local updates), allows for better communication efﬁciency [14] than the more common consensus-based methods [9, 17, 25, 28], where asymptotic agreement is reached through successive local averaging.

Original: Yet, the convergence rateof token methods degrades even faster for larger and sparser networks, due to a lack of parallelcommunications.
Revised: Yet, the convergence rate of token methods degrades even faster for larger and sparser networks, due to a lack of parallel communications.

Original: Multi-token methods [18–20] mitigate this problem by running multiple tokenssimultaneously and combining them.
Revised: Multi-token methods [6, 14, 34] mitigate this problem by running multiple tokens simultaneously and combining them.

Revised: Motivated by the above observations, we propose a Semi-Decentralized FL (SDFL) multi-token algorithm for vertical FL.

Revised: By using both client-server and client-client communications, we reduce the communications at the server while mitigating the slow convergence of decentralized algorithms in sparse and large networks.

Revised: Our algorithm might be of particular interest for applications, for example, using time series data measured by personal devices to learn a model of some “cross-client” phenomenon of interest (e.g. meteorological).

Revised: Here, each sample would correspond to the data collected across the devices at a given timestamp.

Original: As our main contributions, we:
Revised: Our main contributions are as follows.

Original: • Formalize the multi-token semi-decentralized federated learning scheme, which is robustto server failures and ﬂexible in the degree of dependence on the server, recovering both client-server and decentralized FL as particular cases;
Revised: • We formalize the multi-token semi-decentralized federated learning scheme, which is robust to server failures and ﬂexible in the degree of dependence on the server, recovering both client-server and decentralized FL as particular cases.

Original: • Design a Multi-Token Coordinate Descent (MTCD) algorithm for vertical FL with improvedcommunication efﬁciency over state-of-the-art methods;• Run numerical experiments  on both synthetic and real data, for a variety of communication setups, showing the improved communication efﬁciency of MTCD as well as theeffectiveness of resorting to multiple tokens.
Revised: • We further design a Multi-Token Coordinate Descent (MTCD) algorithm for vertical FL with improved communication efﬁciency over state-of-the-art methods. • Numerical experiments are performed on both synthetic and real data for a variety of communication setups, showing the improved communication efﬁciency of MTCD as well as the effectiveness of resorting to multiple tokens.

Original: Related works.
Revised: Related works.

Original: Recently, SDFL approaches have been proposed to lower communication costsand deal with data heterogeneity [21, 22] and to handle intermittent connections, latency, andstragglers [23, 24].
Revised: Recently, SDFL approaches have been proposed to lower communication costs and deal with data heterogeneity [11, 20] and to handle intermittent connections, latency, and stragglers [4, 35].

Original: Some SDFL works deal with (multi-layered) hierarchical networks [25, 26].
Revised: Some SDFL works deal with (multi-layered) hierarchical networks [15, 36].

Original: Coordinate Descent (CD) methods [27], where (blocks of) coordinates are updated sequentially,rather than simultaneously, are natural candidates for optimization in feature-distributed learning.
Revised: Coordinate Descent (CD) methods [33], where (blocks of) coordinates are updated sequentially, rather than simultaneously, are natural candidates for optimization in feature-distributed learning.

Original: Theblock selection is most often cyclic [28] or independent and identically distributed random [29, 30].
Revised: The block selection is most often cyclic [2] or independent and identically distributed random [27, 30].

Original: In contrast, [31] considers block selection following a Markov chain.
Revised: In contrast, [32] considers block selection following a Markov chain.

Original: Several extensions to CD havebeen proposed, such as acceleration and parallelization [32] and distributed CD methods [20, 33].
Revised: Several extensions to CD have been proposed, such as acceleration and parallelization [10] and distributed CD methods [6, 21].

Original:  Single-token coordinate descent (STCD)
Revised:  Single-token coordinate descent (STCD)

Original: We now introduce a simple, particular case of our algorithm, closely related to [31] and the applicationmentioned therein, taken from [17].
Revised: We now introduce a simple, particular case of our algorithm, closely related to [32] and the application mentioned therein, taken from [22].

Original: Yet, we work in the primal domain and on a feature-distributedsetting.
Revised: Yet, we work in the primal domain and on a feature-distributed setting.

Original: In this section, we use the terms client and agent interchangeably.
Revised: In this section, we use the terms client and agent interchangeably.

Original: 2.1 Problem statement
Revised: 2.1 Problem statement

Original: Model.
Revised: Model.

Original: Our goal is to minimize the following regularized generalized linear model:[Equation]
Revised: Our goal is to minimize the following regularized generalized linear model:

Original: [Equation]
Revised: [Equation]

Original: ) where θ is a parameter of interest and we have N samples x n , n ∈ { 1 , . ..
Revised: where θ is a parameter of interest and we have N samples x n , n ∈ { 1 , . .

Original:  , N } .
Revised: . , N } .

Original: Our loss function (cid:96)is a (possibly nonconvex) smooth and block smooth loss function with a nonempty set of minimizersand r is a separable (possibly nonsmooth) closed proper function, r ( θ )
Revised: Our loss function (cid:96) is a (possibly nonconvex) smooth and block smooth loss function with a nonempty set of minimizers and r is a separable (possibly nonsmooth) closed proper function, r ( θ )

Original: = (cid:80)
Revised: = (cid:80)

Original: Kk =
Revised: Kk =

Original: 1 r k ( θ k ) , which weassume to have a simple proximal mapping.
Revised: 1 r k ( θ k ) , which we assume to have a simple proximal mapping.

Original: Setup.
Revised: Setup.

Original: In this section, we do not require the existence of a server.
Revised: In this section, we do not require the existence of a server.

Original: The clients [ K ] (cid:44) { 1 , . . .
Revised: The clients [ K ] (cid:44) { 1 , . . .

Original: , K }learn the model in a fully decentralized way, communicating through channels described by a static,undirected communication graph G =
Revised: , K } learn the model in a fully decentralized manner, communicating through channels described by a static, undirected communication graph G =

Original: ( V , E ) , where V =
Revised: ( V , E ) , where V =

Original: [ K ] is the vertex set and E the edge set.
Revised: [ K ] is the vertex set and E the edge set.

Original: We assume G is connected and denote the set of neighbors of agent k by N k (cid:44) { i : { i, k } ∈ E} .
Revised: We assume G is connected and denote the set of neighbors of agent k by N k (cid:44) { i : { i, k } ∈ E} .

Original: Our dataset X ∈ R N × d with samples x n ∈ R d , n ∈ [ N ] , is distributed by features across the clients.
Revised: Our dataset X ∈ R N × d with samples x n ∈ R d , n ∈ [ N ] , is distributed by features across the clients.

Original: More precisely, each sample x n is partitioned as x n = ( x nk )
Revised: More precisely, each sample x n is partitioned as x n = ( x nk )

Original: Kk =1 ∈ R d , where x nk ∈ R d k and64 (cid:80)K k =
Revised: Kk =1 ∈ R d , where x nk ∈ R d k and (cid:80)

Original: 1 d k = d .
Revised: Kk =1 d k = d .

Original: Consequently, we can equivalently deﬁne the partitioning as X =
Revised: Consequently, we can equivalently deﬁne the partitioning as X =

Original: [ X 1 . . .
Revised: [ X 1 . . .

Original: X K ] ,where X k ∈ R N × d k .
Revised: X K ] , where X k ∈ R N × d k .

Original: Our parameter θ ∈ R
Revised: Our parameter θ ∈ R

Original: d is partitioned similarly to x n . That is, θ =( θ k )
Revised: d is partitioned similarly to x n , that is, θ =

Original:  Kk =1 ∈ R d with θ k ∈ R d k .
Revised: ( θ k ) Kk =1 ∈ R d with θ k ∈ R d k .

Original: The labels y ∈
Revised: The labels y ∈

Original: R N are part of the loss function, which we assume all agents toknow.
Revised: R N are part of the loss function, which we assume all agents to know.

Original: 2.2 Algorithm
Revised: 2.2 Algorithm

Original: We now propose a decentralized token method algorithm to minimize (1).
Revised: We now propose a decentralized token method algorithm to minimize (1).

Original: Our token carries z (cid:44)
Revised: Our token carries z (cid:44)

Original: Xθ ∈ R N
Revised: Xθ ∈ R N

Original: and performs a random walk over the graph, being updated and communicated by eachagent after performing local computations.
Revised: and performs a random walk over the graph, being updated and communicated by each agent after performing local computations.

Original: We denote by k t the agent holding the token at iteration t .
Revised: We denote by k t the agent holding the token at iteration t .

Original: The token sufﬁces to compute partial gradients.

Original: From the deﬁnition of z , we have at iteration tthat z t =

Original: (cid:80) Kk = X k θ tk . Letting L ( θ ) (cid:44) (cid:80) Nn
Revised: Algorithm

Original: =1 (cid:96) (cid:0) x (cid:62) n θ (cid:1) , we see that, given z t , agent k can computethe gradient of the loss function with respect to (w.r.t.)

Original: its local parameter θ k :

Original: [Equation]

Original: Initialize θ 

Revised: Initialize θ

Original: = 0 , z 0 = 0 ,
Revised: = 0 , z 0 = 0 ,

Original: arbitrary k 0 , and choose step-size µ for t ≥ 0 do if t mod Q = 0 then
Revised: arbitrary k 0 , and choose step-size µ for t ≥ 0 do if t mod Q = 0 then

Original: Agent k t sends z t +1 to agent k t +1 ∼ U ( N k t ) else k t +1 =
Revised: Agent k t sends z t +1 to agent k t +1 ∼ U ( N k t ) else k t +1 =

Original: k t end if θ t +1 k 
Revised: k t end if θ t +1 k =

Original: = (cid:26) proxµr k
Revised: [Equation] prox

Original: ( θ tk − µ ∇ k L ( θ t )) 
Revised: [Equation] t )) if k = k t

Original: if k = k t θ tk if k (cid:54) =
Revised: [Equation] if k (cid:54) =

Original: k t { z t used when computing ∇ k t L ( θ t ) }
Revised: k t { z t used when computing ∇ k t L ( θ t ) }

Original: [Equation]
Revised: [Equation]

Revised: ) end for

Revised: The token sufﬁces to compute partial gradients.

Revised: From the deﬁnition of z , we have at iteration t that z t = (cid:80)

Revised: Kk =

Revised: X k θ tk .

Revised: Letting L ( θ ) (cid:44) (cid:80) Nn

Revised: =1 (cid:96) (cid:0) x (cid:62) n θ (cid:1) , we see that, given z t , agent k can compute the gradient of the loss function with respect to (w.r.t.)

Revised: its local parameter θ k :

Revised: [Equation]

Revised: (cid:44)

Revised: [Equation]

Revised: (cid:88)

Revised: n

Revised: [Equation]

Original: end for allowing agent k to (locally) perform a proximal CD step w.r.t.
Revised: (cid:96) (cid:0) x (cid:62) n [Equation] (cid:1) = N (cid:88) n = ∇ z (cid:96) ( z n ) x nk , z n = x (cid:62) n θ , allowing agent k to (locally) perform a proximal CD step w.r.t.

Original: θ k , as long as it also holds θ tk :
Revised: θ k , as long as it also holds θ tk :

Original: [Equation]
Revised: [Equation]

Original: Given that we only update θ k at agent k , k t always knows θ tk t .
Revised: Given that we only update θ k at agent k , k t always knows θ tk t .

Original: We now describe STCD, summarized in Algorithm 1, where U denotes the uniform distribution.
Revised: We now describe STCD, summarized in Algorithm 1, where U denotes the uniform distribution.

Original: Initialization.
Revised: Initialization.

Original: To start STCD at a given client k 0 , we need to ensure that z 0 which, as we saw,contains all the information needed to do a (proximal) CD step, and θ 0 verify z 

Revised: To start STCD at a given client k 0 , we need to ensure that z 0 which, as we saw, contains all the information needed to do a (proximal) CD step, and θ 0 verify z

Original: = Xθ 0 .
Revised: = Xθ 0 .

Original: To do thiswithout requiring any information from the other clients, we must initialize θ 0 = 0 and z 0 = 0 .
Revised: To do this without requiring any information from the other clients, we must initialize θ 0 = 0 and z 0 = 0 .

Original: Updating and communicating the token.
Revised: Updating and communicating the token.

Original: After performing a local (proximal) coordinate descentstep, which gives agent k t the iterate θ t +1 k t , we need to update the token to ensure that z t +1 =
Revised: After performing a local (proximal) coordinate descent step, which gives agent k t the iterate θ t +1 k t , we need to update the token to ensure that z t +1 =

Original: Xθ t +1 .
Revised: Xθ t +1 .

Original: Observing that z t allows for a local update z t +1 =
Revised: Observing that z t allows for a local update z t +1 =

Original: z t + X k t
Revised: z t + X k t

Original: θ t +1 k t − θ tk t ) whose result can then besent to agent k t
Revised: θ t +1 k t − θ tk t ) whose result can then be sent to agent k t

Original: +1 , we see that, by induction, z t can be kept up-to-date throughout our algorithm.
Revised: +1 , we see that, by induction, z t can be kept up-to-date throughout our algorithm.

Original: Additionally, this also means that an agent can actually perform Q multiple coordinate descent stepsw.r.t.
Revised: Additionally, this also means that an agent can actually perform Q multiple coordinate descent steps w.r.t.

Original: its local parameters, θ k .
Revised: its local parameters, θ k .

Original: In essence, STCD is a technique allowing for Markov Chain Coordinate Descent [31] to be performedin feature-distributed setups.
Revised: In essence, STCD is a technique allowing for Markov Chain Coordinate Descent [32] to be performed in feature-distributed setups.

Original: In terms of the progress made in the parameter space, Algorithm 1differs from [31] only in that it requires initializing θ 

Revised: In terms of the progress made in the parameter space, Algorithmdiffers from [32] only in that it requires initializing θ

Original: = 0 and in that we allow for Q > 1 .
Revised: = 0 and in that we allow for Q > 1 .

Original: Let f be a function that is L g
Revised: Let f be a function that is L g

Original: -smooth
Revised: -smooth

Original: and block L -smooth for all blocks k , with a nonempty setof minimizers, and let r be a separable closed proper function.
Revised: and block L -smooth for all blocks k , with a nonempty set of minimizers, and let r be a separable closed proper function.

Original: It is shown in [31] that, for Q = 1 ,
Revised: It is shown in [32] that, for Q = 1 ,

Original: Algorithm 1 converges almost surely to a solution of (1).
Revised: Algorithm 1 converges almost surely to a solution of (1).

Original: 2.3 Limitations
Revised: 2.3 Limitations

Original: The decentralized token algorithm in this section has an appealing simplicity.
Revised: The decentralized token algorithm in this section has an appealing simplicity.

Original: However, while itoutperforms state-of-the-art FD learning algorithms in a variety of setups, as we will see in Section 4,its performance deteriorate faster with network connectivity than these decentralized consensus-basedalgorithms.
Revised: However, while it outperforms state-of-the-art feature-distributed learning algorithms in a variety of setups, as we will see in Section 4, its performance deteriorate faster with network connectivity than these decentralized consensus-based algorithms.

Original: Yet, this simple algorithm acts as a stepping stone to the more general multi-tokenalgorithm we now present in Section 3, which mitigates this problem.
Revised: Yet, this simple algorithm acts as a stepping stone to the more general multi-token algorithm we now present in Section 3, which mitigates this problem.

Original:  Multi-token coordinate descent (MTCD)
Revised:  Multi-token coordinate descent (MTCD)

Original: In this section, instead of a fully decentralized setup, we deal with SDFL and generalize the algorithmused in the previous section in a direction allowing us to leverage the existence of a server.
Revised: In this section, instead of a fully decentralized setup, we deal with SDFL and generalize the algorithm used in the previous section in a direction allowing us to leverage the existence of a server.

Original: 3.1 Problem statement
Revised: 3.1 Problem statement

Original: In this section, the set of clients and the data and model partitioning is similar to Section 2.
Revised: In this section, the set of clients and the data and model partitioning are similar to Section 2.

Original: However,we now consider a server-enhanced setup (that is, SDFL), illustrated in Figure 1, where we also take into account a central server with links to all the clients.
Revised: However, we now consider a server-enhanced setup (that is, SDFL), illustrated in Figure 1, where we also take into account a central server with links to all the clients.

Original: In this section, we do not require the graphdescribing the communications between the clients to be connected.
Revised: In this section, we do not require the graph describing the communications between the clients to be connected.

Original: Considering the same model (1) as in the previous section, we now develop an algorithm to leveragethe existence of this server, exploiting it to mitigate the deterioration of the performance of Algorithm 1in poorly connected networks.
Revised: Considering the same model (1) as in the previous section, we now develop an algorithm to leverage the existence of this server, exploiting it to mitigate the deterioration of the performance of Algorithmin poorly connected networks.

Original: 3.2 Algorithm
Revised: 3.2 Algorithm

Original: The algorithm introduced in this section can be seen as having two parts.
Revised: The algorithm introduced in this section can be seen as having two parts.

Original: The roaming part is similarto the procedure in Algorithm 1, except for the fact that we run multiple tokens in parallel, each withan associated model.
Revised: The roaming part is similar to the procedure in Algorithm 1, except for the fact that we run multiple tokens in parallel, each with an associated model.

Original: The novelty lies in the averaging part, where we exploit the access to the serverby periodically averaging these tokens there.
Revised: The novelty lies in the averaging part, where we exploit the access to the server by periodically averaging these tokens there.

Original: More precisely, we use Γ tokens z γ ∈ R N ,
Revised: More precisely, we use Γ tokens z γ ∈ R N ,

Original: γ ∈
Revised: γ ∈

Original: [Γ] , each with an associated model instance θ γ andz γ (cid:44) Xθ γ ∈ R N .
Revised: [Γ] , each with an associated model instance θ γ and z γ (cid:44) Xθ γ ∈ R N .

Original: Similarly to what we had in Algorithm 1, we initialize θ 0 γ = 

Revised: Similarly to what we had in Algorithm 1, we initialize θ 0 γ =

Original: and z 0 γ = 0 ,γ ∈ [Γ] , and for S × Q iterations, each such token undergoes a process similar to the one in
Revised: and z 0 γ = 0 , γ ∈ [Γ] , and for S × Q iterations, each such token undergoes a process similar to the one in

Original: Algorithm 1.
Revised: Algorithm 1.

Original: The difference lies in the fact that now, every S hops of the tokens, between each ofwhich each agent performs Q local steps, the Γ tokens are averaged at the server and the Γ modelsare averaged locally, at the clients.
Revised: The difference lies in the fact that now, every S hops of the tokens, between each of which each agent performs Q local steps, the Γ tokens are averaged at the server and the Γ models are averaged locally, at the clients.

Original: Crucially, these averaging operations allow us to preserve therelationship deﬁning Z tγ .
Revised: Crucially, these averaging operations allow us to preserve the relationship deﬁning Z tγ .

Original: (This relationship is also preserved during the roaming part, as explained inthe previous section.)
Revised: (This relationship is also preserved during the roaming part, as explained in the previous section.)

Original: This method, a parallel Markov Chain Coordinate Descent for feature-distributed SDFL setups, issummarized in Algorithm 2, where k tγ denotes the agent holding token γ at time t .
Revised: This method, a parallel Markov Chain Coordinate Descent for feature-distributed SDFL setups, is summarized in Algorithm 2, where k tγ denotes the agent holding token γ at time t .

Original: Recovering client-server and decentralized setups.
Revised: Recovering client-server and decentralized setups.

Original: If no client-server communications areavailable ( S → ∞ ) our algorithm is reduced to a decentralized one.
Revised: If no client-server communications are available ( S → ∞ ) our algorithm is reduced to a decentralized one.

Original: In this setting, even if at a lowerrate, our algorithm will still converge, as long as G is connected, being reduced to Γ simultaneous runsof Algorithm 1.
Revised: In this setting, even if at a lower rate, our algorithm will still converge, as long as G is connected, being reduced to Γ simultaneous runs of Algorithm 1.

Original: While a decentralized asynchronous token averaging is possible (by exploring therandom intersections of multiple tokens are at the same client), this is signiﬁcantly less effective thanserver averaging in terms of mitigating the performance deterioration caused by poor connectivity.
Revised: While a decentralized asynchronous token averaging is possible (by exploring the random intersections of multiple tokens are at the same client), this is signiﬁcantly less effective than server averaging in terms of mitigating the effect of poor connectivity.

Original: Incontrast, if the set of edges E is empty and we assign a token per agent, we recover the client-serversetting.
Revised: In contrast, if the set of edges E is empty and we assign a token per agent, we recover the client-server setting.

Original: Extension to locally nonlinear models.
Revised: Extension to locally nonlinear models.

Original: We can generalize our model while keeping an additivestructure w.r.t.
Revised: We can generalize our model while keeping an additive structure w.r.t.

Original: { X k , θ k } , capturing nonlinearities between X k and θ k .
Revised: { X k , θ k } , capturing nonlinearities between X k and θ k .

Original: Letting h k : R | θ k |× d k → R E ,we would minimize:
Revised: Letting h k : R | θ k |× d k → R E , we would minimize:

Original: [Equation]
Revised: [Equation]

Original: which has generalized linear models as the particular case h k ( θ k , x nk )
Revised: which includes generalized linear models as the particular case h k ( θ k , x nk )

Original: = (cid:104)
Revised: = (cid:104)

Original: θ k , x nk (cid:105) , 
Revised: θ k , x nk (cid:105) , where

Original: where E = 1and | θ k | = | x nk | =
Revised: E = 1 and | θ k | = | x nk | =

Original: d k .
Revised: d k .

Original: However, this would require additional communications, since a separateaveraging of θ t and z t would no longer preserve their relation, leading to a need to recompute
Revised: However, this would require additional communications, since γ ∼ U ( ¯ N k tγ ) , where ¯ N (cid:44) N k tγ ∪ { k tγ } .

Original: Initialize θ 0 γ = 0 , z 0 γ = 0 , and arbitrary k 0 γ , for γ ∈ [Γ] , and choose step-size µ for t ≥ 0 do if t mod Q = 0 and (cid:98) t/Q (cid:99) mod S = 

Revised: Initialize θ 0 γ = 0 , z 0 γ = 0 , and arbitrary k 0 γ , for γ ∈ [Γ] , and choose step-size µ for t ≥ 0 do if t mod Q = 0 and (cid:98) t/Q (cid:99) mod S =

Original: then clients send z tγ to the server server computes z t 1 , . .
Revised: then clients send z tγ to the server server computes z t 1 , . .

Original: z t Γ = 1Γ (cid:80) Γ γ =

Revised: z t Γ = 1Γ (cid:80) Γ γ =

Original: z tγ server sends updated z tγ to the clients for k ∈ [ K ] in parallel do θ 1 k , . .
Revised: z tγ server sends updated z tγ to the clients for k ∈ [ K ] in parallel do θ 1 k , . .

Original: , θ Γ k = 1Γ (cid:80) Γ γ =1 θ γk end for end if if t mod Q = 0 then for γ ∈ [Γ] in parallel do
Revised: , θ Γ k = 1Γ (cid:80) Γ γ =1 θ γk end for end if if t mod Q = 0 then for γ ∈ [Γ] in parallel do

Original: Agent k tγ sends z t +1 γ to agent k t +1 γ ∼ U ( N k tγ ) end for else k t +1 γ
Revised: Agent k tγ sends z t +1 γ to agent k t +1 γ ∼ U ( N k tγ ) end for else k t +1 γ

Original: =
Revised: =

Original: k tγ , γ ∈ [Γ] end if for γ ∈ [Γ] in parallel do θ t +1 γk = (cid:40) proxµr k
Revised: k tγ , γ ∈ [Γ] end if for γ ∈ [Γ] in parallel do θ t + [Equation] = (cid:40) prox

Original: ( θ tγk − µ ∇ k L ( θ tγ ))
Revised: [Equation] [Equation]

Original: if k = k tγ θ
Revised: L

Original: [Equation]
Revised: [Equation]

Original:  if k (cid:54) =
Revised: ) if k = k tγ θ [Equation] if k (cid:54) =

Original: k tγ { z tγ used when computing ∇ k tγ L ( θ tγ ) }
Revised: k tγ { z tγ used when computing ∇ k tγ L ( θ tγ ) }

Original: z t +
Revised: z t +

Original: [Equation]
Revised: [Equation]

Original: γ
Revised: γ

Original: [Equation]
Revised: [Equation]

Original: end for end for  the tokens, instead of simply updating them in an online manner.
Revised: end for end for a separate averaging of θ t and z t would no longer preserve their relation, leading to a need to recompute the tokens, instead of simply updating them in an online manner.

Original: This would require all clients tocommunicate with the server.
Revised: This would require all clients to communicate with the server.

Original:  Experiments
Revised:  Experiments

Original: In this section, we compare STCD to Dual Consensus Proximal Algorithm (DCPA) [4], a state-ofthe-art decentralized method used as a baseline, and to MTCD.
Revised: In this section, we compare STCD to Dual Consensus Proximal Algorithm (DCPA) [1], a state-ofthe-art decentralized method used as a baseline, and to MTCD.

Original: We use CVXPY [34] to obtain f (cid:63) ,which we use to compute the suboptimality gap f ( θ t ) − f (cid:63) throughout our experiments.

Original: Models and datasets.
Revised: Models and datasets.

Original: We perform ridge regression on a dataset generated following the sameprocess as [4], 2 but 10 times larger both in number of samples and in dimensionality ( N = 1000 andd = 2000 ).
Revised: We perform ridge regression on a dataset generated following the same process as [1], 2 where the number of samples and the dimensionality are N = 1000 and d = 2000 , respectively.

Original: We have the following objective function, with α = 10 :
Revised: We have the following objective function, with α = 10 :

Original: [Equation]
Revised: [Equation]

Original: We also perform binary classiﬁcation on the GisetteDigit recognition (training) dataset [35], where N = 6000 and d = 5000 .
Revised: We also perform sparse logistic regression on the Gisette dataset [12], where N = 6000 and d = 5000 .

Original: Let s ( z ) (cid:44) (1 + e − z ) − 1 , we do sparse logistic regression, with β = 1 :
Revised: Letting s ( z ) (cid:44) (1 + e − z ) − 1 , we consider the following objective function, with β = 1 :

Original: [Equation]
Revised: [Equation]

Original: Metrics.
Revised: Metrics.

Revised: We use CVXPY [8] to obtain f (cid:63) , and then compute the suboptimality gap f ( θ t ) − f (cid:63) throughout our experiments.

Original: In the previous sections, we considered each local CD step as an STCD iteration, to avoidthe visual clutter of having a double counter (for hops and local CD updates).
Revised: In the previous sections, we considered each local CD step as an STCD iteration, to avoid the visual clutter of having a double counter (for hops and local CD updates).

Original: In this section, to allowfor a fairer comparison with DCPA, where only communication rounds count as a round (despite thefact that we need to compute a proximal operator with a gradient descentsubroutine), we count as an
Revised: In this section, to allow for a fairer comparison with DCPA, where only communication rounds count as a round (despite the fact that we need to compute a proximal operator with a gradient descent

Original: STCD iteration each hop in the network.

Original: We use the same deﬁnition of iteration for MTCD.

Original: [Figure]
Revised: [Figure]

Original: Comparison with state-of-the-art.
Revised: Comparison with state-of-the-art.

Original: For ridge regression, we randomly generate a graph as in [36],with connectivity ratio p = 0 .
Revised: For ridge regression, we randomly generate a graph as in [31], with connectivity ratio p = 0 .

Original: 4 , and a chain graph, both with K = 20 nodes and d k = 100 for all k .
Revised: 4 , and a chain graph, both with K = 20 nodes and d k = 100 for all k .

Original: For STCD, we use µ = 10 − 5 and Q = 20 .
Revised: For STCD, we use µ = 10 − 5 and Q = 20 .

Original: For DCPA, we use µ w = 0 .
Revised: For DCPA, we use µ w = 0 .

Original: 01 , µ y = 0 .
Revised: 01 , µ y = 0 .

Original: 0003 , µ x = 0 .
Revised: 0003 , µ x = 0 .

Original: For sparse logistic regression, we similarly generate a random graph and chain graph, both with
Revised: For sparse logistic regression, we similarly generate a random graph and chain graph, both with

Original: K = 20 nodes (now d k = 250 for all k ).
Revised: K = 20 nodes (now d k = 250 for all k ).

Original: For STCD, we use µ = 10 − 4 and Q = 30 .
Revised: For STCD, we use µ = 10 − 4 and Q = 30 .

Original: For DCPA, weuse µ w = 0 .
Revised: For DCPA, we use µ w = 0 .

Original: 001 , µ y = 0 .
Revised: 001 , µ y = 0 .

Original: 00003 , µ x = 0 .
Revised: 00003 , µ x = 0 .

Original: In Figure 2, we see that, while STCD does not improve upon DCPA in terms of progress per iteration,it signiﬁcantly outperforms it in terms of communication efﬁciency.
Revised: In Figure 2, we see that, while STCD does not improve upon DCPA in terms of progress per iteration, it signiﬁcantly outperforms it in terms of communication efﬁciency.

Original: Yet, we can also see that STCDis particularly vulnerable to poorly connected networks, as evidenced by a performance deteriorationwhen going from a random network to a chain network.
Revised: Yet, we can also see that STCD is particularly vulnerable to poorly connected networks, as evidenced by a performance deterioration when going from a random network to a chain network.

Original: [Figure]
Revised: [Figure]

Original: Comparing STCD with MTCD.
Revised: Comparing STCD with MTCD.

Original: We again deal with the ridge regression problem above andresort to the same dataset.
Revised: We again deal with the ridge regression problem above and resort to the same dataset.

Original: However, we now focus on chain graphs (for MTCD, complemented byclient-server communications), as they are poorly connected setups where the drawback of STCD isevident.
Revised: However, we now focus on chain graphs (for MTCD, complemented by client-server communications), as they are poorly connected setups where the drawback of STCD is evident.

Original: We use µ = 10 − 5 and Q = 20 for both STCD and MTCD and use Γ = 10 , S = 10 for thelatter throughout all three experiments, varying only the number of nodes K ∈ { 20 , 40 , 80 } .
Revised: We use µ = 10 − 5 and Q = 20 for both STCD and MTCD and use Γ = 10 , S = 10 for the latter throughout all three experiments, varying only the number of nodes K ∈ { 20 , 40 , 80 } .

Original: In Figure 3, we see that, while both algorithms see a drop in performance for graphs with a lowerconnectivity, MTCD mitigates this effect, consistently outperforming STCD.
Revised: In Figure 3, we see that, while both algorithms see a drop in performance for graphs with a lower connectivity, MTCD mitigates this effect, consistently outperforming STCD.

Original:  Conclusions
Revised:  Conclusions

Original: We formalize the multi-token SDFL scheme and propose a communication efﬁcient SDFL algorithmfor feature-distributed data.
Revised: We formalize the multi-token SDFL scheme and propose a communication-efﬁcient SDFL algorithm for feature-distributed data.

Original: Numerical results show the improved communication efﬁciency of ouralgorithm as well as the power of endowing decentralized methods with periodical client-servercommunications.
Revised: Numerical results show the improved communication efﬁciency of our algorithm as well as the power of endowing decentralized methods with periodical client-server communications.

Original: A natural extension to this work is a detailed study of how the number of tokens andthe frequency of their averaging inﬂuences the convergence rate, both empirically and analytically.Checklist
Revised: A natural extension to this work is a detailed study of how the number of tokens and the frequency of their averaging inﬂuence the convergence rate, both empirically and analytically.

Original: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope? [Yes] (b) Did you describe the limitations of your work? [Yes]
Revised: Acknowledgments and Disclosure of Funding

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A](d)

Original: Have you read the ethics review guidelines and ensured that your paper conforms to them?

Original: [Yes]2.

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A](b)

Original: Did you include complete proofs of all theoretical results?

Original: [N/A]

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimentalresults (either in the supplemental material or as a URL)?

Original: [No] The code and data will be madeavailable in a later submission.

Original: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters, how they were chosen)?

Original: [Yes]

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experimentsmultiple times)?

Original: [No]

Original: (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,internal cluster, or cloud provider)?

Original: [No]

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes]

Original: (b) Did you mention the license of the assets?

Original: [No]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [No] (d) Did you discuss whether and how consent was obtained from people whose data you’re us- ing/curating? [N/A]
Revised: This work is supported in part by the Fundação para a Ciência e a Tecnologia through the Carnegie

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: informa-

Original: tion

Original: or offensive content? [N/A]5. If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?[N/A](b
Revised: Mellon Portugal Program; by the grant U.S. National Science Foundation CCF-2007911; and by the

Original: Did you describe any potential participant risks, with links to Institutional Review Board (IRB)approvals, if applicable? [N/A](c) Did you include the estimated hourly wage paid to participants and the total amount spent onparticipant compensation? [N/A]
Revised: CMU-Portugal project CMU/TIC/0016/2021.