LightVeriFL : Lightweight and Verifiable Secure Federated Learning
Abstract
Secure aggregation protocols are implemented in federated learning to protect the local models of the participating users so that the server does not obtain any information beyond the aggregate model at each iteration.
However, existing secure aggregation schemes fail to protect the integrity, i.e., correctness, of the aggregate model in the possible presence of a malicious server forging the aggregation result, which motivates the need for verifiable aggregation in federated learning.
Existing verifiable aggregation schemes either have a complexity that linearly grows with the model size or require time-consuming reconstruction at the server, that is quadratic in the number of users, in case of likely user dropouts.
To overcome these limitations, we propose LightVeriFL , a lightweight and communication-efficient secure verifiable aggregation protocol, that provides the same guarantees for verifiability against a malicious server, data privacy, and dropout-resilience as the state-ofthe-art protocols without incurring substantial communication and computation overheads.
The proposed LightVeriFL protocol utilizes homomorphic hash and commitment functions of constant length, that are independent of the model size, to enable verification at the users.
In case of dropouts, LightVeriFL uses a one-shot aggregate hash recovery of the dropped users, instead of a one-by-one recovery based on secret sharing, making the verification process significantly faster than the existing approaches.
We evaluate LightVeriFL through experiments and show that it significantly lowers the total verification time in practical settings.
Introduction
Federated learning (FL) is a distributed learning paradigm proposed to address the growing concerns about user data privacy in distributed learning systems [1].
In FL, a group of users jointly train a global model without sending their local data to a central server (see Fig. 1(a)).
Even though user datasets stay private, local models sent by the users can potentially cause data leakage from the users [2, 3, 4, 5].
Secure aggregation frameworks have been implemented to protect the users’ individual local models as well as tolerate likely dropouts in FL [6, 7, 8, 9, 10].
Secure aggregation schemes hide individual local models from the server, which only learns the aggregate model.
Despite their benefits in protecting the user models, none of these secure aggregation schemes enable the users to verify the correctness of the aggregate model received from the server at each iteration.
The typical FL framework, even in the presence of secure aggregation schemes, is prone to a malicious server forging the aggregation results for its own benefit or a lazy server sending incorrect results
[Figure]
Incorrect aggregation
[Figure]
to reduce its computation cost (see Fig. 1(b)).
Since the users are oblivious to the aggregation procedure, such incorrect computations at the server can potentially alter the learning procedure.
From a trustworthiness standpoint, without verifiable aggregation, users cannot make sure whether their contributions are included in the global model, also motivating the study of verifiability in FL.
Related Works.
In [11], the server generates a proof for the aggregation computation to enable verification at the users.
The proof utilized in [11] has a communication overhead that is linear in the model size, which makes it impractical in modern FL systems with hundreds of thousands of parameters.
Authors in [12] propose a communication-efficient verifiable aggregation scheme, which utilizes hashes of the local model updates of the users for verification at the expense of huge computation overhead in the presence of dropouts (see Appendix A for further discussion).
In this work, we propose a lightweight verifiable aggregation scheme that provides the same guarantees for input privacy, dropout resilience and verifiability in the presence of a malicious server as the state-of-the-art protocols without incurring substantial computation and communication overheads.
Contributions.
The main bottleneck in [12] is that the server recovers the hashes of the dropped users one-by-one in the verification stage (see Appendix C for details).
These hashes are later utilized by the users to check the integrity of the aggregate model.
In large systems with frequent dropouts, this one-by-one reconstruction incurs a significant quadratic burden on the verification time.
In this work, inspired by the design of LightSecAgg [8], we propose a verifiable aggregation scheme named
LightVeriFL for faster verification in the presence of dropouts in FL systems.
The proposed LightVeriFL scheme utilizes linearly homomorphic hashes of the local models of the users for verification.
As shown in Fig.
2, after the local training, each user generates a hash, which is protected by a random mask generated by the respective user.
These masks are encoded carefully such that once received sufficient responses from the users, the server is able to generate the aggregate hash of all participating users in one-shot (as opposed to one-by-one reconstruction of [12]).
That is, even if certain number of users drop in the verification stage, the server is able to recover the aggregate hash of all users that have sent model updates in that iteration.
Finally, enabled by the linear homomorphism of the hashes, each surviving user verifies the integrity of the aggregation.
Key features of the proposed LightVeriFL protocol are listed as follows:
(1) Verifiability and trustworthiness.
With LightVeriFL users make sure that i) their contributions are reflected in the global model and ii) the aggregation result provided by the server is exact.
(2) Compatibility with the existing secure aggregation schemes.
LightVeriFL is compatible with federated averaging-based secure aggregation schemes, e
g., SecAgg [6], LightSecAgg [8].
(3) Input privacy and dropout resilience.
LightVeriFL does not leak any private information of the users and is resilient to dropouts during the verification procedure.
Combined with a secure aggregation scheme, LightVeriFL forms a secure verifiable aggregation scheme and guarantees input privacy and dropout resilience both in model aggregation and aggregate model verification.
(4) One-shot aggregate hash recovery at the server.
In LightVeriFL , the server is able to recover the aggregate hash of the participating users (whose models are included in the aggregation) in one-shot even in the presence of dropouts thanks to the employed mask encoding strategy.
[Figure]
(5) Reduced overheads and faster verification.
With the one-shot aggregate hash recovery, computation overhead at the server drastically decreases in case of dropouts compared to [12], which results in much faster verification, also confirmed by our empirical results for realistic model sizes, particularly in systems with large number of users.
Our experiments indicate that the primary source of this gain is the complexity reduction at the server.
(6) A novel encoding scheme.
Existing encoding strategies for secure aggregation aim at recovering the sum of the individual models.
In the case of linear homomorphic hashes, one needs the product of the hashes of the users for verification.
Inspired by the encoding strategy in [8], we propose a novel encoding strategy that utilizes elliptic curves to recover the “aggregate" product of the user hashes.
Notation.
Z p denotes the ring of integers modulo p .
We use Z ∗ p to denote all invertible elements of
Z p , i.e., Z ∗ p =
Z p \{ 0 } .
G denotes a cyclic group of order q .
We have [ N ] ≜ { 1 , 2 , . . .
, N } .
Preliminaries
Linear homomorphic hash.
Following [12, 13], we let G be a cyclic group with prime order q and generator g .
Given d distinct elements g 1 , g 2 , . . .
, g d , the hash of a model x ∈ F dq is given by h ( x ) ←− d (cid:89)
j = g x [
[Equation]
where x [ j ] denotes the j th element of the model vector x .
The hash construction in (1) has collision resistance [13] such that it is computationally infeasible to find two distinct vectors x 1 , x 2 ∈ F dq that satisfy h ( x 1 )
=
h ( x 2 ) .
The resulting hash in (1) is of constant length independent of d and satisfies an additivity (in exponent) property for x 1 , x 2 ∈ F d q such that h ( x 1 + x 2 )
=
h ( x 1 ) h ( x 2 ) .
Commitment.
A commitment scheme COM, takes an input message h and uniform randomness r to produce a commitment string c such that c =
COM.Commit ( h, r ) .
When it is time to decommit, i.e., reveal the hidden committed value, the committer sends the claimed committed message h ′ and the claimed committed randomness r ′ to the interested party, which then checks c
= COM.Commit ( h ′ , r ′ ) .
If this holds, the interested party accepts the committed value h .
A well-known commitment scheme is the Pedersen commitment scheme [14].
Given a subgroup G of Z ∗ p of order q , with p = 2 q + 1 , in the Pedersen commitment scheme, the committer sends c = g h t r , where g is the generator of the subgroup and t is selected such that t = g a with a unknown to the receiver.
Here,
r ∈
Z ∗ p is randomly selected and called the blinding factor.
Pedersen commitment schemes are perfectly hiding and computationally binding [14] and are additively homomorphic such that for commitment pairs ( h 1 , r 1 ) and ( h 2 , r 2 ) , we have c ( h 1 + h 2 , r 1 + r 2 )
=
c ( h 1 , r 1 ) c ( h 2 , r 2 ) .
In LightVeriFL , we use a variation of the original Pedersen commitment scheme (see Appendix D) and utilize its homomorphic property to verify the integrity of the hashes reconstructed by the server.
Elliptic curve (EC).
An EC over F p , denoted by E ( F p ) , consists of points P =
( x, y ) , x, y ∈ F p that satisfy y
= x 3 + ax + b , together with the point at infinity O [15].
p > 3 is an odd prime and a, b ∈ F p satisfy 4 a 3 + 27 b 2 ̸ = 0 .
Two operations are defined on ECs: point addition and scalar multiplication.
Given an integer k , the scalar multiplication kP corresponds to adding point P to itself k times and is analogous to the exponentiation operation in multiplicative groups.
ECs are well-suited for resource constrained environments such as FL since significantly smaller parameters are required to achieve the same level of security, compared to the classical public-key schemes [15].
Problem Setting 3.1 Federated Learning
FL is a distributed learning framework, in which a global model x of dimension d is jointly trained by a group of users on their own privately held datasets D i .
The FL framework aims to minimize the global loss function
L ( x ) = 1 N (cid:80) Ni
=
L i ( x ) , where L i ( x ) denotes the local loss function of the i th user and without loss of generality, |D i | = n for all i ∈
[ N ] .
Training in FL is an iterative process.
At each iteration, the server sends out the current global model x ( t ) to the participating users.
Each user i updates its local model x i ( t ) and sends it to the server.
We let U a ( t ) denote the surviving users at iteration t during aggregation.
The server aggregates the results with x ( t +1) =
[Equation]
(cid:80)
[Equation]
x
[Equation]
t ) and pushes the updated global model, x ( t + 1) back to the users for the next iteration.
3.2 Threat Model and Privacy & Verifiability Guarantees
All users and the server are honest but curious.
Up to T of the users can collude with each other as well as the server to obtain information on the inputs of the honest users.
Corrupted parties follow the protocol and report their local models honestly but they may try to infer the local models of the honest users.
Additionally, we allow a corrupted server to forge the aggregation results arbitrarily in an effort to convince the users of a wrong aggregation result.
Thus, the goal is to protect the confidentiality of the user inputs as well as give each user the capability of verifying the integrity of the server aggregation.
3.3 Dropout Resilience
In FL, users may sometimes drop from the protocol execution due to communication/connection issues, battery problems etc.
The proposed secure verifiable aggregation protocol should be resilient to these random dropouts.
We assume that at most D users drop during the verification protocol such that we have at least N − D surviving users that want to verify the aggregation.
Since the existing secure aggregation schemes [6, 7, 8] provide resilience for dropouts occurring during model aggregation, in this work, we focus on the dropouts occurring during the verification of the aggregate model.
Thus, we want the proposed protocol to tolerate D dropouts such that the remaining N − D clients can correctly verify the integrity of the aggregation that includes local models of N users.
Goal.
We want to design a lightweight and communication-efficient verifiable aggregation protocol that simultaneously provides input privacy against T = N 2 colluding users and resilience to D = N 2 −dropouts as well as a verifiability guarantee in the presence of a malicious server spoofing the aggregation results.
The proposed verifiable aggregation protocol should be compatible with the existing secure aggregation protocols to protect the confidentiality of the users’ local models.
Overview of the LightVeriFL Protocol
Our protocol utilizes certain cryptographic primitives as in [6, 8, 11, 12] so that all the operations are performed over a finite field.
In order to implement LightVeriFL , users perform the following additional operations during an FL iteration (for the detailed description, complexity analysis, and the pseudo code of LightVeriFL see Appendices D, G, and E, respectively).
During the aggregation phase.
Each user i i) generates a mask
z i , encodes it according to (8) and shares the encoded mask with the other users, ii) generates its hash h i based on its local model
x i according to (1), iii) commits its hash h i and exchanges its commitment
c
i with the other users, iv) sends its masked hash
˜ h
i =
h
i
+ z
i to the server.
At the end of the aggregation phase, the server recovers the aggregate model y and sends it back to the users.
After the aggregation phase.
Upon the aggregation phase, some users may drop and the verification is performed by the surviving users.
Each of the surviving users sends the encoded masks it has received from the other users (surviving and dropped) to the server for reconstruction.
Server reconstructs the aggregate mask and recovers the aggregate hashes of the surviving and dropped users (decommitting round).
Then, each surviving user i) verifies the correctness of the recovered hashes coming from the server using the commitments it has received from every other user during the aggregation phase, ii) verifies the integrity of the aggregate model y by computing its hash and comparing it against the aggregate of the individual hashes of the users, reconstructed by the server and accepted by the user in step i) (batch checking round).
If a user encounters an incorrect result either in step i) or ii) during the verification phase, it regards the result as forged and rejects the aggregation result y computed by the server in that iteration.
Improved LightVeriFL with amortized verification:
In LightVeriFL , the most time-consuming operation is the two hash computations over a model of dimension d .
First, in the aggregation phase, each user computes the hash of its local model.
Then, in the batch checking round, each user computes the hash of the aggregate model.
The former hash computation is necessary, however, the latter one can be amortized in order to cut down the computational overhead of the entire scheme.
For this, we implement the amortized verification technique [12] with batch size L such that users verify aggregations of the past L iterations all at once by performing a single hash computation using (13).
Theoretical Guarantees
In this section, we show the correctness of the verification along with input privacy guarantee and dropout resilience in LightVeriFL .
Proofs are deferred to Appendix F.
Theorem 1.
[Correctness of Verification] Under the LightVeriFL scheme, users accept the aggregation results y ( ℓ ) , ℓ ∈
[ L ] if and only if these results are correctly aggregated by the server with probability (almost) 1 .
Theorem 2.
[Input Privacy Guarantee]
The proposed LightVeriFL protocol provides input privacy against up to T colluding users.
Theorem 3.
[Dropout Resilience in Verification]
The proposed LightVeriFL scheme guarantees dropout resilience up to any D dropped users during the verification phase such that N ≥ T + D +1 .
Theorem 4.
The proposed LightVeriFL scheme guarantees successful aggregation integrity verification in the presence of any D user dropouts during the verification phase without sacrificing input privacy against up to any T colluding users for T + D < N .
When LightVeriFL is implemented together with a secure aggregation scheme, a secure verifiable aggregation scheme is obtained.
Experimental Results 6.1 Experimental Setup
Implementation.
We implement the linearly homomorphic hash as well as the commitments using the NIST P-256 elliptic curve [16].
This curve has a 256 -bit subgroup order n .
We fix q to 2 31 −such that all user models lie in F dq .
As in [12], we simulate the clients and the server on our in-house 64-bit Ubuntu 20.04.2 LTS machine equipped with AMD EPYC 7502 CPU.
Baseline.
We use the VeriFL [12] scheme described in Appendix C as our baseline.
We do not consider other aggregation verification methods such as [11] and [17] as baselines since these works are not communication-efficient, i.e., the required communication scales with the model size d , and/or do not support user dropouts.
In the experiments, we implement both VeriFL and LightVeriFL on top of SecAgg and utilize amortized verification in both schemes with L = 10 .
Number of users and dropout rate.
We have up to N = 200 users in our experiments.
For the dropped users, we consider a worst-case scenario for verification, in which we assume all N users successfully participated in the model aggregation and some pN portion of these users drop in the verification phase.
For this, we artificially drop pN users at each iteration.
Following the observations made in [8] and [18], we take p = 0 .
1 , p = 0 .
3 , and p = 0 .
[Figure]
[Table]
Model size.
By default, we use d = 100 K as our model size in the experiments as in [12] (see
Appendix H for experiment results with varying d )
[Table]
6.2 Performance Analysis
We consider a single training round and measure the total running time of LightVeriFL and VeriFL .
We do not include model training in the results shown in this section.
In Fig.
3 we increase the number of users to N = 200 for varying dropout rates with d = 100 K and L = 10 .
In Fig. 3(a), we see that, unlike LightVeriFL , the total running time of the VeriFL scheme is affected by larger dropouts as its reconstruction complexity increases quadratically with
N .
To better observe this, in Fig. 3(b), we present the running times of the verification phases of the two schemes.
Verification phase in both schemes involves the decommitting and batch checking.
In Fig. 3(b), we observe that while the verification time in the proposed LightVeriFL scheme is largely unaffected by the increasing dropout rates, the verification time in VeriFL significantly increases as N gets larger and as the dropout rate increases.
This is due to the fact that VeriFL performs one-by-one reconstruction of the dropped user hashes whereas in LightVeriFL the server reconstructs the aggregate hash of all users all at once, independent of the dropout rate.
Next, we present the verification time breakdown of the two schemes for different dropout rates with
N = 200 , d = 100 K , and L = 10 in Table 1 as LightVeriFL specifically targets improving the verification time.
In Table 1, we observe that the proposed LightVeriFL protocol achieves up to 9 .
38 × improvement in the verification time (see Appendix H for further discussion).
Conclusion and Future Directions
Unlike the existing verifiable aggregation schemes which suffer from large communication and computation overheads, the proposed LightVeriFL scheme is lightweight and communication- efficient, which are enabled by the use of constant-length hashes for aggregation verification and one-shot aggregate hash recovery at the server instead of one-by-one recovery of the dropped user hashes.
LightVeriFL achieves significantly faster aggregate model integrity verification at the users in the presence of a malicious server forging the aggregation results while guaranteeing the same input privacy and dropout-resilience as the state-of-the-art protocols.
Despite its benefits, limitations still exist.
We plan to improve LightVeriFL in the following aspects: 1) performing end-to-end experiments by also considering model training to investigate the gain achieved by LightVeriFL over baselines, 2) extending LightVeriFL to asynchronous FL, 3) considering the verification of user data and models as well as design of a Byzantine-robust secure verifiable aggregation scheme
Appendix A Other Related Works
Besides [11] and [12], there are other works in verifiable federated learning focusing on verifying the computations of the server and/or the local information and model updates of the users (see the survey in [19]).
Among these, a related work is [17], which uses Lagrange interpolation and the Chinese Remainder Theorem to encrypt the model updates of the users.
The server aggregates the encrypted models, which are then verified by the users.
A major disadvantage of this scheme is that it cannot support user dropouts, which is one of our main considerations in this work.
The aggregate model verification problem that we consider in this work can be cast as a secure multiparty computation in the presence of malicious parties.
The existing techniques in the secure multiparty computation domain utilize garbled circuit-based approaches through Cut-and-Choose [20, 21] and
Commit-and-Prove [22, 23] techniques, which incur significant communication overheads and thus are not suitable for the FL setting.
Secret sharing-based solutions as in [24] require secret sharing the inputs, which again induces a huge communication complexity that depends linearly on the model size.
In the privacy-preserving machine learning literature the main focus is on the input privacy of the users.
Aside from the secure aggregation techniques [6, 7, 8, 9, 10, 25], another privacy-preserving approach is differential privacy [26].
In FL, employing DP usually entails adding artificial noises to the local models of the users before sending them out for aggregation [27, 28, 29, 30].
None of these works in the DP literature enable computation verification at the users beyond achieving input privacy.
We note that clients can still implement local DP in the proposed LightVeriFL scheme.
In LightVeriFL the focus is on the verifiability of the server computations.
Even though
LightVeriFL provides secure verifiable aggregation, it does not verify the integrity of the user inputs.
That is, LightVeriFL cannot detect the malicious inputs of Byzantine users.
In this sense,
Byzantine-robust aggregation literature [31, 32, 33, 34] complements the proposed secure verifiable aggregation scheme.
It is a great open problem to design a verifiable Byzantine-robust aggregation mechanism, considering various different aggregation rules other than federated averaging.
B Overview of Secure Aggregation
In the original FL framework described in Section 3.1, local models of the participating users are received in plain by the server at each iteration.
However, these local models carry significant information about the respective users’ datasets and using certain attacks, e.g., model inversion, private local data-points of the users can be recovered from their models [2, 3, 4, 5].
To remedy this, authors in [6] propose a secure aggregation scheme, named SecAgg , in which the server recovers the aggregate model y ( t ) =
(cid:80) i ∈U a ( t ) x
i ( t ) at each iteration t without obtaining any information about the individual local models x i ( t ) ∈
U a ( t ) .
In SecAgg , the users protect their local models with two separate masks.
The first mask is a pairwise mask that is agreed upon between each pair ( i, j ) of users, i, j ∈ [ N ] .
For this, before the training starts user pair ( i, j ) agrees on a random seed
a
i,j , where a i,j = Key.
Agree ( sk i , pk j ) =
Key.
Agree ( sk j , pk i ) .
Here, sk i and pk i denote the private and public keys of user i , respectively.
Each user i generates another mask based on a private random seed b i .
With these, each user i masks its local model x i and sends the masked version ˜x i to the server, where
[Equation]
where, PRG stands for a pseudo-random generator.
We note that, in (2), the pairwise random masks protect the confidentiality of each user’s local model and cancel out at the server upon aggregation.
When a user i ’s local model is only delayed, but not dropped, in order to prevent privacy breaches during this cancellation, the additional mask based on the private seed b i is used.
Each user i secret shares its private seed b i and private key sk i with the other users via Shamir’s secret sharing [35].
In the aggregation step, the server collects the shares of the dropped users’ private keys as well as the shares of the private seeds of the surviving users to reconstruct the pairwise seeds of each dropped user and the private seed of each surviving user, respectively.
The server then performs the following to obtain the aggregate model y , which is equal to
[Equation]
where U a and D a represent the set of surviving and dropped users at the aggregation stage, respectively.
The SecAgg scheme guarantees local model privacy as long as the number of dropped users at aggregation D a and the number of colluding users T satisfy N − D a > T .
A major performance bottleneck of the SecAgg scheme is the fact that it requires the server to reconstruct the seeds one by one (the private seeds of the surviving users as well as the pairwise seeds of the dropped users), which incurs a computation overhead of O ( N 2 ) .
To remedy this, more efficient secure aggregation schemes have been proposed [7, 8].
In [8], authors propose
LightSecAgg , in which the server recovers the aggregate mask of the surviving users in one-shot, thus overcoming the aforementioned O ( N 2 ) bottleneck.
In LightSecAgg , users still protect their models with local masks but these masks are encoded and shared with the other users such that once the server receives sufficiently many responses from the users, it can reconstruct the aggregate mask of these surviving users and hence recover the aggregate model.
The approach in LightSecAgg constitutes the basis of our approach in designing a lightweight and verifiable aggregation protocol.
Limitations of the secure aggregation schemes.
With secure aggregation alone, users cannot verify whether i) the aggregation result provided by the server is correct and ii) their individual models are accounted for in the aggregation.
Thus, without a verifiable aggregation scheme, users are vulnerable to a malicious/lazy server forging the aggregation result and/or not incorporating local models of all users.
C Overview of the Baseline Protocol: VeriFL
In this section, we give an overview of the existing verifiable aggregation protocols for FL.
The first work on verifiable aggregation is [11].
In that work, authors propose VerifyNet that utilizes a zero knowledge proof technique such that upon aggregation, the server sends a proof to the users indicating the correctness of the aggregation.
Users may accept or reject this proof.
The major bottleneck in this system is the fact that the size of this proof increases linearly with d , the dimension of the model.
Thus, the time required for verification using the zero knowledge proof increases linearly with d , making the VerifyNet scheme impractical for real-life learning models with large number of parameters.
Reference, [12] proposes a verifiable aggregation scheme named VeriFL , which is the baseline scheme we consider in this work.
The VeriFL scheme utilizes homomorphic hash functions of the local models as well as commitments to design a communication-efficient verification scheme.
In
VeriFL , the communication overhead is independent of d , thus making the VeriFL scheme more efficient than VerifyNet .
Below we give a brief overview of the VeriFL scheme.
In VeriFL , training iterations are performed as described in Section 3.1.
The VeriFL scheme starts with a preparation stage.
Before sending out its updated local model, each participating user i generates a linearly homomorphic hash h i of its local model x i .
Based on this hash h
i , user i then generates a commitment
c
i using a commitment scheme COM such that c
i =
COM.Commit ( h i , r i ) , where r i is a uniformly random string privately sampled by user i .
Next, each user i forwards its commitment string c
i to all the other participating users.
Once this step is completed for all user pairs, each user sends its local model to the server.
In VeriFL , in sending the local models to the server for aggregation, users follow the SecAgg protocol described in Appendix B. Once the server recovers the aggregate model y and pushes it to the users, the verification stage of the VeriFL scheme commences with the decommitment step, in which each user i receives the decommitment strings ( h j , r j ) from all the other users j ∈ [ N ] to check if c j received in the preparation step satisfies
[Equation]
j , r j ) .
If (4) is not satisfied for at least one other user j , user i terminates the process and regards the aggregation y as forged.
If no issue is detected at this step for any of the users, user i proceeds and checks the equality of the hashes.
For this, user i first computes the hash of the aggregate model, h agg and then check if the following holds
[Equation]
If (5) does not hold, user i raises a flag and regards the aggregate result as forged, otherwise user i accepts the result and starts generating its updated local model for the next iteration.
We note that thanks to SecAgg , the above VeriFL scheme can tolerate dropouts during aggregation.
However, dropouts occurring during the verification step need to be handled to successfully verify the integrity of the aggregation.
For this, each user i secret shares its decommitment string ( h i , r i ) with the other users in the preparation step.
In the verification step, after decommitting, surviving users send the shares they have received from the dropped users to the server, which recovers the decommitment strings ( h j , r j ) of the dropped users and sends them back to the surviving users.
Equipped with these, along with the decommitment strings of the other surviving users, each surviving user i performs (4) and (5).
Limitations of the VeriFL scheme in [12].
i)
As in the original SecAgg scheme, the VeriFL scheme suffers from the one-by-one reconstruction of the dropped users’ decommitment strings ( h j , r j ) , which are then used by the surviving users in performing the verification through (4) and (5).
This one-by-one reconstruction at the server incurs an O ( N 2 ) computation bottleneck and significantly slows down the system for large N .
For example, when N = 500 and d = 100 K , when the dropout rate is %30 , while the actual verification steps in (4) and (5) take around 5 seconds (majority of which is the hash computation of the aggregate model), the reconstruction of the dropped decommitment strings takes approximately 150 seconds, incurring a significant burden on the verification procedure.
ii)
In VeriFL , the homomorphic hashes of the honest users are revealed in the verification phase (in decommitment for surviving users and during the reconstruction at the server for the dropped users).
Since the homomorphic hash of a model vector is a deterministic function of the inputs, an adversary may use the revealed hash result to guess the local model of an honest user.
Thus, in VeriFL , the local model confidentiality of the honest users may be broken, particularly when the gradient vector has only a few non zero entries [36].
Motivated by these limitations, in this work, we propose LightVeriFL , which forgoes one-by-one reconstruction of the dropped users’ decommitment strings ( h j , r j ) .
Instead, in LightVeriFL , the server performs a one-shot reconstruction of the aggregate decommitment strings of the dropped and surviving users.
By this way, not only we avoid the major O ( N 2 ) computation bottleneck in reconstruction, thus making the verification significantly faster but also avoid revealing the individual hashes of any of the users (dropped or surviving) to any of the parties (users and the server), circumventing the aforementioned privacy breach.
D Detailed Description of LightVeriFL
In this section, we describe the LightVeriFL scheme in detail.
We require the elements of the local models of the users and the aggregate model to lie in F dq , q is a prime number.
The key feature of the LightVeriFL scheme is that the server is able to reconstruct the aggregate hashes of the dropped users in one-shot as opposed to VeriFL [12], which has the server reconstructing the dropped hashes one-by-one.
In LightVeriFL , the server additionally recovers the aggregate hash of the surviving users so that these surviving users never reveal their hashes in plain to other users.
The proposed LightVeriFL scheme has two phases: aggregation phase and verification phase.
In the aggregation phase, additional operations are performed on top of SecAgg to enable verification later on.
The verification phase happens after the users receive the aggregate model from the server.
In order to protect the privacy of the local models, in LightVeriFL , users mask their hashes before sending them to the server.
Then, in the verification phase, they make use of the additive homomorphism of the constructed hashes described in Section 2 to verify the integrity of the aggregation.
That is, after receiving the aggregation, each user needs to be able to check whether the following holds:
[Equation]
where y is the aggregate model obtained from the server and U and D show the set of surviving and dropped users during verification, respectively.
As mentioned earlier, in LightVeriFL , unlike
VeriFL [12], we do not want the server to reconstruct the hashes one-by-one.
Then, in order for users to perform (6), the server needs to reconstruct the product of the hashes of the surviving and dropped users as in LightVeriFL no user sees another user’s hash in plain.
This is when elliptic curves comes into play, which have the following property.
Property 1.
The order of a point P on the EC is the smallest integer n such that nP
= O .
Given
E ( F p ) , one can find a base point P that generates a cyclic subgroup of order n , where n < p is a prime number.
That is, if k and ℓ are integers kP = ℓP if and only if k ≡ l in modulo n [15].
By utilizing this cyclic subgroup property, we perform all encoding and decoding operations in
LightVeriFL on ECs with modulus n .
When the linearly homomorphic hash described in (1) is implemented using an EC 5 , the condition in (6) becomes
[Equation]
using the EC analogues of the operations in multiplicative groups.
That is, in this case the server needs to recover the aggregate hashes of the users analogous to the secure aggregation problem [6, 8].
Thus, when the hashes are constructed over the EC, surviving users need the aggregation of the hashes of all users to perform the verification step in (7).
In this case, existing secure aggregation tools become applicable to the problem at hand.
We note that we implement a variant of the Pedersen commitment scheme on the EC such that c
= αh + βr , where h is hash of a user, denoted by a point on the EC, whereas r is a randomly chosen point on the curve.
Here, α and β are integer coefficients agreed upon by the users before the start of the protocol.
Once the commitments are implemented over the EC, the additive homomorphic property described in Section 2 becomes c ( h 1 + h 2 , r 1 + r 2 ) =
c ( h 1 , r 1 ) + c ( h 2 , r 2 ) .
Formally, the LightVeriFL scheme assumes that out of the N users, at most T of them colludes with each other while D users drops during the verification phase of the LightVeriFL scheme.
Here, we have 0 ≤ T ≤ N − 1 and 0 ≤ D ≤ N − 1 .
We let U denote the targeted number of surviving users during the verification step where N − D ≥
U = T
+ 1 .
We detail the two phases of the LightVeriFL scheme next.
The pseudo code of LightVeriFL is given in Algorithm 1 in
Appendix E.
The Aggregation Phase:
In order to verify the aggregation computed by the server, users perform the following additional operations during the aggregation phase in addition to the implemented secure aggregation protocol, i.e., SecAgg .
Round A.0: Advertising Keys.
LightVeriFL starts with the setup operations of the linearly homomorphic hash and commitment schemes, e.g., fixing an EC with generator g and subgroup order n , setting up d distinct points on the EC for the hash computation in (1), and so on.
The operations in this round do not depend on the local models of the users.
For the SecAgg protocol, during this phase users agree on pairwise masks as well as described in Appendix B.
Round A.1: Offline Encoding and Sharing Local Masks.
This step is inspired by the LightSecAgg scheme in [8] except that all the masks are sampled from an EC, i.e., masks are points on the curve.
First each user i ∈ [ N ] , generates a mask z i , which is a point on the EC.
Next, user i randomly samples jointly uniform points [ n i ] k from the EC for k ∈ { 2 , . . .
U } .
Using these randomly picked [ n i ] k , user i encodes the mask
z i as follows
[Equation] [Equation]
Here, W j is the j th column of the T -private MDS matrix W ∈ F U × N n , where n is the order of the subgroup on EC.
The use of matrix W in encoding protects the generated masks from any subset of colluding T users.
We can always generate such T -private MDS matrix for a given U , N , and T [35, 37, 8].
After the encoding, each user i ∈ [ N ] , sends its encoded mask [˜ z i ] j to user j ∈ [ N ] \{ i } so that in the end of this step, each user i ∈ [ N ] has [˜ z j ] i from all users j ∈ [ N ] .
Round A.2: Hash Generation and Uploading Masked Hashes.
In this step, each user i ∈ [ N ] updates its local model
x i and then based on this model computes its hash h
i 7 according to (1).
Next, each user computes its commitment c i based on its hash according to the aforementioned variant of the Pedersen commitment.
In generation of the commitment, each user uniformly samples a point r i on the EC to be used in decommitting.
Each user i ∈
[ N ] sends its commitment c
i to users j ∈ [ N ] \{ i } such that at the end of this step, each user has commitments of every other user.
Once the commitment exchange is completed, each user i ∈ [ N ] uploads its masked hash ˜
h
i =
h
i + z i and masked randomness ˜ r
i = r
i
+ z
i to the server along with its masked local model that is generated according to the employed secure aggregation scheme.
Round A.3: Aggregate Model Recovery.
The server recovers the aggregate model y and sends it back to the users.
This concludes the LightVeriFL operations during the aggregation phase.
Verification Phase: Having received the aggregate model y at the end of the aggregation phase, users perform the verification phase to check the integrity of the aggregation.
Round V.0:
Aggregate Decommitting.
In this step, the server performs one-shot recovery of the aggregate decommitment strings ( h i , r i ) of the dropped and surviving users.
As mentioned earlier, we let U and D denote the set of surviving and dropped users in the verification stage, respectively.
In order to recover the aggregate mask of the surviving users, (cid:80) i ∈U
z i , each surviving user i is notified to send the aggregate encoded mask it has received from other surviving users, (cid:80) j ∈U [˜ z j ] i .
Upon receiving U such messages, the server is able to decode the aggregate mask of the surviving users (cid:80) i ∈U
z i due to the MDS property of the encoding in (8).
Next, a similar one-shot decoding is repeated for the aggregate mask of the dropped users and the server reconstructs (cid:80) k ∈D z k .
Finally, the server is able to recover the aggregate decommitment strings of the surviving and users with
[Equation]
The same one-shot recovery steps in (9)-(10) are performed for the decommitment strings of the dropped users in D as well.
The server sends these decommitment strings back to the surviving users.
The next step of the users is to verify the integrity of the reconstructions performed by the server.
For this, we utilize the homomorphic additive property of the described commitment scheme such that each user checks
[Equation]
using the commitments received from the users in Round A. 1 during the aggregation phase.
In (11) we have (cid:80) Ni =1 h i =
(cid:80) i ∈U h
i + (cid:80) k ∈D h k .
If (11) does not hold for a user, then that user raises a flag and rejects the aggregate hash recovery performed by the server.
Otherwise, users proceed with the next round.
Round V.1: Batch Checking.
Having accepted the reconstructed hash results received from the server, in this round users verify the correctness of the aggregation result y .
First, each user computes the hash of the aggregate model y , denoted by h agg , using the construction in (1) over the EC.
Next, each user checks if the following condition is satisfied:
[Equation]
If the condition in (12) is satisfied, then users accept the aggregated model and proceed with the next iteration of training.
Otherwise, they regard the aggregate model computed by the server as forged and reject the result.
Improved LightVeriFL with Amortized Verification:
In the amortized verification, Round V. 0 of the verification phase stays the same.
On the other hand, Round V. 1 of the verification phase is only performed at every L iterations, where L is the predetermined batch size of the verification protocol.
That is, during iterations ℓ ∈ [ L ] , each user samples a random coefficient α ℓ and stores the sum of the hashes of the users reconstructed by the server as h ℓ = (cid:80)
Ni =1 h ℓi as well as the aggregate model in the ℓ th iteration y ( ℓ ) .
Then, in the L th iteration, it checks if the following relationship holds:
[Equation]
If (13) holds for a user, then that user verifies the aggregations of the past L iterations all at once by performing only one hash computation in verification as opposed to performing L expensive hash computations during verification, one for each of the L iterations.
E Pseudo Code of LightVeriFL
The pseudo code of LightVeriFL is given in Algorithm 1.
F Proofs of Theoretical Guarantees
Theorem 1.
[Correctness of Verification] Under the LightVeriFL scheme, users accept the aggregation results y ( ℓ ) , ℓ ∈
[ L ] if and only if these results are correctly aggregated by the server with probability (almost) 1 .
Proof.
In the final amortized verification step, as described earlier, each user finds the hash of the weighted sum of the aggregate models from last L iterations, i.e., (cid:80) ℓ ∈
[ L ] α ℓ y ( ℓ ) .
During the verification, the server acts alone without colluding with the users.
During the entire execution of
LightVeriFL , users receive two computation results from the server.
The first one is the aggregate hash reconstruction of the surviving and dropped users.
If the server sends incorrectly reconstructed aggregate hashes in this step, during the aggregate commitment check step in (11) users detect the error in the aggregate hash reconstruction and regard the result as forged.
Assuming that the reconstructed hashes are accepted by the users, the second computation to be verified by the users is the aggregate models y ( ℓ ) , ℓ ∈ [ L ] .
Let us assume, as in [12], that the server successfully spoofs the aggregation result in K iterations, K ⊆
[ L ] and sends ¯y ( k ) instead of y ( k ) , with y ( k ) ̸ = ¯y ( k ) for k ∈ K .
Then, for the users to accept the incorrectly aggregated results, for the following needs to be satisfied
[Equation]
where y ( k )
[ i ] denotes the i the element of the aggregated model, i ∈
[ d ] .
We note that α k s are uniformly selected numbers by each user from elliptic curve subgroup of order n such that the following condition holds
[Equation]
with probability 1 n , even when y ( k ) ̸
= ¯y ( k ) for k ∈ K , i ∈ [ d ] .
Here, since the elliptic curves are designed to have a large subgroup order n , this event is negligible.
9 With that in mind, the
20: uploads
: receives the aggregate model y from the server 26: end for 27: //
Verification Phase 28: identifies set of surviving users U ⊆
[ N ] in the verification phase 29: //
Round V.0:
Aggregate Decommitting 30: for each user i ∈ U in parallel do 31: computes aggregated encoded masks (cid:80) j ∈ [ N ] [˜ z
j ] i 32: uploads aggregated encoded masks (cid:80) j ∈ [ N ] [˜ z j ] i to the server 33: end for 34: collects U messages of aggregated encoded masks (cid:80) j ∈
[ N ] [˜ z j ] i from user i ∈
U 35: // recovers the aggregate mask 36: (cid:80) i ∈
[ N ] z i ← obtained by decoding the received U messages 37: // recovers the aggregate hash and randomness of the users 38: (cid:80) i ∈
[ N ] h i ← (cid:80) i ∈
[ N ] ˜ h
i − (cid:80) i ∈
[ N ] z i and (cid:80) i ∈
[ N ] r i ← (cid:80) i ∈
[ N ] ˜ r i − (cid:80) i ∈
[ N ] z i 39: sends (cid:80) i ∈
[ N ] h i and (cid:80) i ∈
[ N ] r i to U 40: for each user i ∈ U in parallel do 41: receives (cid:80) i ∈
[ N ] h i and (cid:80) i ∈
[ N ] r i from the server 42: // verifies the integrity of (cid:80) i ∈
[ N ] h i and (cid:80) i ∈
[ N ] r i 43: checks COM.Commit (cid:16)(cid:80) i ∈
[ N ] h i , (cid:80) i ∈ [ N ] r i (cid:17) ?
= (cid:80) i ∈
[ N ] c i 44: if above is true, accepts the aggregate hash and randomness recovered by the server and moves to the next round 45: otherwise, regards the result as forged 46: end for 47: //
Round V.1: Batch Checking condition in (14) does not hold due to the collision resistance property of the linearly homomorphic hashes we use as described in (1).
That is, no two different vectors produce the same hash with overwhelming probability such that the aggregate hash verification step in (12) detects any spoofing attempts launched by the server.
Thus, given the server acts independently to forge the aggregation results, the aggregate commitment check in (11) along with the collision property of the hash guarantees verification of the aggregation results provided by the server.
Theorem 2.
[Input Privacy Guarantee]
The proposed LightVeriFL protocol provides input privacy against up to T colluding users.
Proof.
The use of a secure aggregation scheme such as SecAgg [6] or LightSecAgg [8] takes care of the input privacy in the aggregation protocol.
Additionally, we need show that hashes and commitments used in the LightVeriFL protocol do not violate input privacy of the users.
As discussed in [36], if revealed, the linearly homomorphic hashes used in verification schemes can help an adversary guess the input vector of a user, particularly if the input vector has a few non-zero elements.
That is, when the hash of a user is revealed, the distinguisher can tell the difference between the simulated case and the actual case.
Thus, in our scheme, no party (including the server and up to T colluding users) has access to the hash of an honest user.
More formally, the proposed LightVeriFL protocol guarantees the following mutual information condition given in (16) for an arbitrary set T
I 
[Equation]
of T colluding users and a surviving user
set U such that U ⊆
[ N ] , |U| ≥ U , U
= T + 1 .
In order to show (16) we use similar steps as in [8, Proof of Theorem 1].
In addition to the hashes, we need to consider the commitments exchanged during the execution of
LightVeriFL in the Round A. 2 of the aggregation phase as described in Appendix D. During this round, each user receives the commitments of all the other users.
The commitment scheme we use is perfectly hiding.
That is even though an adversarial user, given that it has enough compute resources, can find multiple decommitment pairs leading to the same commitment, that adversarial user has no way of determining the actual hash of another user from the received commitment, beyond random guessing.
Thus, exchanging commitments does not violate the input privacy of users.
Theorem 3.
[Dropout Resilience in Verification]
The proposed LightVeriFL scheme guarantees dropout resilience up to any D dropped users during the verification phase such that N ≥ T + D +1 .
Proof.
In LightVeriFL , each user i encodes its mask z i using the same T − private MDS matrix such that the aggregate encoded mask that a surviving user j sends to the server in aggregate decommitting satisfies
[Equation]
where W j is the j th column of the MDS matrix W .
From (17), we see that (cid:80) i ∈D [
˜ z
i ] j is the encoded version of the desired aggregate mask (of the dropped users) (cid:80) i ∈D z i .
By construction, we have
N − D ≥ U so that the server receives (cid:80) i ∈D
˜
z
i ] j from at least U users.
Thus, through MDS decoding, the server successfully recovers the aggregate mask of the dropped users (cid:80) i ∈D z i and performs (cid:80) i ∈D h i =
(cid:80) i ∈D ( h i
+ z i ) − (cid:80)
i ∈D z i .
A similar encoding/decoding strategy holds for the aggregate mask of the surviving users (cid:80) i ∈U
[˜ z i ] j .
Thus, the server is able to reconstruct the aggregate hashes of the users when up to D users drop where N − D ≥ U
= T + 1 .
Using the aggregate hash of the users, the surviving users perform the verification operation even in the presence of dropouts.
Theorem 4.
The proposed LightVeriFL scheme guarantees successful aggregation integrity verification in the presence of any D user dropouts during the verification phase without sacrificing input privacy against up to any T colluding users for T + D < N .
When LightVeriFL is implemented together with a secure aggregation scheme, such as SecAgg [6] and LightSecAgg [8], a secure verifiable aggregation scheme is obtained.
Theorem 4 simply follows from Theorems 1-3.We note that in LightVeriFL , as in the secure aggregation schemes, the guarantee in Theorem 4 is for a single FL iteration only.
Since all the randomness in LightVeriFL is generated independently across all iterations, this guarantee can be extended to the entire FL protocol by invoking the proposed LightVeriFL scheme at each iteration.
G Complexity Analysis
In this section, we provide the complexity analysis of the LightVeriFL scheme for N − D ≥ U =
T + 1 as in [8].
We do not include the complexity of the SecAgg scheme as our LightVeriFL can be implemented in a standalone manner without any secure aggregation protocol.
We note that the hashes and commitments have constant lengths that are independent of the model size d and N .
Offline storage cost.
In LightVeriFL , each user generates a random mask
z i (a point on the EC) of constant length and stores coded masks of all the other users as well as their constant-length commitments.
In addition, in the amortized verification, each user stores the past L aggregate models.
Thus, the total storage cost of LightVeriFL at each user is O ( N + Ld ) .
Offline communication and computation loads.
In LightVeriFL , each user computes its coded masks in offline manner before the local model is computed.
To compute the coded mask, each user performs an ( N, U ) MDS coding, where the size of each data block is constant (since z i and [ n i ] k , k = { 2 , . .
, U } are all points on the EC).
Thus, the offline computation load at each user at each iteration is O ( N log N ) .
Then, still in the offline mode, each user shares each of the N coded segments with the other users, which induces a communication load of O ( N ) .
Online communication load.
Each user sends its masked hash to the server as well as the corresponding commitment to the other users during the aggregation phase.
Since either hash and commitment has a constant length, the total communication load for the users during the aggregation phase is
O ( N ) .
In addition, in the verification phase, each surviving user sends the aggregate coded masks it has received from the other users (which has a constant length) to the server.
Thus, the total online communication load for a user is O ( N ) .
Correspondingly, the online communication load at the server is O ( N + U ) since it receives the masked hashes from N users in the aggregation phase as well as the coded masks from the surviving U users in the verification phase.
Online computation load.
After computing its local model, each user computes its hash, which is the most expensive operation in LightVeriFL at the users and introduces an O ( d ) computation load, d is the model size.
Then, in the amortized verification each surviving user computes the hash of the aggregate model at every L iterations, where L is the batch size, inducing another O ( dL ) computation load, making the total online computation load
O ( L +1 L d ) at each user The server, on the other hand, performs the reconstruction from the coded masks it has received from U surviving users to recover (cid:80) i ∈U
z i and (cid:80) i ∈D z
i , which is the most time-consuming operation performed at the server in LightVeriFL .
For this reconstruction, the server decodes a U dimensional MDS code using U coded messages it has received from the surviving users.
Since each message is of a constant length, the total computation complexity here is O
( U log U ) operations in F n , where n is the subgroup order of the EC.
Next, we compare the complexity of the proposed LightVeriFL with the baseline VeriFL [12] in
Table 2.
We consider a scenario, where T = N 2 , U = T + 1 , and D = pN , where 0 ≤ p < 12 .
From Table 2, we see that in the proposed LightVeriFL scheme, the recovery complexity at the server is almost linear in the number of users N , whereas in the VeriFL scheme reconstruction has quadratic complexity in N , as in VeriFL the server reconstructs each of the dropped users’ hashes one-by-one.
Thus, LightVeriFL significantly improves the computation time at the server, thereby speeding up the entire verification process especially when the number of users N grows.
Further, in LightVeriFL , the masks that are used to hide the hashes are prepared in advance in an offline manner as these masks are independent of the local models as well as the hashes.
In VeriFL , however, to tolerate dropouts, generated hashes are secret shared among the users, which can be performed in
[Table] [Table]
an online manner after the local training is completed.
Thus, the online computation complexity at the users and the online communication complexity at the server is higher than those of LightVeriFL , which does not require secret sharing among the users and one-by-one reconstruction at the server.
We compare the complexities of standalone implementations of SecAgg [6] and our proposed
LightVeriFL scheme in Table 3.
We observe from Table 3 that the complexity of standalone implementation of LightVeriFL is not significant compared to the SecAgg scheme, as in LightVeriFL , hashes and commitments are of constant length independent of d .
That is, when the LightVeriFL protocol is implemented on top of SecAgg [6], the complexity of the overall scheme is no more than
SecAgg alone.
Thus, achieving a secure verifiable aggregation scheme through LightVeriFL is not infeasible in practical scenarios, where SecAgg is implemented.
A similar argument applies to more efficient secure aggregation schemes such as SecAgg + [7] and LightSecAgg [8].
H Experimental Details and Additional Experiment Results
In our implementation, we use the open-source fastecdsa Python library for fast elliptic curve cryptography [38].
The implementation of the proposed LightVeriFL scheme can be found in https://github.com/bbuyukates/LightVeriFL-fast .
In Tables 1 and 4, we use L = 10 for the batch size of the amortization and report the average of 5 independent runs along with the standard error.
In Table 1, we observe that in LightVeriFL decommitting round is not affected in all three dropout scenarios, since the server reconstructs the aggregate hash of all N = 200 users in one-shot whereas in the VeriFL scheme the time needed for decommitting increases as more and more users drop the protocol.
The batch checking round in both of the schemes is not affected by varying dropout rates aside from a minor straggler effect.
That is, as the dropout rate decreases, more number of users stay in the system for verification, which in turn increases the completion time of the verification phase due to the straggler effect.
To report the performance results for realistic FL settings we show the performance of LightVeriFL for varying d as well.
For this, we use d = 10 K (similar to a logistic regression model on MNIST [39] which requires d = 7 , 850 ) and d = 1 M (similar to training a CNN [1] on FEMNIST [40], which requires d = 1 , 206 , 590 ).
[Table]
In Table 4, we investigate the affect of the model size on the proposed verification algorithm.
Considering practical FL systems, we take d = 10 K , d = 100 K , and d = 1 M for N = 200 users, 30% dropout rate, and L = 10 .
In this case, decommitting time stays (almost) the same for both of the schemes across all three d values since hashes are of constant length independent of d in either of the schemes.
In both of the schemes, the batch checking time, which requires the computation of the hash of the aggregate model, increases linearly with d .
In Table 4, we see that for smaller d values the verification time is dominated by the decommitting step in VeriFL .
In these cases, the proposed LightVeriFL scheme achieves the biggest gain.
In fact, we observe around 23 .
34 × gain when d = 10 K .
From Tables 1 and 4 we deduce that the gain achieved by the proposed LightVeriFL scheme over the VeriFL scheme is more prominent when the verification time is dominated by the decommitting step.
This happens when the model size is smaller for moderate number of users, e.g., d = 10 K and
N = 200 or d = 100 K and N = 200 .
Such a large gain is expected when the number of users gets larger in more complex models with larger model sizes.