  Due to the resource consumption for transmitting massive data and the concern
for exposing sensitive data, it is impossible or undesirable to upload clients'
local databases to a central server. Thus, federated learning has become a hot
research area in enabling the collaborative training of machine learning models
among multiple clients that hold sensitive local data. Nevertheless,
unconstrained federated optimization has been studied mainly using stochastic
gradient descent (SGD), which may converge slowly, and constrained federated
optimization, which is more challenging, has not been investigated so far. This
paper investigates sample-based and feature-based federated optimization,
respectively, and considers both the unconstrained problem and the constrained
problem for each of them. We propose federated learning algorithms using
stochastic successive convex approximation (SSCA) and mini-batch techniques. We
show that the proposed algorithms can preserve data privacy through the model
aggregation mechanism, and their security can be enhanced via additional
privacy mechanisms. We also show that the proposed algorithms converge to
Karush-Kuhn-Tucker (KKT) points of the respective federated optimization
problems. Besides, we customize the proposed algorithms to application examples
and show that all updates have closed-form expressions. Finally, numerical
experiments demonstrate the inherent advantages of the proposed algorithms in
convergence speeds, communication costs, and model specifications.
