  The paper considers a distributed version of deep reinforcement learning
(DRL) for multi-agent decision-making process in the paradigm of federated
learning. Since the deep neural network models in federated learning are
trained locally and aggregated iteratively through a central server, frequent
information exchange incurs a large amount of communication overheads. Besides,
due to the heterogeneity of agents, Markov state transition trajectories from
different agents are usually unsynchronized within the same time interval,
which will further influence the convergence bound of the aggregated deep
neural network models. Therefore, it is of vital importance to reasonably
evaluate the effectiveness of different optimization methods. Accordingly, this
paper proposes a utility function to consider the balance between reducing
communication overheads and improving convergence performance. Meanwhile, this
paper develops two new optimization methods on top of variation-aware periodic
averaging methods: 1) the decay-based method which gradually decreases the
weight of the model's local gradients within the progress of local updating,
and 2) the consensus-based method which introduces the consensus algorithm into
federated learning for the exchange of the model's local gradients. This paper
also provides novel convergence guarantees for both developed methods and
demonstrates their effectiveness and efficiency through theoretical analysis
and numerical simulation results.
