  In a cell-free wireless network, distributed access points (APs) jointly
serve all user equipments (UEs) within the their coverage area by using the
same time/frequency resources. In this paper, we develop a novel downlink
cell-free multiple-input multiple-output (MIMO) millimeter wave (mmWave)
network architecture that enables all APs and UEs to dynamically self-partition
into a set of independent cell-free subnetworks in a time-slot basis. For this,
we propose several network partitioning algorithms based on deep reinforcement
learning (DRL). Furthermore, to mitigate interference between different
cell-free subnetworks, we develop a novel hybrid analog beamsteering-digital
beamforming model that zero-forces interference among cell-free subnetworks and
at the same time maximizes the instantaneous sum-rate of all UEs within each
subnetwork. Specifically, the hybrid beamforming model is implemented by using
a novel mixed DRL-convex optimization method in which analog beamsteering
between APs and UEs is conducted based on DRL while digital beamforming is
modeled and solved as a convex optimization problem. The DRL models for network
clustering and hybrid beamsteering are combined into a single hierarchical DRL
design that enables exchange of DRL agents' experiences during both network
training and operation. We also benchmark the performance of DRL models for
clustering and beamsteering in terms of network performance, convergence rate,
and computational complexity.
