  As distributed systems grow in scale and complexity, the need for flexible
automation of systems management functions also grows. We outline a framework
for building tools that provide distributed, scalable, declarative, modular,
and continuous automation for distributed systems. We focus on four points of
design: 1) a state-management approach that prescribes source-of-truth for
configured and discovered system states; 2) a technique to solve the
declarative unification problem for a class of automation problems, providing
state convergence and modularity; 3) an eventual-consistency approach to state
synchronization which provides automation at scale; 4) an event-driven
architecture that provides always-on state enforcement.
  We describe the methodology, software architecture for the framework, and
constraints for these techniques to apply to an automation problem. We overview
a reference application built on this framework that provides state-aware
system provisioning and node lifecycle management, highlighting key advantages.
We conclude with a discussion of current and future applications.
