Repository: kubernetes
Issue: Official support for restoring Kubernetes cluster
Labels: sig/api-machinery,kind/feature,help wanted,triage/accepted

### What would you like to be added?

I would like to propose for Kubernetes to provide an official support for a restore operation. As Kubernetes stores all it's data in etcd, this feature would be mostly about defining an official guided way for users to restore their etcd. 

## Background

Restoring Kubernetes cluster requires 2 things:
* Restoring etcd cluster using some previously created backup
* Invalidating cache for all Kubernetes clients

Etcd supports a cluster restore as described in https://etcd.io/docs/latest/op-guide/recovery/#restoring-a-cluster. Restoring cluster requires a previously saved backup which generation we leave up to user preferences. 

Restoring an etcd is essentially "going back in time" which causes a problem for controllers. Kubernetes controllers use Informers which works like a local cache. Etcd going back in time means that some changes cached in Informers are no longer valid. Not having a proper client cache invalidation strategy can lead to unpredictable and inconsistent behavior of controllers.

## Proposal
* Etcd restore operation should support bumping revision by an arbitrary number.
* Etcd restore operation should support marking old revisions as compacted.

As defined in [The Implicit Kubernetes-Etcd Contract](https://docs.google.com/document/d/1NUZDiJeiIH5vo_FMaTWf0JtrQKCx0kpEaIIuPoj9P6A/edit?usp=sharing) etcd revision is used to establish a total order of operation. Kubernetes expects it to be non-decreasing. Problem is that Etcd restore operation breaks this assumption. If we could bump revision by an arbitrary number we could ensure that restore operation will not decrease the revision. For example for 1 week backup I can bump the revision by 1'000'000'000 assuming that etcd I run has less then 1500 QPS. Note: Kubernetes doesn't require revision to be continuous, it can include arbitrary gaps.

To ensure all kubernetes clients clear their cache I propose to mark to mark all revisions (including the bump) as compacted. This prevents etcd from responding to requests about revisions that were lost (executed after backup but before restore) and ensures that all clients will invalidate their cache. Using compaction window to mark revisions as unavailable seems like a natural mechanism for this case.

### Tracking work
* [x] https://github.com/etcd-io/etcd/issues/16028
* [ ] Add e2e tests for Kubernetes restore operation.
* [ ] Document the procedure

This was originally proposed and discussed on SIG-Apimachinery meeting May 31st 2023. https://youtu.be/sueYXW89ZHg
SIG leads confirmed that there is no KEP needed as most of the work will be done on etcd side.

cc @ahrtr @wojtek-t @logicalhan @jpbetz @deads2k 

### Why is this needed?

Without zero official guidance each vendor/administrator had a different plan for etcd restore many of which were broken and incorrect. Having an official guided restore operation allow whole community to work together and share their experience leading to improvements in Kubernetes disaster recovery handling.