Repository: kubernetes
Issue: [client-go] Resource leak in kubernetes/client-go/tools/leaderelection
Labels: kind/bug,sig/api-machinery,triage/accepted

### What happened?

**Opinion 1**

Lock resource is created in [tryAcquireOrRenew()](https://github.com/kubernetes/client-go/blob/master/tools/leaderelection/leaderelection.go#L333), but not deleted when [release()](https://github.com/kubernetes/client-go/blob/master/tools/leaderelection/leaderelection.go#L294).
 
If resource cannot be deleted in pkg, it shall not be created. Or, at lease a choice to `ForbiddenCreation`.

```
type Interface interface {
	// Get returns the LeaderElectionRecord
	Get(ctx context.Context) (*LeaderElectionRecord, []byte, error)

	// Create attempts to create a LeaderElectionRecord
	Create(ctx context.Context, ler LeaderElectionRecord) error

	// Update will update and existing LeaderElectionRecord
	Update(ctx context.Context, ler LeaderElectionRecord) error

	// RecordEvent is used to record events
	RecordEvent(string)

	// Identity will return the locks Identity
	Identity() string

	// Describe is used to convert details on current resource lock
	// into a string
	Describe() string
}
```

**Opinion 2**

Different lock resources react differently with `update` method.

Some resources can be created via `update` method, but some cannot. Shall we change `update` method to `patch`?

LockResource | AllowCreateOnUpdate
--- | --- 
Configmaps | [false](https://github.com/kubernetes/kubernetes/blob/master/pkg/registry/core/configmap/strategy.go#L75)
Leases | [true](https://github.com/kubernetes/kubernetes/blob/master/pkg/registry/coordination/lease/strategy.go#L67)
Endpoints | [true](https://github.com/kubernetes/kubernetes/blob/master/pkg/registry/core/endpoint/strategy.go#L69)

### What did you expect to happen?

Add a new `bool` parameter `ForbiddenCreation` in `LeaderElectionConfig` struct.

```
type LeaderElectionConfig struct {
	// Lock is the resource that will be used for locking
	Lock rl.Interface

	// LeaseDuration is the duration that non-leader candidates will
	// wait to force acquire leadership. This is measured against time of
	// last observed ack.
	//
	// A client needs to wait a full LeaseDuration without observing a change to
	// the record before it can attempt to take over. When all clients are
	// shutdown and a new set of clients are started with different names against
	// the same leader record, they must wait the full LeaseDuration before
	// attempting to acquire the lease. Thus LeaseDuration should be as short as
	// possible (within your tolerance for clock skew rate) to avoid a possible
	// long waits in the scenario.
	//
	// Core clients default this value to 15 seconds.
	LeaseDuration time.Duration
	// RenewDeadline is the duration that the acting master will retry
	// refreshing leadership before giving up.
	//
	// Core clients default this value to 10 seconds.
	RenewDeadline time.Duration
	// RetryPeriod is the duration the LeaderElector clients should wait
	// between tries of actions.
	//
	// Core clients default this value to 2 seconds.
	RetryPeriod time.Duration

	// Callbacks are callbacks that are triggered during certain lifecycle
	// events of the LeaderElector
	Callbacks LeaderCallbacks

	// WatchDog is the associated health checker
	// WatchDog may be null if it's not needed/configured.
	WatchDog *HealthzAdaptor

	// ReleaseOnCancel should be set true if the lock should be released
	// when the run context is cancelled. If you set this to true, you must
	// ensure all code guarded by this lease has successfully completed
	// prior to cancelling the context, or you may have two processes
	// simultaneously acting on the critical path.
	ReleaseOnCancel bool

	// Name is the name of the resource lock for debugging
	Name string

	// ForbiddenCreation provides a choice to manage lock resource
        // lifecycle outside of leaderElection pkg. Note that if set to true,
        // you should create lock resource before run leaderElector.
	ForbiddenCreation bool
}
```

By default, `ForbiddenCreation` is `false`, which doesn't impact current users. It's a compatible change.

### How can we reproduce it (as minimally and precisely as possible)?

Write a `main()` to take `leaderelection` into use.
```
package main

import (
	"context"
	"time"

	"k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/client-go/kubernetes"
	"k8s.io/client-go/rest"
	"k8s.io/client-go/tools/leaderelection"
	"k8s.io/client-go/tools/leaderelection/resourcelock"
)

func main() {
	kc, _ := rest.InClusterConfig()
	clientSet, _ := kubernetes.NewForConfig(kc)
	elector, _ := leaderelection.NewLeaderElector(leaderelection.LeaderElectionConfig{
		Lock: &resourcelock.ConfigMapLock{
			ConfigMapMeta: v1.ObjectMeta{
				Name:      "myLocker",
				Namespace: "myNamespace",
			},
			Client: clientSet.CoreV1(),
			LockConfig: resourcelock.ResourceLockConfig{
				Identity: "myPod",
			},
		},
		ReleaseOnCancel: true,
		LeaseDuration:   11 * time.Second,
		RenewDeadline:   7 * time.Second,
		RetryPeriod:     2 * time.Second,
	})
	elector.Run(context.Background())
}
```

### Anything else we need to know?

Code: https://github.com/kubernetes/client-go/tree/master/tools/leaderelection

### Kubernetes version

<details>

```console
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v0.21.0-beta.1", GitCommit:"e1bfb408ac5421021cd2f8c676ca398c9430674a", GitTreeState:"clean", BuildDate:"2022-02-15T19:14:53Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.3+6b15b0f", GitCommit:"3a0f2c90b43e6cffd07f57b5b78dd9f083e47ee2", GitTreeState:"clean", BuildDate:"2022-02-23T20:12:54Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}
```

</details>


### Cloud provider

N/A

### OS version

N/A


### Install tools

N/A

### Container runtime (CRI) and version (if applicable)

N/A


### Related plugins (CNI, CSI, ...) and versions (if applicable)

N/A