Repository: kubernetes
Issue: ELB Security Group references not cleaned up after deleting service
Labels: kind/bug,lifecycle/stale,sig/cloud-provider,needs-triage

### What happened?

Kubernetes fails to clean up AWS resources it created when creating a Service which leads to a build-up of resources, inability to delete the environment this happens within without manual intervention, etc.

I'm using EKS and creating a Service. When I create service:

- Kubernetes creates an ELB based on the annotations in my service, included below.
- Kubernetes creates a Security Group for the ELB (which allows traffic to the ELB on the port specified for the service)
- Kubernetes creates a reference to this new security group on the security group associated with nodes

This is all OK.

When I delete a service:

- Kubernetes deletes the ELB and waits for its deletion.
- Kubernetes attempts to delete the Security Group for the ELB, but fails! It fails because it needs to first remove the reference on the security group associated with nodes.
- Kubernetes eventually gives up and leaves the reference and security group behind.

Service:
```
apiVersion: v1
kind: Service
metadata:
  annotations:
    external-dns.alpha.kubernetes.io/hostname: nginx.test.domain.
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
  labels:
    app: nginx
  name: nginx
  namespace: default
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: nginx
  type: LoadBalancer
```

### What did you expect to happen?

Kubernetes deletes the resources it created.

### How can we reproduce it (as minimally and precisely as possible)?

- Create the included service on an EKS cluster
  - I'm unsure if there needs to be a live pod behind the service or if creating the service alone can trigger this.
- Delete it
- Observe the resources left behind
- Observe the error message in cloudwatch stating that the reference needs to be deleted first
- Observe the error message from kube-controller-manager stating that it is assuming the delete fails because "assuming because LB is in process of deleting", which is wrong.
- After 10 minutes, it gives up

### Anything else we need to know?

Here is controller-manager logs showing the bad behavior.

- `sg-0a2d3dae08175283c` is the Security Group created for the ELB for the service, that towards the end of the log Kube fails to delete
- `sg-0f3139a0bfc62b18d` is the Security Group associated with nodes that the reference to the above SG is added to. The log clearly describes this group being manipulated on create but not on delete.

```
kube-controller-manager I0427 17:15:47.851479      10 controller.go:368] Ensuring load balancer for service default/nginx
kube-controller-manager I0427 17:15:47.851512      10 controller.go:853] Adding finalizer to service default/nginx
kube-controller-manager I0427 17:15:47.852272      10 event.go:291] "Event occurred" object="default/nginx" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"
kube-controller-manager I0427 17:15:47.886020      10 utils.go:413] couldn't find ipfamilies for service: default/nginx. This could happen if controller manager is connected to an old apiserver that does not support ip families yet. EndpointSlices for this Service will use IPv4 as the IP Family based on familyOf(ClusterIP:172.20.39.211).
kube-controller-manager I0427 17:15:47.886711      10 aws.go:4035] EnsureLoadBalancer(kubernetes, default, nginx, us-east-1, , [{http TCP <nil> 80 {0 8080 } 31460}], map[external-dns.alpha.kubernetes.io/hostname:nginx.autotest1.matterport.com. service.beta.kubernetes.io/aws-load-balancer-backend-protocol:http service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout:60 service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled:true])
kube-controller-manager I0427 17:15:48.105777      10 aws.go:3594] Ignoring private subnet for public ELB "subnet-008ca525728263ed1"
kube-controller-manager I0427 17:15:48.105799      10 aws.go:3594] Ignoring private subnet for public ELB "subnet-017d4e20871fe1997"
kube-controller-manager I0427 17:15:48.105811      10 aws.go:3594] Ignoring private subnet for public ELB "subnet-0eb2817fab493821f"
kube-controller-manager I0427 17:15:48.105819      10 aws.go:3594] Ignoring private subnet for public ELB "subnet-0b83856cc1b80d911"
kube-controller-manager I0427 17:15:48.827905      10 aws.go:3256] Existing security group ingress: sg-0a2d3dae08175283c []
kube-controller-manager I0427 17:15:48.827949      10 aws.go:3287] Adding security group ingress: sg-0a2d3dae08175283c [{
  FromPort: 3,
  IpProtocol: "icmp",
  IpRanges: [{
      CidrIp: "0.0.0.0/0"
    }],
  ToPort: 4
} {
  FromPort: 80,
  IpProtocol: "tcp",
  IpRanges: [{
      CidrIp: "0.0.0.0/0"
    }],
  ToPort: 80
}]
kube-controller-manager I0427 17:15:48.877881      10 utils.go:413] couldn't find ipfamilies for service: default/nginx. This could happen if controller manager is connected to an old apiserver that does not support ip families yet. EndpointSlices for this Service will use IPv4 as the IP Family based on familyOf(ClusterIP:172.20.39.211).
kube-controller-manager I0427 17:15:49.177491      10 aws_loadbalancer.go:1009] Creating load balancer for default/nginx with name: a9054651877d5446abd3790c7c48a48f
kube-controller-manager I0427 17:15:50.075877      10 aws_loadbalancer.go:1212] Updating load-balancer attributes for "a9054651877d5446abd3790c7c48a48f"
kube-controller-manager I0427 17:15:50.787220      10 aws.go:4654] Adding rule for traffic from the load balancer (sg-0a2d3dae08175283c) to instances (sg-0f3139a0bfc62b18d)
kube-controller-manager I0427 17:15:50.891505      10 aws.go:3331] Existing security group ingress: sg-0f3139a0bfc62b18d [{
  IpProtocol: "-1",
  UserIdGroupPairs: [{
      Description: "Allow node to communicate with each other",
      GroupId: "sg-0f3139a0bfc62b18d",
      UserId: "246224679784"
    },{
      GroupId: "sg-0a551a8743394356b",
      UserId: "246224679784"
    }]
} {
  FromPort: 22,
  IpProtocol: "tcp",
  IpRanges: [{
      CidrIp: "10.0.0.0/8",
      Description: "Allow ssh protocol from private subnet"
    }],
  ToPort: 22
} {
  FromPort: 1025,
  IpProtocol: "tcp",
  ToPort: 65535,
  UserIdGroupPairs: [{
      Description: "Allow worker Kubelets and pods to receive communication from the cluster control plane",
      GroupId: "sg-0c8c55fd92d597a28",
      UserId: "246224679784"
    }]
} {
  FromPort: 443,
  IpProtocol: "tcp",
  ToPort: 443,
  UserIdGroupPairs: [{
      Description: "Allow nodes to recieve traffic from control plane via 443",
      GroupId: "sg-0c8c55fd92d597a28",
      UserId: "246224679784"
    }]
}]
kube-controller-manager I0427 17:15:50.891598      10 aws.go:3228] Comparing sg-0a2d3dae08175283c to sg-0f3139a0bfc62b18d
kube-controller-manager I0427 17:15:50.891606      10 aws.go:3228] Comparing sg-0a2d3dae08175283c to sg-0a551a8743394356b
kube-controller-manager I0427 17:15:50.891613      10 aws.go:3359] Adding security group ingress: sg-0f3139a0bfc62b18d [{
  IpProtocol: "-1",
  UserIdGroupPairs: [{
      GroupId: "sg-0a2d3dae08175283c"
    }]
}]
kube-controller-manager I0427 17:15:51.303264      10 aws_loadbalancer.go:1460] Instances added to load-balancer a9054651877d5446abd3790c7c48a48f
kube-controller-manager I0427 17:15:51.303300      10 aws.go:4420] Loadbalancer a9054651877d5446abd3790c7c48a48f (default/nginx) has DNS name a9054651877d5446abd3790c7c48a48f-324500995.us-east-1.elb.amazonaws.com
kube-controller-manager I0427 17:15:51.303338      10 controller.go:894] Patching status for service default/nginx
kube-controller-manager I0427 17:15:51.303914      10 event.go:291] "Event occurred" object="default/nginx" kind="Service" apiVersion="v1" type="Normal" reason="EnsuredLoadBalancer" message="Ensured load balancer"
kube-controller-manager I0427 17:15:51.321299      10 utils.go:413] couldn't find ipfamilies for service: default/nginx. This could happen if controller manager is connected to an old apiserver that does not support ip families yet. EndpointSlices for this Service will use IPv4 as the IP Family based on familyOf(ClusterIP:172.20.39.211).
kube-controller-manager I0427 17:20:18.363810      10 utils.go:413] couldn't find ipfamilies for service: default/nginx. This could happen if controller manager is connected to an old apiserver that does not support ip families yet. EndpointSlices for this Service will use IPv4 as the IP Family based on familyOf(ClusterIP:172.20.39.211).
kube-controller-manager I0427 17:20:18.417328      10 controller.go:353] Deleting existing load balancer for service default/nginx
kube-controller-manager I0427 17:20:18.417602      10 event.go:291] "Event occurred" object="default/nginx" kind="Service" apiVersion="v1" type="Normal" reason="DeletingLoadBalancer" message="Deleting load balancer"
kube-controller-manager I0427 17:20:19.263481      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:20:19.263535      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:20:29.633775      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:20:29.633812      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:20:39.838207      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:20:39.838238      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:20:50.275770      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:20:50.275801      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:21:00.633427      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:21:00.633459      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:21:10.832384      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:21:10.832416      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:21:21.270275      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:21:21.270308      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:21:31.503893      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:21:31.503924      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:21:41.739099      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:21:41.739133      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:21:52.141328      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:21:52.141359      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:22:02.541573      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:22:02.541617      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:22:12.951735      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:22:12.951765      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:22:23.390254      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:22:23.390296      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:22:33.982859      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:22:33.982892      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:22:44.360603      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:22:44.360632      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:22:54.925773      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:22:54.925807      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:23:05.514117      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:23:05.514149      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:23:15.981766      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:23:15.981942      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:23:26.414644      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:23:26.414669      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:23:36.928175      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:23:36.928220      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:23:47.431531      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:23:47.431599      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:23:58.047224      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:23:58.047255      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:24:08.732782      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:24:08.732825      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:24:19.343409      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:24:19.343439      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:24:29.827647      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:24:29.827677      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:24:40.321840      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:24:40.321869      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:24:50.837983      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:24:50.838009      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:25:01.404284      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:25:01.404319      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:25:12.052302      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:25:12.052329      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:25:22.605780      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:25:22.605810      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:25:33.161196      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:25:33.161230      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:25:43.633677      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:25:43.633779      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:25:54.146598      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:25:54.146630      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:26:04.752927      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:26:04.752959      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:26:15.258513      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:26:15.258548      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:26:25.772997      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:26:25.773029      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:26:36.340700      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:26:36.340744      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:26:46.751887      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:26:46.751933      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:26:57.095918      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:26:57.095968      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:27:07.543908      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:27:07.543939      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:27:18.002933      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:27:18.002967      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:27:28.381027      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:30:06.452078      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:30:06.452108      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:30:17.094200      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager I0427 17:30:17.094225      10 aws.go:4869] Waiting for load-balancer to delete so we can delete security groups: nginx
kube-controller-manager I0427 17:30:27.669374      10 aws.go:4845] Ignoring DependencyViolation while deleting load-balancer security group (sg-0a2d3dae08175283c), assuming because LB is in process of deleting
kube-controller-manager E0427 17:30:27.669449      10 controller.go:275] error processing service default/nginx (will retry): failed to delete load balancer: timed out deleting ELB: nginx. Could not delete security groups sg-0a2d3dae08175283c
kube-controller-manager I0427 17:30:27.669901      10 event.go:291] "Event occurred" object="default/nginx" kind="Service" apiVersion="v1" type="Warning" reason="SyncLoadBalancerFailed" message="Error syncing load balancer: failed to delete load balancer: timed out deleting ELB: nginx. Could not delete security groups sg-0a2d3dae08175283c"
kube-controller-manager I0427 17:30:27.703308      10 controller.go:868] Removing finalizer from service default/nginx
kube-controller-manager I0427 17:30:27.715212      10 controller.go:894] Patching status for service default/nginx
kube-controller-manager I0427 17:30:27.715667      10 event.go:291] "Event occurred" object="default/nginx" kind="Service" apiVersion="v1" type="Normal" reason="DeletedLoadBalancer" message="Deleted load balancer"
kube-controller-manager I0427 17:30:27.715969      10 garbagecollector.go:471] "Processing object" object="default/nginx-fklnh" objectUID=e6725567-5621-459c-86d7-550b926b3690 kind="EndpointSlice" virtual=false
kube-controller-manager I0427 17:30:28.274273      10 garbagecollector.go:580] "Deleting object" object="default/nginx-fklnh" objectUID=e6725567-5621-459c-86d7-550b926b3690 kind="EndpointSlice" propagationPolicy=Background
```

Here is an excerpt from the Cloudwatch event displaying the reason the delete is failing:

```
    "eventTime": "2022-04-27T17:30:27Z",
    "eventSource": "ec2.amazonaws.com",
    "eventName": "DeleteSecurityGroup",
    "awsRegion": "us-east-1",
    "sourceIPAddress": "eks.amazonaws.com",
    "userAgent": "eks.amazonaws.com",
    "errorCode": "Client.DependencyViolation",
    "errorMessage": "resource sg-0a2d3dae08175283c has a dependent object",
    "requestParameters": {
        "groupId": "sg-0a2d3dae08175283c"
    },
```

I am unable to delete the SG manually. After manually removing the reference described above I am able to manually delete the SG without issue.

This seems similar, but not identical, to #73480.

### Kubernetes version

<details>

```console
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.4", GitCommit:"e6c093d87ea4cbb530a7b2ae91e54c0842d8308a", GitTreeState:"clean", BuildDate:"2022-02-16T12:30:48Z", GoVersion:"go1.17.6", Compiler:"gc", Platform:"darwin/arm64"}
Server Version: version.Info{Major:"1", Minor:"20+", GitVersion:"v1.20.15-eks-14c7a48", GitCommit:"09ff856a7a3e49ba29b28b5532cde35fdac5ec60", GitTreeState:"clean", BuildDate:"2022-04-01T03:14:50Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}
```

</details>


### Cloud provider

<details>
AWS EKS
</details>


### OS version

<details>

```console
N/A
```

</details>


### Install tools

<details>
N/A
</details>


### Container runtime (CRI) and version (if applicable)

<details>
N/A
</details>


### Related plugins (CNI, CSI, ...) and versions (if applicable)

<details>
aws-k8s-cni v1.11.0
</details>