Repository: kubernetes/kops

Issue #3605: Kubelet fail to register node arbitrarily using Spot Instances
Created: 2017-10-11T12:29:05Z
kops 1.8.0-alpha.1
kubernetes: 1.7.8
provider: AWS
To reproduce it create a cluster and add and instance group with maxPrice value
Some instances not register with apiserver, some of them works fine so it seems some kind of race condition.
If you kill kubelet process, after respawn all works fine.
The relevant part of logs i could find in kubelet are this lines repeated in loop:
Oct 11 08:39:47 ip-172-20-104-75 kubelet[1281]: I1011 08:39:47.987315    1281 kubelet.go:1894] SyncLoop (DELETE, "api"): "kube-proxy-ip-172-20-104-75.eu-west-1.compute.internal_kube-system(b1664e35-ae5f-11e7-8c91-062a9a3fee2c)"
Oct 11 08:39:47 ip-172-20-104-75 kubelet[1281]: W1011 08:39:47.990681    1281 kubelet.go:1596] Deleting mirror pod "kube-proxy-ip-172-20-104-75.eu-west-1.compute.internal_kube-system(b1664e35-ae5f-11e7-8c91-062a9a3fee2c)" because it is outdated
Oct 11 08:39:47 ip-172-20-104-75 kubelet[1281]: I1011 08:39:47.990693    1281 mirror_client.go:85] Deleting a mirror pod "kube-proxy-ip-172-20-104-75.eu-west-1.compute.internal_kube-system"
Oct 11 08:39:47 ip-172-20-104-75 kubelet[1281]: I1011 08:39:47.994312    1281 kubelet.go:1888] SyncLoop (REMOVE, "api"): "kube-proxy-ip-172-20-104-75.eu-west-1.compute.internal_kube-system(b1664e35-ae5f-11e7-8c91-062a9a3fee2c)"
Oct 11 08:39:48 ip-172-20-104-75 kubelet[1281]: I1011 08:39:48.008283    1281 kubelet.go:1878] SyncLoop (ADD, "api"): "kube-proxy-ip-172-20-104-75.eu-west-1.compute.internal_kube-system(bd58ce30-ae5f-11e7-8c91-062a9a3fee2c)"
[...]

Cluster yaml
kind: Cluster
metadata:
  creationTimestamp: 2017-10-11T08:54:21Z
  name: test
spec:
  api:
    loadBalancer:
      type: Public
  authorization:
    rbac: {}
  channel: stable
  cloudProvider: aws
  configBase: s3://kubernetes-artifacts/test
  dnsZone: test
  etcdClusters:
  - enableEtcdTLS: true
    etcdMembers:
    - instanceGroup: master-eu-west-1a
      name: a
    - instanceGroup: master-eu-west-1b
      name: b
    - instanceGroup: master-eu-west-1c
      name: c
    name: main
    version: 3.1.10
  - enableEtcdTLS: true
    etcdMembers:
    - instanceGroup: master-eu-west-1a
      name: a
    - instanceGroup: master-eu-west-1b
      name: b
    - instanceGroup: master-eu-west-1c
      name: c
    name: events
    version: 3.1.10
  iam:
    legacy: false
  kubeAPIServer:
    auditLogMaxAge: 10
    auditLogMaxBackups: 1
    auditLogMaxSize: 100
    auditLogPath: /var/log/kube-apiserver-audit.log
  kubelet:
    featureGates:
      ExperimentalCriticalPodAnnotation: "true"
  kubernetesApiAccess:
  - 0.0.0.0/0
  kubernetesVersion: 1.7.8
  masterInternalName: api.internal.test
  masterPublicName: api.test
  networkCIDR: 172.20.0.0/16
  networking:
    weave:
      mtu: 8912
  nonMasqueradeCIDR: 100.64.0.0/10
  sshAccess:
  - 0.0.0.0/0
  subnets:
  - cidr: 172.20.32.0/19
    name: eu-west-1a
    type: Private
    zone: eu-west-1a
  - cidr: 172.20.64.0/19
    name: eu-west-1b
    type: Private
    zone: eu-west-1b
  - cidr: 172.20.96.0/19
    name: eu-west-1c
    type: Private
    zone: eu-west-1c
  - cidr: 172.20.0.0/22
    name: utility-eu-west-1a
    type: Utility
    zone: eu-west-1a
  - cidr: 172.20.4.0/22
    name: utility-eu-west-1b
    type: Utility
    zone: eu-west-1b
  - cidr: 172.20.8.0/22
    name: utility-eu-west-1c
    type: Utility
    zone: eu-west-1c
  topology:
    bastion:
      bastionPublicName: bastion.test
    dns:
      type: Public
    masters: private
    nodes: private

Instance group spotrequests
apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: 2017-10-11T09:21:48Z
  labels:
    kops.k8s.io/cluster: test
  name: ephimeral
spec:
  image: kope.io/k8s-1.7-debian-jessie-amd64-hvm-ebs-2017-07-28
  machineType: m4.2xlarge
  maxSize: 7
  minSize: 3
  maxPrice: "0.15"
  role: Node
  subnets:
  - eu-west-1b
  - eu-west-1c
  - eu-west-1a

Duplicate issue #4090: kube-proxy - Failed to retrieve node info - cluster does not validate
Created: 2017-12-15T18:16:23Z
What kops version are you running? The command kops version, will display
this information.

Version 1.8.0

What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.

Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.5", GitCommit:"cce11c6a185279d037023e02ac5249e14daa22bf", GitTreeState:"clean", BuildDate:"2017-12-07T18:09:00Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.4", GitCommit:"9befc2b8928a9426501d3bf62f72849d5cbcd5a3", GitTreeState:"clean", BuildDate:"2017-11-20T05:17:43Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}


What cloud provider are you using?
AWS


What commands did you run?  What is the simplest way to reproduce this issue?
kops replace --force -f ig.yaml where the yaml configuration is below for a new instance group


apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  labels:
    kops.k8s.io/cluster: name-redacted
  name: m4.large-us-west-2b
spec:
  image: kope.io/k8s-1.8-debian-jessie-amd64-hvm-ebs-2017-12-02
  machineType: m4.large
  maxPrice: "0.1"
  maxSize: 4
  minSize: 1
  nodeLabels:
    k8s.io/cluster-autoscaler/enabled: ""
    kops.k8s.io/instancegroup: m4.large-us-west-2b
  role: Node
  subnets:
  - us-west-2b



What happened after the commands executed?
Node comes up and cluster fails to validate. On the node, the kube proxy.log includes ""Failed to retrieve node info: nodes "ip-172-20-90-65.us-west-2.compute.internal" not found"


What did you expect to happen?
I expected the node to bootstrap successfully and the cluster to validate after creating a new instance group.


Please provide your cluster manifest. Execute


apiVersion: kops/v1alpha2
kind: Cluster
metadata:
  creationTimestamp: 2017-12-12T19:58:05Z
  name: name-redacted
spec:
  additionalPolicies:
    node: |
      [
        {
          "Effect": "Allow",
          "Action": ["sts:AssumeRole"],
          "Resource": ["*"]
        }, {
            "Effect": "Allow",
            "Action": [
                "autoscaling:DescribeAutoScalingGroups",
                "autoscaling:DescribeAutoScalingInstances",
                "autoscaling:DescribeTags",
                "autoscaling:SetDesiredCapacity",
                "autoscaling:TerminateInstanceInAutoScalingGroup"
            ],
            "Resource": "*"
        }
      ]
  api:
    dns: {}
  authorization:
    rbac: {}
  channel: stable
  cloudProvider: aws
  configBase: bucket-redacted
  dnsZone: dnszone-redacted
  etcdClusters:
  - etcdMembers:
    - instanceGroup: master-us-west-2a
      name: a
    name: main
  - etcdMembers:
    - instanceGroup: master-us-west-2a
      name: a
    name: events
  iam:
    legacy: false
  kubernetesApiAccess:
  - 0.0.0.0/0
  kubernetesVersion: 1.8.4
  masterInternalName: name-redacted
  masterPublicName: name-redacted
  networkCIDR: 172.20.0.0/16
  networking:
    weave:
      mtu: 8912
  nonMasqueradeCIDR: 100.64.0.0/10
  sshAccess:
  - 0.0.0.0/0
  subnets:
  - cidr: 172.20.32.0/19
    name: us-west-2a
    type: Public
    zone: us-west-2a
  - cidr: 172.20.64.0/19
    name: us-west-2b
    type: Public
    zone: us-west-2b
  topology:
    dns:
      type: Public
    masters: public
    nodes: public

---

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: 2017-12-15T16:59:21Z
  labels:
    kops.k8s.io/cluster: name-redacted
  name: m4.large-us-west-2a
spec:
  image: kope.io/k8s-1.8-debian-jessie-amd64-hvm-ebs-2017-12-02
  machineType: m4.large
  maxPrice: "0.1"
  maxSize: 4
  minSize: 1
  nodeLabels:
    k8s.io/cluster-autoscaler/enabled: ""
    kops.k8s.io/instancegroup: m4.large-us-west-2a
  role: Node
  subnets:
  - us-west-2a

---

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: 2017-12-15T17:02:44Z
  labels:
    kops.k8s.io/cluster: name-redacted
  name: m4.large-us-west-2b
spec:
  image: kope.io/k8s-1.8-debian-jessie-amd64-hvm-ebs-2017-12-02
  machineType: m4.large
  maxPrice: "0.1"
  maxSize: 4
  minSize: 1
  nodeLabels:
    k8s.io/cluster-autoscaler/enabled: ""
    kops.k8s.io/instancegroup: m4.large-us-west-2b
  role: Node
  subnets:
  - us-west-2b

---

apiVersion: kops/v1alpha2
kind: InstanceGroup
metadata:
  creationTimestamp: 2017-12-12T19:58:05Z
  labels:
    kops.k8s.io/cluster: name-redacted
  name: master-us-west-2a
spec:
  image: kope.io/k8s-1.8-debian-jessie-amd64-hvm-ebs-2017-12-02
  machineType: t2.small
  maxSize: 1
  minSize: 1
  nodeLabels:
    kops.k8s.io/instancegroup: master-us-west-2a
  role: Master
  subnets:
  - us-west-2a


Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.
Here is an excerpt from kube-proxy.log which shows the failed to retrieve node info error.

Flag --resource-container has been deprecated, This feature will be removed in a later release.
W1215 17:04:25.559841       5 server.go:191] WARNING: all flags other than --config, --write-config-to, and --cleanup are deprecated. Please begin using a config file ASAP.
I1215 17:04:25.562012       5 iptables.go:192] Could not connect to D-Bus system bus: dial unix /var/run/dbus/system_bus_socket: connect: no such file or directory
time="2017-12-15T17:04:25Z" level=warning msg="Running modprobe ip_vs failed with message: `modprobe: ERROR: ../libkmod/libkmod.c:557 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.102-k8s/modules.dep.bin'`, error: exit status 1"
time="2017-12-15T17:04:25Z" level=error msg="Could not get ipvs family information from the kernel. It is possible that ipvs is not enabled in your kernel. Native loadbalancing will not work until this is fixed."
W1215 17:04:25.566843       5 server_others.go:268] Flag proxy-mode="" unknown, assuming iptables proxy
I1215 17:04:25.567845       5 server_others.go:122] Using iptables Proxier.
W1215 17:04:25.582103       5 server.go:580] Failed to retrieve node info: nodes "ip-172-20-90-65.us-west-2.compute.internal" not found
W1215 17:04:25.582158       5 proxier.go:468] invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP
I1215 17:04:25.582327       5 server_others.go:157] Tearing down inactive rules.


Anything else do we need to know?
The specific error message is the same as in #2915.

Closing comment (2018-01-02T09:40:04Z): This is same as #3605. There is a suggested workaround there in one of the comments.