Repository: openshift/openshift-ansible

Issue #9446: 3.10 upgrade fails converting master config
Created: 2018-08-06T17:42:12Z
Description
During upgrade from 3.9 to 3.10, upgrade fails attempting to convert master config to new format, at step:
TASK [openshift_control_plane : Migrate old master env file to new location]
Critically, attempting to re-try the upgrade after that point fails, because the master is no longer running:
fatal: [openshift-master-dev-01]: FAILED! => {"changed": false, "msg": "Configmap for node-config-master must be present"}
We are unsure how to recover from this, cannot get any further than this step after the initial failure, and the cluster is currently totally down.
Version
$ ansible --version
ansible 2.6.1
$ git describe
openshift-ansible-3.10.27-2-32-g8b7bbde6e

Steps To Reproduce

Attempt upgrade: playbooks/byo/openshift-cluster/upgrades/v3_10/upgrade_control_plane.yml 

Expected Results
Upgrade of control plane to succeed
Example command and output or error messages

Observed Results
Control plane upgrade fails at this step:
TASK [openshift_control_plane : Migrate old master env file to new location] ****************************
fatal: [openshift-master-dev-01]: FAILED! => {"changed": false, "module_stderr": "Shared connection to openshift-master-dev-01 closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n  File \"/tmp/ansible_ofJYPd/ansible_module_master_env_config_migrate.py\", line 234, in <module>\r\n    main()\r\n  File \"/tmp/ansible_ofJYPd/ansible_module_master_env_config_migrate.py\", line 230, in main\r\n    run_module()\r\n  File \"/tmp/ansible_ofJYPd/ansible_module_master_env_config_migrate.py\", line 223, in run_module\r\n    create_file(src, dest)\r\n  File \"/tmp/ansible_ofJYPd/ansible_module_master_env_config_migrate.py\", line 197, in create_file\r\n    config.readfp(open(src))\r\n  File \"/usr/lib64/python2.7/ConfigParser.py\", line 324, in readfp\r\n    self._read(fp, filename)\r\n  File \"/tmp/ansible_ofJYPd/ansible_module_master_env_config_migrate.py\", line 115, in _read\r\n    inline_prefixes = {p: -1 for p in self._inline_comment_prefixes}\r\nAttributeError: SectionlessParserOld instance has no attribute '_inline_comment_prefixes'\r\n", "msg": "MODULE FAILURE", "rc": 1}

Additional Information
operating system
rhel atomic host  7.5.1.1
inventory file
{
  "_meta": {
    "hostvars": {
      "openshift-master-dev-01": {
        "openshift_schedulable": true,
        "openshift_node_group_name": "node-config-master"
      },
      "openshift-node-dev-01": {
        "openshift_schedulable": true,
        "openshift_node_group_name": "node-config-infra"
      },
      "openshift-node-dev-02": {
        "openshift_schedulable": true,
        "openshift_node_group_name": "node-config-infra"
      },
      "openshift-node-dev-03": {
        "openshift_node_group_name": "node-config-compute"
      },
      "openshift-node-dev-04": {
        "openshift_node_group_name": "node-config-compute"
      },
      "openshift-node-dev-05": {
        "openshift_node_group_name": "node-config-compute"
      },
      "openshift-node-dev-06": {
        "openshift_node_group_name": "node-config-compute"
      },
      "openshift-node-dev-07": {
        "openshift_node_group_name": "node-config-compute"
      },
      "openshift-node-dev-08": {
        "openshift_node_group_name": "node-config-compute"
      },
      "openshift-node-dev-09": {
        "openshift_node_group_name": "node-config-compute"
      },
      "openshift-node-dev-10": {
        "openshift_node_group_name": "node-config-compute"
      }
    }
  },
  "OSEv3": {
    "children": [
      "masters",
      "nodes",
      "etcd"
    ],
    "vars": {
      "openshift_node_groups": [
        {
          "name": "node-config-master",
          "labels": [
            "node-role.kubernetes.io/master=true"
          ],
          "edits": []
        },
        {
          "name": "node-config-infra",
          "labels": [
            "node-role.kubernetes.io/infra=true"
          ],
          "edits": []
        },
        {
          "name": "node-config-compute",
          "labels": [
            "node-role.kubernetes.io/compute=true"
          ],
          "edits": []
        }
      ],
      "containerized": true,
      "deployment_type": "origin",
      "dynamic_volumes_check": false,
      "openshift_cloudprovider_kind": "vsphere",
      "openshift_cloudprovider_vsphere_datacenter": "Main Datacenter",
      "openshift_cloudprovider_vsphere_datastore": "UNI-ATC-01-BRONZE-OPENSHIFT",
      "openshift_cloudprovider_vsphere_folder": "REDACTED",
      "openshift_cloudprovider_vsphere_host": "vmware-test",
      "openshift_cloudprovider_vsphere_password": "REDACTED",
      "openshift_cloudprovider_vsphere_username": "os-disk-prov",
      "openshift_enable_service_catalog": true,
      "openshift_hosted_registry_routehost": "registry",
      "openshift_logging_install_logging": true,
      "openshift_logging_storage_kind": "dynamic",
      "openshift_master_cluster_public_hostname": "REDACTED",
      "openshift_master_cluster_hostname": "openshift-master-dev-01",
      "openshift_master_default_subdomain": "REDACTED",
      "openshift_master_identity_providers": [
        {
          "name": "openshift-dev",
          "challenge": "false",
          "login": "true",
          "mappingMethod": "claim",
          "kind": "OpenIDIdentityProvider",
          "clientID": "openshift_dev",
          "clientSecret": "REDACTED",
          "claims": {
            "id": [
              "sub"
            ],
            "preferredUsername": [
              "sub",
              "email"
            ],
            "name": [
              "nickname",
              "given_name",
              "name",
              "sub"
            ],
            "email": [
              "email",
              "sub"
            ]
          },
          "urls": {
            "authorize": "https://oauth/oidc/authorize",
            "token": "https://oauth/oidc/token",
            "userInfo": "https://oauth/oidc/userinfo"
          }
        }
      ],
      "openshift_master_named_certificates": [
        {
          "cafile": "REDACTED-ca.crt",
          "certfile": "REDACTED.combined.crt",
          "keyfile": "REDACTED.key"
        }
      ],
      "openshift_master_overwrite_named_certificates": true,
      "openshift_provisioners_install_provisioners": true,
      "openshift_release": "v3.10",
      "openshift_rolling_restart_mode": "system",
      "openshift_version": "3.10.0",
      "openshift_web_console_image_name": "web-console",
      "os_firewall_use_firewalld": false,
      "os_sdn_network_plugin_name": "redhat/openshift-ovs-multitenant",
      "template_service_broker_install": true,
      "openshift_prometheus_namespace": "openshift-metrics",
      "openshift_prometheus_node_selector": {
        "region": "infra"
      }
    }
  },
  "all": {
    "vars": {
      "openshift_disable_check": "disk_availability"
    }
  },
  "etcd": {
    "hosts": [
      "openshift-master-dev-01"
    ]
  },
  "masters": {
    "hosts": [
      "openshift-master-dev-01"
    ]
  },
  "nodes": {
    "hosts": [
      "openshift-master-dev-01",
      "openshift-node-dev-01",
      "openshift-node-dev-02",
      "openshift-node-dev-03",
      "openshift-node-dev-04",
      "openshift-node-dev-05",
      "openshift-node-dev-06",
      "openshift-node-dev-07",
      "openshift-node-dev-08",
      "openshift-node-dev-09",
      "openshift-node-dev-10"
    ]
  }
}

Duplicate issue #9571: 3.10 Upgrade Fails for All In One Deployment
Created: 2018-08-13T20:35:17Z
Description
With a cluster set up similar to https://github.com/openshift/openshift-ansible/blob/release-3.10/inventory/hosts.localhost upgrade fails with error: Configmap for node-config-all-in-one must be present.
Version
ansible 2.5.3
  config file = /home/centos/openshift-ansible/ansible.cfg
  configured module search path = [u'/home/centos/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Apr 11 2018, 07:36:10) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)]

openshift-ansible-3.10.27-2-69-gd96b19f

Steps To Reproduce

SSH to a host running 3.9 from https://github.com/openshift/openshift-ansible/blob/release-3.10/inventory/hosts.localhost
Run ansible-playbook playbooks/byo/openshift-cluster/upgrades/v3_10/upgrade_control_plane.yml

Expected Results
Cluster should upgrade without any errors.
Observed Results
Ansible throws an error:
Monday 13 August 2018  16:30:33 -0400 (0:00:00.754)       0:00:29.752 ********* 
ok: [devserver.localdomain] => {
    "configout": {
        "changed": false, 
        "failed": false, 
        "results": {
            "cmd": "/bin/oc get configmap node-config-all-in-one -o json -n openshift-node", 
            "results": [
                {}
            ], 
            "returncode": 0, 
            "stderr": "Error from server (NotFound): configmaps \"node-config-all-in-one\" not found\n", 
            "stdout": ""
        }, 
        "state": "list"
    }
}

TASK [openshift_node_group : fail] ********************************************************************************************************************************************************************************************************************************************************************************************
Monday 13 August 2018  16:30:33 -0400 (0:00:00.137)       0:00:29.889 ********* 
fatal: [devserver.localdomain]: FAILED! => {"changed": false, "msg": "Configmap for node-config-all-in-one must be present"}

Additional Information

Closing comment (2018-08-13T20:35:25Z): May be related to #9446