Repository: ansible/molecule

Issue #3404: ANSIBLE_ROLES_PATH not set correctly?
Created: 2022-01-18T22:32:40Z
Issue Type

Bug report

Molecule and Ansible details
ansible --version && molecule --version

5.1.0 and 3.5.1
Molecule installation method (one of):

source
pip

Have tried both methods.
Ansible installation method (one of):

source
pip
OS package

pip
Desired Behavior
Molecule tests that have been passing, should continue to be passing.  When running in CI, if roles are installed, the ANSIBLE_ROLES_PATH points to the roles.
Actual Behaviour
Running molecule ansible tests in CircleCI.
Certain tests that succeeded one month ago are failing now, even though nothing apparently changed.  It's not impossible, that this is being caused by something else entirely such as Ansible itself or another package.   However at the moment it looks like molecule so your feedback would be helpful to point in the right direction.

Case 1:
https://app.circleci.com/pipelines/github/cloudalchemy/ansible-prometheus?branch=master&filter=all
When molecule ran on Dec 27, 2021, it succeeded.
Here is a interesting snippet from the log output:
INFO     Added ANSIBLE_ROLES_PATH=/home/circleci/.cache/ansible-compat/dda4ac/roles:/home/circleci/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles
INFO     Using /home/circleci/.ansible/roles/cloudalchemy.prometheus symlink to current repository in order to enable Ansible to find the role using its expected full name.

Notice that it's creating a symlink.
Now,
INFO     Set ANSIBLE_ROLES_PATH=/home/circleci/.cache/ansible-compat/dda4ac/roles:/home/circleci/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles

The wording changed from "Added" to "Set".   So, something changed.  And it does not mention the symlink anymore.  Probably no symlink is being created. Because of that the test fails, during  "molecule syntax".  It can't find the role.

Case 2:
This repo used to succeed, about a month ago.
https://app.circleci.com/pipelines/github/prometheus/demo-site/833/workflows/39439543-5173-4027-8be0-3d1817db7b7c/jobs/3397
Now it's failing:
https://app.circleci.com/pipelines/github/prometheus/demo-site/857/workflows/82accec5-9ae4-4c17-ba50-9c25d35b550e/jobs/3516
When does it fail? During the tests, it gets to this step:
run molecule --debug syntax

Again, it can't find the roles.  Similar to Case 1.
Where are the roles being installed? It can be seen that ansible-galaxy installs roles in /home/circleci/project/roles/
- extracting cloudalchemy.prometheus to /home/circleci/project/roles/cloudalchemy.prometheus

If we test with ansible 5.1 and molecule version 3.4.1, the tests succeed. Here is log output regarding ANSIBLE_ROLES_PATH
INFO     Added ANSIBLE_ROLES_PATH=~/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:roles
ANSIBLE_ROLES_PATH: /home/circleci/.cache/molecule/project/default/roles:/home/circleci:/home/circleci/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:~/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:roles
ANSIBLE_ROLES_PATH=/home/circleci/.cache/molecule/project/default/roles:/home/circleci:/home/circleci/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:~/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:roles 

Next, test with ansible 5.1 and molecule 3.5.1.  This fails.
The wording has changed from "Added" to "Set".   Why?  And, more importantly, the "Set" did not accomplish anything because it didn't set the path.
The debug information (lines 3 and 4) shows something different from the "Set" (line 1).
INFO     Set ANSIBLE_ROLES_PATH=/home/circleci/.cache/ansible-compat/dda4ac/roles:roles:/home/circleci/project/roles
DEBUG: ANSIBLE ENVIRONMENT:
ANSIBLE_ROLES_PATH: '/home/circleci/.cache/molecule/project/default/roles:/home/circleci:/home/circleci/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:'
ANSIBLE_ROLES_PATH=/home/circleci/.cache/molecule/project/default/roles:/home/circleci:/home/circleci/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles: ANSIBLE_VAULT_PASSWORD=******* MOLECULE_DEBUG=True

Duplicate issue #3581: ERROR! the role 'acme.myrole' was not found. "symlink to current repository" not used?!
Created: 2022-06-10T15:35:06Z
Issue Type

Bug report

Molecule and Ansible details
ansible --version && molecule --version
ansible [core 2.12.6]
  config file = None
  configured module search path = ['/Users/maarten/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/maarten/.pyenv/versions/3.10.4/envs/test/lib/python3.10/site-packages/ansible
  ansible collection location = /Users/maarten/.ansible/collections:/usr/share/ansible/collections
  executable location = /Users/maarten/.pyenv/versions/test/bin/ansible
  python version = 3.10.4 (main, Jun  4 2022, 20:32:38) [Clang 13.0.0 (clang-1300.0.29.30)]
  jinja version = 3.1.2
  libyaml = True
molecule 3.6.1 using python 3.10
    ansible:2.12.6
    delegated:3.6.1 from molecule
    docker:1.1.0 from molecule_docker requiring collections: community.docker>=1.9.1

Molecule installation method (one of):

pip

Ansible installation method (one of):

pip

test setup:
❯ pwd
/Users/maarten/test
❯ pyenv virtualenv test
❯ pyenv activate test
❯ pip install -U pip setuptools
❯ pip install  "molecule[docker]" ansible
❯ pyenv which molecule
/Users/maarten/.pyenv/versions/test/bin/molecule
❯ rm -rf ~/.cache/molecule
❯ molecule init role -d docker acme.myrole
❯ cd myrole
❯ cat << EOF > meta/main.yml
galaxy_info:
  author: My Name
  role_name: myrole
  namespace: acme
  description: My role description
  license: MIT
EOF
❯ molecule create

Detail any linters or test runners used:

none

Desired Behavior
Have molecule converge find the role without  setting the ANSIBLE_ROLES_PATH environment variable on command line explicitly:

❯  ANSIBLE_ROLES_PATH=/Users/maarten/.cache/ansible-compat/a8f7d1/roles:/Users/maarten/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles molecule --debug converge
DEBUG    Validating schema /Users/maarten/test/myrole/molecule/default/molecule.yml.
INFO     default scenario test matrix: dependency, create, prepare, converge
INFO     Performing prerun...
INFO     Set ANSIBLE_LIBRARY=/Users/maarten/.cache/ansible-compat/a8f7d1/modules:/Users/maarten/.ansible/plugins/modules:/usr/share/ansible/plugins/modules
INFO     Set ANSIBLE_COLLECTIONS_PATH=/Users/maarten/.cache/ansible-compat/a8f7d1/collections:/Users/maarten/.ansible/collections:/usr/share/ansible/collections
INFO     Using /Users/maarten/.cache/ansible-compat/a8f7d1/roles/acme.myrole symlink to current repository in order to enable Ansible to find the role using its expected full name.
INFO     Running default > dependency
WARNING  Skipping, missing the requirements file.
WARNING  Skipping, missing the requirements file.
INFO     Running default > create
WARNING  Skipping, instances already created.
INFO     Running default > prepare
WARNING  Skipping, prepare playbook not configured.
INFO     Running default > converge
INFO     Sanity checks: 'docker'
DEBUG: ANSIBLE ENVIRONMENT:
ANSIBLE_COLLECTIONS_PATH: /Users/maarten/.cache/molecule/myrole/default/collections:/Users/maarten/.ansible/collections:/usr/share/ansible/collections:/etc/ansible/collections
ANSIBLE_CONFIG: /Users/maarten/.cache/molecule/myrole/default/ansible.cfg
ANSIBLE_FILTER_PLUGINS: /Users/maarten/.pyenv/versions/3.10.4/envs/test/lib/python3.10/site-packages/molecule/provisioner/ansible/plugins/filter:/Users/maarten/.cache/molecule/myrole/default/plugins/filter:/Users/maarten/test/myrole/plugins/filter:/Users/maarten/.ansible/plugins/filter:/usr/share/ansible/plugins/filter
ANSIBLE_FORCE_COLOR: '1'
ANSIBLE_LIBRARY: /Users/maarten/.pyenv/versions/3.10.4/envs/test/lib/python3.10/site-packages/molecule/provisioner/ansible/plugins/modules:/Users/maarten/.cache/molecule/myrole/default/library:/Users/maarten/test/myrole/library:/Users/maarten/.ansible/plugins/modules:/usr/share/ansible/plugins/modules
ANSIBLE_ROLES_PATH: /Users/maarten/.cache/molecule/myrole/default/roles:/Users/maarten/test:/Users/maarten/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:/Users/maarten/.cache/ansible-compat/a8f7d1/roles:/Users/maarten/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles

DEBUG: MOLECULE ENVIRONMENT:
MOLECULE_DEBUG: 'True'
MOLECULE_DEPENDENCY_NAME: galaxy
MOLECULE_DRIVER_NAME: docker
MOLECULE_ENV_FILE: /Users/maarten/test/myrole/.env.yml
MOLECULE_EPHEMERAL_DIRECTORY: /Users/maarten/.cache/molecule/myrole/default
MOLECULE_FILE: /Users/maarten/.cache/molecule/myrole/default/molecule.yml
MOLECULE_INSTANCE_CONFIG: /Users/maarten/.cache/molecule/myrole/default/instance_config.yml
MOLECULE_INVENTORY_FILE: /Users/maarten/.cache/molecule/myrole/default/inventory/ansible_inventory.yml
MOLECULE_PROJECT_DIRECTORY: /Users/maarten/test/myrole
MOLECULE_PROVISIONER_NAME: ansible
MOLECULE_SCENARIO_DIRECTORY: /Users/maarten/test/myrole/molecule/default
MOLECULE_SCENARIO_NAME: default
MOLECULE_STATE_FILE: /Users/maarten/.cache/molecule/myrole/default/state.yml
MOLECULE_VERIFIER_NAME: ansible
MOLECULE_VERIFIER_TEST_DIRECTORY: /Users/maarten/test/myrole/molecule/default/tests

DEBUG: SHELL REPLAY:
ANSIBLE_COLLECTIONS_PATH=/Users/maarten/.cache/molecule/myrole/default/collections:/Users/maarten/.ansible/collections:/usr/share/ansible/collections:/etc/ansible/collections ANSIBLE_CONFIG=/Users/maarten/.cache/molecule/myrole/default/ansible.cfg ANSIBLE_FILTER_PLUGINS=/Users/maarten/.pyenv/versions/3.10.4/envs/test/lib/python3.10/site-packages/molecule/provisioner/ansible/plugins/filter:/Users/maarten/.cache/molecule/myrole/default/plugins/filter:/Users/maarten/test/myrole/plugins/filter:/Users/maarten/.ansible/plugins/filter:/usr/share/ansible/plugins/filter ANSIBLE_FORCE_COLOR=1 ANSIBLE_LIBRARY=/Users/maarten/.pyenv/versions/3.10.4/envs/test/lib/python3.10/site-packages/molecule/provisioner/ansible/plugins/modules:/Users/maarten/.cache/molecule/myrole/default/library:/Users/maarten/test/myrole/library:/Users/maarten/.ansible/plugins/modules:/usr/share/ansible/plugins/modules ANSIBLE_ROLES_PATH=/Users/maarten/.cache/molecule/myrole/default/roles:/Users/maarten/test:/Users/maarten/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:/Users/maarten/.cache/ansible-compat/a8f7d1/roles:/Users/maarten/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles MOLECULE_DEBUG=True MOLECULE_DEPENDENCY_NAME=galaxy MOLECULE_DRIVER_NAME=docker MOLECULE_ENV_FILE=/Users/maarten/test/myrole/.env.yml MOLECULE_EPHEMERAL_DIRECTORY=/Users/maarten/.cache/molecule/myrole/default MOLECULE_FILE=/Users/maarten/.cache/molecule/myrole/default/molecule.yml MOLECULE_INSTANCE_CONFIG=/Users/maarten/.cache/molecule/myrole/default/instance_config.yml MOLECULE_INVENTORY_FILE=/Users/maarten/.cache/molecule/myrole/default/inventory/ansible_inventory.yml MOLECULE_PROJECT_DIRECTORY=/Users/maarten/test/myrole MOLECULE_PROVISIONER_NAME=ansible MOLECULE_SCENARIO_DIRECTORY=/Users/maarten/test/myrole/molecule/default MOLECULE_SCENARIO_NAME=default MOLECULE_STATE_FILE=/Users/maarten/.cache/molecule/myrole/default/state.yml MOLECULE_VERIFIER_NAME=ansible MOLECULE_VERIFIER_TEST_DIRECTORY=/Users/maarten/test/myrole/molecule/default/tests


PLAY [Converge] ****************************************************************

TASK [Gathering Facts] *********************************************************
ok: [instance]

TASK [Include acme.myrole] *****************************************************

PLAY RECAP *********************************************************************
instance                   : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Actual Behaviour
Converge (without setting the ANSIBLE_ROLES_PATH environment variable on command line explicitly):
❯ molecule --debug converge
DEBUG    Validating schema /Users/maarten/test/myrole/molecule/default/molecule.yml.
INFO     default scenario test matrix: dependency, create, prepare, converge
INFO     Performing prerun...
INFO     Set ANSIBLE_LIBRARY=/Users/maarten/.cache/ansible-compat/a8f7d1/modules:/Users/maarten/.ansible/plugins/modules:/usr/share/ansible/plugins/modules
INFO     Set ANSIBLE_COLLECTIONS_PATH=/Users/maarten/.cache/ansible-compat/a8f7d1/collections:/Users/maarten/.ansible/collections:/usr/share/ansible/collections
INFO     Set ANSIBLE_ROLES_PATH=/Users/maarten/.cache/ansible-compat/a8f7d1/roles:/Users/maarten/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles
INFO     Using /Users/maarten/.cache/ansible-compat/a8f7d1/roles/acme.myrole symlink to current repository in order to enable Ansible to find the role using its expected full name.
INFO     Running default > dependency
WARNING  Skipping, missing the requirements file.
WARNING  Skipping, missing the requirements file.
INFO     Running default > create
WARNING  Skipping, instances already created.
INFO     Running default > prepare
WARNING  Skipping, prepare playbook not configured.
INFO     Running default > converge
INFO     Sanity checks: 'docker'
DEBUG: ANSIBLE ENVIRONMENT:
ANSIBLE_COLLECTIONS_PATH: /Users/maarten/.cache/molecule/myrole/default/collections:/Users/maarten/.ansible/collections:/usr/share/ansible/collections:/etc/ansible/collections
ANSIBLE_CONFIG: /Users/maarten/.cache/molecule/myrole/default/ansible.cfg
ANSIBLE_FILTER_PLUGINS: /Users/maarten/.pyenv/versions/3.10.4/envs/test/lib/python3.10/site-packages/molecule/provisioner/ansible/plugins/filter:/Users/maarten/.cache/molecule/myrole/default/plugins/filter:/Users/maarten/test/myrole/plugins/filter:/Users/maarten/.ansible/plugins/filter:/usr/share/ansible/plugins/filter
ANSIBLE_FORCE_COLOR: '1'
ANSIBLE_LIBRARY: /Users/maarten/.pyenv/versions/3.10.4/envs/test/lib/python3.10/site-packages/molecule/provisioner/ansible/plugins/modules:/Users/maarten/.cache/molecule/myrole/default/library:/Users/maarten/test/myrole/library:/Users/maarten/.ansible/plugins/modules:/usr/share/ansible/plugins/modules
ANSIBLE_ROLES_PATH: '/Users/maarten/.cache/molecule/myrole/default/roles:/Users/maarten/test:/Users/maarten/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:'

DEBUG: MOLECULE ENVIRONMENT:
MOLECULE_DEBUG: 'True'
MOLECULE_DEPENDENCY_NAME: galaxy
MOLECULE_DRIVER_NAME: docker
MOLECULE_ENV_FILE: /Users/maarten/test/myrole/.env.yml
MOLECULE_EPHEMERAL_DIRECTORY: /Users/maarten/.cache/molecule/myrole/default
MOLECULE_FILE: /Users/maarten/.cache/molecule/myrole/default/molecule.yml
MOLECULE_INSTANCE_CONFIG: /Users/maarten/.cache/molecule/myrole/default/instance_config.yml
MOLECULE_INVENTORY_FILE: /Users/maarten/.cache/molecule/myrole/default/inventory/ansible_inventory.yml
MOLECULE_PROJECT_DIRECTORY: /Users/maarten/test/myrole
MOLECULE_PROVISIONER_NAME: ansible
MOLECULE_SCENARIO_DIRECTORY: /Users/maarten/test/myrole/molecule/default
MOLECULE_SCENARIO_NAME: default
MOLECULE_STATE_FILE: /Users/maarten/.cache/molecule/myrole/default/state.yml
MOLECULE_VERIFIER_NAME: ansible
MOLECULE_VERIFIER_TEST_DIRECTORY: /Users/maarten/test/myrole/molecule/default/tests

DEBUG: SHELL REPLAY:
ANSIBLE_COLLECTIONS_PATH=/Users/maarten/.cache/molecule/myrole/default/collections:/Users/maarten/.ansible/collections:/usr/share/ansible/collections:/etc/ansible/collections ANSIBLE_CONFIG=/Users/maarten/.cache/molecule/myrole/default/ansible.cfg ANSIBLE_FILTER_PLUGINS=/Users/maarten/.pyenv/versions/3.10.4/envs/test/lib/python3.10/site-packages/molecule/provisioner/ansible/plugins/filter:/Users/maarten/.cache/molecule/myrole/default/plugins/filter:/Users/maarten/test/myrole/plugins/filter:/Users/maarten/.ansible/plugins/filter:/usr/share/ansible/plugins/filter ANSIBLE_FORCE_COLOR=1 ANSIBLE_LIBRARY=/Users/maarten/.pyenv/versions/3.10.4/envs/test/lib/python3.10/site-packages/molecule/provisioner/ansible/plugins/modules:/Users/maarten/.cache/molecule/myrole/default/library:/Users/maarten/test/myrole/library:/Users/maarten/.ansible/plugins/modules:/usr/share/ansible/plugins/modules ANSIBLE_ROLES_PATH=/Users/maarten/.cache/molecule/myrole/default/roles:/Users/maarten/test:/Users/maarten/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles: MOLECULE_DEBUG=True MOLECULE_DEPENDENCY_NAME=galaxy MOLECULE_DRIVER_NAME=docker MOLECULE_ENV_FILE=/Users/maarten/test/myrole/.env.yml MOLECULE_EPHEMERAL_DIRECTORY=/Users/maarten/.cache/molecule/myrole/default MOLECULE_FILE=/Users/maarten/.cache/molecule/myrole/default/molecule.yml MOLECULE_INSTANCE_CONFIG=/Users/maarten/.cache/molecule/myrole/default/instance_config.yml MOLECULE_INVENTORY_FILE=/Users/maarten/.cache/molecule/myrole/default/inventory/ansible_inventory.yml MOLECULE_PROJECT_DIRECTORY=/Users/maarten/test/myrole MOLECULE_PROVISIONER_NAME=ansible MOLECULE_SCENARIO_DIRECTORY=/Users/maarten/test/myrole/molecule/default MOLECULE_SCENARIO_NAME=default MOLECULE_STATE_FILE=/Users/maarten/.cache/molecule/myrole/default/state.yml MOLECULE_VERIFIER_NAME=ansible MOLECULE_VERIFIER_TEST_DIRECTORY=/Users/maarten/test/myrole/molecule/default/tests


PLAY [Converge] ****************************************************************

TASK [Gathering Facts] *********************************************************
ok: [instance]

TASK [Include acme.myrole] *****************************************************
ERROR! the role 'acme.myrole' was not found in /Users/maarten/test/myrole/molecule/default/roles:/Users/maarten/.cache/molecule/myrole/default/roles:/Users/maarten/test:/Users/maarten/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:/Users/maarten/test/myrole:/Users/maarten/test/myrole/molecule/default

The error appears to be in '/Users/maarten/test/myrole/molecule/default/converge.yml': line 7, column 15, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

      include_role:
        name: "acme.myrole"
              ^ here

PLAY RECAP *********************************************************************
instance                   : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

WARNING  Retrying execution failure 2 of: ansible-playbook --diff --inventory /Users/maarten/.cache/molecule/myrole/default/inventory --skip-tags molecule-notest,notest /Users/maarten/test/myrole/molecule/default/converge.yml
CRITICAL Ansible return code was 2, command was: ['ansible-playbook', '--diff', '--inventory', '/Users/maarten/.cache/molecule/myrole/default/inventory', '--skip-tags', 'molecule-notest,notest', '/Users/maarten/test/myrole/molecule/default/converge.yml'][

Closing comment (2022-06-14T07:54:11Z): Reproduce the issue with Molecule-3.6.1
# optional: remove all user-installed Python packages
# python3 -m pip list --user | awk '{print $1}' | tail +3 | xargs python3 -m pip uninstall -y

git clone https://github.com/kilip/ansible-role-workstation
cd ansible-role-workstation

python3 -m pip install --user ansible molecule molecule-docker docker
python3 -m molecule test

The error
ERROR! the role 'kilip.workstation' was not found in /home/o/prj/ansible-role-workstation/molecule/default/playbooks/roles:/home/o/.cache/molecule/ansible-role-workstation/default/roles:/home/o/prj:/home/o/.ansible/roles:/usr/share/ansible/roles:/etc/ansible/roles:/home/o/prj/ansible-role-workstation:/home/o/prj/ansible-role-workstation/molecule/default/playbooks

The error appears to be in '/home/o/prj/ansible-role-workstation/molecule/default/playbooks/base.yml': line 8, column 7, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

  roles:
    - "kilip.workstation"
      ^ here
WARNING  Retrying execution failure 1 of: ansible-playbook --inventory /home/o/.cache/molecule/ansible-role-workstation/default/inventory --skip-tags molecule-notest,notest --syntax-check /home/o/prj/ansible-role-workstation/molecule/default/playbooks/base.yml

Install molecule v4.0.0a0
$ python3 -m pip install --user --index-url https://test.pypi.org/simple --extra-index-url https://pypi.org/simple molecule==4.0.0a0
[...]
Installing collected packages: molecule
  Attempting uninstall: molecule
    Found existing installation: molecule 3.6.1
    Uninstalling molecule-3.6.1:
      Successfully uninstalled molecule-3.6.1
Successfully installed molecule-4.0.0a0

Success
python3 -m molecule test

Release molecule-4.0.0 ASAP
Thank you @ssbarnea @zhan9san for your great and very useful work. But, please, release molecule-4.0.0 as soon as possible because this is a blocking issue for many Molecule users.
Similar issue
If think this issue is the same as #3404