Writing files permission in bitbucket-pipeline
Asked by Dorin Zohar:

Hello,

I'm using bitbucket-pipeline for system tests.

One of the test script (python) is calling: **os.system("python3 run_system.py')**

**run_system.py** - is a script that walks through files in the input folder, then writes file into the output folder.

When I run the system test locally, on my computer, it run without any problem and the test passes.

When I run the system test on bitbucket-pipelines - it fails to write the file into the output folder. What made me think this is a permissions issue.

\* Note that if I run **run_system.py** directly on bitbucket-pipelines (running the script itself and not the test script that runs **run_system.py** using os.system) - it runs perfectly, and writes file without a problem.  
The problem occurs only when I use os.system.

\* I tried using subprocess.Popen instead of os.system - same problem


Answers:
author: Caroline R
body: Hi, [@Dorin Zohar](/t5/user/viewprofilepage/user-id/4661321)! Welcome to the community!

In order to further investigate this issue, could you please confirm:

* What is the exact error you are getting when running the test script in Pipeline?

* How are you running the test script in the Pipeline? Could you please share the syntax of how you are running it?

* Will it be possible to share the Pipeline config, test script, and system script with us? Please remove the sensitive data from the scripts before sharing it with us.

Looking forward to hearing from you.

Kind regards,  
Caroline

comments: author: Dorin Zohar
body: [@Caroline R](/t5/user/viewprofilepage/user-id/3934006) thanks, as for your questions:

* I don't get a specific error - it is just a test and it fails. When I tried to investigate why did it fail, I found out the output files weren't written.
* The test is running using pylint (see config file)
* Config is attached
* Tests are also attached, for simplicity I added only the relevant test (test_system_clean.py) and the runner script (clean_runner.py) with example configuration (clean_config.yaml).   
  It depends on other components (not attached) but the flow is:
  * Creating 10 files in input dir (files created successfully)
  * Creating "test" config (with a structure similar to clean_config.yaml)
  * Running:

    ```
    os.system('python3 ./tools/clean_runner.py')
    ```

    **from inside the test**
  * The clean_runner.py do whatever to the files, then needs to save the new files into the output dir (configured in the config.yaml)

#### bitbucket-pipelines.yml {#toc-hId--507573767}

```
definitions:
    steps:
        - step: &run-test  # step id
            script:  # shell commands to run
                - sed '/.*infra.*$/d' requirements.txt > requirements_without_infra.txt
                - pip install -r requirements_without_infra.txt
                - curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash
                - apt-get install git-lfs
                - pip install pytest-cov
                # run the tests and report the code coverage
                - python -m pytest --cov-report term-missing --cov=. --junitxml=./test-reports/junit.xml ./tests ./tests_solutions

        - step: &run-pylint-pr
              script:
                  - sed '/.*infra.*$/d' requirements.txt > requirements_without_infra.txt
                  - pip install -r requirements_without_infra.txt
                  - pip install pytest-pylint
                  - curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash
                  - apt-get install git-lfs
                  - git checkout $BITBUCKET_PR_DESTINATION_BRANCH
                  - git checkout $BITBUCKET_BRANCH
                  - diff_files=$(git diff-tree -r --diff-filter=D $BITBUCKET_BRANCH $BITBUCKET_PR_DESTINATION_BRANCH --name-only ":(exclude)setup.py" ":(exclude)autodoc.py" | grep -E \.py$ | tr "\n" " ")
                  - echo ${diff_files}  # print the files that were changed
                  - python -m pytest --junitxml=./test-reports/junit.xml --pylint ${diff_files}



clone:
  depth: full
  # you can clone lfs files if you set lfs: true

options:
    max-time: 7 # maximum minutes to run the tests

pipelines:
    pull-requests:  # run when pull request is created (or updated)
        "**":  # on all branches
            - parallel: # steps bellow will be run in parallel
                - step:
                    <<: *run-test  # concatenate the setup-test step from the "definitions" section
                    name: Tests  # name to show on the pipelines web page
                    image: python:3.8.7  # docker to use (from Docker Hub)
```

#### clean_config.yml {#toc-hId-1979939066}

```
augmentor:
  config:
    augmentations:
    - flip
    - rotation
  name: Consecutive
walker:
  config:
    input_dir: /tmp/pytest-of-dorin/pytest-39/test_system0/input
  name: FSWalker
writer:
  config:
    output_dir: /tmp/pytest-of-dorin/pytest-39/test_system0/output
  name: ImageFS
```

#### clean_runner.py {#toc-hId-172484603}

```
import hydra
from omegaconf import DictConfig
from clean_code.manager import Manager


def image_augmentor_runner(config):
    manager = Manager(config)
    manager.run()


@hydra.main(config_path="configs", config_name="clean_config")
def main(config: DictConfig):
    image_augmentor_runner(config)


if __name__ == '__main__':
    main()
```

#### test_system_clean.py {#toc-hId--1634969860}

```
def test_system(tmp_path):
    """
    This is a system test, that checking the output of the entire program (main())
    We create input files (known images), and apply the program on them (with the randomness of the augmentations).
    Then we check that the output files are as expected:
        1. Number of output files == number of input files
        2. Size of each output image == size of its corresponding input image
        3. Sum of pixels of each output image == sum of pixels of its corresponding input image (our augmentations change
        only the order of the pixels, not the values)
    """
    input_path = os.path.join(tmp_path, 'input')
    os.makedirs(input_path, exist_ok=True)

    output_path = os.path.join(tmp_path, 'output')
    os.makedirs(output_path, exist_ok=True)

    num_files_to_test = 10
    test_file_handler = TestFilesHandler(num_files_to_test, input_path)
    test_files_dict = test_file_handler.get_path_to_image_dict()

    config_handler = TestConfigHandler(tmp_path)
    yaml_config = config_handler.test_config

    with open('./tools/configs/clean_config.yaml', 'w') as outfile:
        yaml.dump(yaml_config, outfile, default_flow_style=False)

    os.system('python3 ./tools/clean_runner.py')

    assert len(os.listdir(output_path)) == num_files_to_test

    for out_image_name in os.listdir(output_path):
        out_image_path = os.path.join(output_path, out_image_name)
        out_image = cv2.imread(out_image_path)

        in_image_path = os.path.join(input_path, out_image_name)
        in_image = test_files_dict[in_image_path]

        assert np.sum(in_image) == np.sum(out_image)
        assert in_image.size == out_image.size
```


author: Caroline R
body: Hi, [@Dorin Zohar](/t5/user/viewprofilepage/user-id/4661321),

We are not experts in python, but we performed some tests and managed to create files using os.system in Bitbucket, so we believe this is not a permissions issue. In this case, we will share the steps we have followed:

**bitbucket-pipelines.yml**

    definitions:    steps:        - step: &run-test              script:                  - python test_system_clean.py                - cat /tmp/input/test.txt              pipelines:    branches:          "**":                  - step:                    <<: *run-test                      name: Tests                      image: python:3.8.7                      

**test_system_clean.py**

    import osos.system('python3 clean_runner.py')

**clean_runner.py**

    import osinput_path = os.path.join("/tmp", 'input')os.makedirs(input_path, exist_ok=True)f= open(input_path + "/test.txt","w+")print(input_path + "/test.txt")for i in range(10):     f.write("This is line %d\r\n" % (i+1))

I hope this helps, but do let me know if you have questions.

Also, I see that you have access to workspaces in the Standard plan, so in case this is happening in a repo under these workspaces, you should be able to open a support ticket.

You can open it via <https://support.atlassian.com/contact/#/>, in "What can we help you with?" select "Technical issues and bugs" and then Bitbucket Cloud as the product. Please make sure to inform the corresponding workspace.

Kind regards,  
Caroline


Tags: bitbucket-cloud

bitbucket-pipelines

cloud