fts3 3.14.2
FTS Python3 CLI and libraries
License: Apache License 2.0
Requires Python: >=3.9

FTS3-REST-Flask
        ===============
        
        FTS3-REST-Flask is the RESTful API for the FTS3 Service.  
        
        This is the front-facing component to communicate with the FTS3 Service,
        allowing submission of transfers, querying transfer updates, as well as configuration management.  
        
        The project consists of an Apache module running Python3 + Flask. 
        It is also a direct continuation of the former [fts-rest][1], where the migration 
        decision and evaluation can be seen [here][2].
        
        The officially supported platform is Alma9. The server part is provided via RPMs
        for the official platform. The client is provided via EPEL for multiple platforms,
        as well as via [PyPi/fts3][3].
        
        ## Installation
        
        The project should be installed via RPMs, provided via the FTS repository.
        
        ### Installing the client
        
        Make sure to have the FTS repositories enabled:
        - [production][4] repository: official RPMs deemed ready for production
        - [development][5] repository: RPMs built on the latest `develop` branch commit
        
        #### Installing the client:
        ```shell
        $ dnf install -y fts-rest-client
        ```
        
        #### Installing the server
        
        The project relies on `mod_wsgi` built for Python 3.9. Some of the dependencies
        require the `crb` (CodeReady Linux Builder) repository to be enabled.
        All requirements are specified and should be brought-in via the spec file.
        ```shell
        $ dnf config-manager --enable crb
        $ dnf install -y fts-rest-server
        ```
        
        Configuring the server is done via the following two configuration files:
        - `/etc/fts3/fts3restconfig`
        - `/etc/htpd/conf.d/fts3rest.conf`
        
        ## Development
        
        The project is tracked using JIRA, under the [FTS Project][6] (requires CERN log-in). 
        Soon the project will be mirrored on GitHub, where issues and pull requests are also accepted (and encouraged!). 
        
        For development purposes, using a virtual environment is encouraged.
        
        ### Git Workflow
        
        This project follows a simplified [GitFlow][7] model. The main branch is the `develop` branch,
        with `master` being reserved only for tagged released.
        
        Feature branches are developed separately and merged into the `develop` branch.
        When preparing a release, either `develop` is merged directly into `master`
        or a release branch is created. Production hotfixes are allowed only in case of necessity.
        Both release and hotfix branch changes must be merged back into `develop`.
        
        ### Development workflow
        
        When taking on a task, the following workflow model should be followed:
        1. Create a JIRA ticket for the identified problem
        2. Estimate the magnitude of the task:
            - Small changes can go directly on the `develop` branch
            - Larger changes must go into a separate feature branch
        3. Implement and test your changes
        4. Merge them into the `develop` branch:
            - Merge large changes via merge requests and request code review
            - Changes affecting the database must also be code reviewed!
        5. Close the JIRA ticket and mark the appropriate `fixVersion`
        
        ### Continuous Integration
        
        The project uses Gitlab-CI for CI/CD. The [pipeline][8] runs for every push, in every branch:
        - black: checks code must be formatted with `black`
        - pylint: checks for syntax errors (runs for every supported Python3 version)
          - If you are sure that pylint is mistaken, add `# pylint: skip-file` at the beginning of the relevant file
        - functional tests: run for every supported Python3 version
        - bandit: detects potential security issues in the code
          - Allowed to fail as it flags many false positives.
          - To ignore a false positive, append `# nosec` to the offending line
        - build: RPM for the client and server, plus `sdist` and `wheel` for the client
        - deploy: upload client and server RPM to the FTS testing repository
        
        Merge requests will proceed only if the pipeline succeeds.  
        In case of emergency the pipeline can be [skipped][9].
        
        The pipeline runs in a container from the image tagged as `ci`. The dockerfile is in the `.gitlab-ci` directory and the 
        image is hosted in the container registry of this project. The image contains the pre-installed Python environment 
        in order to speed up CI execution. When new environment is desired (such as new or upgraded dependencies), 
        a new image must be created via the `.gitlab-ci/docker/create-ci-image.sh` script.
        
        #### During development
        
        Developers may add the `.gitlab-ci/pre-commit` hook to their local repository, 
        enabling the following checks before every commit:
        - Run `black` to format the changed files
        - Run `pylint` only on the changed files
          - Note: As pylint works better when it is run on the full project, some rules have been disabled
        - Run `radon` and `bandit` only on the changed files
          - In case of false positives, the hook can be skipped with the commit option `--no-verify`
        
        A convenience script is provided to install the `pre-commit` hook:
        ```shell
        $ cd .gitlab-ci/
        $ ./precommit_install.sh
        ```
        
        ### Python dependencies
        
        The project uses [pip-tools][10] to manage dependencies:
        - `requirements.in`: list of dependencies for the production app
        - `dev-requirements.in`: extra list of packages used for development (e.g. static code analysis)
        
        ### Setting up the development environment
        
        The `pipcompile.sh` script is provided to help set up a development environment:
        ```shell
        $ virtualenv venv
        $ source venv/bin/activate
        (venv) $ pip install --upgrade pip
        (venv) $ pip install pip-tools
        (venv) $ ./pipcompile.sh --sync
        ```
        
        Running `pipcompile.sh` will generate `requirements.txt`. Running the command with `--sync`
        will also synchronize the virtual environment with the requirements.
        
        Additionally, you will need to configure the HTTPd config file. 
        A template version for development is offered at `src/fts3rest/fts3rest_dev_conf.in`. 
        To fill it with the right paths and install it, run the `install-httpd-dev.sh` script:
        ```shell
        $ cd src/fts3rest/
        $ ./install-httpd-dev.sh
        ```
        
        ### Functional tests
        
        The functional tests reside under `src/fts3rest/fts3rest/tests/functional`. 
        
        In oder to run the tests, a fully functional MySQL-compatible database is required.
        The database must have the appropriate FTS schema and the connection details configured 
        in the FTS3 REST config file. 
        
        Example on Centos 7 with locally installed `mariadb`: 
        ```shell
        $ wget https://gitlab.cern.ch/fts/fts3/-/raw/v3.11.0/src/db/schema/mysql/fts-schema-7.0.0.sql
        $ mysql -u root --execute "CREATE DATABASE ftsflask;"
        $ mysql -u root --execute "CREATE USER 'fts3'@'%';"
        $ mysql -u root --execute "GRANT ALL PRIVILEGES ON ftsflask.* TO 'fts3'@'%' IDENTIFIED BY 'ftsflaskpass';"
        $ mysql -u fts3 --password=ftsflaskpass ftsflask
        MariaDB [ftsflask]> source fts-schema-7.0.0.sql
        ```
        
        Before running the tests, make sure to include the `fts3rest` project in the `PYTHONPATH`.  
        Additionally, the `FTS3TESTCONFIG` environment variable can also be set to use a different config file during testing:
        ```shell
        $ export PYTHONPATH="${PROJECT_DIR}/src:${PROJECT_DIR}/src/fts3rest"
        $ export FTS3TESTCONFIG="${PROJECT_DIR}/src/fts3rest/fts3rest/tests/fts3testconfig"
        ```
        
        For ease-of-use, the `runtests.sh` script is provided:
        ```shell
        $ cd src/fts3rest
        $ ./runtests.sh
        ```
        
        **Note**: OIDC tests are not run in the CI as the container would need a client registered and this has not yet been set up.   
        To run OIDC tests in a development environment, the environment variables `xdc_ClientId` and `xdc_ClientSecret` must be set.
        
        ### Building packages
        
        In order to build RPM packages, two utility scripts are provided 
        for the client and server: `fts-rest-client.sh` and `fts-rest-server.sh`.
        
        The following example shows how to build server packages:
        ```shell
        $ ./packaging/server/fts-rest-server.sh
        $ tree "build/server/RPMS"
        ```
        
        ## Useful links
        
        - Documentation: https://fts3-docs.web.cern.ch/fts3-docs/fts-rest/docs/index.html
        - FTS Repository: https://fts-repo.web.cern.ch/fts-repo/
        - Apache License, version 2.0: https://www.apache.org/licenses/LICENSE-2.0
        - Support channels: fts-support@cern.ch
        - Development team contact: fts-devel@cern.ch 
        
        
        [1]: https://gitlab.cern.ch/fts/fts-rest
        [2]: https://its.cern.ch/jira/browse/FTS-1496
        [3]: https://pypi.org/project/fts3/
        [4]: https://fts-repo.web.cern.ch/fts-repo/el9/x86_64/
        [5]: https://fts-repo.web.cern.ch/fts-repo/testing/el9/x86_64/
        [6]: https://its.cern.ch/jira/projects/FTS/issues
        [7]: https://nvie.com/posts/a-successful-git-branching-model/
        [8]: https://gitlab.cern.ch/fts/fts-rest-flask/-/pipelines
        [9]: https://docs.gitlab.com/ee/ci/yaml/#skipping-jobs
        [10]: https://github.com/jazzband/pip-tools