Multiple repositories Pull Request chaos, crawl them all in one single place
Tags: react, devops, github, tooling
Published: 2022-09-16 06:42:25

It is very frequent to find software engineering projects where multiple repositories are involved for the same or different projects, somehow related between them, a lot of people push their pull requests to any of them and it is very normal to lose tracking of the situation or you have to constantly browse them all to have a clearer picture about what is going on. That’s the situation we had here at the Red Hat Business Automation team and we solved it by creating a helpful tool you can easily use for your set of projects, easy, quick and for free.

## The cross-repo PRs problem
This is already covered by Cross-repo Pull Requests? build-chain tool to the rescue! entry, so feel free to read it in case you are not familiar with this kind of situation or concepts.

## The Chain-Status solution
So we said to ourselves, what if we would have a centralized place, a web page for instance, to be able to see in a quick look what’s the situation about all the pull requests for all of our repositories? **Chain Status** was the solution.

Prerequisites:

- It has to solve not only our particular problem, so anyone can use it.
- It has to be public, no authentication required.
- It has to be fast, we can’t wait for the whole pull request set to be crawled everytime anyone gets into the application.
- Multiple streams or different project set can be handled in different views, like different products or product versions from the same place.
- The content can be filtered out.

So the conclusion was to create in one hand a React web page to consume the pull request information from a static report and another tool to generate that report based on Github information. This way:

- The information will be produced asynchronously, the frequency will be up to the user/developer and Github API rate limit problems will be avoided.
- The information can be taken even from private repositories and be exposed publicly and no authentication will be required. 
- No waiting time while information is requested from Github service.
- The webpage (HTML+JS files) can be stored on any web service, even on free services like Github Pages or Netlify.
- No backend server is required.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/h3xkkp8didiki0c20o11.png)

## Running example

You can check KIE RHBA status web page at https://kiegroup.github.io/droolsjbpm-build-bootstrap/status/kiegroup-status

## How can I add it to my organization?

The best way to integrate this tool in your organization or set of repositories is by using the provided configurable [Github actions](https://docs.github.com/en/actions). In particular this tool comes with two main easy-to-use [actions](https://github.com/kiegroup/chain-status/tree/main/.ci/actions):

- Generate App: this action aims to build and copy the React web application inside your repository and publish it using [gh-pages](https://www.npmjs.com/package/gh-pages) NPM tool.
- Generate Data: given a project structure and some project information as input, this action is focused on generating the data report gathering the information using the Github API. This report is then used by the web application as a content source.

Thus, in order to use these actions on your organization, you only have to add two [Github workflows](https://docs.github.com/en/actions/using-workflows) (one per action) on your main repository as follows:

1. Prerequisites: having a Github token properly configured in your organization, see more details on how to configure it.
1. Generate app workflow (generate_status_page.yaml): add the Github workflow for the web page generation, this should generally be run only once (or whenever there are changes on the web app look and feel).

    ```yaml
    name: Generate status page
    on: workflow_dispatch
    
    jobs:
      generate-status-page:
        if: github.repository_owner == '<OWNER>'
        concurrency:
          group: generate-status-page
          cancel-in-progress: true
        strategy:
          matrix:
            os: [ubuntu-latest]
          fail-fast: true
        runs-on: ubuntu-latest
        name: Generate status page
        steps:
          - name: Generate status page
            uses: kiegroup/chain-status/.ci/actions/generate-app@main
            with:
              info-md-url: "<PATH-TO-INFO>"
              github-token: "${{ secrets.GITHUB_TOKEN }}"
              gh-pages-branch: "gh-pages"
    ```

1. Generate data workflow (generate_status_page_data.yaml): add the periodic workflow that will continuously generate the data fetched by the web application.

    ```yaml
    name: Generate status page data
    
    on:
      workflow_dispatch:
      schedule:
        - cron: '0 * * * *'
    jobs:
      generate-status-page-data:
        if: github.repository_owner == '<OWNER>'
        concurrency:
          group: generate-status-page-data
          cancel-in-progress: true
        strategy:
          matrix:
            os: [ubuntu-latest]
          fail-fast: true
        runs-on: ubuntu-latest
        name: Generate status page data
        steps:
          - name: Generate status page data
            uses: kiegroup/chain-status/.ci/actions/generate-data@main
            with:
              definition-file: <PATH-TO-DEFINITION-FILE>
              # projects: <PROJECTS-LIST>
              title: <TITLE>
              subtitle: <SUBTITLE>
              base-branch-filter: <BRANCH-LIST>
              created-by: Github Action
              created-url: https://github.com/${{ github.repository }}/actions/runs/${{ github.run_id }}
              logger-level: debug
              github-token: "${{ secrets.GITHUB_TOKEN }}"
              gh-pages-branch: "gh-pages" 
    ```

As already introduced, the generate data flow relies on a project structure definition which can be provided either using build-chain definition file or a projects list:

- Build-chain definition file (using ‘definition-file’ field), a YAML definition file for cross-related inter-dependent projects which was introduced for Github action build chain. This tool is already covered by Cross-repo Pull Requests? build-chain tool to the rescue!, so feel free to read it if you want to get more details on it and on its definition files.
- Projects list (using ‘projects’ field), a comma-separated list of projects for which you would like to provide Pull Requests statuses. [Still a Work in Progress PR-35]

This was a brief explanation on how you could integrate this tool in your organization, if you need more details on this feel free to reach the [chain-status homepage](https://github.com/kiegroup/chain-status), where you can find a step-by-step guide on how to integrate it with some links to running examples.

## Additional functionalities
Additionally to the pull request summary functionality, it is also possible to add multiple Jenkins status reports.

The main advantage of this feature is that you can check the status of all your Jenkins jobs in a single place, making it easier to check what runs succeeded/failed and also the time and average time jobs are consuming.

As an example you can check the KIE RHBA daily builds page https://kiegroup.github.io/droolsjbpm-build-bootstrap/job/daily-builds 

To configure the Jenkins status reports feature, you can create a Jenkins pipeline that will generate and update the data periodically. You can schedule the Jenkins pipeline to run and keep the status updated based on your required demand.

You can add the following steps as part of your Jenkins pipeline to generate and update the status report:

1. Clone the GitHub pages repository
    ```JSON
    stage('Clone gh-pages repository') {
      steps {
        script {
          println "Checking out https://github.com/${ghPagesRepository}:${ghPagesBranch} into ${ghPagesRepoFolder} folder"
          sh "git clone -b ${ghPagesBranch} --single-branch https://github.com/${ghPagesRepository} ${ghPagesRepoFolder}"
        }
      }
    }
    ```
 
1. Install the chain-status tool
    ```JSON
    stage('Install chain-status tool') {
      steps {
        script {
          try {
            sh "npm install -g @kie/chain-status-action"
          } catch(e) {
            println '[WARNING] Error installing @kie/chain-status-action.'
          }
        }
      }
    }
    
    ```

1. Generate the updated data
    ```JSON
    stage('Generate data') {
      steps {
        script {
          dir(ghPagesRepoFolder) {
            sh "build-chain-status-report --jenkinsUrl ${jenkinsURL} --jobUrl ${jenkinsJobPath} -t ${projectTitle} -st ${projectSubtitle} --certFilePath ${jenkinsCertFile} --outputFolderPath ./data/ --skipZero -cb \"Jenkins Job\" -cu \"${env.BUILD_URL}\" --order 1001"
          }
        }
      }
    }
    ```

1. Push changes to update the status report
    ```JSON
    stage('Push changes to repository') {
      steps {
        script {
          println "Pushing changes to ${ghPagesRepository}:${ghPagesBranch}"
            dir(ghPagesRepoFolder) {
              withCredentials([usernamePassword(credentialsId: "${githubCredentialsId}", usernameVariable: 'GITHUB_USER', passwordVariable: 'GITHUB_TOKEN')]) {
    githubscm.setUserConfig("${GITHUB_USER}")
              sh("git config --local credential.helper \"!f() { echo username=\\$GITHUB_USER; echo password=\\$GITHUB_TOKEN; }; f\"")
              sh 'git add data/*'
              sh 'git commit -m "Generate Jenkins Data"'
              sh "git push origin ${ghPagesBranch}"                            
            }
          }
        }
      }
    }
    ```

## Useful links
* [Chain status] https://github.com/kiegroup/chain-status/ 
* [Build chain tool] https://github.com/kiegroup/github-action-build-chain
* [Build chain npm package] https://www.npmjs.com/package/@kie/build-chain-action 
* [Configuration reader] https://github.com/kiegroup/build-chain-configuration-reader
* [RHBA definition and project tree files] https://github.com/kiegroup/droolsjbpm-build-bootstrap/tree/e1e15170d85cc9c9c6b67bd02106fd89c9d3f603/.ci 
* [RHBA flows] https://github.com/kiegroup/droolsjbpm-build-bootstrap/tree/e1e15170d85cc9c9c6b67bd02106fd89c9d3f603/.github/workflows
* More info [KIE Blog](https://blog.kie.org/?p=16323)

> Featured photo by https://www.flickr.com