vania-janet/multiturn-rag-retrieval-data huggingface dataset:
The following example script runs tests and then uploads Mocha results to DevOps Insights:

 Run tests and generate a test results file here.
istanbul cover --report lcov --report json-summary _mocha -- -R xunit -O output=./test/results/mocha.xml 'test//.js'

install the DevOps Insights plugin
ibmcloud plugin install -f doi

 Login to IBMCloud if you are not already logged in. Assumes that $API_KEY environment variable has been set as a secured property
ibmcloud login --apikey $API_KEY --no-region

Publish test results. Assumes that MY_APP_NAME and MY_BUILD_NUMBER environment variables are already set
ibmcloud doi testrecord-publish --logicalappname="$MY_APP_NAME" --buildnumber="$MY_BUILD_NUMBER" --filelocation /test/results/mocha.xml --type unittest
ibmcloud doi testrecord-publish --logicalappname="$MY_APP_NAME" --buildnumber="$MY_BUILD_NUMBER" --filelocation ./coverage/coverage-summary.json --type code
Show more

The command supports the following type values:



Table 1. Test record types

 Type Description 

 unittest Unit test results 
 fvt Functional verification test results 
 code Code coverage results 
 sonarqube SonarQube scan results 
 staticsecurityscan Static security scan results from IBM Application Security on Cloud 
 dynamicsecurityscan Dynamic security scan results from IBM Application Security on Cloud 
 vulnerabilityadvisor Vulnerability Advisor results from IBM Vulnerability Advisor on Cloud 



Test records must provide data in one of the following supported formats:



Table 2. Supported formats

 Test Type Supported Formats 

 Unit test Mocha, xUnit, Karma/Mocha 
 Functional verification test Mocha, xUnit, Karma/Mocha 
 Code coverage Cobertura, lcov, JaCoCo 
 SonarQube Scan data that is provided by SonarQube scans
or
ibmcloud doi testrecord-publish --filelocation "tests/fvt/.json" --type fvt --logicalappname testapp --buildnumber master:199 --toolchainid b531487c-9c22-4f3b-9d20-5be408d57891 --region ca-tor

The following test types are supported:



Table 3. Test record types

 Type Description 

 unittest Unit test results 
 fvt Functional verification test (FVT) results 
 code Code coverage results 
 sonarqube SonarQube scan results 
 vulnerabilityadvisor Vulnerability Advisor results from IBM Vulnerability Advisor on Cloud 
 cratf Terraform report generated by Code Risk Analyzer 
 crabom Bill of Materials (BOM) report generated by Code Risk Analyzer 
 cradeploy Deployment report generated by Code Risk Analyzer 
 cracve Vulnerability report generated by Code Risk Analyzer 
 zapscan OWASP Zed Attack Proxy (ZAP) scan reports 



IBM Application Security on Cloud 1.0.0 is no longer published (staticsecurityscan and dynamicsecurityscan test types). All IBM Application Security on Cloud 1.0.0 support is provided by HCL. For more information, see the [HCL AppScan documentation](https://help.hcltechsw.com/appscan/Welcome.html).







 Publishing a deployment record 

The following command publishes a deployment record to DevOps Insights:

ibmcloud doi deployrecord-publish --env ENV --status STATUS --logicalappname LOGICALAPPNAME --buildnumber BUILDNUMBER --toolchainid TOOLCHAINID [--joburl JOBURL] [--appurl APPURL] [--region REGION]



Table 4. Command options for publishing a deployment record

 Command Options Required or Optional Description 

 -E, --env Required The environment where the pipeline job deployed the app. 
 -S, --status Required The deployment status. This value must be either pass or fail. 
 -L, --logicalappname Required Name of the application.
On Debian and Ubuntu, to determine the version of a package that is installed in your image, you can use the following commands, where <package_name> is the name of your package.



* To list the metadata for a specific installed package, run either of the following commands:

apt show <package_name>

dpkg-query -l <package_name>
* To list all installed packages and their versions, run either of the following commands:

apt list

dpkg-query -W







 Red Hat and CentOS package manager commands 

On Red Hat® OpenShift® and CentOS, to determine the version of a package that is installed in your image, you can use the following commands, where <package_name> is the name of your package.



* To list the metadata for a specific installed package, run either of the following commands:

rpm -qi <package_name>

yum info <package_name>
* To list all installed packages and their versions, run either of the following commands:

rpm -qa

yum list installed









 Does Vulnerability Advisor have versions? 

Vulnerability Advisor is available in two versions, version 3 and version 4. For more information, see [Managing image security with Vulnerability Advisor](https://cloud.ibm.com/docs/Registry?topic=Registry-va_index&interface=ui).

Vulnerability Advisor version 3 is deprecated from 19 June 2023. For more information about how to update to version 4, see [Update Vulnerability Advisor to version 4 by 19 June 2023](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_notices_va_v4&interface=ui).
* Applies exemption policies to reports at an account, [namespace](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_overviewoverview_elements_namespace), [repository](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_overviewoverview_elements_repository), or [tag](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_overviewoverview_elements_tag) level to mark when issues that are flagged do not apply to your use case.



The following functions are available in version 4:



* Scans images for issues.
* Provides an evaluation report that is based on security practices that are specific to IBM Cloud Kubernetes Service.
* Provides instructions about how to fix a reported [vulnerable package](https://cloud.ibm.com/docs/Registry?topic=Registry-va_indexpackages) in its reports.
* Applies exemption policies to reports at an account, [namespace](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_overviewoverview_elements_namespace), [repository](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_overviewoverview_elements_repository), or [tag](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_overviewoverview_elements_tag) level to mark when issues that are flagged do not apply to your use case.



The Security status column in the Images tab of the Container Registry dashboard displays the number of issues that are associated with each image. To find out more about the issues, click the link in the Security status column.

The Vulnerability Advisor dashboard provides an overview and assessment of the security for an image. If you want to find out more about the Vulnerability Advisor dashboard, see [Reviewing a vulnerability report](https://cloud.ibm.com/docs/Registry?topic=Registry-va_indexva_reviewing).

Encrypted images aren't scanned by Vulnerability Advisor.



 Data protection
* Applies exemption policies to reports at an account, [namespace](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_overviewoverview_elements_namespace), [repository](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_overviewoverview_elements_repository), or [tag](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_overviewoverview_elements_tag) level to mark when issues that are flagged do not apply to your use case.



The following functions are available in version 4:



* Scans images for issues.
* Provides an evaluation report that is based on security practices that are specific to IBM Cloud Kubernetes Service.
* Provides instructions about how to fix a reported [vulnerable package](https://cloud.ibm.com/docs/Registry?topic=Registry-va_index&interface=uipackages) in its reports.
* Applies exemption policies to reports at an account, [namespace](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_overviewoverview_elements_namespace), [repository](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_overviewoverview_elements_repository), or [tag](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_overviewoverview_elements_tag) level to mark when issues that are flagged do not apply to your use case.



The Security status column in the Images tab of the Container Registry dashboard displays the number of issues that are associated with each image. To find out more about the issues, click the link in the Security status column.

The Vulnerability Advisor dashboard provides an overview and assessment of the security for an image. If you want to find out more about the Vulnerability Advisor dashboard, see [Reviewing a vulnerability report](https://cloud.ibm.com/docs/Registry?topic=Registry-va_index&interface=uiva_reviewing).

Encrypted images aren't scanned by Vulnerability Advisor.



 Data protection
For more information about the expected output from user script stages, see [Custom scripts](https://cloud.ibm.com/docs/devsecops?topic=devsecops-custom-scripts).





 Build 

In the Build stage, you can build your own artifacts. Although the pipeline provides some default features for Docker image type artifacts, you can build any type of artifact in this stage.

Use the environment variables from the Pipeline UI to provide credentials, secrets, and parameters for your build. You can access these environment variables in this stage, and in all custom stages. For more information about how to access parameters and secrets in custom script stages, see [Custom scripts](https://cloud.ibm.com/docs/devsecops?topic=devsecops-custom-scriptscd-devsecops-scripts-secrets).





 Artifact scan and sign 

The artifact scan and sign stages provide a default behavior for Docker images, with customizable steps:



* Image signing by using GPG Key.
* Container Registry Vulnerability Advisor scanning.



To get started with these stages, provide your artifacts for the pipeline to use the pipelinectl interface. You are not required to update the build scripts and the .pipeline-config.yaml configuration.

To use a different scan or sign process, or to process artifacts other than Docker images in icr.io, you can customize these stages by using the .pipeline-config.yaml configuration in your project.





 Deploy to dev 

The Deploy stage deploys built artifacts into a dev environment. You can provide your variables and credentials for this stage from [variables in the pipeline UI](https://cloud.ibm.com/docs/devsecops?topic=devsecops-custom-scriptscd-devsecops-scripts-secrets) and the [pipeline trigger webhook payload](https://cloud.ibm.com/docs/devsecops?topic=devsecops-cd-devsecops-webhook-payloads).





 Dynamic scan 

Dynamic code scan is a form of black box vulnerability scanning that allows software teams to scan running applications and identify vulnerabilities.
Publishing test results (deprecated) 

You can publish test results by using idra (deprecated) to integrate your Delivery Pipeline with DevOps Insights. Test results inform IBM Cloud® DevOps Insights about the tests that are run during the build process. These tests must generate some kind of JSON or XML-based file. The generated test file is published as a test record. Along with the test file, this record contains the application name, build number, types of test, and other fields. The application name and build ID in the test record must match the ones that are used in the build record for a specific build. After you run a test in a pipeline, you can upload the results to DevOps Insights.



 Before you begin 

Before you publish test results, you must publish your build record. For more information, see [publishing a build record by using idra (deprecated)](https://cloud.ibm.com/docs/ContinuousDelivery?topic=ContinuousDelivery-publish-build-idra).





 Publishing test results 

Test results must provide data in one of these supported formats:



Table 1. Test type and format

 Test Type Supported Formats 

 Functional verification test Mocha, xUnit, Karma/Mocha 
 Unit test Mocha, xUnit, Karma/Mocha 
 Code coverage Cobertura, lcov, JaCoCo 
 SonarQube Scan data that is provided by SonarQube scans 
 Static AppScan Static App Scans that are provided by IBM Application Security on Cloud 
 Dynamic AppScan Dynamic App Scans that are provided by IBM Application Security on Cloud 
 Vulnerability Advisor Vulnerability Advisor results from IBM Vulnerability Advisor on Cloud 



The following example script uploads FVT test results to DevOps Insights:

 Run tests and generate a test results file here.
 Add user api key to stage environment variable as a secured property
export LOGICAL_APP_NAME="SampleApp"
export BUILD_PREFIX="master"

 node 4.x or above is needed
export PATH=/opt/IBM/node-v4.2/bin:$PATH
npm install -g grunt-idra3
For more information about Object Storage buckets, see [IBM Cloud Object Storage buckets as evidence locker](https://cloud.ibm.com/docs/devsecops?topic=devsecops-cd-devsecops-cos-bucket-evidence).







 v1 evidence collection 

The v1 evidence collection is the legacy evidence collection. To opt out of v1 evidence collection, see [Turning off legacy v1 evidence collection](https://cloud.ibm.com/docs/devsecops?topic=devsecops-turn-off-v1-evidence).

According to the [DevOps lifecycle](https://www.ibm.com/cloud/learn/devops-a-complete-guidetoc-how-devops-u5vApdai), the continuous integration and continuous deployment flows can be divided into three stages: code, build, and deploy.

The DevOps architecture divides these flows into four stages by distinguishing between pre-production and production deployment. The DevSecOps reference architecture currently includes the pre-production and production deployment within the same stage.

Evidence is collected from the steps in each stage [collect-evidence](https://cloud.ibm.com/docs/devsecops?topic=devsecops-devsecops-collect-evidence) script. The following image shows the current steps in the reference pipelines that produce evidence.

Zoom

![Evidence collection](https://cloud.ibm.com/docs-content/v1/content/5eb88a2575e30e316dbb83829c7f4c8b1ddbc67c/devsecops/images/CI-CD-evidence.png)

Figure 3. Evidence collection



 Code stage 

In the Code stage, evidence is collected for the following steps:



* Detect secrets
* Unit test results
* Code Vulnerability scan, CIS check, and Bill of Material check by way of Code Risk Analyzer







 Build stage 

In the Build stage, evidence is collected for the following steps:



* Vulnerability Advisor scan
* Image signing







 Deploy stage
ibmcloud cr trash-list [--restrict NAMESPACE] [--json]



 Prerequisites 

To find out about the required permissions, see [Access roles for using Container Registry](https://cloud.ibm.com/docs/Registry?topic=Registry-iamaccess_roles_using).





 Command options 

--restrict NAMESPACE
: (Optional) Limit the output to display only images in the specified namespace.

--json
: (Optional) Outputs JSON that contains the details of the contents of the trash.





 Example 

Display the images that are in the trash in the birds namespace.

ibmcloud cr trash-list --restrict birds







 ibmcloud cr va-version 

Find out which version of Vulnerability Advisor you're using.

ibmcloud cr va-version



 Prerequisites 

None.







 ibmcloud cr va-version-set 

Set the version of Vulnerability Advisor. You can set the version to either v3, the default, or v4. If you want to use version 4 temporarily, see [Setting the Vulnerability Advisor version](https://cloud.ibm.com/docs/Registry?topic=Registry-va_index&interface=uiva_set_version).

ibmcloud cr va-version-set VERSION



 Prerequisites 

None.





 Command options 

VERSION
: The version of Vulnerability Advisor that you want to use. Valid values are v3 and v4.





 Example 

To set the Vulnerability version to version 4, run the following command:

ibmcloud cr va-version-set v4







 ibmcloud cr vulnerability-assessment (ibmcloud cr va) 

View a vulnerability assessment report for your images.

ibmcloud cr vulnerability-assessment [--extended | -e] [--vulnerabilities | -v] [--configuration-issues | -c] [--output FORMAT | -o FORMAT] IMAGE [IMAGE...]
6.3 Ensure Toolchain has Code Risk Analyzer configured that collects a bill of materials for pipeline run Ensure DevSecOps Toolchain collects software bills of materials (SBOM) to provide transparency in build artifacts 
 6.4 Ensure Toolchain is configured with image signing Ensure DevSecOps Toolchain signs build artifacts to attest their provenance 
 6.5 Ensure Toolchain source code meets Center for Internet Security Docker benchmarks Ensure DevSecOps Toolchain validates code against Center for Internet Security (CIS) Docker benchmarks to ensure container runtimes are configured securely 
 6.6 Ensure Toolchain has branch protection rules enabled Ensure DevSecOps Toolchain verifies source code branch protection rules to enforce security policies 
 6.7 Ensure Toolchain has secret detection scans enabled for source code Ensure DevSecOps Toolchain source code contains no secrets 
 6.8 Ensure Toolchain production change request exists and is approved Ensure DevSecOps Toolchain deployment has approved change documentation including security impact analysis 
 6.9 Ensure Toolchain Container Registry Vulnerability Advisor scans images for OS vulnerability detection Ensure DevSecOps Toolchain scans build artifacts to identify vulnerabilities 
 6.12 Ensure Toolchain acceptance tests exist and have passed Ensure DevSecOps Toolchain passes acceptance tests to validate every deployment 
 6.16 Ensure that only the tool integrations within the toolchain are included in the allow list parameter array Ensure Toolchain is configured only with the allowed integration tools 
 6.17 Ensure a Toolchain static scan exists and has passed Ensure DevSecOps Toolchain passes static code scan to identify vulnerabilities in source code 
 6.18 Ensure a Toolchain dynamic scan exists and has passed Ensure DevSecOps Toolchain passes dynamic code scan to identify vulnerabilities in deployed artifacts 







 13 March 2023 

The following new controls have been added to the IBM Cloud Security Best Practices library and profile as of 17 March 2023.



Table. Summary of IBM Cloud Security Best Practices profile changes for Version 1.1.0

 Control ID Control Description 

 6.1 Ensure Toolchain scans during continuous integration the source code to identify vulnerabilities 
 6.2 Ensure Toolchain has unit tests that are continuously run to validate source code changes
Publishing test results 

You can publish test results by using Jenkins to integrate your projects with IBM Cloud® DevOps Insights. Test results inform DevOps Insights about the tests that are run during the build process. These tests must generate some kind of JSON or XML-based file.

The generated test file is published as a test record. Along with the test file, this record contains the application name, build number, types of test, and other fields. The application name and build ID in the test record must match the ones that are used in the build record for a specific build. After you run a test in a pipeline, you can upload the results to DevOps Insights.



 Before you begin 

Before you publish test results, you must publish a build record. For more information, see [publishing a build record by using Jenkins](https://cloud.ibm.com/docs/ContinuousDelivery?topic=ContinuousDelivery-publish-build-jenkins).





 Publishing test results 

The following are types of tests that can be used and the formats that are supported.



Table 1. Supported tools and formats

 Type Formats 

 Unit Test JUnit (JSON), xUnit (xml), Mocha (JSON), KarmaMocha (JSON) 
 Function Tests JUnit (JSON), xUnit (xml), Mocha (JSON), KarmaMocha (JSON) 
 Code Coverage Cobertura (xml), lcov, JaCoCo (xml) 
 Sonarqube Sonarqube 
 Static Security Scan IBM Application Security on Cloud (xml) 
 Dynamic Security Scan IBM Application Security on Cloud (xml) 
 IBM Vulnerability Advisor IBM Vulnerability Advisor (JSON) 
 Custom Data Sets - Tests JUnit (JSON), xUnit (xml), Mocha (JSON), KarmaMocha (JSON) 
 Custom Data Sets - Code Coverage Cobertura (xml), lcov, JaCoCo (xml) 



Publish build records with the publishTestResult step. This step requires two parameters. It can also accept one optional parameter.



Table 2. Publishing build records parameters and definitions

 Parameter Definition 

 type The type of test result.
* [Managing user access](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_release_notes4oct2018_access)



* [7 August 2018](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_release_notesregistry-7aug2018)



* [Exemption policies available in Vulnerability Advisor](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_release_notes7aug2018_exemption)



* [25 July 2018](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_release_notesregistry-25jul2018)



* [IBM Cloud Activity Tracker available for Vulnerability Advisor](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_release_notes25jul2018_at)



* [12 July 2018](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_release_notesregistry-12jul2018)



* [Vulnerability Advisor API version 3](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_release_notes12jul2018_va3)



* [31 May 2018](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_release_notesregistry-31may2018)



* [Use Helm for Passport Advantage images](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_release_notes31may2018_helm)



* [21 March 2018](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_release_notesregistry-21mar2018)



* [Container Scanner](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_release_notes21mar2018_cs)



* [16 March 2018](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_release_notesregistry-16mar2018)
CC pipeline uses the [pipelinectl save_repo](https://cloud.ibm.com/docs/devsecops?topic=devsecops-devsecops-pipelinectlsave_repo) interface to register repos found in the inventory entries, then uses the [list_repos](https://cloud.ibm.com/docs/devsecops?topic=devsecops-devsecops-pipelinectllist_repos) and [load_repo](https://cloud.ibm.com/docs/devsecops?topic=devsecops-devsecops-pipelinectlload_repo) commands to iterate over repos and send them to scanners.

For more information about the expected output from user script stages, see [Custom scripts](https://cloud.ibm.com/docs/devsecops?topic=devsecops-custom-scripts).





 Artifact scan and sign 

The artifact scan stage provides default behavior for Docker images, with a customizable step:



* Container Registry Vulnerability Advisor scanning



CC pipeline uses the [pipelinectl save_artifact](https://cloud.ibm.com/docs/devsecops?topic=devsecops-devsecops-pipelinectlsave_artifact) interface to register artifacts found in the inventory entries, then iterates over these artifacts, by using the [list_artifacts](https://cloud.ibm.com/docs/devsecops?topic=devsecops-devsecops-pipelinectllist_artifacts) and [load_artifact](https://cloud.ibm.com/docs/devsecops?topic=devsecops-devsecops-pipelinectlload_artifact) commands.

To get started with this stage, provide your artifacts for the pipeline by using the [pipelinectl](https://cloud.ibm.com/docs/devsecops?topic=devsecops-devsecops-pipelinectl) interface. You are not required to update the build scripts and the .pipeline-config.yaml configuration.