vania-janet/multiturn-rag-retrieval-data huggingface dataset:
tfvars-branch main Branch of the Git repository that contains the tfvars files. 
 tfvars-files Files that contain Terraform variables values. 
 terraform-version 1.2.9 Terraform CLI tool version to install if not present in the image used for the stages. 



The same parameters apply for the scripts that are used in an IaC CD deployment process. Because the CD process can process multiple inventory entries, you can scope a parameter for an inventory entry. To specify Terraform context and variables for a scope (inventory entry), prefix the property with the inventory entry name, for example: <inventory_entry>_. This prefix applies for the environment entries tf-dir, TF_VAR_<XXXX>, tfvars-repository, tfvars-branch, and tfvars-files.

Example:

hello-iac-sample_TF_VAR_resource_group : Default





 Static code scan 

The static code scan stage runs a number of static code analyzer tools on the specified IaC repositories. The repos that are provided by the [pipelinectl save_repo](https://cloud.ibm.com/docs/devsecops?topic=devsecops-devsecops-pipelinectlsave_repo) command and the default app repo are scanned.

You can use any of the methods that are defined for [static code scan](https://cloud.ibm.com/docs/devsecops?topic=devsecops-devsecops-ci-pipelinedevsecops-ci-pipeline-static-codescan) configurable for application relate continuous integration pipeline.

The IaC continuous integration pipeline defines more tools that are enabled by using the opt-in-* parameters from Table 2 set to 1.



Table 3. IaC static scan tools parameters

 Name Type Default Description Required or optional 

 opt-in-terraform-fmt-validate text Runs the terraform fmt and terraform validate commands in static-scan stage. optional 
 opt-in-tflint text Runs the tflint command in static-scan stage. optional
deploy Deploy the built artifacts to the dev environment. Yes 
 dynamic-scan Run dynamic scan on the application. Yes 
 acceptance-test Run acceptance and integration tests on the deployed built artifacts on the dev environment. Yes 
 scan-artifact Scan the built artifacts. Yes 
 release Add the built artifacts to the inventory. Yes 
 finish Collect, create, and upload the logs files, artifacts, and evidence to the evidence locker. Yes 



For more information about how to customize stages by using the .pipeline-config.yaml file, see [Custom scripts](https://cloud.ibm.com/docs/devsecops?topic=devsecops-custom-scripts) and [Pipeline parameters](https://cloud.ibm.com/docs/devsecops?topic=devsecops-cd-devsecops-pipeline-parmcd-ci-parameters) lists.





 Static code scan 

The static code scan stage runs a number of static code analyzer tools on the specified app repositories. The repos that are provided by the [pipelinectl save_repo](https://cloud.ibm.com/docs/devsecops?topic=devsecops-devsecops-pipelinectlsave_repo) command and the default app repo are scanned.

You can use any of the following methods to add static code to your pipeline:



* Provide an already running SonarQube instance name, URL, and credentials by adding the SonarQube tool to your toolchain. The static-scan task runs a scan on the specified repos.
* If you don't have your own SonarQube instance, the pipeline creates a SonarQube instance during the pipeline run. You can access this instance after the static-scan stage successfully runs.
* Add your own static scan code to the static-scan custom stage in your .pipeline-config.yaml file for a custom implementation.





 Adding SonarQube scan to your pipelines
This parameter is for any static scan that you want to run in a pipeline. These scans use API calls, which can take a short time to work. Therefore, retry these calls if needed. You can change the amount of seconds to wait with this parameter. By default, this parameter is set to 10.





 summary-retry-attempts 

Use a retry method to upload the evidence summary to make sure it is successful. You can change the times of retries with this parameter. By default, this parameter is set to 5.





 summary-max-retry-time 

Use a retry method to upload the evidence summary to make sure it is successful. You can change the amount of seconds to wait with this parameter. By default, this parameter is set to 32.





 target-environment 

The target environment that the app is deployed to, which is the target inventory branch of the promotion. The default value is prod. This value can be overwritten with the usage of a webhook trigger by adding the target-environment property to the payload.





 target-environment-purpose 

This parameter is for the CD pipeline. target-environment-purpose determines the context of the target environment and how it is used. Valid values are pre_prod and production. Mark anything that is not a production related change as pre_prod.





 target-environment-detail 

This parameter is for the CD pipeline, it describes the target environment.





 sysdig-scan 

This parameter is for the pipelines that use scan artifact stages to run the Sysdig scan by using sysdig inline scanner on icr.io images.
For more information, see [Adding compliance details](https://cloud.ibm.com/docs/account?topic=account-catalog-format-controls).



1. Click Add controls.
2. Choose a profile.
3. Select the controls that you want to add to your version.
4. Click Add.
5. In the Code Risk Analyzer scan section, click Run scan.
6. Wait for the scan to finish.
7. In the Security and Compliance Center scan section, select the profile that you scanned.
8. Select the Security and Compliance Center scan.
9. Click Add scan.
10. Click Next.







 Review requirements 

You must complete validation and any other requirements to share to your account. When you're ready to make your product available to all users who have access to your private catalog, click Ready to share.





 Sharing the software 

If you want to share your product to your account or enterprise, click the name of the product in the navigation to go to the product details page. From the Actions menu, click Share. Select where you want to share your product, and click Share.





 Onboarding software to your catalog by using the CLI 

Complete the following steps to add your software by using the CLI. You can use this task in a CI/CD process.



1. Create a private catalog. Private catalogs provide a way for you to manage access to products for users in your account. For more information, see the [cli documentation](https://cloud.ibm.com/docs/cli?topic=cli-manage-catalogs-plugincreate-catalog) for creating a private catalog.

ibmcloud catalog create --name CATALOG [--catalog-description "DESCRIPTION"]
2. Add software to your private catalog. For more information, see the [cli documentation](https://cloud.ibm.com/docs/cli?topic=cli-manage-catalogs-plugincreate-offering) for adding software to your private catalog.
Select the profile that you scanned.
2. Select the Security and Compliance Center scan.
3. Click Add scan.
4. Click Next.









 Review requirements 

You must complete validation and any other requirements to publish your product to your account.





 Next steps 

After you onboard and validate your virtual server image, you're ready to publish it to your account. From the Actions menu, select Publish to account. As a result, the virtual server image is available only to users who have access to the Sample virtual server image private catalog in your account.
To run the SonarQube scan, the pipeline needs the following continuous integration parameters:



Table 1. Continuous integration pipeline parameters

 Name Type Description Required or Optional 

 cluster-name Text The name of the Docker build cluster. Required 
 dev-region Text The IBM Cloud region that hosts the cluster. Required 
 opt-in-sonar Text The option to enable the SonarQube scan. Required 
 sonarqube Tool integration The SonarQube tool integration. Optional 
 sonarqube-config Text Runs a SonarQube scan in an isolated Docker-in-Docker container (default configuration) or in an existing development Kubernetes cluster (cluster configuration). Alternatively, you can bring your own SonarQube instance and configure the SonarQube tool integration (custom configuration). Options: default, cluster, or custom. Default is default. For more information, see ([Adding SonarQube to the continuous integration pipeline](https://cloud.ibm.com/docs/devsecops?topic=devsecops-sonarqubesonarqube-ci-pipeline)). Required 
 opt-in-sonar-hotspots Text The Sonarqube scan for detecting hotspots. Optional 



For more information about pipelines parameters, see [Pipeline parameters](https://cloud.ibm.com/docs/devsecops?topic=devsecops-cd-devsecops-pipeline-parm).

If you add multiple SonarQube tool integrations to your pipeline, you can switch between them by changing the value of the SonarQube pipeline parameter, which is a tool integration parameter.





 Updating the quality gate 

If you use the SonarQube instance that the pipeline created, you can update the default quality gate.



1. Go to the SonarQube dashboard that was created by the URL from the pipeline logs in the static-scan task.

Zoom

!
[Part 3: Map additional credentials](https://cloud.ibm.com/docs/security-compliance?topic=security-compliance-ibm-credential-map)Part 3: Map additional credentials

This tutorial is part 3 of a 4-part tutorial series that guides you through configuring the Security and Compliance Center to scan your IBM Cloud resources for compliance.

Security and Compliance Center



* 20 minutes
* 2022-12-13



[Step 4: Scan your resources and view results](https://cloud.ibm.com/docs/security-compliance?topic=security-compliance-ibm-scan)Step 4: Scan your resources and view results

This tutorial is part 4 of a 4-part tutorial series that guides you through configuring the Security and Compliance Center to scan your IBM Cloud resources for compliance.

Security and Compliance Center



* 10 minutes
* 2022-12-06



[Evaluate the compliance of your Red Hat OpenShift on IBM Cloud clusters ](https://cloud.ibm.com/docs/security-compliance?topic=security-compliance-osco-scan)Evaluate the compliance of your Red Hat OpenShift on IBM Cloud clusters

By using the OpenShift Compliance Operator (OSCO) through Security and Compliance Center, you can run scans to validate your level of compliance to a grouping of controls, also known as a profile. In this scenario, you configure your cluster to be able to run the OSCO scan.

Security and Compliance Center Red Hat OpenShift on IBM Cloud



* 20 minutes
* 2023-06-28



Solutions[Deploying and configuring F5 BIG-IP](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-vpc-architecture-connectivity-f5-tutorial)Deploying and configuring F5 BIG-IP

If you want to use a client to site full tunnel VPN to connect to your management VPC or need a web application firewall (WAF) to enable consumers to connect to your workload VPC over the public internet, you need to install your own software solution. One approach is to use F5 BIG-IP Virtual Edition.
echo "APP_URL :- ${APP_URL}"
go run acceptance-test/acceptance-test.test.go





 release 

Define this stage to upload evidence and artifacts that are generated from the previous stages.

release:
abort_on_failure: false
image: icr.io/continuous-delivery/toolchains/devsecops/compliance-baseimage:2.26. 1@sha256:a780174a64474187b01b5e40a1721d8307f02897ac6f3eba2d482d4f4926edf1
script: 
!/usr/bin/env bash
source scripts/release.sh





 scan-artifact 

Define this stage to perform a scan for vulnerabilities in the generated artifacts.

scan-artifact:
abort_on_failure: false
image: icr.io/continuous-delivery/pipeline/pipeline-base-image:2. 6@sha256:7f588468622a981f89cf5e1212aaf75fface9da6169b5345ca52ab63d8215907
script: 
!/usr/bin/env bash
source scripts/va_scan.sh





 dynamic-scan 

Define this stage to run dynamic scan on the deployed application.

dynamic-scan:
dind: true
abort_on_failure: false
image: icr.io/continuous-delivery/pipeline/pipeline-base-image:2.12@sha256:ff4053b0bca784d6d105fee1d008cfb20db206011453071e86b69ca3fde706a4
script: 
!/usr/bin/env bash
echo "Please insert script to invoke/execute dynamic scan tool like OWASP ZAP on the built and deployed application."





 Related information
* The footer key:

parameter name: "parquet.encryption.footer.key"
parameter value: "<master key ID>"

For example:

dataFrame.write
.option("parquet.encryption.footer.key" , "k1")
.option("parquet.encryption.column.keys" , "k2:SSN,Address;k3:CreditCard")
.parquet("<path to encrypted files>")

Important: If neither the "parquet.encryption.column.keys" parameter nor the "parquet.encryption.footer.key" parameter is set, the file will not be encrypted. If only one of these parameters is set, an exception is thrown, because these parameters are mandatory for encrypted files.
* The class implementing EncryptionPropertiesFactory:

parameter name: "parquet.crypto.factory.class"
parameter value: "com.ibm.parquet.key.management.IBMKeyToolsFactory"

Important: If "parquet.crypto.factory.class" is not set, the file won't be encrypted by a crypto factory.





 Optional parameters 

The following optional parameters can be used when writing encrypted data:



* The encryption algorithm AES-GCM-CTR

By default, Parquet encryption uses the AES-GCM algorithm that provides full protection against tampering with data and metadata in Parquet files. However, as Spark 2.3.0 runs on Java 8, which doesn’t support AES acceleration in CPU hardware (this was only added in Java 9), the overhead of data integrity verification can affect workload throughput in certain situations.

To compensate this, you can switch off the data integrity verification support and write the encrypted files with the alternative algorithm AES-GCM-CTR, which verifies the integrity of the metadata parts only and not that of the data parts, and has a lower throughput overhead compared to the AES-GCM algorithm.

parameter name: "parquet.encryption.algorithm"
parameter value: "AES_GCM_CTR_V1"
Enabling continuous compliance in your application software development cycle 

Taking advantage of DevSecOps to automate the integration of security at every phase of your enterprise application development lifecycle addresses security issues as they emerge, making them easier, faster, and less expensive to fix. By using the [DevSecOps Application Lifecycle Management](https://cloud.ibm.com/docs/devsecops-alm?topic=devsecops-alm-devsecops-alm-overview) deployable architecture, you can take advantage of the two main benefits of using DevSecOps, which are speed and security.

Throughout the development cycle, the code is reviewed, scanned, and tested for security issues. This integrated security allows your team to catch issues early, and reduces duplicative reviews and unnecessary rebuilds, resulting in more secure code that is delivered faster. As DevSecOps integrates vulnerability scanning and patching into the development and release cycle, the ability to identify and patch common vulnerabilities and exposures (CVE) is increased. This limits the window a threat actor has to take advantage of vulnerabilities in public-facing production systems. Additionally, continuous compliance with periodic scanning of applications in production is included.

Implementing automated scanning and testing can ensure that incorporated software dependencies are at appropriate patch levels, and confirm that software passes security unit testing. Plus, it can test and validate code for security with static and dynamic analysis. Evidence of scan results is securely stored, collected, and summarized as part of an automated change management process that gates deployments, before changes are promoted to production.

Review the following sections to get an overview of the essential steps for ensuring that your application development lifecycle is secure by using the IBM Cloud DevSecOps tools.



 Setting up your secure software development architecture 

The DevSecOps Application Lifecycle Management deployable architecture streamlines the process to set up your secure software supply chain by using a set of continuous integration (CI), continuous deployment (CD), and continuous compliance (CC) toolchain templates. These templates use a collection of tool integrations and customizable reference Tekton pipelines for build, scan, test, change management, and deploy applications.
Evidence summary 

Evidence summary refers to a collection of evidence pieces that are relevant for a set of artifacts. The pipeline currently uses two approaches to collect and summarize evidence, v1 and v2. The intention is that v1 is superseded by v2.



 v2 evidence collection 

The evidence locker stores three kinds of entities: Assets, Evidence, and Attachments. For more information, see [v2 evidence format](https://cloud.ibm.com/docs/devsecops?topic=devsecops-devsecops-evidencedevsecops-v2-evidence-format).



 Asset 

An asset represents something that you can test and scan, such as a Git commit in a repository or a docker image.





 Evidence 

A piece of evidence represents the outcome of a scan or test. Evidence is always connected to at least a single asset. Multiple assets are allowed. For example, a single end-to-end test suite probably tests multiple assets together.





 Attachment 

An attachment is arbitrary data that can be attached to a piece of evidence. It's usually the raw output of the tool that performed the scan or test.





 Collecting evidence 

Use the [collect-evidence](https://cloud.ibm.com/docs/devsecops?topic=devsecops-devsecops-collect-evidence) script to collect evidence.





 Evidence summary 

In contrast to v1, v2 evidence is stored in a flat hierarchy, where each piece of evidence is identified by its own hash. This hash provides a layer of integrity protection. That is, any modification of the evidence content can be detected. As each piece of evidence is related to one or more assets, the evidence summarization algorithms discover the relevant evidence based on assets (as opposed to pipelines run IDs in v1).

Evidence is summarized in the CD pipeline, based on the provenance field of each inventory item. For more information, see [Inventory](https://cloud.ibm.com/docs/devsecops?topic=devsecops-cd-devsecops-inventory).
rclone mkdir RemoteName:newbucket





 List available buckets 

rclone lsd RemoteName:





 List contents of a bucket 

rclone ls RemoteName:newbucket





 Copy a file from local to remote 

rclone copy /Users/file.txt RemoteName:newbucket





 Copy a file from remote to local 

rclone copy RemoteName:newbucket/file.txt /Users/Documents/





 Delete a file on remote 

rclone delete RemoteName:newbucket/file.txt





 List Commands 

There are several related list commands



* ls to list size and path of objects only
* lsl to list modification time, size, and path of objects only
* lsd to list directories only
* lsf to list objects and directories in easy to parse format
* lsjson to list objects and directories in JSON format









 rclone sync 

The sync operation makes the source and destination identical, and modifies the destination only. Syncing doesn’t transfer unchanged files, testing by size and modification time or MD5SUM. Destination is updated to match source, including deleting files if necessary.

Since this operation can cause data loss, test first with the --dry-run flag to see exactly what would be copied and deleted.

Files in the destination aren't deleted if there are errors at any point.

The contents of the directory are synced, not the directory itself. When source:path is a directory, it’s the contents of source:path that are copied, not the directory name and contents. For more information, see the extended explanation in the copy command.

If dest:path doesn’t exist, it is created and the source:path contents go there.

rclone sync source:path dest:path [flags]



 Using rclone from multiple locations at the simultaneously 

You can use rclone from multiple places simultaneously if you choose different subdirectory for the output:

Server A> rclone sync /tmp/whatever remote:ServerA
Server B> rclone sync /tmp/whatever remote:ServerB
For more information, see [Scheduling a scan](https://cloud.ibm.com/docs/security-compliance?topic=security-compliance-schedule-scan).



1. Select the profile that you scanned.
2. Select the Security and Compliance Center scan.
3. Click Add scan.
4. Click Next.









 Step 9: Review requirements 

You must complete validation and any other requirements to share your product to your account. When you're ready to share your product, click Ready to share. As a result, the virtual server image is available only to users who have access to the Sample virtual server image private catalog in your account.





 Next steps 

If you want to share your product to your account or enterprise as well as your private catalog, click the name of the product in the navigation to go to the product details page. From the Actions menu, click Share. Select where you want to share your product, and click Share.

You can also use the Partner Center to publish your product to the IBM Cloud catalog. If you publish your product to the IBM Cloud catalog, it will be publicly available to all IBM Cloud users. For more information, see [Publishing your software](https://cloud.ibm.com/docs/sell?topic=sell-sw-publish).
ibmcloud doi testrecord-publish --logicalappname "$MY_APP_NAME" --buildnumber "$MY_BUILD_NUMBER" --filelocation "./test-sample-folder/unit_test_mocha.json" --type=unittest

For a sample test result files that you can upload to DevOps Insights, see [this GitHub repository with dummy data](https://github.com/devops-insights/example-upload-data-format).





 Uploading Code Coverage Results 

To upload Code Coverage results or custom code data, use the following command:

ibmcloud doi testrecord-publish --logicalappname "$MY_APP_NAME" --buildnumber "$MY_BUILD_NUMBER" --filelocation "./test-sample-folder/code_coverage_cobertura.xml" --type=code

For a sample test result files that you can upload to DevOps Insights, see [this GitHub repository with dummy data](https://github.com/devops-insights/example-upload-data-format).





 Uploading IBM Application Scan reports 

Users can get the report from the IBM AppScan CLI tool. Scan the application from the CLI to get the results in compressed file format.

appscan.sh get_result -i $job_id -d "$APPSCAN_SCAN_NAME.zip" -t zip

Unzip the results in a folder and go to that folder, then upload a file called Report-final.xml

ibmcloud doi testrecord-publish --logicalappname "$MY_APP_NAME" --buildnumber "$MY_BUILD_NUMBER" --filelocation "./Report-final.xml" --type staticsecurityscan

Change the type to dynamicsecurityscan if the report is from a dynamic scan.





 Uploading SonarQube results 

To upload data from SonarQube, you must provide the SonarQube server token by using --token. Your SonarQube server is required to be accessible from your CI/CD tool.