vania-janet/multiturn-rag-retrieval-data huggingface dataset:
* --primary-network-interface: PRIMARY_NETWORK_INTERFACE_JSON|@PRIMARY_NETWORK_INTERFACE_JSON_FILE, primary network interface in JSON or JSON file. For the data schema, see the primary_network_interface property in the [API documentation](https://cloud.ibm.com/apidocs/vpccreate-instance).
* --network-interface: NETWORK_INTERFACE_JSON|@NETWORK_INTERFACE_JSON_FILE, network interface attachment in JSON or JSON file. For the data schema, see the network_interfaces property in the [API documentation](https://cloud.ibm.com/apidocs/vpccreate-instance).
* --default-trusted-profile: ID or name of the trusted profile.
* --default-trusted-profile-auto-link: If set to true, the system creates a link to the specified target trusted profile during instance creation. Regardless of whether a link is created by the system or manually by using the IAM Identity service, it automatically deletes when the instance is deleted. One of: true,false. (default: true).
* --metadata-service, --ms: Enable or disable the Instance Metadata Service. One of: true, false.
* --metadata-service-protocol, --msp: The communication protocol for the metadata service endpoint. Applies only when the metadata service is enabled. One of: http, https. (default: http).
* --metadata-service-response-hop-limit, --msrhl: The hop limit (IP time to live) for IP response packets from the metadata service.
* --host-failure-policy: The action to perform if the compute host experiences a failure. One of: restart, stop.
* --resource-group-id: ID of the resource group. This ID is mutually exclusive with --resource-group-name.
* --resource-group-name: Name of the resource group. This name is mutually exclusive with --resource-group-id.
* --output: Specify output format, only JSON is supported.
For more information, see [cocoa locker commands](https://cloud.ibm.com/docs/devsecops?topic=devsecops-cd-devsecops-clicocoa-locker-commands).

The evidence can be configured by using the following flags:



* --evidence-type-id: a string that identifies the type of the evidence (for example com.ibm.unit_test)
* --evidence-type-version: a string that identifies the evidence details schema (for example 1.0.0, v2, and so on)



Some optional flags can be specified as well:



* --date: evidence creation date (defaults to the current timestamp), accepts a string that can be parsed with [Date.parse](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/parse).
* --details: evidence details can be specified as key=value pairs, the flag can be specified multiple times (for example --details result=success --details foo=bar, and so on).
* --origin: details about the origin of the evidence can be specified by using key=value pairs, for example --origin job_id=123, and so on.
* --asset: specifies which assets (by their URI or internal ID) are relevant for the evidence, see cocoa locker asset commands. The flag can be specified multiple times. The asset must be already present in the locker.
* --attachment: specifies file attachments to the evidence. Each attachment must point to an existing file, and each attachment name must be unique (that is, --attachment a/foo.json --attachment b/foo.json is not permitted).
* --issue: list of issue URLs, which can be used to track activities that are related to the findings. The flag can be passed multiple times.
* --scope: a unique identifier that can be used to correlate evidence from multiple sources.
* --comment-on-issues: add a comment with details about the created evidence to the incident issues provided by using --issue flag
You can select from three predefined [IOPS tiers](https://cloud.ibm.com/docs/vpc?topic=vpc-block-storage-profilestiers) that offer reliable IOPS performance for your workload requirements. You can also define [custom IOPS](https://cloud.ibm.com/docs/vpc?topic=vpc-block-storage-profilescustom) and specify a range of IOPS for a volume size that you choose. Custom IOPS is a good option when you have well-defined performance requirements that do not fall within a predefined IOPS tier. If you choose a custom IOPS profile, also define a minimum and maximum range for the volume size.

Maximum IOPS for data volumes varies based on volume size and the type of profile you select.

IOPS is measured based on a load profile of 16-KB blocks with random 50% read and 50% writes. Workloads that differ from this profile might experience reduced performance. If you use a smaller block size, maximum IOPS can be obtained, but throughput is less. For more information, see [How block size affects performance](https://cloud.ibm.com/docs/vpc?topic=vpc-capacity-performancehow-block-size-affects-performance).
* What typical network performance might I expect between my compute instances and the Block Storage for VPC service?

 What typical network performance might I expect between my compute instances and the Block Storage for VPC service? 

Block Storage for VPC is connected to compute instances on a shared network, so the exact performance latency depends on the network traffic within a specific timeframe. Target latency for a 16-KB block size volume is under 1 millisecond for random reads and under 2 milliseconds for writes.
* What mechanisms are used to avoid data storage contention?

 What mechanisms are used to avoid data storage contention? 

Data storage contention is a common issue when multiple instances compete for access to the same Block Storage for VPC volume. Block Storage for VPC uses Rate Limiting at the hypervisor for optimal bandwidth between the hypervisor and Block Storage for VPC service.
The sharded configuration allows for Elasticsearch to run concurrent operations on your data across all the nodes. To gain extra concurrency and performance, [add nodes to Elasticsearch cluster](https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=databases-for-elasticsearch-horizontal-scaling). When you add nodes, your shards are automatically rebalanced across the cluster to spread resource usage across all the nodes and increasing performance.





 Memory Management 

Elasticsearch memory is divided into to categories, JVM heap size and system memory. It uses heap for internal caching, and the rest of the system memory for the operating system, file system caches, and garbage collection. The more memory that is allocated to the heap, the less is allocated to the rest of the system.

Databases for Elasticsearch deployments have their memory allocation policy set at 50% heap and 50% system memory, with a max heap size of 32 GB. In some cases, it is useful to scale your deployment above 64 GB of RAM even with the heap limit as Elasticsearch does make use of the file system cache and alleviate pressure on disk I/O utilization. You can configure autoscaling to increase memory when disk I/O utilization reaches a certain threshold.





 Disk IOPS 

The number of Input/Output Operations per second (IOPS) is limited by the type of storage volume. Storage volumes for Databases for Elasticsearch deployments are provisioned on [Block Storage Endurance Volumes in the 10 IOPS per GB tier](https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-orderingthroughConsoleorderingthroughConsoleEndurance). Hitting IOPS limits can cause your databases to respond slowly or appear unresponsive.

Indexing uses disk, so if your use-case is write-heavy, your indexing speed can be limited by the IOPS available to your deployment. Some bottlenecks can be ameliorated by [tuning your indexes for disk usage](https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-disk-usage.html).
Validating evidence 

In the DevSecOps architecture, before deploying the assets, all the pieces of evidence that were collected in the CI pipeline must be validated so that the assets meet compliance requirements.

Collecting evidence is an essential aspect of the DevSecOps reference architecture. The CI pipeline collects the pieces of evidence for all the scans or steps for the assets that are performed by the CI pipeline. The CD pipeline deploys the assets that are produced by the CI pipeline.

You can validate the evidence in one of the following ways:



1. Using the Security and Compliance Center (SCC) profile.
2. Using a config file.





 Using the SCC profile 

Use SCC to embed security checks into your everyday workflows to monitor for security and compliance. By monitoring for risks, you can identify security vulnerabilities, work to mitigate the impact, and fix the issues. The toolchain must have SCC integration and Use profile with attachment enabled, based on the SCC profile and version. All controls are validated based on the collected evidence for all the resources in the attachment. The toolchain supports both the IBM Cloud® Security Best Practices with version 1.0.0 or above or the IBM Cloud® for Financial Services with version 1.2.0 or above profile. If you want a subset of specified controls, create a custom profile using these profiles and select a subset of controls. The toolchain is validated against that profile. For more information, see [Security and Compliance Center](https://cloud.ibm.com/docs/ContinuousDelivery?topic=ContinuousDelivery-scc).





 Using a config file 

Validating evidence by using a config file works as follows as follows:



* Pre-deployment checks validate the evidence that is produced by the CI pipeline and gates the deployment based on these checks. If all the checks are validated successfully, the change request is auto-approved. If any of the checks fail, the change request is not auto-approved and deployment is blocked. In the case of an emergency change request, the deployment is not blocked. The config file is part of the change request.
Configuring Redis as a Cache 

IBM Cloud® Databases for Redis supports changing the [Redis database configuration](https://cloud.ibm.com/docs/databases-for-redis?topic=databases-for-redis-changing-configuration), and you can use it to configure [Redis as a cache](https://redis.io/topics/lru-cache). When configured as a cache, Redis evicts old data in favor of new data according to the cache settings you define. Even when configured as a cache, Databases for Redis deployments still take a daily backup snapshot. It is not currently possible to disable backups on your deployment. They also write some data to disk for high-availability. Redis relies on copying over an .rdb file to resync followers.



 Cache Settings 

To configure the cache, you adjust the maxmemory and the maxmemory settings of your deployment. maxmemory defines the size of the cache. The maxmemory-policy defines the eviction behavior when the maxmemory limit is reached. In addition, other settings take care of database operations and tuning.



 maxmemory 

By default, maxmemory is set to 80% of a data node's available memory, so your node doesn't run out of system resources. You can adjust this setting, but set a reasonable limit. Otherwise, your data can take all the available memory and your deployment runs out of resources.





 maxmemory-policy 



Table 1. Available Redis eviction policies

 Policy Behavior 

 noeviction Does not evict keys and returns an error when the maxmemory limit is reached. 
 allkeys-lru Evicts less recently used (LRU) keys first. 
 volatile-lru Evicts less recently used (LRU) keys from the set of keys that expire first. 
 allkeys-random Evicts keys randomly. 
 volatile-random Evicts keys randomly from the set of keys that expire.
EXPECTED_FINGERPRINT="SHA256:PEAncMcnz8jNEOmBabCtJ13cg0oGI0YxLOMWVOkDgjc"
ssh-keyscan -t rsa $HOST > /tmp/hostkey
FINGERPRINT=$(ssh-keygen -lf /tmp/hostkey | cut -d ' ' -f 2)
if [ "$EXPECTED_FINGERPRINT" == "$FINGERPRINT" ]; then
cat /tmp/hostkey >> /.ssh/known_hosts
fi





 Configuring your firewall to allow Git webhook messages 

You can update your firewall configuration to allow webhook messages from Git Repos and Issue Tracking to reach endpoints that are behind the firewall. To allow Git webhook messages, open traffic to the [subnet ranges](https://cloud.ibm.com/docs/services/ContinuousDelivery?topic=ContinuousDelivery-git-subnet-ranges) for your specific region.





 Updating your display name 

You can update the display name that appears for you throughout Git Repos and Issue Tracking.



1. On the Git Repos and Issue Tracking User Settings dashboard, from the menu, click your avatar and select Edit profile. Click Profile. On the Profile page, in the Main settings section, update your full name.
2. Click Update profile settings to change the name that is displayed for you throughout Git Repos and Issue Tracking.









 Physical file and repo size limits 

Files are strictly limited to 100 MB. The suggested repo size limit is 1 GB. If your repo exceeds 1 GB, you might receive an email with a request to reduce the size of the repo.
If the LUN is being presented to the hypervisor to serve VHD files, choose Hyper-V for its OS type.





* Is the allocated IOPS limit enforced by instance or by volume?

IOPS is enforced at the volume level. Said differently, two hosts connected to a volume with 6000 IOPS share that 6000 IOPS.
* Measuring IOPS

IOPS is measured based on a load profile of 16-KB blocks with random 50 percent read and 50 percent writes. Workloads that differ from this profile can experience inferior performance. To improve performance, you can try [adjusting the host queue depth settings](https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-hostqueuesettings) or [enabling Jumbo frames](https://cloud.ibm.com/docs/BlockStorage?topic=FileStorage-jumboframes).
* What happens when a smaller block size is used to measure performance?

Maximum IOPS can still be obtained when you use smaller block sizes. However, throughput becomes smaller. For example, a volume with 6000 IOPS would have the following throughput at various block sizes:



* 16 KB * 6000 IOPS == 93.75 MB/sec
* 8 KB * 6000 IOPS == 46.88 MB/sec
* 4 KB * 6000 IOPS == 23.44 MB/sec



* How can I tell how much of the Storage is being used? Why are the usage details of Block Storage for Classic not displayed in the UI?

Block Storage for Classic is yours to format and manage the way that you want to. IBM Cloud® can't see the contents of the LUN, and so the UI can't provide information about the disk space usage. You can obtain more information about the volume, such as how much disk space is taken and how much is available, from your Compute host's operating system.

You can use the following commands.



* Linux®:

df -h

The command provides an output that shows how much space is available space and the percentage used.

$ df -hT /dev/sda1
Filesystem Type Size Used Avail Use% Mounted on
--snapshot-size INTEGER Optional parameter for ordering snapshot
space along with endurance file storage;
specifies the size (in GB) of snapshot space
to order
--service-offering [storage_as_a_service|enterprise|performance]
The service offering package to use for
placing the order [optional, default is
'storage_as_a_service']
--billing [hourly|monthly] Optional parameter for Billing rate (default
to monthly)
-h, --help Show this message and exit.



 Example order 

The following example shows how to order a 10 GB File Storage for Classic volume with 100 IOPS per GB.

 slcli file volume-order --storage-type performance --size 20 --location dal10 --iops 100
Order 32076317 placed successfully!
> Storage as a Service
> File Storage
> 20 GBs
> 100 IOPS

For more information about ordering through the IBM Cloud CLI, see [Working with the File Storage service (ibmcloud sl file)](https://cloud.ibm.com/docs/cli?topic=cli-sl-file-storage-servicesl_file_volume_order).

By default, you can provision a combined total of 700 File Storage for Classic volumes. To increase the number of your volumes, contact your sales representative. Read about increasing limits [here](https://cloud.ibm.com/docs/FileStorage?topic=FileStorage-managinglimits). For more information about the limit on simultaneous authorizations, see the [FAQs](https://cloud.ibm.com/docs/FileStorage?topic=FileStorage-file-storage-faqsauthlimit).







 Ordering File Storage for Classic with the API 

The method order_file_volume (storage_type, location, size, iops=None, tier_level=None, snapshot_size=None, service_offering='storage_as_a_service', hourly_billing_flag=False) places an order for a file volume.

You must specify the following parameters for a successful order.
The instructions in this topic are available for VPC worker nodes only. If you want to attach raw, unformatted block storage to a classic worker node, you must install the [IBM Cloud Block Storage attacher plug-in](https://cloud.ibm.com/docs/containers?topic=containers-utilitiesblock_storage_attacher).

Before you begin:

[Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.](https://cloud.ibm.com/docs/containers?topic=containers-access_cluster)



1. List your storage volumes and note the ID of the volume that you want to attach.

ibmcloud is vols
2. List the worker nodes in your cluster and note the ID of the worker node where you want to attach your volume.

ibmcloud ks worker ls -c <cluster_name_or_ID>
3. Attach your Block Storage for Classic to your VPC worker node.

ibmcloud ks storage attachment create --cluster <cluster_name_or_ID> --volume <volume> --worker <worker_ID>





 Removing raw Block Storage for VPC from VPC worker nodes by using the CLI 

You can remove storage from your worker node by using the ibmcloud ks storage attachment rm command.

[Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.](https://cloud.ibm.com/docs/containers?topic=containers-access_cluster)



1. List your storage volumes and note the ID of the volume that you want to remove.

ibmcloud is vols
2. Get the details of your volume such as the worker-id where the volume is attached. The worker-id is listed as the Instance name in the Volume Attachment Instance Reference section of the command output.

ibmcloud is vol <volume-ID>

Example output

Volume Attachment Instance Reference Attachment type Instance ID Instance name Auto delete Attachment ID Attachment name
If CI finds the issue and the Due Date is not available, the value of the Due Date is n/a.





 Processing results into issues 

Issues for problems that are found are created according to the results of specific tools. The collect-evidence script attempts to process the result files in the attachments and create a list of issues.

Issues are bound to assets, which can be commits in a repository or a docker image with a digest.

An issue is created for every issue ID - an issue ID is composed of the following components:



* Bound asset
* The tool that found the vulnerability
* The vulnerability identifier (the CVE ID, for example)



For example, if CVE-2022-001 is found by two separate tools, the process creates two issues today.



 Supported tools 

Currently, the following tools are supported for issue processing:



* CRA
* VA
* Detect secrets
* OWASP-ZAP API scanner
* OWASP-ZAP UI scanner
* SonarQube







 Unsupported tools or result formats 

If the collect-evidence script receives an attachment from an unsupported tool, or the result file format is not recognized during processing, the script skips creating issues and uses the result file as a simple attachment to the evidence.





 Issue contents 



* Issue is the name of the problem or vulnerability.
* Due Date shows the date when the fix is due or n/a if no Due Date is available.
* Subject is the asset that the issue is bound to.
* URL is the link to the exact version of the asset.
* Tool Type is the tool that produced the contents for the issue



For issues that are created on Git Repos and Issue Tracking, the due date is set in the GitLab native due date field instead of the issue description.

The issue description contains the timestamp when the issue was first discovered. For example, First found on 2022-04-07. The date is in YYYY-MM-DD format. The locations where the problem occurs are listed in the comments of the issue.









 Postponing the due date of an incident issue
To view a complete list of certifications for Secrets Manager, see section 5.4 of the [Secrets Manager software product compatibility report](https://www.ibm.com/software/reports/compatibility/clarity-reports/report/html/softwareReqsForProduct?deliverableId=FDDE7600BDD511EA92B4FC8223E18670).



Security and Compliance Center



* How do I start monitoring a rule?

Security and Compliance Center supports two types of rules - predefined and custom. Predefined rules are associated with specifications by default and are automatically monitored when you create an attachment to a profile. Custom rules can be monitored after they are associated with a specification by customizing a control library. Then, you select those controls when you create an attachment.
* Can I create a rule for a service that I didn't provision?

Yes, you can create rules for services or resources that are not already provisioned in your accounts. When the service or resource is created, it is automatically evaluated according to your rule definition.



Solutions IBM Cloud Framework for Financial Services



* What are the supported reference architectures for the IBM Cloud for Financial Services?

See [references architectures](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-reference-architecture-overview).
* What is a consumer and what is an application provider?

We refer to the entity that uses (consumes) the application as the consumer and the entity that deploys (provides) the application as the application provider. This documentation generally assumes that you are a an application provider.

The application provider and consumer might be the same when the organization providing the application also consumes it (for example, a financial institution deploys an application for internal use by others in their company). Or, they can be different when a third-party ISV provides an application for a financial institution to use.

Application providers who are within financial institutions are not beholden to the IBM Cloud Framework for Financial Services control requirements.
* What does it mean for a service to be IBM Cloud for Financial Services Validated?
The global_allocation_limit is set in the [memorymanager] section to limit memory allocation for the SAP HANA production system and the nonproduction system that is running on NODE2.

On NODE2, define an environment variable with the wanted memory limit for the secondary HANA production instance.

export GLOBAL_ALLOCATION_LIMIT=<memory_size_in_mb_for_hana_secondary>

Then, run the following command to update the global.ini configuration file.

sudo -i -u ${sid}adm -- <<EOT
python $DIR_INSTANCE/exe/python_support/setParameter.py -set SYSTEM/global.ini/system_replication/preload_column_tables=false -set SYSTEM/global.ini/memorymanager/global_allocation_limit=$GLOBAL_ALLOCATION_LIMIT
EOT

Verify the configuration file.

cat /hana/shared/${SID}/global/hdb/custom/config/global.ini

You cannot use hdbsql and ALTER SYSTEM ALTER CONFIGURATION ... statements on the secondary, no SQL connect is possible in this state. Activate the change by using the hdbnsutil –reconfig command.

sudo -i -u ${sid}adm -- hdbnsutil -reconfig

Update the global.ini configuration file of the nonproduction instance to allow for the memory resource usage of the secondary.

On NODE2, define an environment variable with the wanted memory limit for the nonproduction HANA instance.

export NON_PROD_GLOBAL_ALLOCATION_LIMIT=<memory_size_in_mb_for_non_prod_hana>

Then, run the following command to update the global.ini configuration file.

sudo -i -u ${sid_np}adm -- <<EOT
python $DIR_INSTANCE/exe/python_support/setParameter.py -set SYSTEM/global.ini/memorymanager/global_allocation_limit=$NON_PROD_GLOBAL_ALLOCATION_LIMIT -reconfigure
EOT

Verify the configuration file.