vania-janet/multiturn-rag-retrieval-data huggingface dataset:
User data 

When you create an IBM Cloud® Virtual Servers for Virtual Private Cloud instance, you can specify optional user data that automatically performs common configuration tasks or runs scripts. For more information about user data and usernames, see [IAM access](https://cloud.ibm.com/docs/account?topic=account-userroles).

VPC uses Cloud-init technology to configure virtual server instances. The User Data field on the New virtual server for VPC page allows users to put in custom configuration options by using cloud-init. Cloud-init supports several formats for configuration data, including yaml in a cloud-config file.

You can specify cloud-config data directly in the User Data field, or you can include the cloud-config data in a text file and specify the file name when you create your instance. For example, if you save the cloud-config data in userdata.blob, specify -user-data @userdata.blob when you create an instance by using the CLI.

When you create a virtual server instance by using the IBM Hyper Protect Container Runtime image, you must pass a contract as part of the User Data field. Since the image is a locked down image with SSH access disabled, the only way to interact with the instance is by passing the contract.

The size limit of the User Data field (or file) is 64 K bytes.



 User data examples for Linux 



 Adding a user and SSH key 

The following cloud-init example shows how a Linux user can add a user and provide the user with an authorized SSH key. The Name field has the public key that is added to /.ssh/authorized_keys.

For Linux images, the Ed25519 SSH key type can be used only if the SSH server for the operating system supports this key type. For more information, see [Getting started with SSH keys](https://cloud.ibm.com/docs/vpc?topic=vpc-ssh-keys).

cloud-config
users:
- name: demouser
gecos: Demo User
sudo: ALL=(ALL) NOPASSWD:ALL
That is, it must be in relative formats, such as config/myconfigfile.conf or ./myconfigfile.conf, or in an absolute path such as /path/to/config/myconfigfile.conf.

Additionally, you can set the limit for the number of enqueued URLs that are displayed as an option for the testit command, for example: crawler testit --limit [number]





 Running a crawl 

Run the following command: crawler crawl

This runs a crawl with the default configuration file (crawler.conf).

You can also specify a custom configuration file as an option for the crawl command, for example: crawler crawl --config [config/myconfigfile.conf]

The path to the configuration file passed in the --config option must be a qualified path. That is, it must be in relative formats, such as config/myconfigfile.conf or ./myconfigfile.conf, or in an absolute path such as /path/to/config/myconfigfile.conf.





 Restarting a crawl 

Run the following command: crawler restart

This command runs a restart crawl by starting a new crawl with the default configuration file (crawler.conf).

You can also specify a custom configuration file as an option for the restart command, for example: crawler restart --config [config/myconfigfile.conf]

The path to the configuration file passed in the --config option must be a qualified path. That is, it must be in relative formats, such as config/myconfigfile.conf or ./myconfigfile.conf, or in an absolute path such as /path/to/config/myconfigfile.conf.





 Resuming a crawl 

Run the following command: crawler resume

This command resumes a crawl from where it stopped with the default configuration file (crawler.conf).
If no size is specified, the size of the original volume is used.

-i, --duplicate-iops
: Performance Storage IOPS. Range 100-6000, and in multiples of 100. If no IOPS is specified, the IOPS of the original volume is used.

-t, --duplicate-tier
: Endurance Storage Tier. If no tier is specified, the tier of the original volume is used.

-n, --duplicate-snapshot-size
: The size of snapshot space to order for the duplicate. If no snapshot space size is specified, the snapshot space size of the original volume is used.

-d, --dependent-duplicate
: Whether this duplicate is a dependent duplicate of the origin volume.

-f, --force
: Force operation without confirmation.

Examples:

ibmcloud sl file volume-duplicate 12345678

This command shows order a new volume by duplicating the volume with ID 12345678.





 ibmcloud sl file volume-order 

Order a file storage volume:

ibmcloud sl file volume-order [OPTIONS]

Command options:

-t, --storage-type
: Required. Type of storage volume, options are: performance, endurance.

-s, --size
: Required. Size of storage volume in GB.

-i, --iops
: Performance Storage IOPs. Range 100-6000, and in multiples of 100 [required for storage-type performance].

-e, --tier
: Endurance Storage Tier (IOP per GB) [required for storage-type endurance]. Options are: 0.25,2,4,10.

-d, --datacenter
: Required. Data center short name.

-n, --snapshot-size
: Optional parameter for ordering snapshot space along with the volume.

-b, --billing
: Optional parameter for Billing rate (default to monthly), options are: hourly, monthly.

-f, --force
: Force operation without confirmation.

Examples:
To configure the log level, you must customize the log section in the dragent.yaml file.

The monitoring agent generates log entries in /opt/draios/logs/draios.log.



* The log file rotates when it reaches 10MB in size.
* The 10 most recent log files are kept. The date-stamp that is appended to the filename is used to determine which files to keep.
* Valid log levels are: none, error, warning, info, debug, trace
* The default log level is info, where an entry is created for each aggregated metrics transmission to the backend servers, once per second, in addition to entries for any warnings and errors.
* You can customize the type of log and the entries that are collected by configuring the monitoring agent configuration file /opt/draios/etc/dragent.yaml. After you edit the file, you must restart the agent at the shell with docker restart sysdig-agent to activate the changes.





Table 2. Log section entries

 Use cases Log section entry 

 Troubleshoot agent behavior file_priority: debug 
 Reduce container console output console_priority: warning 
 Filtering events by severity event_priority: warning 
 Verify what metrics are included or excluded metrics_excess_log: true
myvol Bound pvc-01ac123a-123b-12c3-abcd-0a1234cb12d3 20Gi RWX ibmc-file-bronze 147d
2. Retrieve the StorageType, volumeId, and the server of the physical File Storage for Classic that is associated with your PVC by listing the details of the PV that your PVC is bound to. Replace <pv_name> with the name of the PV that you retrieved in the previous step. The storage type, volume ID, and the server name are shown in the Labels section of your CLI output.

kubectl describe pv <pv_name>

Example output

Name: pvc-4b62c704-5f77-11e8-8a75-b229c11ba64a
Labels: CapacityGb=20
Datacenter=dal10
Iops=2
StorageType=ENDURANCE
Username=IBM02SEV1543159_6
billingType=hourly
failure-domain.beta.kubernetes.io/region=us-south
failure-domain.beta.kubernetes.io/zone=dal10
path=IBM01SEV1234567_8ab12t
server=fsf-dal1001g-fz.adn.networklayer.com
volumeId=12345678
...
3. Modify the size or IOPS of your volume in your IBM Cloud infrastructure account.

Example for performance storage.

ibmcloud sl file volume-modify <volume_ID> --new-size <size> --new-iops <iops>

Example for endurance storage.

ibmcloud sl file volume-modify <volume_ID> --new-size <size> --new-tier <iops>

volume_ID
: Enter the ID of the volume that you retrieved earlier.

new-size
: Enter the new size in gigabytes (Gi) for your volume.
Once built, they can be used for selection by key, lists of keys, ranges of keys, and aggregation of data. For example, Find orders between two dates, and calculate the total value of the orders, which are grouped by month.

IBM Cloudant includes the four built-in reducers (or five if you count none).

_count
: For counting things.

_sum
: For totalizing values.

_stats
: For providing counts and totals suitable for calculating means, variances, and standard deviations.

_approx_count_distinct
: For approximate counting of unique values of the key.

The design document's MAP function is passed a doc - the function is called once per document in the database. Any key-value pairs that emit from the MAP function create the index.

The KEY is the thing (or things) you want to select on (for example, date).

The VALUE is the thing that you need to report on (for example, total sales).

The Reducer is _sum so that the VALUE is totaled for matching keys (for example, orders on the same date).

See what defining a MapReduce looks like in the IBM Cloudant Dashboard.

When the MapReduce view is built, it can be queried to see each KEY-VALUE pair stored in the index.

Or, if the reducer is switched on, the result set can be grouped by the value of each key. In this case, we are totalizing each day's sales.

The view can be queried for individual keys (for example, sales on a specific date), all keys, or a range of keys (for example, between two dates).

MapReduce views are built asynchronously and can take some time to be ready for large data sets.

See some tips:

Use logic in your JavaScript to include only data that makes sense, for example, totalize only completed orders. Indexed keys don't have to be strings. A common pattern is to use array keys, for example, an array of year, month, day. These index keys allow query-time grouping by elements in the array.
If a self-hosted CouchDB service does _not* support HTTPS, CouchDB can mediate replication jobs instead of IBM Cloudant. For example, stop the replication job that runs on the IBM Cloudant side and set up a replication job on a self-hosted Apache CouchDB service to "pull" the data from IBM Cloudant.

IBM Cloudant supports only https:// traffic, so if IBM Cloudant is to be the source or target in a self-hosted replication definition, it must be configured with an https:// prefix.







 IBM Cloudant Correction to Provisioned Throughput accounting for partioned _find API notice 

An accounting error exists in IBM Cloudant that results in undercounting of the operations that are associated with this API against the [provisioned throughput](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-ibm-cloud-publicprovisioned-throughput-capacity) capacity of an IBM Cloudant instance. You can find the correct accounting in the IBM Cloudant API documentation under [Query a list of all documents in a database partition](https://cloud.ibm.com/apidocs/cloudantgetpartitionalldocs-partitioned-databases). Until now, this accounting operation was not implemented correctly.

Effective 1 October 2023, this accounting error is corrected. Customers performing _find queries that filter through large numbers of documents might need to increase the provisioned throughput capacity for their IBM Cloudant instances. Otherwise, immediately after this correction deploys on 1 October 2023, IBM Cloudant returns 429 errors for read operations.
The [RHEL for SAP Subscriptions and Repositories](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_for_sap_solutions/8/html/rhel_for_sap_subscriptions_and_repositories/asmb_enable_repo_rhel-for-sap-subscriptions-and-repositories-8con_hana_rhel-for-sap-subscriptions-and-repositories-8) documenation describes how to enable the required repositories.
* A nonproduction SAP HANA System is installed on NODE2 with a different SID and Instance Number than the production system. The nonproduction system needs its own dedicated storage volumes and file systems. Restrict the Global Memory Allocation Limit for the nonproduction system to ensure sufficient memory for the HANA system replication workload on the secondary. The limit is set with the global_allocation_limit parameter in the [memorymanager] section of the global.ini configuration file.
* Optional, a virtual IP address is reserved for the nonproduction system as described in [Reserving virtual IP addresses](https://cloud.ibm.com/docs/sap?topic=sap-ha-vsiha-vsi-reserve-virtual-ip-addresses).







 Setting up the cost optimized scenario 

The cost optimized scenario is an extension of the setup that is described in [Configure SAP HANA Scale-Up System Replication in a RHEL HA Add-On cluster](https://cloud.ibm.com/docs/sap?topic=sap-ha-rhel-hana-sr). Complete the setup for the production system System Replication cluster before you continue with the following steps.



 Configuring the nonproduction SAP HANA Instance on NODE2 

To simplify the setup, prepare the following environment variables for user ID root on NODE2. These environment variables are used in subsequent commands in the remainder of the instructions.

On NODE2, create a file with the following environment variables. Then, adapt them according to the configuration of the nonproduction system.

export SID_np=<SID> SAP HANA System ID of non-production system (uppercase)
0111-1a111aaa-1111-1111-111a-aaa1a1a11a11 volume-attachment attached boot a001-f11ed1e1-1aa1-11dc-b11d-a0dc111b1111 dissuade-anointer-errand-handbrake kube-aa1111aa11aaaaa11aa1-cluster-name-default-00000110
4. Remove the storage attachment.

ibmcloud oc storage attachment rm --attachment <attachment-ID> -c <cluster> --worker <worker-ID>
5. Verify that your storage was removed from the worker node.

ibmcloud oc storage attachment ls -c <cluster-ID> --worker <worker-id>









 Backing up and restoring PVC data for file and block storage 

With the IBM Cloud Backup Restore Helm chart, you can create a one-time or scheduled backup for data that is stored in a file storage or block storage persistent volume claim (PVC). Your data is stored in an IBM Cloud Object Storage service instance that you create and own. You can use existing backups in your IBM Cloud Object Storage service instance to restore data to a PVC in your cluster.

What happens when I install the Helm chart?
: When you install the Helm chart, a Kubernetes pod is created in your cluster that performs a one-time or periodic backup of your PVC data, or restores data from IBM Cloud Object Storage to a PVC. You configure your backup or restore in the values.yaml file that is provided with the Helm chart or by setting options in the helm install command.

What limitations do I need to be aware of?
: If you want to back up or restore data of a block storage PVC, your PVC must not be mounted to an app. Block storage is mounted with a RWO access mode. This access allows only one pod to be mounted to the block storage at a time. To back up or restore your data, you must remove the pod that uses the storage to unmount the PVC.
0111-1a111aaa-1111-1111-111a-aaa1a1a11a11 volume-attachment attached boot a001-f11ed1e1-1aa1-11dc-b11d-a0dc111b1111 dissuade-anointer-errand-handbrake kube-aa1111aa11aaaaa11aa1-cluster-name-default-00000110
4. Remove the storage attachment.

ibmcloud ks storage attachment rm --attachment <attachment-ID> -c <cluster> --worker <worker-ID>
5. Verify that your storage was removed from the worker node.

ibmcloud ks storage attachment ls -c <cluster-ID> --worker <worker-id>









 Backing up and restoring PVC data for file and block storage 

With the IBM Cloud Backup Restore Helm chart, you can create a one-time or scheduled backup for data that is stored in a file storage or block storage persistent volume claim (PVC). Your data is stored in an IBM Cloud Object Storage service instance that you create and own. You can use existing backups in your IBM Cloud Object Storage service instance to restore data to a PVC in your cluster.

What happens when I install the Helm chart?
: When you install the Helm chart, a Kubernetes pod is created in your cluster that performs a one-time or periodic backup of your PVC data, or restores data from IBM Cloud Object Storage to a PVC. You configure your backup or restore in the values.yaml file that is provided with the Helm chart or by setting options in the helm install command.

What limitations do I need to be aware of?
: If you want to back up or restore data of a block storage PVC, your PVC must not be mounted to an app. Block storage is mounted with a RWO access mode. This access allows only one pod to be mounted to the block storage at a time. To back up or restore your data, you must remove the pod that uses the storage to unmount the PVC. After the backup or restoring of data is finished, you can re-create your pod and mount the backed up or restored PVC.
If successful, you get a one-time passcode. Copy this passcode and paste it as a response on the prompt. After successful authentication, you are prompted to choose your account. If you have access to multiple accounts, select the account that you want to log in as. Respond to any remaining prompts to finish logging in.
2. Select the current generation of VPC.

ibmcloud is target --gen 2







 Expand volume capacity from the CLI 

From the CLI, use the ibmcloud is volume-update command with the --capacity option to indicate the new size of the volume in GBs.

ibmcloud is volume-update VOLUME_ID --capacity CAPACITY_GB

The following example expands the capacity of a general-purpose volume to 8,000 MB.

$ ibmcloud is volume-update demo-volume-update --capacity 8000
Updating volume demo-volume-update under account Test Account as user test.user@ibm.com...

ID r014-dee9736d-08ee-4992-ba8d-3b64a4f0baac
Name demo-volume-update
CRN crn:v1:bluemix:public:is:us-east-1:a/a10d63fa66daffc9b9b5286ce1533080::volume:r014-dee9736d-08ee-4992-ba8d-3b64a4f0baac
Status updating
Attachment state attached
Capacity 100
IOPS 3000
Bandwidth(Mbps) 3145
Profile general-purpose
Encryption key -
Encryption provider_managed
Resource group defaults
Created 2023-06-29T16:14:59+00:00
Zone us-east-1
Health State ok
Volume Attachment Instance Reference Attachment type Instance ID Instance name Auto delete Attachment ID Attachment name
List your storage volumes and note the ID of the volume that you want to remove.

ibmcloud is vols
2. Get the details of your volume such as the worker-id where the volume is attached. The worker-id is listed as the Instance name in the Volume Attachment Instance Reference section of the command output.

ibmcloud is vol <volume-ID>

Example output

Volume Attachment Instance Reference Attachment type Instance ID Instance name Auto delete Attachment ID Attachment name
data 0727_e18c10d7-7f18-48aa-b5ef-5ed163e54198 kube-bsaucubd07dhl66e4tgg-cluster-default-00000a19 false 0727-3bfe90b0-dc2d-498a-946b-8837a5dad7bc volume-attachment
3. List the storage attachments on a worker node in your cluster and make a note of the attachment ID that your want to remove.

ibmcloud oc storage attachment ls -c <cluster> --worker <worker-id>

Example output:

Listing volume attachments...
OK
ID Name Status Type Volume ID Volume Name Worker ID
0111-1a111aaa-1111-1111-111a-aaa1a1a11a11 volume-attachment attached boot a001-f11ed1e1-1aa1-11dc-b11d-a0dc111b1111 dissuade-anointer-errand-handbrake kube-aa1111aa11aaaaa11aa1-cluster-name-default-00000110
4. Remove the storage attachment.

ibmcloud oc storage attachment rm --attachment <attachment-ID> -c <cluster> --worker <worker-ID>
5. Verify that your storage was removed from the worker node.

ibmcloud oc storage attachment ls -c <cluster-ID> --worker <worker-id>









 Backing up and restoring PVC data for file and block storage
List your storage volumes and note the ID of the volume that you want to remove.

ibmcloud is vols
2. Get the details of your volume such as the worker-id where the volume is attached. The worker-id is listed as the Instance name in the Volume Attachment Instance Reference section of the command output.

ibmcloud is vol <volume-ID>

Example output

Volume Attachment Instance Reference Attachment type Instance ID Instance name Auto delete Attachment ID Attachment name
data 0727_e18c10d7-7f18-48aa-b5ef-5ed163e54198 kube-bsaucubd07dhl66e4tgg-cluster-default-00000a19 false 0727-3bfe90b0-dc2d-498a-946b-8837a5dad7bc volume-attachment
3. List the storage attachments on a worker node in your cluster and make a note of the attachment ID that your want to remove.

ibmcloud ks storage attachment ls -c <cluster> --worker <worker-id>

Example output:

Listing volume attachments...
OK
ID Name Status Type Volume ID Volume Name Worker ID
0111-1a111aaa-1111-1111-111a-aaa1a1a11a11 volume-attachment attached boot a001-f11ed1e1-1aa1-11dc-b11d-a0dc111b1111 dissuade-anointer-errand-handbrake kube-aa1111aa11aaaaa11aa1-cluster-name-default-00000110
4. Remove the storage attachment.

ibmcloud ks storage attachment rm --attachment <attachment-ID> -c <cluster> --worker <worker-ID>
5. Verify that your storage was removed from the worker node.

ibmcloud ks storage attachment ls -c <cluster-ID> --worker <worker-id>









 Backing up and restoring PVC data for file and block storage
Increasing capacity of an existing boot volume from the CLI 

From the CLI, locate the boot volume that you want to expand. You can use the ibmcloud is volumes command filter the results by specifying the resource group. Also, if you know the name or ID of the instance, you can view instance details and get information about the boot volume.

After you located the volume, use the volume-update command and provide the ID or name of the boot volume. Use the --capacity parameter to indicate the new size of the boot volume in GBs.

For example, this example increases the capacity of my-boot-vol1 to 200 GB. The existing capacity displays as the boot volume capacity is being expanded. Run the ibmcloud is volume command and specify the volume name to see the new capacity.

ibmcloud is volume update my-boot-vol-1 --capacity 200
Updating volume my-boot-vol1 under account VPC1 as user myuser@mycompany.com...

ID 9d60ba27-170c-4e2a-9bf6-6dbb11f95c38
Name my-boot-vol1
CRN crn:v1:staging:public:is:us-south-1:a/efe5afc483594adaa8325e2b4d1290df::volume:9d60ba27-170c-4e2a-9bf6-6dbb11f95c38
Status updating
Capacity 190
IOPS 3000
Bandwidth(Mbps) 393
Profile general-purpose
Encryption key -
Encryption provider_managed
Resource group Default
Created 2022-02-24T16:43:47+05:30
Zone us-south-1
Volume Attachment Instance Reference Attachment type Instance ID Instance name Auto delete Attachment ID Attachment name
boot 0716_84f99419-554d-4c05-bea0-7034d1c40ed3 vsi-1 true 0716-ee0ca315-7a21-42e2-99f7-b68377bbffe0 boot-vol-name
Workspace failures 

Review the following sections to help debugging workspace new failures.



  Workspace create fails with message Request Entity Too Large 

  What’s happening 

When you create a workspace from an existing terraform.tfstate file, it fails with error code 413 and a message Required Entiry Too Large.

  Why it’s happening 

Your workspace create did not reach the Schematics, due to the size limitation of the terraform.tfstate file.

Example error message

FAILED
Could not create the workspace. Please verify that your request is correct. If the problem persists, contact IBM Cloud support.

Status Code: 413
Message:

<title>413 Request Entity Too Large</title>

  How to fix it 

The terraform.tfstate file size must be less than 2 MB. When you create workspace from an existing Terraform state file, the terraform.tfstate file size must be less than 2 MB. Greater than 2 MB state file size is not supported in the Schematics. Rerun the workspace create operation with a size that is less than 2 MB state file.