vania-janet/multiturn-rag-retrieval-data huggingface dataset:
* Allow all outbound traffic from the virtual server instance
* Allow all traffic in the security group



* 1 subnet to enable networking in your VPC
* 2 virtual server instances with SAP certified storage and network configurations
* 2 floating IP address that you use to access your VPC virtual server instance over the public network



During the second phase, the Ansible Playbook is called and the SAP architecture is installed for both dedicated VSI’s SAP App VSI machine and dedicated SAP HANA VSI box. The SAP architecture that is deployed is the SAP NW 7.x release on stand-alone dedicated SAP HANA 2.0 box release. For more information about this architecture, see [Automating SAP HANA stand-alone virtual server instance on IBM Cloud® VPC by using Terraform and Ansible](https://cloud.ibm.com/docs/sap?topic=sap-automate-terraform-sap-hana-vsi).





 Single-host SAP HANA system 

A single-host system is the simplest system installation type that runs an SAP HANA system entirely on one host. You can scale the system up as needed. The single-host system has these components:

Zoom

![Figure 1. Sample reference architecture](https://cloud.ibm.com/docs-content/v1/content/f89cc64b006ca59b35404693959c83d5747fee3e/sap/images/refarch-sap-hana-single-host-only.svg)

Figure 1. SAP NetWeaver 7.x with SAP HANA standard installation with AAS on VSI to VPC IBM Cloud

The scripts are designed to create a new VPC and install SAP (SAP NW 7.x release) solution together with its dedicated DB SAP HANA box in one task flow.





 SAP Kits 

For each IBM Cloud region, IBM allocates temporary storage on a dedicated temporary Deployment server (Bastion Server) that is used for terraform environment. It is your responsibility to download the necessary SAP and DB kits to your Deployment (Bastion) Server. All file archives are decompressed by Ansible during the automatic deploying process.
* The architecture fully separates compute, metadata, and storage to provide ultimate flexibility.
* Fit for purpose query engines designed to handle modern data formats that are highly elastic and scalable.
* Data sharing between watsonx.data, Db2® Warehouse on Cloud, and Netezza or any other data management solution through common Iceberg table format support, connectors, and a shareable metadata store.



* How can I create a watsonx.data service instance?

To create a watsonx.data service instance, see [Provision an instance](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-getting-started) in [Getting started with watsonx.data](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-getting-started).
* How can I access watsonx.data web console?

To access the watsonx.data web console, see [Open the web console](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-getting-started) in [Getting started with watsonx.data](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-getting-started).
* Which data formats for ingestion job are supported in watsonx.data?

Data ingestion in watsonx.data supports .CSV and .Parquet formats.
* How can I delete the watsonx.data instance?

To delete the watsonx.data instance, see [Deleting watsonx.data instance](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-delete_lh).
* How can I manage IAM access for watsonx.data?

IBM Cloud® Identity and Access Management (IAM) controls access to watsonx.data service instances for users in your account. Every user that accesses the watsonx.data service in your account must be assigned an access policy with an IAM role.
COS_INSTANCE_CRN = "<service-instance-id>" eg "crn:v1:bluemix:public:cloud-object-storage:global:a/3bf0d9003xxxxxxxxxx1c3e97696b71c:d6f04d83-6c4f-4a62-a165-696756d63903::"

 Create resource
cos = ibm_boto3.resource("s3",
ibm_api_key_id=COS_API_KEY_ID,
ibm_service_instance_id=COS_INSTANCE_CRN,
config=Config(signature_version="oauth"),
endpoint_url=COS_ENDPOINT
)
Show more

A client provides a low-level interface to the COS S3 API. This allows for processing HTTP responses directly, rather than making use of abstracted methods and attributes provided by a resource to access the information contained in headers or XML response payloads.

import ibm_boto3
from ibm_botocore.client import Config, ClientError

 Constants for IBM COS values
COS_ENDPOINT = "<endpoint>" Current list avaiable at https://control.cloud-object-storage.cloud.ibm.com/v2/endpoints
COS_API_KEY_ID = "<api-key>" eg "W00YixxxxxxxxxxMB-odB-2ySfTrFBIQQWanc--P3byk"
COS_INSTANCE_CRN = "<service-instance-id>" eg "crn:v1:bluemix:public:cloud-object-storage:global:a/3bf0d9003xxxxxxxxxx1c3e97696b71c:d6f04d83-6c4f-4a62-a165-696756d63903::"

 Create client
cos = ibm_boto3.client("s3",
print("CLIENT ERROR: {0}n".format(be))
except Exception as e:
log_error("Unable to update metadata: {0}".format(e))
Show more





 Using COPY to update metadata 

def update_metadata_copy(bucket_name, item_name, key, value):
try:
 set the new metadata
new_metadata = {
key: value
}

 set the copy source to itself
copy_source = {
"Bucket": bucket_name,
"Key": item_name
}

cos_cli.copy_object(Bucket=bucket_name, Key=item_name, CopySource=copy_source, Metadata=new_metadata, MetadataDirective="REPLACE")

print("Metadata update (COPY) for {0} Complete!n".format(item_name))
except ClientError as be:
print("CLIENT ERROR: {0}n".format(be))
except Exception as e:
log_error("Unable to update metadata: {0}".format(e))
Show more







 Using Immutable Object Storage 



 Add a protection configuration to an existing bucket 

Objects written to a protected bucket cannot be deleted until the protection period has expired and all legal holds on the object are removed. The bucket's default retention value is given to an object unless an object-specific value is provided when the object is created. Objects in protected buckets that are no longer under retention (retention period has expired and the object does not have any legal holds), when overwritten, will again come under retention. The new retention period can be provided as part of the object overwrite request or the default retention time of the bucket will be given to the object.

The minimum and maximum supported values for the retention period settings MinimumRetention, DefaultRetention, and MaximumRetention are a minimum of 0 days and a maximum of 365243 days (1000 years).

def add_protection_configuration_to_bucket(bucket_name):
try:
new_protection_config = {
"Status": "Retention",
* --default-trusted-profile-auto-link: If set to true, the system creates a link to the specified target trusted profile during instance creation. Regardless of whether a link is created by the system or manually by using the IAM Identity service, it automatically deletes when the instance is deleted. One of: true,false. (default: true).
* --metadata-service, --ms: Enable or disable the Instance Metadata Service. One of: true, false.
* --metadata-service-protocol, --msp: The communication protocol for the metadata service endpoint. Applies only when the metadata service is enabled. One of: http, https. (default: http).
* --metadata-service-response-hop-limit, --msrhl: The hop limit (IP time to live) for IP response packets from the metadata service.
* --host-failure-policy: The action to perform if the compute host experiences a failure. One of: restart, stop. (default: restart).
* --resource-group-id: ID of the resource group. This ID is mutually exclusive with --resource-group-name.
* --resource-group-name: Name of the resource group. This name is mutually exclusive with --resource-group-id.
* --output: Specify output format, only JSON is supported. One of: JSON.
* --interactive, -i:
* -q, --quiet: Suppress verbose output.



--------------------







 ibmcloud is instance-create-from-template 

Create a virtual server instance from instance template.
For details about a single snapshot, use the VPC API to make a GET /snapshots call and specify the snapshot ID.

curl -X GET "$vpc_api_endpoint/v1/snapshots/7528eb61-bc01-4763-a67a-a414a103f96d?version=2022-12-16&generation=2" -H "Authorization: $iam_token"

A successful response looks like the following example.

{
"bootable": true,
"clones": [],
"captured_at": "2022-12-16T20:19:12Z",
"created_at": "2021-12-16T20:18:18Z",
"crn": "crn:[...]",
"deletable": true,
"encryption": "user_managed",
"encryption_key": {
"crn": "crn:[...]"
},
"href": "https://us-south.iaas.cloud.ibm.com/v1/snapshots/7528eb61-bc01-4763-a67a-a414a103f96d",
"id": "7528eb61-bc01-4763-a67a-a414a103f96d",
"lifecycle_state": "stable",
"minimum_capacity": 100,
"name": "my-snapshot",
"operating_system": {
"architecture": "amd64",
"dedicated_host_only": false,
"display_name": "Ubuntu Linux 20.04 LTS Focal Fossa Minimal Install (amd64)",
"family": "Ubuntu Linux",
DNS Services Maps IBM Cloud services FQDN to IP addresses that provides access to those services within the customer's VPC. 
 Pulsar Notifies of encryption key changes at IBM Key Protect or Hyper Protect Crypto Services that affect volumes, instances, or images in VPC. 
 New Relic Stores a set of metrics events that are used by IBM's internal infrastructure team. 
 LaunchDarkly Manages the rollout of new features in VPC Infrastructure Services. LaunchDarkly provides feature flags used to control visibility and availability of a feature to a selected user base. 



The following diagram depicts and classifies the dependencies of the VPC service:

Zoom

![VPC Dependency Diagram](https://cloud.ibm.com/docs-content/v1/content/f02b9c9adcbb7328d95a45e105cec371d50f15eb/vpc/images/vpc-dependencies-architecture.svg)

Figure 2. VPC dependencies

For purposes of classification, a dependency is considered "hard" if a failure can impact the availability of some of or all of the VPC service. For example, failure of the Red Hat Licensing Service prevents activation of compute instances that are provisioned with IBM Cloud-provided Red Hat Enterprise Linux images in that region.

The data that you provide to the VPC service in your region is exchanged only with logging and data services in the same region. Data backups are stored in IBM Cloud Object Storage in the same region.





 Working with Approved Scanning Vendors (ASVs) 

It is solely the responsibility of the customer to have ASV scans of LBaaS and VPN data-plane appliances completed on a quarterly basis (a PCI requirement). IBM does not use ASVs to scan data-plane appliances because the scans can negatively impact customer workload functions and performance.
{endpoint}/{object-name}?uploadId={uploadId}= virtual host style

Example request

DELETE /some-bucket/multipart-object-123?uploadId=0000015a-df89-51d0-2790-dee1ac994053 HTTP/1.1
Authorization: Bearer {token}
Host: s3.us.cloud-object-storage.appdomain.cloud

Example response

HTTP/1.1 204 No Content
Date: Thu, 16 Mar 2017 22:07:48 GMT
X-Clv-Request-Id: 06d67542-6a3f-4616-be25-fc4dbdf242ad
Accept-Ranges: bytes
Server: Cleversafe/3.9.1.114
X-Clv-S3-Version: 2.5





 Using S3cmd (CLI) 

[S3cmd](https://s3tools.org/s3cmd) is a free Linux and Mac command-line tool and client for uploading, retrieving, and managing data in cloud storage service providers that use the S3 protocol. It is designed for power users who are familiar with command-line programs and is ideal for batch scripts and automated backup. S3cmd is written in Python. It's an open source project available under GNU Public License v2 (GPLv2) and is free for both commercial and private use.

S3cmd requires Python 2.6 or newer and is compatible with Python 3. The easiest way to install S3cmd is with the Python Package Index (PyPi).

pip install s3cmd
What’s happening 

You are receiving a refresh token error in the generate a plan, apply a plan, and destroy resources requests: Error: The provided Refresh Token is invalid. Please provide a proper refresh token for Terraform to run the configuration. Code: 400

 Why it’s happening 

You didn't provide the correct refresh token, or you didn't provide a refresh token altogether.

 How to fix it 

Check to see whether the refresh token that was generated by using the curl command is correct; otherwise, regenerate the refresh token.





 Why am I receiving an error when I apply a change to my workspace? 

 What’s happening 

You are receiving the following error when you try to apply a change to your workspace: Apply failed due to "Error: Error Deleting Volume : The volume is still attached to an instance."

 Why it’s happening 

After reconfiguring the volume profile, capacity, or IOPS, your workspace needs to be cleaned up before applying the change.

 How to fix it 

You need to destroy your existing resources and try applying the change again. Your data on the storage node will be deleted if you destroy your existing resources.





 Why does cluster creation fail with SSH issues? 

You are receiving the following error messages when the Ansible provisioner tries to set up the Spectrum Scale function on the worker and storage node resources:

 What’s happening 



* msg": "Failed to connect to the host via ssh, Connection closed by UNKNOWN port 65535", "unreachable": true}
* Error: Failed to connect to the host via ssh: Connection timed out during banner exchange", "unreachable



 Why it’s happening 

While Schematics deploys the infrastructure resources, the automation code is configured with a few Ansible playbooks, which are required to set up the Spectrum Scale function on the virtual server instance nodes with the help of the Ansible provisioner. When the Ansible provisioner tries to SSH to these nodes to se the Spectrum Scale feature, the nodes go to an unreachable state.

To fix the issue, you can:

 How to fix it 



1.
You can find the Client ID in the APIs & Services section of the project dashboard. true N/A 
 Authorization URI auth_uri Secret Authorization URI for the service account. You can find this in the JSON service account key file. true N/A 
 Token URI token_uri Secret Token URI for the service account. You can find this in the JSON service account key file. true N/A 
 URL for the authorization provider certificate auth_provider_x509_cert_url Secret URL for the authorization provider certificate. You can find this in the JSON service account key file. true N/A 
 URL for the client certificate client_x509_cert_url Secret URL for the client certificate. You can find this in the JSON service account key file. true N/A 









 Storage class reference for Compute Engine 

Review the Satellite storage classes for Google compute engine persistent disk storage. You can describe storage classes in the command line with the oc describe sc <storage-class-name> command.



Table 2. Storage class reference for Google compute engine persistent disk storage

 Storage class name Default Read IOPS per GB Default Write IOPS per GB Size range (per disk) Hard disk Reclaim policy Volume Binding Mode 

 sat-gce-block-platinum NA NA 500 GB - 64 TB SSD Delete Immediate 
 sat-gce-block-platinum-metro NA NA 500 GB - 64 TB SSD Delete WaitForFirstConsumer 
 sat-gce-block-gold 30 30 10 GB - 64 TB SSD Delete Immediate 
 sat-gce-block-gold-metroDefault 30 30 10 GB - 64 TB SSD Delete WaitForFirstConsumer 
 sat-gce-block-silver 6 30 10 GB - 64 GB SSD Delete Immediate 
 sat-gce-block-silver-metro 6 6 10 GB - 64 GB SSD Delete WaitForFirstConsumer 
 sat-gce-block-bronze 0.75 1.5 10 GiB - 64 TiB HDD Delete Immediate
This step customizes the IBM Analytics Engine instance to make the Databases for PostgreSQL certificate available to all Spark workloads run against the instance through the library set.



1. Upload the customization_script.py from the page in [Script based customization](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-cust-script) to a IBM Cloud Object Storage bucket.
2. Run postgres-cert-customization-submit.json that uses the spark-submit REST API to customize the instance. Note that the code references postgres.cert that you uploaded to IBM Cloud Object Storage.

{
    "application_details": 
    {
       "application": "/opt/ibm/customization-scripts/customize_instance_app.py",
"arguments": ["{"library_set":{"action":"add","name":"certificate_library_set","script":{"source":"py_files","params":"https://s3.direct.<CHANGME>.cloud-object-storage.appdomain.cloud","<CHANGEME_BUCKET_NAME>","postgres.cert","<CHANGEME_ACCESS_KEY>","<CHANGEME_SECRET_KEY>"]}}}"],
"py-files": "cos://CHANGEME_BUCKET_NAME.mycosservice/customization_script.py"
    }
} 

Note that the library set name certificate_library_set must match the value of the Databases for PostgreSQL metastore connection parameter ae.spark.librarysets that you specified.



4. Specify the following Databases for PostgreSQL metastore connection parameters as part of the Spark application payload or as instance defaults. Make sure that you use the private endpoint for the "spark.hadoop.javax.jdo.option.ConnectionURL" parameter below:
* --catalog-offering: The CRN for the IBM Cloud catalog offering. If specified, the latest version of that offering is used. For more information about creating a catalog offering, see [Onboarding software to your account](https://cloud.ibm.com/docs/account?topic=account-create-private-catalog&interface=cli).
* --catalog-offering-version: The CRN for the version of a IBM Cloud catalog offering. For more information about creating a version for the catalog offering, see [Onboarding software to your account](https://cloud.ibm.com/docs/account?topic=account-create-private-catalog&interface=cli).
* --total-volume-bandwidth: The amount of bandwidth (in megabits per second) that is allocated exclusively to instance storage volumes. An increase in this value results in a corresponding decrease to total network bandwidth.
* --boot-volume: BOOT_VOLUME_JSON|@BOOT_VOLUME_JSON_FILE, boot volume attachment in JSON or JSON file. For the data schema, see the boot_volume_attachment property in the [API documentation](https://cloud.ibm.com/apidocs/vpccreate-instance).
* --volume-attach: VOLUME_ATTACH_JSON|@VOLUME_ATTACH_JSON_FILE, volume attachment in JSON or JSON file, list of volumes. For the data schema, see the volume_attachments property in the [API documentation](https://cloud.ibm.com/apidocs/vpccreate-instance).
* --keys: Comma-separated IDs or names of SSH keys. SSH keys can either be of type RSA or Ed25519. Ed25519 can be used only if the operating system supports this key type. Ed25519 can't be used with Windows or VMware images.
* --dedicated-host: ID or name of the host destination where the instance is placed.
* --dedicated-host-group: ID or name of the host group destination where the instance is placed.
For more information, see [Storage types](https://cloud.ibm.com/docs/hpc-spectrum-scale?topic=hpc-spectrum-scale-storage-types).
* Why are there two different resource group parameters that can be specified in the IBM Cloud catalog tile?

The first resource group parameter entry in the Configure your workspace section in the IBM Cloud catalog applies to the resource group where the Schematics workspace is provisioned on your IBM Cloud account. The value for this parameter can be different than the one used for the second entry in the Parameters with default values section in the catalog. The second entry applies to the resource group where VPC resources are provisioned.
* Where are the Terraform files used by the IBM Spectrum Scale tile located?

The Terraform-based templates can be found in this [GitHub repository](https://github.com/IBM/ibm-spectrum-scale-ibm-cloud-schematics).
* Where can I find the custom image name to image ID mappings for each cloud region?

The mappings can be found in the image-map.tf file in this [GitHub repository](https://github.com/IBM/ibm-spectrum-scale-ibm-cloud-schematics).
* Can I use my own custom image in the Spectrum Scale cluster by specifying the image name in the deployment value bootstrap_osimage_name?

No, you can't use your own custom image for the bootstrap node at this time. The bootstrap node image is configured with all of the required functions to set up the Spectrum Scale compute and storage resources.
* Can I connect directly through SSH to the bootstrap, compute, or storage nodes from a system external to IBM Cloud?

No, any SSH connection to the bootstrap, compute, or storage nodes is only possible through the bastion node for security reasons. You would use the following command to connect to your bootstrap, compute, or storage nodes (the IP address is specific to your particular node): ssh -J ubuntu@<bastion_IP_address> vpcuser@<IP_address>
"name": "ibm_customer_number",
"value": "12345",
},
{
"name": "compute_cluster_gui_username",
"value": "admin",
},
{
"name": "compute_cluster_gui_password",
"value": "P@ssw0rd",
},
{
"name": "storage_cluster_gui_username",
"value": "admin",
},
{
"name": "storage_cluster_gui_password",
"value": "P@ssw0rd",
},
{
"name": "storage_cluster_key_pair",
"value": "scale-hpcc-key",
},
{
"name": "compute_cluster_key_pair",
"value": "scale-hpcc-key",
}]

template_repo_request_model = {}
template_repo_request_model['url'] = 'https://github.com/IBM/ibm-spectrum-scale-ibm-cloud-schematics'

logging.info("Started Creating Schematic Workspace")

workspace_response = schematics_service.create_workspace(
description="HPC Cluster schematic workspace using API",
name="Sample Schematic API workspace",
template_data=[template_source_data_request_model],
template_repo=template_repo_request_model,
type=['terraform_v1.0.0'],
location="us-south",
resource_group="Default",
tags=[""]
).get_result()

print(json.dumps(workspace_response, indent = 2))

Show more





 Example Python response 

The following Python response is a generic example for reference.
If not set, the context instance specified by ibmcloud cis instance-set INSTANCE is used.--output: Specify output format, only JSON is supported.<-- </section "id="section-update-glb-pool-options" "> --><-- <section "id="section-update-glb-pool-examples" "> --> Examples Update GLB pool 17b5962d775c646f3f9725cbc7a53df4. ibmcloud cis glb-pool-update 17b5962d775c646f3f9725cbc7a53df4 --json '{"description":"application server pool in US", "name":"us-pool", "enabled":true, "check_regions":"WNAM"], "minimum_origins":1,"monitor":"f1aba936b94213e5b8dca0c0dbf1f9cc", "origins":{"name":"us-app-dal01","address":"1.1.1.1","enabled":true,"header":{"host":"example.com"]}}, {"name":"us-app-dal02","address":"2.2.2.2","enabled":true}] ! ! ! , "notification_email":"someone@example.com"}'-i "cis-demo"
<-- </section "id="section-update-glb-pool-examples" "> --><-- </section "id="section-update-glb-pool" "> --><-- <section "id="section-list-glb-monitors" "> --> ibmcloud cis glb-monitors List GLB monitors for a given service instance. ibmcloud cis glb-monitors -i, --instance INSTANCE] --output FORMAT] ! ! ! ! ! !