vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Even parquet files in some circumstances are staged like, Parquet file is local, Parquet files in local folder, a huge Parquet file(local or S3) all of which may not fit in memory, source file is S3 parquet file, and file from S3 parquet folder which may contain special column types such as date or decimal. Example: <s3://lh-target/staging> ibm-lh data-copy --staging-location <STAGING_LOCATION> 
 staging-s3-creds S3 credentials of staging location ibm-lh data-copy --staging-s3-creds <STAGING_S3_CREDS> 
 staging-hive-catalog If the default hive catalog cannot be used due to credential mismatch with source or staging locations, it is possible to configure a custom hive catalog using this option ibm-lh data-copy --staging-hive-catalog <catalog_name> 
 system-config This parameter can be used to specify system related parameters such as memory limit ibm-lh data-copy --system-config <file_path> 
 log-directory This option is used to specify the location of log files. The ingest log files will be created undeneath the given directory. Default ingest log file is /tmp/ingest.log ibm-lh data-copy --ingest-config <ingest_config_file> --log-directory <directory_path> 
 target-catalog-uri Target catalog uri ibm-lh data-copy --target-catalog-uri <TARGET_CATALOG_URI> 
 target-table-storage Target table file storage location ibm-lh data-copy --target-table-storage <TARGET_TABLE_STORAGE> 
 target-tables Data migration target table. <catalog>.<schema>.<table1>. Example: <iceberg.demo.customer1> ibm-lh data-copy --target-tables <TARGET_TABLES> 
 schema Schema file that includes CSV specifications, and so on ibm-lh data-copy --schema <SCHEMA>
x-amz-meta-key2: value2

Content-Length: 533

The 'queen' bee is developed from larvae selected by worker bees and fed a
substance referred to as 'royal jelly' to accelerate sexual maturity. After a
short while the 'queen' is the mother of nearly every bee in the hive, and
the colony will fight fiercely to protect her.

Example response

HTTP/1.1 200 OK
Date: Thu, 25 Aug 2016 18:30:02 GMT
X-Clv-Request-Id: 9f0ca49a-ae13-4d2d-925b-117b157cf5c3
Accept-Ranges: bytes
Server: Cleversafe/3.9.0.121
X-Clv-S3-Version: 2.5
x-amz-request-id: 9f0ca49a-ae13-4d2d-925b-117b157cf5c3
ETag: "3ca744fa96cb95e92081708887f63de5"
Content-Length: 0





 Using COPY to update metadata 

The complete details about the COPY request are [here](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-object-operationsobject-operations-copy).

Syntax

PUT https://{endpoint}/{bucket-name}/{object-name} path style
PUT https://{bucket-name}.{endpoint}/{object-name} virtual host style

Example request

PUT /apiary/queen-bee HTTP/1.1
Authorization: Bearer {token}
Content-Type: text/plain; charset=utf-8
Host: s3.us.cloud-object-storage.appdomain.cloud
x-amz-copy-source: /apiary/queen-bee
x-amz-metadata-directive: REPLACE
x-amz-meta-key1: value1
x-amz-meta-key2: value2
Red Hat Licensing Service Activates a compute instance that was provisioned with an IBM-provided RHEL image. 
 Windows Licensing Service Activates a compute instance that was provisioned with an IBM-provided Windows image. 
 Ubuntu Package Update Service Provides OS packages to a compute instance provisioned with an IBM-provided CentOS image. 
 CentOS Package Update Service Provides OS packages to a compute instance provisioned with an IBM-provided CentOS image. 
 Debian Package Update Service Provides OS packages to a compute instance provisioned with an IBM-provided Debian image. 
 IBM Cloud Object Storage Provides interim storage of customer-provided VPC images, recent storage of log data, and backups of all VPC resource metadata. Also stores collected flow logs. 
 Infrastructure Management Service (IMS) Provisions, manages and shows information about storage volumes that are provisioned by VPC, and handles the management of our infrastructure racks. 
 IBM Key Protect Hosts customer root keys that wrap the data encryption keys for encrypted storage. 
 Hyper Protect Crypto Services (HPCS) Also hosts customer root keys that wrap the data encryption keys for encrypted storage. 
 Global Search and Tagging - Search (GhoST) Provides a global view of an account's IBM cloud resources, which is used by VPC to retrieve information such as which IBM KMS hosts a customer root key. 
 IBM Cloud Internet Services Serves as the initial ingress layer for all VPC API requests, providing DNS, global load balancing, firewalling, DDoS, and other security features. 
 Hypersync and Hyperwarp Enables VPC resource and lifecycle events to be published to GhoST and other subscribers. 
 IBM Event Streams Enables VPC resource and lifecycle events to be published to GhoST and other subscribers. 
 IBM Log Analysis Collects logs that are generated by the VPC service. Depending on the type of log, it might be consumed by IBM support and operations, customers, or both 
 IBM Cloud Activity Tracker Forwards service audit events to the IBM Cloud® Activity Tracker instance that is set up in the region by the cloud account owner. For more information, see [Activity Tracker events](https://cloud.ibm.com/docs/vpc?topic=vpc-at-events).
ibmcloud is instance-create my-instance r134-35b9cf35-616e-462e-a145-cf8db4062fcf us-south-2 bx2-2x8 0726-198db988-3b9b-4cfa-9dec-0206420d37d0 --catalog-offering crn:v1:public:globalcatalog-collection:global:a/efe5afc483594adaa8325e2b4d1290df:0b322820-dafd-4b5e-b694-6465da6f008a:offering:136559f6-4588-4af2-8585-f3c625eee09d
--keys r134-89ec781c-9630-4f76-b9c4-a7d204828d61 --placement-group c5f1f366-b92a-4080-991a-aa5c2e33d96b

Where the following argument and option values are used



* INSTANCE_NAME: my-instance
* VPC: r134-35b9cf35-616e-462e-a145-cf8db4062fcf
* ZONE_NAME: us-south-2
* PROFILE_NAME: bx2-2x8
* SUBNET: 0726-198db988-3b9b-4cfa-9dec-0206420d37d0
* CATALOG-OFFERING: is crn:v1:public:globalcatalog-collection:global:a/efe5afc483594adaa8325e2b4d1290df:0b322820-dafd-4b5e-b694-6465da6f008a:offering:136559f6-4588-4af2-8585-f3c625eee09d
Satellite for resellers 

As a hybrid or distributed cloud reseller, I want to set up Satellite locations and deploy Red Hat OpenShift on IBM Cloud clusters for my customers.



 Account overview 

In this model, you create an enterprise account and separate client accounts for each client. Within each client account, you create a Satellite location and deploy an Red Hat OpenShift on IBM Cloud cluster.

Zoom

![Concept overview of Satellite tenancy setup](https://cloud.ibm.com/docs-content/v1/content/c8335d66af691d19e837116fd633481c477061eb/satellite//images/account_overview_graphic.svg)

Figure 1. Enterprise account setup for resellers.





 Managing client locations and clusters 

Review the best practices for managing client locations and clusters.

Understand the client's requirements and use case.
: Gather requirements for your client's use case before creating their location and cluster. Understand the types of workloads your clients want to run in their location and the compute, storage, and other requirements that go along with their use case.

Keep extra hosts available and unassigned in the location
: For faster scale up, plan to keep extra hosts available and ready for assignment to clusters and services in the location.





 Understanding responsibilities for your enterprise and your clients 

Create a responsibility matrix for your enterprise and your clients. The following example shows a responsibility matrix that you can adapt for your purposes.



Example responsibility matrix for resellers.

 Your enterprise Client cluster administrator Client cluster operator Client developer 

 <br><br> * Prepare requirements template & review client requirements.<br> * Prepare quote.<br> * Create client accounts, locations, and clusters, and API keys.<br><br><br> <br><br> * Perform requirements assessment.<br> * Complete the requirements template.<br> * Review and approve quote.<br> * Review and confirm environment setup.<br> * Assign cluster operator and developer roles.<br><br><br> Verify cluster operator permissions Verify cluster developer permissions 







 Setting up your account
- name: customer-auth
secret:
secretName: storage-secret-store
- configMap:
name: cluster-info
name: cluster-info
- apiVersion: apps/v1
kind: StatefulSet
metadata:
annotations:
version: 2.0.3_354
labels:
addonmanager.kubernetes.io/mode: Reconcile
app: ibm-vpc-block-csi-driver
name: ibm-vpc-block-csi-controller
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
app: ibm-vpc-block-csi-driver
serviceName: ibm-vpc-block-service
template:
metadata:
labels:
app: ibm-vpc-block-csi-driver
spec:
securityContext:
runAsNonRoot: true
runAsUser: 2121
containers:
- args:
- --v=5
- --csi-address=$(ADDRESS)
- --timeout=600s
- --feature-gates=Topology=true
env:
- name: ADDRESS
value: /csi/csi.sock
image: quay.io/k8scsi/csi-provisioner:v1.6.0
securityContext:
privileged: false
allowPrivilegeEscalation: false
imagePullPolicy: Always
name: csi-provisioner
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 20Mi
volumeMounts:
- mountPath: /csi
name: socket-dir
- args:
- --v=5
- --csi-address=/csi/csi.sock
- --timeout=900s
image: quay.io/k8scsi/csi-attacher:v2.2.0
securityContext:
privileged: false
OLAP:



* BI.S4.H8.12000
* BI.S4.H8.6000
* BI.S4.H4.3000
* BI.S4.H4.6000
* BI.S4.H2.3000
* BI.S2.H8401
* BI.S2.H4101
* BI.S2.H4201



Check [SAP' Certified and Supported SAP HANA Hardware Directory](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=v:deCertified;iaas;ve:28;v:b046dad8-7aa0-457a-ade5-286ebaf88a2f&id=s:2014) for details of the supported configurations.





 Network layout for Scale-out configurations 

For Intel Bare Metal scale-out configurations, contact IBM Cloud support for assisting you to set-up the required networking. Depending on the hardware used, the choice of networks might be restricted, or special configurations might have to be adapted. See the following diagram, for the layout to use. The diagram describes the use of three fully redundant (LACP config), physically separate networks, for:



* Storage traffic,
* Internal SAP HANA inter-node communication
* Communication with the client(s), for example SAP ABAP application servers or SAP HANA Studio for administration purposes.



Use the network that holds the default route of your environment to pass the NFS traffic through it, the storage servers are reachable through that gateway, only.





 Storage for Scale-out configurations 

For scale-out configuration, the ability of storage volumes to be accessed from different server nodes is required for failover purposes. Thus, local storage is out of scope, and NFS volumes need to be deployed.
PUT /apiary/queen-bee HTTP/1.1
Authorization: Bearer {token}
Content-Type: text/plain; charset=utf-8
Host: s3.us.cloud-object-storage.appdomain.cloud
x-amz-copy-source: /apiary/queen-bee
x-amz-metadata-directive: REPLACE
x-amz-meta-key1: value1
x-amz-meta-key2: value2

Example request

PUT /apiary/queen-bee HTTP/1.1
Authorization: 'AWS4-HMAC-SHA256 Credential={access-key}/{date}/{region}/s3/aws4_request,SignedHeaders=host;x-amz-date;,Signature={signature}'
x-amz-date: {timestamp}
Content-Type: text/plain
Host: s3.us.cloud-object-storage.appdomain.cloud
x-amz-copy-source: /apiary/queen-bee
x-amz-metadata-directive: REPLACE
x-amz-meta-key1: value1
x-amz-meta-key2: value2

Example response

HTTP/1.1 200 OK
Date: Thu, 25 Aug 2016 18:30:02 GMT
X-Clv-Request-Id: 9f0ca49a-ae13-4d2d-925b-117b157cf5c3
Accept-Ranges: bytes
Server: Cleversafe/3.9.0.121
X-Clv-S3-Version: 2.5
x-amz-request-id: 9f0ca49a-ae13-4d2d-925b-117b157cf5c3
ETag: "3ca744fa96cb95e92081708887f63de5"
Content-Length: 0







 Next Steps
* UI
* CLI
* API







 How third-party services use the IBM Cloud platform 

A third-party service uses the IBM Cloud platform for authentication, access, self-service instance creation, metering, and billing. This topic provides a high-level overview of the platform components that your service uses, and pulls these concepts together into an end-to-end provisioning scenario.



 The IBM Cloud provisioning layer 

The provisioning layer manages the lifecycle of IBM Cloud resources. The provisioning layer is responsible for controlling and tracking the lifecycle of resources in a customer account. Resources are physical or logical components that can be created or reserved for an application or service instance. Examples of resources include database, accounts, processor, memory, and storage limits. In general, resources that are tracked by the provisioning layer are intended to associate usage metrics and billing, but that isn’t always the case. In some cases, the resource might be associated to the provisioning layer to ensure that the resource lifecycle can be managed along with the account lifecycle.



 Resource lifecycle management 

The provisioning layer provides common APIs to control the lifecycle of resources from creation (creating an instance) to binding (creating access credentials) to unbinding (removing access) to deletion (deleting an instance). Additionally, the IBM Cloud platform provides CLIs and a UI that can manage the lifecycle of these resources that don’t require you to create your own facilities.

The provisioning layer provides APIs to help you manage the following elements of your resource lifecycle:



* Creating a resource instance
* Updating a resource instance
* Binding
* Resource keys
* Unbinding
* Deleting a resource instance









 IBM Cloud Identity and Access Management (IAM) 

Identity Access Management (IAM) enables you to securely authenticate users and control access to all cloud resources consistently across IBM Cloud. The IBM Cloud provisioning layer adopted IAM for authentication and authorization of actions that are taken against the provisioning layer. Third-party product providers use IAM to create an authentication flow (OAuth). See [What is IAM?]
* The cluster releases the virtual IP address on NODE1, and acquires it on the new primary on NODE2.
* If an application, such as SAP NetWeaver, is connected to a tenant database of SAP HANA, the application automatically reconnects to the new primary.



On NODE2, run the following commands to check that the global_allocation_limitand preload_column_tables are unset.

sudo -i -u ${sid}adm -- hdbcons "mm globallimit" | grep limit

grep -E "global_allocation_limit|preload_column_tables" /hana/shared/${SID}/global/hdb/custom/config/global.ini





 Test1 - Recovery procedure 

As the cluster resource SAPHana_${SID}_${INSTNO} is configured with AUTOMATED_REGISTER=false, the cluster doesn't restart the failed HANA database, and doesn't register it against the new primary. The status on the new primary (NODE2) shows the secondary in status 'CONNECTION TIMEOUT'.

On NODE1, run the following commands to register the previous primary as new secondary.

sudo -i -u ${sid}adm -- <<EOT
hdbnsutil -sr_register --name=${DC1} --remoteHost=${NODE2} --remoteInstance=${INSTNO} --replicationMode=sync --operationMode=logreplay
EOT

Verify the system replication status.

sudo -i -u ${sid}adm -- <<EOT
hdbnsutil -sr_state
HDBSettings.sh systemReplicationStatus.py
EOT

On NODE1, run the following command to start the cluster node.

pcs cluster start

The new secondary instance restarts and shows up in status synced (SOK).

pcs status --full

Configure cluster resource SAPHana_${SID}_${INSTNO} with AUTOMATED_REGISTER=true.

On NODE1, run the following command.
(Note: If the current profile supports instance storage disks, the requested profile can have a different instance storage disk configuration.) 2. Be compatible with any placement target constraints. For example, if the instance is placed on a dedicated host, the requested profile family must be the same as the dedicated host family.
* --total-volume-bandwidth: The amount of bandwidth (in megabits per second) that is allocated exclusively to instance storage volumes. An increase in this value results in a corresponding decrease to total network bandwidth.
* --dedicated-host: ID or name of the host destination where the instance is placed.
* --dedicated-host-group: ID or name of the host group destination where the instance is placed.
* --metadata-service, --ms: Enable or disable the Instance Metadata Service. One of: true, false.
* --metadata-service-protocol, --msp: The communication protocol for the metadata service endpoint. Applies only when the metadata service is enabled. One of: http, https. (default: http).
* --metadata-service-response-hop-limit, --msrhl: The hop limit (IP time to live) for IP response packets from the metadata service.
* --host-failure-policy: The action to perform if the compute host experiences a failure. One of: restart, stop.
* --output: Specify output format, only JSON is supported. One of: JSON.
* -q, --quiet: Suppress verbose output.



--------------------







 ibmcloud is instance-volume-attachment 

View details of a volume attachment.

ibmcloud is instance-volume-attachment INSTANCE VOLUME_ATTACHMENT [--output JSON] [-q, --quiet]



 Command options 



* INSTANCE: ID or name of the instance.
* VOLUME_ATTACHMENT: ID or name of the volume attachment.
* --output: Specify output format, only JSON is supported. One of: JSON.
* -q, --quiet: Suppress verbose output.
registered client 1: NS-IBMADC690867-d85b
registered client 2: NS-IBMADC690867-4v36
registered client 3: NS-jpmongevsi05win2012vsi-4v36
registered client 4: NS-IBMADC690867-k8ru
registered client 5: NS-IBMADC690867-wnzs
registered client 6: NS-IBMADC690867-s6dr

Command Result : 0 (Success)
8. Assign a partition to the client. Make sure you reference the partition created before. You will also have to ensure the name matches the identifier for the client displayed in the previous step.

[jpmongehsm2] lunash:>client assignPartition -c NS-IBMADC690867-s6dr -p partition6

'client assignPartition' successful.

Command Result : 0 (Success)

The previous output uses the following syntax:

client assignPartition -client <clientname> -partition <partition name
9. Verify connectivity in your Citrus Netscaler VPX:

root@IBMADC690867-s6dr vtl verify

The following Luna SA Slots/Partitions were found:

Slot Serial Label
==== ================ =====
0 534071053 partition6

The output displayed by vtl verify should list the partition's "Slot #", serial number, and the name of the partition bound to this trust link. Any other output indicates a problem.

In addition, it's also a good idea to confirm the paths of the certificates and the server in the Chrystoki file located in the /etc directory. To do so:

root@IBMADC690867-s6dr cd /etc/
root@IBMADC690867-s6dr cat /etc/Chrystoki.conf
Chrystoki2 = {
Note on catalog (GET): This endpoint defines the contract between the broker and the IBM Cloud platform for the services and plans that the broker supports. This endpoint returns the catalog metadata that is stored within your broker. These values define the minimal contract between your service and the IBM Cloud platform. All additional catalog metadata that isn't required is stored within the IBM Cloud catalog. Any updates to catalog display values like links and icons must be updated in the IBM Cloud console, and not housed in your broker. None of metadata that is stored in your broker is displayed in the IBM Cloud console or the IBM Cloud CLI. The console and CLI return what was set within Partner Center Sell and stored in the IBM Cloud catalog. The following section shows the minimal-required values that catalog (GET) returns:

{
"services": [{
"id": "0bc9d744-6f8c-4821-9648-2278bf6925bb",
"name": "ibmcloud-link",
"description": "An IBM provided service that enables aliasing to service instances in the IBM Cloud.",
"bindable": true,
"plan_updateable": false,
"plans":
{
"id": "da40662d-2f72-4a19-8c79-8c77cf285e1",
"name": "ibmcloud-alias",
"free": true,
"description": "The IBM Cloud alias plan used for linking."
}
]
}]
}
Show more





 Required endpoints logic for bindable services 

If your service can be bound to applications in IBM Cloud, it must return API endpoints and credentials to your service consumers. A bindable service must use the bindable operations in the Open Service Broker specification, and implement the following endpoints or paths:

bindings and credentials (PUT)
: Binds your service instance to an application.

bindings and credentials (DEL)
: Unbinds your service instance from an application.





 Required IBM Cloud extension endpoints
x-amz-meta-key2: value2

Content-Length: 533

The 'queen' bee is developed from larvae selected by worker bees and fed a
substance referred to as 'royal jelly' to accelerate sexual maturity. After a
short while the 'queen' is the mother of nearly every bee in the hive, and
the colony will fight fiercely to protect her.

Example response

HTTP/1.1 200 OK
Date: Thu, 25 Aug 2016 18:30:02 GMT
X-Clv-Request-Id: 9f0ca49a-ae13-4d2d-925b-117b157cf5c3
Accept-Ranges: bytes
Server: Cleversafe/3.9.0.121
X-Clv-S3-Version: 2.5
x-amz-request-id: 9f0ca49a-ae13-4d2d-925b-117b157cf5c3
ETag: "3ca744fa96cb95e92081708887f63de5"
Content-Length: 0





 Using COPY to update metadata 

The complete details about the COPY request are [here](https://cloud.ibm.com/docs/cloud-object-storage/api-reference?topic=cloud-object-storage-object-operationsobject-operations-copy).

Syntax

PUT https://{endpoint}/{bucket-name}/{object-name} path style
PUT https://{bucket-name}.{endpoint}/{object-name} virtual host style

Example request

PUT /apiary/queen-bee HTTP/1.1
Authorization: Bearer {token}
Content-Type: text/plain; charset=utf-8
Host: s3.us.cloud-object-storage.appdomain.cloud
x-amz-copy-source: /apiary/queen-bee
x-amz-metadata-directive: REPLACE
x-amz-meta-key1: value1