vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Using IBM Cloud Direct Link to connect to IBM Cloud Object Storage 

You can configure IBM Cloud® Direct Link so that you have access to IBM Cloud Object Storage (COS). Although the methods described here were designed and tested with COS, they can work for certain other IBM Cloud services.

By policy, IBM Cloud Direct Link denies access to IBM Cloud private service endpoints, including those endpoints used by COS. The technique that is described in this document relies on indirect access to COS through servers hosted in a customer's IBM Cloud account. After setup, each customer's server can forward traffic bidirectionally between IBM Cloud private service endpoints and their remote networks that are connected by Direct Link.



 What is IBM Cloud Object Storage? 

IBM Cloud Object Storage (COS) is a web-scale solution that stores unstructured data. It provides reliability, security, availability, and disaster recovery without manual replication.

The information that is stored within IBM Cloud Object Storage is encrypted and dispersed across many geographic locations. It's accessible through an implementation of the S3 API. This service uses distributed storage technologies that are provided by the IBM Cloud Object Storage service.

IBM COS is available in three configurations:



* Cross Region service provides higher durability and availability than using a single region, but at the cost of slightly higher latency. This service is available today in the US and the EU. By using a Virtual Router Appliance (VRA), you also might use Direct Link to connect to COS in the Asia Pacific region.
* Regional service provides the reverse. It distributes objects across many availability zones within a single region. If a region or availability zone is inaccessible, the object store continues to function smoothly. Any missed changes are applied when the inaccessible data center comes back online.
* Single Site service offers affordable access to COS in a selected data center.





 COS private and public endpoints 

Endpoints are URLs that applications use to issue COS commands and exchange data with COS. Every endpoint uses the same application programming interface (API) to interact with COS.

Servers that are provisioned within IBM Cloud use private API endpoints for services, including COS.
IBM Analytics Engine is a compute engine offered in IBM Watson® Studio and can be used to push Watson Studio jobs to IBM Analytics Engine. Data can be written to Cloudant or Db2 Warehouse on Cloud after being processed by using Spark.
* How is IBM Analytics Engine integrated with IBM Watson Studio?

IBM Analytics Engine is a first class citizen in IBM Watson® Studio. Projects (or individual notebooks) in Watson Studio can be associated with IBM Analytics Engine.

Once you have an IBM Analytics cluster running in IBM Cloud, log in to Watson Studio using the same IBM Cloud credentials you used for IBM Analytics Engine, create a project, go to the project's Settings page, and then add the IBM Analytics Engine service instance you created to the project. For details, including videos and tutorials, see [IBM Watson Learning ](https://developer.ibm.com/clouddataservices/docs/analytics-engine/get-started/).

After you have added the IBM Analytics Engine service to the project, you can select to run a notebook on the service. For details on how to run code in a notebook, see [Code and run notebooks](https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/code-run-notebooks.html).
* Can I use Kafka for data ingestion?

IBM Message Hub, an IBM Cloud service is based on Apache Kafka. It can be used to ingest data to an object store. This data can then be analyzed on an IBM Analytics Engine cluster. Message Hub can also integrate with Spark on the IBM Analytics Engine cluster to bring data directly to the cluster.
* Can I set ACID properties for Hive in IBM Analytics Engine?

Hive is not configured to support concurrency. Although you can change the Hive configuration on IBM Analytics Engine clusters, it is your responsibility that the cluster functions correctly after you have made any such changes.
* How much time does it take for the cluster to get started?

When using the Spark software pack, a cluster takes about 7 to 9 minutes to be started and be ready to run applications. When using the Hadoop and Spark software pack, a cluster takes about 15 to 20 minutes to be started and be ready to run applications.
Adding custom service parameters for your service 

When you onboard your service to IBM Cloud®, you might need additional information from your customers. To get the additional information, you can add custom input fields for your product in IBM Cloud Partner Center. Customers can use the custom fields to input any additional information that you need when they create instances of your product.



 Adding custom input fields 

Partner center offers two input options for the required customer information. You can choose a Simple input option to add custom fields like short text and paragraph entries. Or, for advanced use cases that require more complex inputs, you can choose the Advanced option to receive the input as JSON code.



 Simple input option to add custom fields 

To add custom fields for simple use cases, complete the following steps:



1. In the IBM Cloud console, click the Navigation Menu icon ![Navigation Menu icon](https://cloud.ibm.com/docs-content/v1/content/ff5860bfede49db3197c4bbba7f7123769bdc068/icons/icon_hamburger.svg) > Partner Center > Sell > My products.
2. Select your product.
3. Click Product details.
4. To add custom setup options, click Add custom fields.
5. Select Simple.
6. Select Short text entry, Paragraph entry, or Option selection based on how you want to receive the input from customers.
7. Complete each field.

You can test and review how the custom fields are displayed to customers by looking at the preview section of the side panel. This preview is not saved.
8. Click Done. You can edit your custom fields by clicking the Edit icon ![Edit icon](https://cloud.ibm.com/docs-content/v1/content/ff5860bfede49db3197c4bbba7f7123769bdc068/icons/edit-tagging.svg). To delete your custom fields, click the Delete icon !
Audit logging 

Audit logs are available for IBM® Cloudant® for IBM Cloud®. You can find more details on how to obtain audit logs depending on whether you use IBM Cloudant or outside IBM Cloud®.



 IBM Cloudant for IBM Cloud 

Users of IBM Cloudant can use IBM Cloud® Activity Tracker to access audit logs for the service. IBM Cloud Activity Tracker records user-initiated activities that change the state of a service in IBM Cloud. You can use this service to investigate abnormal activity and critical actions and to comply with regulatory audit requirements. You can also be alerted about actions as they happen. The events that are collected comply with the IBM Cloud Auditing Data Federation (CADF) standard. For more information, see [IBM Cloud Activity Tracker events](https://cloud.ibm.com/docs/services/Cloudant?topic=Cloudant-at_events).





 IBM Cloudant not in IBM Cloud 

Users of IBM Cloudant outside of IBM Cloud, such as users of IBM Cloudant Enterprise dedicated clusters, can contact support to request audit logs as described in the following section. Any user who requires frequent access to audit logs must migrate to IBM Cloudant in IBM Cloud.



 How to access audit logs for your account 

Audit logging records the IBM Cloudant principals who accessed data that is stored in IBM Cloudant. For all HTTP API access to IBM Cloudant, the audit log function records the following information about each HTTP request:



Table 1. Recorded audit information

 Information Description 

 Principal Account credentials, API keys, or IBM Cloud IAM credentials, as identified by an HTTP request header. 
 Action The action carried out, for example, document read. 
 Resource Details about the account, database, and document accessed or query made. 
 Timestamp A record of the time and data of the event. 



IBM Cloudant audit logs can be used to understand:



* What and when databases and documents were accessed within an account, and by whom.
* What and when queries were run, and by whom.
* What a specific principal or user that is accessed, updated, or deleted, and when.
* What and when replication documents were created or deleted.



To request access to the audit logs for your account, contact IBM Cloudant support.
* containers-kubernetes.logging-config.refresh
* containers-kubernetes.logging-filter.create
* containers-kubernetes.logging-filter.delete
* containers-kubernetes.logging-filter.update
* containers-kubernetes.logging-autoupdate.changed
* containers-kubernetes.masterlog-retrieve
* containers-kubernetes.masterlog-status
* containers-kubnertes.cluster.rbac.update
* containers-kubernetes.service.create
* containers-kubernetes.service.delete
* containers-kubernetes.subnet.add
* containers-kubernetes.subnet.create
* containers-kubernetes.subnet.update
* containers-kubernetes.vlan.create
* containers-kubernetes.vlan.delete
* containers-kubernetes.worker.create
* containers-kubernetes.worker.delete
* containers-kubernetes.worker.update
* containers-kubernetes.workerpool.create
* containers-kubernetes.workerpool.delete
* containers-kubernetes.workerpool.update
* containers-kubernetes.zone.delete
* containers-kubernetes.zone.update



Deprecated fields across events
: The following fields are deprecated and replaced by or updated with new values across events.



* The correlationID is replaced by correlationId to align with field casing standards.
* The resourceGroupID field is no longer used. Instead, the resource group ID can be found in the target.resourceGroupId field.
* The reason.reasonCode field is now formatted as an integer (int) instead of string (string).
* The requestData field is now formatted as a JSON object instead of a JSON string.
* The responseData field is now formatted as a JSON object instead of string.
* The target.typeURI values are updated for consistency.
* UI
* CLI







 Deploying OpenShift Data Foundation on VPC clusters 

OpenShift Data Foundation is a highly available storage solution that you can use to manage persistent storage for your containerized workloads in Red Hat® OpenShift® on IBM Cloud® clusters.

Installing OpenShift Data Foundation from OperatorHub is not supported on IBM Cloud clusters. To install ODF, complete the following steps to deploy the cluster add-on.

Minimum required permissions
: Administrator platform access role
: Manager service access role for the cluster in IBM Cloud Kubernetes Service.



 Prerequisites 

Review the following prerequisites.



1. [Install](https://cloud.ibm.com/docs/openshift?topic=openshift-cli-install) or update the oc CLI.
2. Create a [VPC cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-clusters) with at least 3 worker nodes. For high availability, create a cluster with at least one worker node per zone across three zones. Each worker node must have a minimum of 16 CPUs and 64 GB RAM.

You can deploy OpenShift Data Foundation on 3 worker nodes of 16 CPUs and 32 GB RAM, but you must taint your worker nodes to run only ODF pods. You can't run any additional app workloads or system pods on your ODF nodes when you use this setup.





 Optional: Setting up an IBM Cloud Object Storage service instance 

Complete the following steps to create an IBM Cloud Object Storage instance which you can use as the default backing store in your ODF deployment. If you don't want to set up IBM Cloud Object Storage, you can skip this step and [install the add-on](https://cloud.ibm.com/docs/openshift?topic=openshift-deploy-odf-vpcinstall-odf-cli-vpc).

If you want to set up IBM Cloud Object Storage as the default backing store in your storage cluster, create an instance of IBM Cloud Object Storage.
However, if ibmcloud-api-key does not have access to git, git-token must be set.



 Configuring the secrets stores 

With IBM Cloud, you can choose from various secrets management and data protection offerings that help you protect your sensitive data and centralize your secrets. You can choose between the vault integrations depending on your requirements as explained in [Managing IBM Cloud secrets](https://cloud.ibm.com/docs/secrets-manager?topic=secrets-manager-manage-secrets-ibm-cloud). This documentation provides information about prerequisites and how to use a list of prescribed secret names that are otherwise known as hints. By using hints in a template, a toolchain can be automatically populated with preconfigured secrets without any need to manually select them from various vault integrations that are attached to the toolchain.

Use [IBM Cloud® Secrets Manager](https://cloud.ibm.com/docs/secrets-manager?topic=secrets-manager-getting-started) to securely store and apply secrets like API keys, Image Signature, or HashiCorp Vault credentials that are part of your toolchain.

Zoom

![Secrets Manager tool integration form](https://cloud.ibm.com/docs-content/v1/content/5eb88a2575e30e316dbb83829c7f4c8b1ddbc67c/devsecops/images/devsecops-secrets-manager.png)

Figure 1. IBM Secrets Manager Tool Integration

The templates also come with a HashiCorp Vault tool integration like the following example:

Zoom

![HashiCorp Vault Tool Integration form with required fields and example values](https://cloud.ibm.com/docs-content/v1/content/5eb88a2575e30e316dbb83829c7f4c8b1ddbc67c/devsecops/images/hc-tool-int.png)

Figure 2. HashiCorp Vault Tool Integration

To use HashiCorp Vault, you must provide the following information:



* Name: A name for this tool integration.
* UI
* CLI







 Deploying OpenShift Data Foundation on VPC clusters 

OpenShift Data Foundation is a highly available storage solution that you can use to manage persistent storage for your containerized workloads in Red Hat® OpenShift® on IBM Cloud® clusters.

Installing OpenShift Data Foundation from OperatorHub is not supported on IBM Cloud clusters. To install ODF, complete the following steps to deploy the cluster add-on.

Minimum required permissions
: Administrator platform access role
: Manager service access role for the cluster in IBM Cloud Kubernetes Service.



 Prerequisites 

Review the following prerequisites.



1. [Install](https://cloud.ibm.com/docs/openshift?topic=openshift-cli-install) or update the oc CLI.
2. Create a [VPC cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-clusters) with at least 3 worker nodes. For high availability, create a cluster with at least one worker node per zone across three zones. Each worker node must have a minimum of 16 CPUs and 64 GB RAM.

You can deploy OpenShift Data Foundation on 3 worker nodes of 16 CPUs and 32 GB RAM, but you must taint your worker nodes to run only ODF pods. You can't run any additional app workloads or system pods on your ODF nodes when you use this setup.





 Optional: Setting up an IBM Cloud Object Storage service instance 

Complete the following steps to create an IBM Cloud Object Storage instance which you can use as the default backing store in your ODF deployment. If you don't want to set up IBM Cloud Object Storage, you can skip this step and [install the add-on](https://cloud.ibm.com/docs/openshift?topic=openshift-deploy-odf-vpc&interface=uiinstall-odf-cli-vpc).

If you want to set up IBM Cloud Object Storage as the default backing store in your storage cluster, create an instance of IBM Cloud Object Storage.
They can also manage all administrator access across the account.



 Sample access policies 

Review the following sample access policies to help you determine how you might want to assign trusted profiles access to resources that are organized in resource groups.



* A policy that grants federated users a platform administrator role on the IBM Cloud Kubernetes Service across the entire account. Federated users are allowed to apply this profile when the federated user's external IdP attributes fulfill the conditions of the trust relationship. For example, if your corporate user directory has a jobrole attribute that identifies administrators by the value admin, then you can create a condition that dynamically adds federated users with that attribute to the trusted profile, along with other conditions. The federated users that are allowed to apply the trusted profile can access all instances of this service and create instances of the service in any resource group that they have at least a viewer role assigned. Trusted profiles with an administrator role for a resource can also grant access to that resource. Conditions can be specified so that only federated users that require the highest privileges can apply this trusted profile. All other federated users can be filtered out based on their SAML attributes.
* A policy that grants compute resources Reader and Writer roles on a resource group. When compute resources authenticate and fulfill the conditions that are specified in the trusted profile, such as location or resource type the trusted profile is automatically applied. This way, any existing or future resources that meet these conditions can have the profile that is automatically applied when authenticated.
* You can also establish trust with specific compute resources, like a single Kubernetes cluster. For example, you might have an application that is running on Kubernetes Service where the application needs to read and write from IBM Cloudant and read and write to a Cloud Object Storage Dedicated IBM Managed bucket. Both the IBM Cloudant and Cloud Object Storage Dedicated IBM Managed instances would be in the same resource group and the trusted profile would be assigned Reader or Writer roles.



Using trusted profiles is a best practice for applications running on IBM Cloud compute resources to gain access to IAM-enabled resources.







 Compare access groups and trusted profiles
Using Helm charts 

You can securely store and share Helm charts with other users in IBM Cloud® Container Registry.



 OCI support for Helm charts 

The [Open Container Initiative (OCI)](https://opencontainers.org) released [Open Container Initiative Distribution Specification v1.0.0](https://specs.opencontainers.org/distribution-spec/?v=v1.0.0) in September 2021. This specification supports other artifact types in addition to container images. One of the [artifact types that is supported](https://github.com/opencontainers/artifacts/blob/main/artifact-authors.mddefining-oci-artifact-types) is the [Helm chart](https://helm.sh/docs/topics/charts/).

[Helm v3.8.0](https://github.com/helm/helm/releases/tag/v3.8.0) provides support to store and work with charts in OCI registries, as an alternative to [Helm chart repositories](https://helm.sh/docs/topics/chart_repository/). For more information, see the [Helm Registries documentation](https://helm.sh/docs/topics/registries/).





 Adding Helm charts to your namespace 

You can securely store and share Helm charts with other users by adding charts to your

namespacein IBM Cloud Container Registry.

Every Helm chart that you want to add to your namespace must exist on your local computer first. You can either download (pull) a chart from another repository to your local computer, or build your own chart by using the [helm create](https://helm.sh/docs/helm/helm_create/) command. To add a chart to your namespace, you must upload (push) the local chart to your namespace in IBM Cloud Container Registry.

Do not put personal information in your charts (for example, in namespace names or description fields) or in any chart or chart configuration data (for example, chart names or chart labels).



 Pulling charts from another registry or Helm repository
For more information, see [Update a document](https://cloud.ibm.com/apidocs/discovery-dataupdatedocument).

If your data is stored in a JSON file, an array in the original document generates a document ID with a number appended to it. For example, original_id_n. To retain the original document ID without the number suffix, remove the array in the JSON file. Change [ {"name": "value"} ] to {"name": "value"}, for example.

If your v1 documents have system-generated IDs, you can submit an empty [search query](https://cloud.ibm.com/apidocs/discoveryquery) to retrieve a list of the documents and their IDs. You can then assign the same ID to each document as you add it to your new collection in v2.





 Recovering documents 

In some cases, the original documents that were ingested into Discovery V1 are no longer available. You can use the Discovery v1 instance to retrieve information from the document. Discovery creates a text copy of each document that it ingests. The copy is text only, so any documents in HTML, PDF, or other nontext formats are converted to a text-only version.

You can recover only the first 10,000 documents in a collection by using this method.

To transfer document information from v1 to v2, complete the following steps:



1. Extract the documents from v1 by using the API to [submit an empty query](https://cloud.ibm.com/apidocs/discoveryquery).

For example, GET {url}/v1/environments/{environment_id}/collections/{collection_id}/query?q=.

The API returns the results. The matching_results field specifies the total number of results. The results object returns the matching documents. Each document is returned as a separate JSON object. It returns a maximum of 10 documents by default.

{
"matching_results": 34,
"session_token": "nnn",
"results": [
-p, --pubkey PUBLIC_KEY
[-a, --hash HASH_ALGORITHM]



 Example 

This example encrypts a key material using the public key created by kp import-token create.

 create an import token that expires in 5 minutes and allows 10 retrievals
$ ibmcloud kp import-token create -e 300 -m 10

 create a random, base64-encoded, 32-byte key material
$ KEY_MATERIAL=$(openssl rand -base64 32)

 extract the public key that was created by the "kp import-token create" command
$ PUBLIC_KEY=$(ibmcloud kp import-token show | jq -r '.["payload"]')

 encrypt the key material using the public key
$ ibmcloud kp import-token key-encrypt -k $KEY_MATERIAL -p $PUBLIC_KEY

Encrypted Key
Ela33aTdDiKVUNryLeM/xwUEaKWvzY+u ...<redacted>... Asv7bZxvyZn9KNU=





 Required parameters 



* -i, --instance-id

The IBM Cloud instance ID that identifies your Key Protect instance.

You can set an environment variable instead of specifying -i with the following command: $ export KP_INSTANCE_ID=<INSTANCE_ID>.
* -k, --key

The base64-encoded key material that you store and manage in the service.

-p, --pubkey

The base64-encoded public encryption key that was distributed by Key Protect. To create new public key, use ibmcloud kp import-token create. To retrieve a public key, use ibmcloud kp import-token show.







 Optional parameters 



* -a, --hash

The hash algorithm used to wrap the key. SHA1 and SHA256 are supported. The default is SHA256. SHA1 is only supported by HPCS.

This option is used to generate an encrypted key that can only be used in a request with the HPCS API.