vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Connecting Event Streams to IBM Cloud Object Storage by using the Kubernetes Service 

The following task walks you through:



* Getting the Kafka Connect runtime to run in an Kubernetes Service cluster.
* Starting the IBM Cloud® Object Storage Sink Connector to archive data from Kafka topics in Event Streams to an instance of the IBM Cloud® Object Storage service.



The Connector consumes batches of messages from Kafka and uploads the message data as objects to a bucket in the Cloud Object Storage service.

Complete the following steps to get set up:



 Step 1. Install the prerequisites 

Ensure you have the following software and services installed:



* An Event Streams instance - Standard or Enterprise plan. You need to create credentials.
* An instance of the Cloud Object Storage service with at least one bucket.
* An IBM Cloud® Kubernetes Service cluster. You can provision a free one for testing purposes.

You also need CLI access to your cluster. For more information, see [Setting up the CLI and API](https://cloud.ibm.com/docs/containers?topic=containers-cs_cli_install).
* A recent version of kubectl.
* [Git](https://git-scm.com/downloads)







 Step 2. Clone the kafka-connect repositories 

Clone the following two repositories that contain the required files:



* [https://github.com/ibm-messaging/kafka-connect-ibmcos-sink](https://github.com/ibm-messaging/kafka-connect-ibmcos-sink)
* [https://github.com/ibm-messaging/event-streams-samples](https://github.com/ibm-messaging/event-streams-samples)







 Step 3. Create your Kafka Connect configuration 



1. You must set up this configuration only once. Event Streams stores it for future use.
Projects, as service instances, with deployable architectures can be managed centrally in one account. By establishing trust through the project's CRN, it can assume the identity of a trusted profile in another account in the same or another enterprise account hierarchy, then deploy a solution pattern with its resources.







 Step 1: Trusted profile with compute resource 

To put theory into praxis, you are going to authorize a containerized app to perform tasks in an IBM Cloud account. The app is deployed to a Kubernetes cluster. It serves as compute resource which is going to be used to establish trust to use the trusted profile. You can perform all the following steps in a web browser with multiple open tabs. Make sure to leave the browser tabs open as instructed.

For security reasons, the app is operating in a read-only mode. It tries to gather a list of your deployed resources. You will assign privileges to the app which determine which resources it can read. Moreover, you will deploy the app in a way, so that it is accessible from within the Kubernetes cluster only, not from the public internet.

The blog post [Turn Your Container Into a Trusted Cloud Identity](https://www.ibm.com/cloud/blog/turn-your-container-into-a-trusted-cloud-identity) discusses the same scenario.





 Step 2: Kubernetes cluster as compute resource 

Kubernetes Service provides an environment to deploy highly available apps in containers that run in Kubernetes clusters.

Skip this section if you have an existing cluster you want to reuse with this tutorial, throughout the remainder of this tutorial the cluster name is referenced as mycluster-tpcr, simply substitute with the name of your cluster. Note the minimum required Kubernetes version of 1.21.

A minimal cluster with one (1) zone, one (1) worker node and the smallest available size (Flavor) is sufficient for this tutorial. A minimum Kubernetes version of 1.21 is required. Make sure to select an appropriate version when creating the cluster.

Open the [Kubernetes clusters](https://cloud.ibm.com/kubernetes/clusters) and click Create cluster.
[More info](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_logs) 
 [IBM Cloud® Kubernetes Service](https://cloud.ibm.com/docs/containers?topic=containers-getting-started) You can use the IBM Cloud Kubernetes Service service to deploy highly available apps in Docker containers that run in Kubernetes clusters. [More info](https://cloud.ibm.com/docs/containers?topic=containers-healthlogdna) 
 [Red Hat® OpenShift® on IBM Cloud®](https://cloud.ibm.com/docs/openshift?topic=openshift-getting-started) With Red Hat OpenShift on IBM Cloud, you can deploy apps on highly available clusters that come installed with the Red Hat OpenShift on IBM Cloud Container Platform software installed on Red Hat Enterprise Linux. [More info](https://cloud.ibm.com/docs/openshift?topic=openshift-health) 
 [IBM Cloud Satellite®](https://cloud.ibm.com/docs/satellite?topic=satellite-getting-started) With IBM Cloud Satellite, you can bring your own compute infrastructure to run IBM Cloud services and consistently deploy, manage, and control your app workloads. [More info](https://cloud.ibm.com/docs/satellite?topic=satellite-health) 



To see the list of IBM Log Analysis locations where these services send logs, see [Container services](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-cloud_services_locationscs_locations_container).





 Database services 

The following table lists services that send logs to IBM Log Analysis:



Table 4. List of database services

 Service Description More info 

 [IBM Cloudant](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-getting-started-with-cloudant) IBM Cloudant is a document-oriented database as a service (DBaaS). It stores data as documents in JSON format.
Connecting an IBM Cloud application 

Applications running in IBM Cloud can be bound to your IBM Cloud® Messages for RabbitMQ deployment.



 Connecting a Kubernetes Service application 

There are two steps to connecting a Cloud databases deployment to a Kubernetes Service application. First, your deployment needs to be bound to your cluster and its connection strings stored in a secret. The second step is configuring your application to use the connection strings.

The sample app in the [Connecting a Kubernetes Service Tutorial](https://cloud.ibm.com/docs/messages-for-rabbitmq?topic=cloud-databases-tutorial-k8s-app) provides a sample application that uses Node.js and demonstrates how to bind the sample application to a Cloud Databases deployment.

Before connecting your Kubernetes Service application to a deployment, make sure that the deployment and cluster are both in the same region and resource group.



 Binding your deployment 

Public Endpoints - If you are using the default public service endpoint to connect to your deployment, you can run the cluster service bind command with your cluster name, the resource group and your deployment name.

ibmcloud ks cluster service bind <your_cluster_name> <resource_group> <your_database_deployment>

OR
Private Endpoints - If you want to use a private endpoint (if one is enabled on your deployment), then first you need to create a service key for your database so Kubernetes can use it when binding to the database.

ibmcloud resource service-key-create <your-private-key> --instance-name <your_database_deployment> --service-endpoint private

The private service endpoint is selected with --service-endpoint private. After that, you bind the database to the Kubernetes cluster through the private endpoint with the cluster service bind command.

ibmcloud ks cluster service bind <your_cluster_name> <resource_group> <your_database_deployment> --key <your-private-key>

Verify - Verify that the Kubernetes secret was created in your cluster namespace. Running the following command, you get the API key for accessing the instance of your deployment that's provisioned in your account.
Service Description Integration docs 

 [IBM Cloud Kubernetes Service](https://cloud.ibm.com/docs/containers?topic=containers-getting-started) You can use the IBM Cloud Kubernetes Service service to deploy highly available apps in Docker containers that run in Kubernetes clusters. [View docs](https://cloud.ibm.com/docs/containers?topic=containers-encryptionkeyprotect) 
 [Red Hat OpenShift on IBM Cloud](https://cloud.ibm.com/docs/openshift?topic=openshift-encryption) You can use the Red Hat OpenShift on IBM Cloud service to deploy secure, highly available apps in OpenShift clusters. [View docs](https://cloud.ibm.com/docs/openshift?topic=openshift-encryption) 







 Ingestion service integrations 

You can integrate Key Protect with the following integration services.



Table 5. Supported integration services.

 Service Description Integration docs 

 [IBM Cloud Monitoring](https://cloud.ibm.com/docs/monitoring?topic=monitoring-getting-started) The IBM Cloud Monitoring service is a container-intelligence management system. You can use it to gain operational visibility into the performance and health of your applications, services, and platforms. [View docs](https://cloud.ibm.com/docs/monitoring?topic=monitoring-mng-data) 
 [IBM Cloud Schematics](https://cloud.ibm.com/docs/schematics?topic=schematics-getting-started) The IBM Cloud Schematics service delivers Terraform-as-a-Service. You can use it to organize your IBM Cloud resources across environments by using workspaces. [View docs](https://cloud.ibm.com/docs/schematics?topic=schematics-secure-datapi-encrypt)
Kubernetes-based microservice usage 

Learn how to deploy microservice applications, store data generated by those microservices in IBM Cloud® Object Storage, and then query that data by using Data Engine:



*  [Building highly available apps with IBM Container Service, Kubernetes, and Rancher 2.0](https://www.ibm.com/cloud/blog/building-apps-ibm-container-service-kubernetes-rancher-2-0)
*  [Connecting to IBM Cloud Object Storage in Kubernetes](https://www.ibm.com/cloud/blog/connecting-ibm-cloud-object-store-kubernetes)
*  [Querying your distributed application data by using IBM Cloud Object Storage and IBM Cloud Data Engine](https://www.ibm.com/cloud/blog/querying-distributed-application-data-using-ibm-cloud-object-storage-ibm-sql-query)
1. Provision an instance of the IBM Log Analysis service. With this step, you configure a centralized log management system where log data is hosted on IBM Cloud.
2. Provision a cluster on the IBM Cloud Kubernetes Service. Kubernetes v1.9+ clusters are supported.
3. Configure the logging agent on every worker (node) in a cluster.



![logging component overview on the IBM Cloud](https://cloud.ibm.com/docs-content/v1/content/f61dbc5d84bf0c89e477a05cab04c72f429e1b7a/log-analysis/images/Log-Analysis-02-Kubernetes.svg)

In this tutorial, you will learn how to configure cluster-level logging.



 Before you begin 

Work in a [supported region](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-regions). Note: You can send data from a Kubernetes cluster that is located in the same region or in a different region.

Read about IBM Log Analysis. For more information, see [About](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-getting-startedgetting-started_ov).

Use a user ID that is a member or an owner of an IBM Cloud account. To get an IBM Cloud user ID, go to: [Registration](https://cloud.ibm.com/login).

Your IBMID must have assigned IAM policies for each of the following resources in the region that your IBM Log Analysis instance is in:



Table 1. List of IAM policies required to complete the tutorial

 Resource Scope of the access policy Role Information 

 Resource group default Resource group Viewer This policy is required to allow the user to see service instances in the Default resource group. 
 IBM Log Analysis service Resource group Editor This policy is required to allow the user to provision and administer the IBM Log Analysis service in the default resource group.
* [IBM Cloud services and third-party integrations](https://cloud.ibm.com/docs/containers?topic=containers-ibm-3rd-party-integrations)



2. Add services to your cluster: Ask your cluster administrator to [add the integration to your cluster](https://cloud.ibm.com/docs/containers?topic=containers-learning-path-adminadmin_integrate).
3. Access services from your app: Ensure that your app can access the service. For example, to access an IBM Cloud service instance from your app, you must [make the service credentials that are stored in the Kubernetes secret available to your app](https://cloud.ibm.com/docs/containers?topic=containers-service-bindingadding_app).



Need help? Check out [Troubleshooting apps and integrations](https://cloud.ibm.com/docs/containers?topic=containers-debug_worker_nodes). {: tip}f
[Using IBM Cloud Kubernetes Service on classic to host the dl-reverse-proxy](https://cloud.ibm.com/docs/satellite?topic=satellite-dl-iks-classic)Using IBM Cloud Kubernetes Service on classic to host the dl-reverse-proxy

Follow these steps to set up the dl-reverse-proxy for IBM Cloud® Direct Link by using IBM Cloud Kubernetes Service on classic.

Satellite Kubernetes service

+1

Direct Link



* 2 hours
* 2023-01-30



[Deploying OpenShift Data Foundation on Satellite clusters with Azure worker nodes](https://cloud.ibm.com/docs/satellite?topic=satellite-odf-tutorial)Deploying OpenShift Data Foundation on Satellite clusters with Azure worker nodes

In this tutorial, you complete the following tasks to set up OpenShift Data Foundation.

Satellite Kubernetes service

+1

Virtual Private Cloud (VPC)



* 2 hours
* 2023-06-02



[Creating an IBM Cloud Kubernetes Service cluster on VPC infrastructure](https://cloud.ibm.com/docs/schematics?topic=schematics-provisioning-terraform-template)Creating an IBM Cloud Kubernetes Service cluster on VPC infrastructure

Use one of the IBM® provided templates to create an IBM Cloud® Kubernetes Service cluster in a Virtual Private Cloud (VPC). Then, you bind the cluster to an IBM Cloud® Object Storage service instance.

Schematics Kubernetes service

+1

Key Protect



* 60 minutes
* 2023-05-26



[Enhance security of your deployed application](https://cloud.ibm.com/docs/solution-tutorials?topic=solution-tutorials-extended-app-security)Enhance security of your deployed application

You already developed and deployed an application on IBM Cloud. You already followed the introductory tutorial on how to apply end to end security to an application. Following "security by design", you now start to look into the design of a new application, or you need to adapt an older application to new security requirements.
Connecting an IBM Cloud application 

Applications running in IBM Cloud can be bound to your IBM Cloud® Databases for Elasticsearch deployment.



 Connecting a Kubernetes Service application 

There are two steps to connecting a Cloud databases deployment to a Kubernetes Service application. First, your deployment needs to be bound to your cluster and its connection strings stored in a secret. The second step is configuring your application to use the connection strings.

The sample app in [Connecting a Kubernetes Service Tutorial](https://cloud.ibm.com/docs/databases-for-elasticsearch?topic=cloud-databases-tutorial-k8s-app) provides a sample application that uses Node.js and demonstrates how to bind the sample application to a Cloud Databases deployment.

Before connecting your Kubernetes Service application to a deployment, make sure that the deployment and cluster are both in the same region and resource group.



 Binding your deployment 

Public Endpoints - If you are using the default public service endpoint to connect to your deployment, you can run the cluster service bind command with your cluster name, the resource group and your deployment name.

ibmcloud ks cluster service bind <your_cluster_name> <resource_group> <your_database_deployment>

OR
Private Endpoints - If you want to use a private endpoint (if one is enabled on your deployment), then first you need to create a service key for your database so Kubernetes can use it when binding to the database.

ibmcloud resource service-key-create <your-private-key> --instance-name <your_database_deployment> --service-endpoint private

The private service endpoint is selected with --service-endpoint private. After that, you bind the database to the Kubernetes cluster through the private endpoint with the cluster service bind command.

ibmcloud ks cluster service bind <your_cluster_name> <resource_group> <your_database_deployment> --key <your-private-key>

Verify - Verify that the Kubernetes secret was created in your cluster namespace. Running the following command, you get the API key for accessing the instance of your deployment that's provisioned in your account.
Assigning cluster access by using Terraform for IBM Cloud Kubernetes Service 

Terraform on IBM Cloud® enables predictable and consistent provisioning of IBM Cloud services so that you can rapidly build complex, multitiered cloud environments following Infrastructure as Code (IaC) principles. Similar to using the IBM Cloud CLI or API and SDKs, you can automate the provisioning, update, and deletion of your IBM Cloud Kubernetes Service resources by using HashiCorp Configuration Language (HCL).

Looking for a managed Terraform on IBM Cloud® solution? Try out [IBM Cloud® Schematics](https://cloud.ibm.com/docs/schematics?topic=schematics-getting-started). With Schematics, you can use the Terraform scripting language that you are familiar with, but you don't have to worry about setting up and maintaining the Terraform command line or the IBM Cloud® Provider plug-in. Schematics also provides pre-defined Terraform templates that you can easily install from the IBM Cloud® catalog.



 Creating a Kubernetes Service cluster by using Terraform 

Before you can assign cluster access by using Terraform, follow the steps to create a Kubernetes Service cluster using Terraform. If you already have a cluster, see [Assigning IAM user access](https://cloud.ibm.com/docs/containers?topic=containers-terraform-setupterraform-cluster-IAM).

Before you begin, make sure that you have the [required access](https://cloud.ibm.com/docs/containers?topic=containers-access_reference) to create and work with IBM Cloud Kubernetes Service resources.



1. Follow the [Terraform on IBM Cloud® getting started tutorial](https://cloud.ibm.com/docs/ibm-cloud-provider-for-terraform?topic=ibm-cloud-provider-for-terraform-getting-started) to install the Terraform CLI and configure the IBM Cloud® Provider plug-in for Terraform. The plug-in abstracts the IBM Cloud® APIs that are used to provision, update, or delete Kubernetes Service service instances and resources.
: Optional: Do not show the message of the day or update reminders.



 Example cluster rm command 

ibmcloud ks cluster rm --cluster my_cluster







 ibmcloud ks cluster service bind 

Virtual Private Cloud

Classic infrastructure

Add an IBM Cloud service to a cluster by binding the service instance to a Kubernetes namespace. This command creates service credentials of an IBM Cloud service and stores these credentials in a Kubernetes secret in your cluster.

To view available IBM Cloud services from the IBM Cloud catalog, run ibmcloud service offerings. Note: You can add only IBM Cloud services that support service keys. For more information about service binding and what services you can add to your cluster, see [Adding services by using IBM Cloud service binding](https://cloud.ibm.com/docs/containers?topic=containers-service-binding).

ibmcloud ks cluster service bind --cluster CLUSTER --namespace KUBERNETES_NAMESPACE [--key SERVICE_INSTANCE_KEY] [--role IAM_SERVICE_ROLE] --service SERVICE_INSTANCE [-q]

Minimum required permissions
: Editor platform access role for the cluster in IBM Cloud Kubernetes Service and Developer Cloud Foundry role

Command options:

-c, --cluster CLUSTER
: Required: The name or ID of the cluster.

-n, --namespace KUBERNETES_NAMESPACE
: Required: The name of the Kubernetes namespace where you want to create the Kubernetes secret for your service credentials.

--key SERVICE_INSTANCE_KEY
: Optional: The name or GUID of an existing service key. When you use the service-binding command, new service credentials are automatically created for your service instance and assigned the IAM Writer service access role for IAM-enabled services. If you want to use an existing service key that you created earlier, use this option. If you define a service key, you can't set the --role option at the same time because your service keys are already created with a specific IAM service access role.

--role IAM_SERVICE_ROLE
: The IBM Cloud IAM role that you want the service key to have.
* See [Best practices for billing and usage](https://cloud.ibm.com/docs/billing-usage?topic=billing-usage-best-practices).
* Organize your billing with [resource groups](https://cloud.ibm.com/docs/account?topic=account-rgs).
* [Add tags to your clusters](https://cloud.ibm.com/docs/containers?topic=containers-add_workerscluster_tags) according to your organizational strategy.



2. Plan the type of cluster that you need.



* [Size your cluster to support your workloads](https://cloud.ibm.com/docs/containers?topic=containers-strategysizing), including the network bandwidth that your workloads need.
* [Decide the cluster environment that you want](https://cloud.ibm.com/docs/containers?topic=containers-strategykube_env).
* [Consider the availability that you want for your cluster](https://cloud.ibm.com/docs/containers?topic=containers-ha_clusters). For example, a basic high availability setup is one multizone cluster with two worker nodes in each of three zones, for a minimum total of 6 worker nodes.



3. Check out other IBM Cloud services, add-ons, operators, and other third-party software that you might use that can increase your cost. To get an idea of what other costs clusters typically incur, review [Understanding costs for your clusters](https://cloud.ibm.com/docs/containers?topic=containers-costscosts-for-clusters).
4. [Estimate your costs](https://cloud.ibm.com/docs/billing-usage?topic=billing-usage-costcost) and review detailed pricing information for the service, see [IBM Cloud Kubernetes Service: Pricing](https://www.ibm.com/cloud/kubernetes-service).
5. Manage the lifecycle of your cluster to control costs.
* Ensure that you have the [Writer or Manager IBM Cloud IAM service access role](https://cloud.ibm.com/docs/containers?topic=containers-userschecking-perms) for the namespace in which you deploy the Kubernetes LoadBalancer service for the VPC NLB.
* [Log in to your account. If applicable, target the appropriate resource group. Set the context for your cluster.](https://cloud.ibm.com/docs/containers?topic=containers-access_cluster)
* To view VPC NLBs, install the infrastructure-service plug-in. The prefix for running commands is ibmcloud is.

ibmcloud plugin install infrastructure-service
* When cluster nodes are reloaded or when a cluster master update includes a new keepalived image, the load balancer virtual IP is moved to the network interface of a new node. When this occurs, any long-lasting connections to your load balancer must be re-established. Consider including retry logic in your application so that attempts to re-establish the connection are made quickly.





1. [Deploy your app to the cluster](https://cloud.ibm.com/docs/containers?topic=containers-deploy_appapp_cli). Ensure that you add a label in the metadata section of your deployment configuration file. This custom label identifies all pods where your app runs to include them in the load balancing.
2. Create a configuration YAML file for your Kubernetes LoadBalancer service. Consider naming the service in the format <app_name>-vpc-nlb-<VPC_zone>.

Mixed protocol load balancer services are not currently supported in IBM Cloud Kubernetes Service. Do not specify both TCP and UDP ports in your load balancer definition, as doing so causes an error that prevents the load balancer service from provisioning. Note that specifying the service.kubernetes.io/ibm-load-balancer-cloud-provider-vpc-health-check-udp annotation with a TCP port does not create this conflict.

apiVersion: v1
kind: Service
metadata: