vania-janet/multiturn-rag-retrieval-data huggingface dataset:
For more information, see the [Java documentation](https://kafka.apache.org/32/javadoc/org/apache/kafka/clients/admin/KafkaAdminClient.htmldescribeClusterorg.apache.kafka.clients.admin.DescribeClusterOptions).

The number of brokers can also be found by using the kafka-configs.sh shell script bundled in the Apache Kafka distribution.

bin/kafka-configs.sh --command-config command-config.properties --bootstrap-server "kafka-0.blah.cloud:9093" --describe --entity-type brokers





 Event Streams authorization 

To be authorized to set client quotas, a user must have the Manager role on the "cluster" resource in Cloud Identity and Access Management.

A set of credentials created with the Event Streams UI with the Manager role on the instance also has the Manager role on cluster. Thus, you are able to create, delete, and alter topics in addition to setting quotas. Any authenticated user has an implied Reader role on cluster and is able to describe quotas.

To create a set of credentials that can manage topics, groups, and participate in transactions but are not authrorized to set quotas, an IAM access policy that has Manager role on resource types "topic", "group", and "txnid" and Reader role on "cluster" must to be associated with the service ID.



 Example: Managing quotas with the kafka-config.sh script (Apache client) 



1. Download a Kafka binary distribution (at least V3.1.0).
2. Create a properties file (named command-config.properties in the following command lines examples), containing the following entries (replacing "myapikey" with the actual API key).



sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="token" password="myapikey";

security.protocol=SASL_SSL
"http_path" : "exampleString",
"jar_uris" : "exampleString",
"jdbc_driver" : "exampleString",
"jdbc_url" : "exampleString",
"password" : "password",
"port" : "50000",
"project_id" : "exampleString",
"properties" : "exampleString",
"refresh_token" : "exampleString",
"role" : "exampleString",
"sap_gateway_url" : "exampleString",
"server" : "exampleString",
"service_name" : "exampleString",
"sid" : "exampleString",
"ssl" : "false",
"ssl_certificate" : "exampleString",
"ssl_certificate_host" : "exampleString",
"ssl_certificate_validation" : "exampleString",
"username" : "db2inst1",
"warehouse" : "exampleString"
}
Show more





 VirtualizeTableSourceTableDefFile 

The following example shows the format of the VirtualizeTableSourceTableDefFile object.

[ {
"column_name" : "Column1",
"column_type" : "INTEGER"
} ]





 VirtualizeTableVirtualTableDefFile 

The following example shows the format of the VirtualizeTableVirtualTableDefFile object.

[ {
"column_name" : "Column_1",
"column_type" : "INTEGER"
} ]
<client name="client1" hostname="localhost" listname="dal13_primary">
</client>
</clientaffinitydefined>
</acr>
</database>
</databases>
</configuration>
Show more



This configuration assumes the primary is in the dal13 zone and the client is in dal12 (therefore, a minimum of 1 network hop will still be required)



* this sample defines the db2 connection details for two different formations in preprod - a single node formation that uses the database alias singledb and an ha formation that uses the database alias hadb
Netezza connector for Presto 

Netezza connector for Presto is a customized connector for IBM® watsonx.data.

For more information about SQL statements supported by Netezza connector, see [Supported SQL statements](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-supported_sql_statements).



  Connectivity 

To connect Presto to your Netezza database, you need the following database details and connect credentials.



  Database details 

To connect Presto to your Netezza database, you need the following database details.



Table 1. Database details

 Database detail  Description                                             

 Host name        The hostname of the server.                             
 Port number      Used by the database manager for TCP/IP communication.  
 Database name    The name of the Netezza database                        







  Connection credentials 

To connect Presto to your Netezza database, you need the following connection credentials.



Table 12. Connection credentials

 Credential                    Description                              

 Netezza database credentials  Use your database user ID and password.  









  Configuring Netezza connector 



1.  Create a netezza.properties file inside the etc/catalog folder of Presto.
2.  Add the following properties to the netezza.properties file.

connector.name=netezza

connection-url=jdbc:netezza://<hostname>:<port>/<database name>

connection-user=<user name>

connection-password=<password>

allow-drop-table=<true/false>



If you want to create multiple catalogs for Netezza, name the properties files with respective catalog names.





  Limitations 

Netezza connector partially supports CREATE VIEW statement. The Presto Supported SQL syntax does not include creating views with custom column names (different than the table column names).
Migrating a Kafka client to version 0.10.2.X or later versions 

From version 0.10.2, you can configure SASL authentication directly in the client's properties instead of using a JAAS file. This simplification lets you to run multiple clients in the same JVM by using different sets of credentials, which is not possible with a JAAS file.

Complete the following steps:



1. Delete the JAAS file. The JVM property java.security.auth.login.config=<PATH TO JAAS> is also no longer required.
2. Add the following to the client's properties:

sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="USERNAME" password="PASSWORD";

USERNAME and PASSWORD are the values from your Event Streams Service Credentials tab in IBM Cloud.
It is recommended to use IAM token for stress workload.



 Getting IBM API Key 



1. Log in to the [IBM Cloud console](http://test.cloud.ibm.com/).
2. In the navigation bar, click Manage and select Access (IAM).
3. Click API keys in the left navigation bar.
4. Click Create +.
5. In the Create IBM Cloud API key window, enter a name for your API key and enter the appropriate description for the key. For example, ibmlhtoken testing
6. Click Create. A window with message API key successfully created displays.
7. Click Download to save the API key to your local machine.
8. Open the downloaded file and copy the API key to a notepad file.







 Getting IBM Access Management (IAM) token 



1. Call the REST endpoint in IAM to get the IAM token.
2. Replace <your-api-key> with your IBM API key.

curl -X POST 'https://iam.cloud.ibm.com/identity/token' --header "Content-Type: application/x-www-form-urlencoded" --data-urlencode "grant_type=urn:ibm:params:oauth:grant-type:apikey" --data-urlencode "apikey=<your-api-key>"











 Connecting to Presto engine using Presto CLI (Remote) 



1. Download the Presto executable jar from [https://prestodb.io/docs/current/installation/cli.html](https://prestodb.io/docs/current/installation/cli.html)
2. Rename the downloaded file as presto. Make it executable with chmod +x and run it.
3. To check whether Presto CLI is installed, run ./presto --version. Presto cli version displays. For example, Presto CLI 0.281-cfbc6eb
4.
[Connected Client](https://cloud.ibm.com/docs-content/v1/content/690bf9a59650e0582967be8899e39840589532f7/SecureGateway/images/connectedClient.png?raw=true)

If you would like to configure the Access Control List for your client now, [click here for more details](https://cloud.ibm.com/docs/services/SecureGateway?topic=SecureGateway-acl).

Now that you have your client connected, you can [create your first destination](https://cloud.ibm.com/docs/services/SecureGateway?topic=SecureGateway-add-dest).
For more information, see [Governance, risk, and compliance](https://www.ibm.com/support/knowledgecenter/SSFC4F_2.1.0/mcm/compliance/compliance_intro.html).



For more information, see [Enabling operators after IBM Cloud Pak for Multicloud Management installation](https://www.ibm.com/support/knowledgecenter/SSFC4F_2.1.0/install/enable_operator.html).
* You must configure and connect an LDAP directory with IBM Cloud Pak for Multicloud Management. You must have an LDAP group in your configuration for IBM Cloud Pak for Multicloud Management with users defined who will access Infrastructure management. For more information, see the [IAM Guide](https://www.ibm.com/support/knowledgecenter/SSFC4F_2.1.0/iam/3.x.x/iam_guide.html).
* cloudctl must be installed and authenticated to your cluster. You must install the IBM Cloud Pak CLI, cloudctl. For more information, see [Installing the IBM Cloud Pak CLI](https://www.ibm.com/support/knowledgecenter/SSFC4F_2.1.0/cloudctl/install_cli.html).

Note: Download the installation file for CLI tools from the console.
* oc must be installed and authenticated to your cluster. If OpenShift Container Platform CLI tools are not installed on the system, you need to download, decompress, and install the OpenShift Container Platform CLI tools oc from [OpenShift Container Platform client binary files](https://mirror.openshift.com/pub/openshift-v4/clients/).







 Step 1. Generate client ID and client secret 

Create unique strings to use for client ID and client secret. You can generate base64-encoded 32 character strings. Examples:

echo There is a huge white elephant in LA zoo | base64 > client_id.txt
If you use a Kafka client at 0.10.2.1 or later, you can use the sasl.jaas.config property for client configuration instead of a JAAS file. To connect to Event Streams, set sasl.jaas.config as in the following example:

sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required 
username="USERNAME" 
password="PASSWORD";</code>

USERNAME and PASSWORD are the values from your Event Streams Service Credentials tab in IBM Cloud.

If you use sasl.jaas.config, clients that run in the same JVM can use different credentials. For more information, see [Configuring Kafka clients](http://kafka.apache.org/documentation/security_sasl_plain_clientconfig).

For an earlier Kafka client, you must use a JAAS configuration file to specify the credentials. This mechanism is less convenient, therefore use the sasl.jaas.config property instead.





 Migrating a Kafka client from version 0.9.X or 0.10.X to later client versions 

If you use the Java clients, you can use the publicly available Kafka clients version 0.10 or later.

You are encouraged to move from version 0.9.X to the latest version. You can download a Kafka client from [https://kafka.apache.org/downloads](https://kafka.apache.org/downloads).



 Migrating a Kafka client to version 0.10.2.X or later versions 

From version 0.10.2, you can configure SASL authentication directly in the client's properties instead of using a JAAS file. This simplification lets you to run multiple clients in the same JVM by using different sets of credentials, which is not possible with a JAAS file.

Complete the following steps:



1. Delete the JAAS file.
.withEndpointConfiguration(new EndpointConfiguration(endpoint_url, location)).withPathStyleAccessEnabled(true)
.withClientConfiguration(clientConfig).build();
return cosClient;
}

}

Show more







 Next Steps 

Expiration is just one of many lifecycle concepts available for IBM Cloud Object Storage. Each of the concepts we've covered in this overview can be explored further at the [IBM Cloud Platform](https://cloud.ibm.com/).
For more information about creating and managing API keys, see: [Managing user API keys](https://cloud.ibm.com/docs/account?topic=account-userapikey).





 IBMid/password 

The IBMid/password can be used to log in to the console and can also be used within the application in the same ways that a user ID/password is allowed. The exception occurs when the IBMid is federated because a redirection to your organization’s login page cannot be done. However, the recommended method for an application to establish a connection to the database service instance is to use an access token.







 Supported interfaces 

The following database client interfaces are supported:



* [ODBC](https://cloud.ibm.com/docs/Db2whc?topic=Db2whc-iamodbc-clpplus)
* [CLP](https://cloud.ibm.com/docs/Db2whc?topic=Db2whc-iamodbc-clpplus)
* [CLPPLUS](https://cloud.ibm.com/docs/Db2whc?topic=Db2whc-iamodbc-clpplus)
* [JDBC](https://cloud.ibm.com/docs/Db2whc?topic=Db2whc-iamjdbc)





 ODBC, CLP, and CLPPLUS 

For an ODBC application or a command-line client (CLP, CLPPLUS) to connect to a Db2 server by using IAM authentication, a data source name (DSN) needs to be configured first in a db2dsdriver.cfg configuration file by running the following command:

db2cli writecfg add -dsn <dsn_alias> -database <database_name> -host <host_name_or_IP_address> -port 50001 -parameter "Authentication=GSSPLUGIN;SecurityTransportMode=SSL"

The db2dsdriver.cfg configuration file is an XML file, typically located in the sqllib/cfg directory, that contains a list of DSN aliases and their properties.
For more information about service IDs, see: [Introducing IBM Cloud IAM Service IDs and API Keys](https://www.ibm.com/blogs/bluemix/2017/10/introducing-ibm-cloud-iam-service-ids-api-keys/) and [Increase Information Security for Db2 on IBM Cloud](https://www.ibm.com/cloud/blog/increase-information-security-for-db2-on-ibm-cloud).







 Roles and actions 

Every user that accesses the Db2 Warehouse on Cloud service in your account must be assigned an access policy with an IAM role. The access policy that you assign to users in your account determines what actions a user can perform within the context of the service or specific instance that you select. The allowable actions are customized and defined by Db2 Warehouse on Cloud as operations that are allowed to be performed on the service. Each action is mapped to an IAM platform or service role that you can assign to a user. If a specific role and its actions don't fit the use case that you're looking to address, you can [create a custom role](https://cloud.ibm.com/docs/account?topic=account-custom-rolescustom-access-roles) and pick the actions to include.

For information about the exact actions mapped to each role, see [IAM roles and actions](https://cloud.ibm.com/docs/account?topic=account-iam-service-roles-actions) and [Db2 Warehouse](https://cloud.ibm.com/docs/account?topic=account-iam-service-roles-actionsdb2-warehouse).





 Prerequisites 



* Db2 Client V11.1 FP3 or later.
* Must use SSL connections through port 50001.
* [Configure your Db2 client](https://cloud.ibm.com/docs/Db2whc?topic=Db2whc-ssl_supportssl_cfg_client).
* Catalog your database.
You also need CLI access to your cluster. For more information, see [Setting up the CLI and API](https://cloud.ibm.com/docs/containers?topic=containers-cs_cli_install).
* A recent version of kubectl
* [Git](https://git-scm.com/downloads)







 Step 2. Clone the kafka-connect repositories 

Clone the following two repositories that contain the required files:



* [https://github.com/ibm-messaging/kafka-connect-mq-source](https://github.com/ibm-messaging/kafka-connect-mq-source)
* [https://github.com/ibm-messaging/event-streams-samples](https://github.com/ibm-messaging/event-streams-samples)







 Step 3. Create your Kafka Connect configuration 



1. You must set up this configuration only once. Event Streams stores it for future use.

From the event-streams-samples project, navigate to the kafka-connect/IKS directory, edit the connect-distributed.properties file, and replace <BOOTSTRAP_SERVERS>; in one place and <APIKEY> in three places with your Event Streams credentials.

Provide <BOOTSTRAP_SERVERS> as a comma-separated list. If they are not valid, you get an error.

Your <APIKEY> appears in clear text on your machine but is secret when pushed to Kubernetes Service.

Kafka Connect can run multiple workers for reliability and scalability reasons. If your Kubernetes Service cluster has more than one node and you want multiple Connect workers, edit the kafka-connect.yaml file, and edit the entry replicas: 1.
2. Then, run the following commands.

To create a secret, run the following command:

kubectl create secret generic connect-distributed-config --from-file=connect-distributed.properties

To create a configmap, run the following command:
* Consult Appendix 2 at the bottom of this tutorial if you do not have this file





3. IBM MQ Client : To complete this tutorial you will require the IBM MQ command line tool 'runmqsc' installed and on your PATH. If you do not have these commands, you can get them by installing the IBM MQ Client. Appendix 3 at the end of this tutorial details how to do this.
4. IBM MQ MFT Component : To install the IBM MQ MFT component, refer to [Installing Managed File Transfer](https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_9.0.0/com.ibm.mq.ins.doc/q129810_.htm) in IBM Knowledge Center.



--------------------





 Setting up your terminal environment 



1. Open a terminal window



* This terminal will be the admin terminal, used to configure your coordination queue manager.



2. Specify the queue manager details in an MQSERVER environment variable in the admin terminal by completing the following steps:



* The runmqsc cli requires an environment variable be set to identify the remote queue manager that it should connect to. In the admin terminal, export MQSERVER variable with the following command:
* For Windows : SET MQSERVER=CLOUD.ADMIN.SVRCONN/TCP/<HOSTNAME>(<PORT>)
* For Linux : export MQSERVER="CLOUD.ADMIN.SVRCONN/TCP/<HOSTNAME>(<PORT>)"



* <HOSTNAME> - this is 'hostname' in the file connection_info.txt
* <PORT> - this is 'listenerPort' in the file connection_info.txt







This step is required on any new terminal that will be used for remote administration of a queue manager, such as when using runmqsc cli

--------------------
[HSM server](https://cloud.ibm.com/docs-content/v1/content/c5288bed34c3820e3a5251d820ee91adcd2591a3/blockchain/images/icon-hsm-2.png) SSH into the HSM as the admin the HSM server and register the client by running one of the following commands.

If the <CLIENT_ADDRESS> is the IP address of the client:

client register -client ${CLIENT_NAME} -ip <CLIENT_ADDRESS>

If the <CLIENT_ADDRESS> is the fully qualified host name of the client:

client register -client ${CLIENT_NAME} -hostname <CLIENT_ADDRESS>

Replace



* {CLIENT_NAME} with the name of the client. This value can be anything meaningful to you.
* <CLIENT_ADDRESS> with either the IP address or fully qualified host name of the client.



2. ![HSM server](https://cloud.ibm.com/docs-content/v1/content/c5288bed34c3820e3a5251d820ee91adcd2591a3/blockchain/images/icon-hsm-2.png) Because network address translation (NAT) exists between the client and the HSM, we need to disable client source IP address validation by the Network Trust Link Server (NTLS) upon Network Trust Link Agent (NTLA) client connection. Disable ip check on the HSM server and then restart the NTLS service on the HSM server by running the following commands:

ntls ipcheck disable
service restart ntls
3. ![HSM server](https://cloud.ibm.com/docs-content/v1/content/c5288bed34c3820e3a5251d820ee91adcd2591a3/blockchain/images/icon-hsm-2.png) Assign a partition to the newly created client on the HSM server by running the following command:
Db2 connector for Presto 

Db2 connector for Presto is a customized connector for IBM® watsonx.data.

For more information about SQL statements supported by Db2 connector, see [Supported SQL statements](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-supported_sql_statements).



 Connectivity 

To connect Presto to your Db2 database, you need the following database details and connect credentials.



 Database details 

To connect Presto to your Db2 database, you need the following database details.



Table 1. Database details

 Database detail Description 

 Host name The hostname of the server. 
 Port number Used by the database manager for TCP/IP communication. 
 Database name The name of the Db2 database. 







 Connection credentials 

To connect Presto to your Db2 database, you need the following connection credentials.



Table 12. Connection credentials

 Credential Description 

 IBM ID IBM ID is the ID that you used to log in to IBM Cloud. NOTE: Do not use the ID that you use to connect applications or tools to your Db2 database. 
 Db2 database credentials Use your database user ID and password. 
 Administrator-created users Db2 Warehouse and some Db2 managed service plans allow administrative users to create new users. These administrator-created user IDs and passwords can be used to log in to the web console URL and connect to the Db2 database from applications or tools. 



If you are not the owner or administrator of your Db2 instance, you can get your database details and connect credentials from your administrator.







 Configuring Db2 connector 



1. Create a db2.properties file inside the etc/catalog folder of Presto.
2. Add the following properties to the db2.properties file.

connector.name=db2

connection-url=jdbc:db2://<hostname>:<port>/<database name>

connection-user=<user name>

connection-password=<password>

allow-drop-table=<true/false>



If you want to create multiple catalogs for Db2, name the properties files with respective catalog names.





 Limitations
Note: The YOUR_CLIENT_ID and YOUR_CLIENT_SECRET values are generated when you register Infrastructure Management as an OIDC client.





 Using the CLI to create the connection: 



1. Create a secret yaml file named imconnectionsecret.yaml with the following content.

apiVersion: v1
kind: Secret
metadata:
  name: imconnectionsecret
  namespace: management-infrastructure-management
type: Opaque
stringData:
  oauth_username: <IM_ADMIN_LDAP_USER>
  oauth_password: <IM_ADMIN_LDAP_PASS>
  cpconsole: <YOUR_CP4MCM_HOST>
  clientid: <YOUR_CLIENT_ID>
  clientsecret: <YOUR_CLIENT_SECRET>

Example secret (for reference only):

apiVersion: v1
kind: Secret
metadata:
  name: imconnectionsecret
  namespace: management-infrastructure-management
type: Opaque
stringData:
  oauth_username: im_admin_user
  oauth_password: myT9XjFjkA2yeL9pzvacaUCj1lN3cyaH
  cpconsole: cp-console.apps.test.ibm.com
  clientid: N3NzNVFsSjlLVkl6Zk5hZ01MRzJVaVdnbFcxNGl5cnQK
  clientsecret: OS40Ni42OC4xMTljbGllbnRfc2VjcmV0Cg==
2. Run the oc command to create the secret.

oc create -n management-infrastructure-management -f imconnectionsecret.yaml
3. Create a yaml file that is named imconnection.yaml based on the following to create the connection CR.

apiVersion: infra.management.ibm.com/v1alpha1
kind: Connection
metadata:
  labels:
    app.kubernetes.io/instance: connection-1
* [HCX on-premises Service Mesh](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-hcxclient-vcs-mesh-deployment)
* [VMware Hybrid Cloud migrations](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-hcxclient-migrations)
* [Monitoring parameters and components](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-hcxclient-monitoring)
* [HCX troubleshooting](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-hcxclient-troubleshooting)