vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Executor logs can become very large, so balance out the options to optimize the amount of logs that get forwarded to your logging instance versus the information you get in logs for troubleshooting purposes.<br> * Follow the best practices of IBM Log Analysis when choosing the right configuration and searching techniques. For example, you might want to configure the IBM Log Analysis instance plan for a 7 day search with the archival of logs to IBM Cloud Object Storage to save on costs. Also refer to the IBM Log Analysis documentation for techniques on searching for logs of your interest based on keywords, point in time, and so on.<br><br><br> <br><br> * [Configuring and viewing logs](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-viewing-logs)<br><br><br> 
 Customize your service instance <br><br> * You might need to customize your service instance to bring in Python or conda packages that are not preinstalled, or bring in some files(certificates or config files) that are to be made available to Spark applications. Based on your needs, customize your instance using library sets and use these library sets when submitting applications.<br> * The size of your library set has a bearing on the application startup time and the executor startup time (when you auto-scale applications). Also note there is an upper limit for the size of a library set, namely 2 GB. So if different applications need different library sets, it is better for you to use separate library sets, so that they can be specified individually at the time the application is submitted.<br> * Use customization only to bring in files that cannot be brought in by the application details parameters. See [Parameters for submitting Spark applications](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-spark-app-rest-apispark-submit-parms). You must use the standard spark-submit equivalent parameter options such as the files, jars, packages and pyFiles options if that fits your use case.
* In the Main tab:



* Fill in Name
* Target Type of File
* Log format of Text
* Fill in File Name to define the output location, for example: logtemp:///sgclient.log
* Select Archive Mode to Rotate



* In the Event Subscription Tab, select the Add button to add following target event subscription



* Fill in the Event Category selecting sgclient and the Minimum Event Priority of debug
* Fill in the Event Category selecting mgmt and the Minimum Event Priority of debug









 Which ports does the Secure Gateway client use? 

The Secure Gateway Client uses outbound port 443 and port 9000 to connect to npm registry and the IBM Cloud environment. See [Network requirements](https://cloud.ibm.com/docs/SecureGateway?topic=SecureGateway-client-requirementsnetwork-requirements) for details.





 Does the Secure Gateway server support High Availability? 

The server does not support High Availability (HA) nor provide redundancy. To avoid interruption during planned maintenance or outages, you could use the gateway server in another region. You can achieve HA with multiple client connections to the same gateway ID as allowed by your [Secure Gateway Service Plan](https://cloud.ibm.com/docs/SecureGateway?topic=SecureGateway-secure-gateway-service-plans). For additional information, see [High Availability](https://cloud.ibm.com/docs/SecureGateway?topic=SecureGateway-about-sgabout-ha).





 What should I do when the security token is expired? 

The security token could be regenerated in the edit board of the gateway panel, you can click the link Regenerate Token which next to Token Expiration to regenerate the token. The Save button cannot be used to regenerate the token. For additional information, see [Regenerate security token](https://cloud.ibm.com/docs/SecureGateway?topic=SecureGateway-add-sg-gwregen-sectoken)





 Why connection is getting error after HTTP(S) redirect?
Why does my Windows Backup Agent appear offline? 

The Windows® backup agent might appear offline in the Portal for multiple reasons. Follow the steps to rule out issues with firewall port settings, and the buagent process on the server.

 What’s happening 

Windows® agent shows as offline in Portal.



 Possible firewall issue 

 Why it’s happening 

Currently, at minimum, your servers must allow access to the 10.200.86.0/24 and 10.2.118.0/24 subnets for TCP ports 8086, 8087. If the port settings are incorrect, traffic is blocked, and the agent cannot communicate with the Portal.

 How to fix it 

Check firewall ports - 8086 & 8087. For more information, see [Configuring Ports to allow communication between the backup agent and Cloud Backup Portal](https://cloud.ibm.com/docs/Backup?topic=Backup-portinfo).



1. Establish Remote Desktop connection to the offline Agent server.
2. Then, run one of the following commands to check the ports.



* Open CMD and run telnet to validate the connection. If TELNET is not yet installed, you can add the Telnet Client through the "Add Roles and Features".



telnet cloudbackupregister.service.softlayer.com 8086

If the port is open, CMD shows a blank screen. If the port is not open, CMD returns the following message.

Connecting to cloudbackupregister.service.softlayer.com… Could not open connection to the host, on port 8086: Connect failed

If so, you must contact your network team to update the firewall.



* If you don't want to install Telnet, you can use test-netconnection in Powershell.



1. Open Powershell as Administrator.
2. Enter the following command.



test-netconnection cloudbackupregister.service.softlayer.com -Port 8086

If the port is open, the last line shows the following.

TcpTestSucceeded : True
Why can't I reuse a host in a different Satellite location? 

  What’s happening 

When you try to reuse an existing host by assigning it to a new location, you see a message similar to the following error.

Unable to assign host. Could not find a host to assign to your Satellite cluster.

  Why it’s happening 

If the host is not removed from the cluster before it is removed from the original location, the cluster master retains the information about the host. When the host is later added to a new location, a duplicate hostname is recognized and the assignment fails.

  How to fix it 

If the cluster master still recognizes the host, you must rename and reload the host before you can attach it to a new location.



1.  Follow your host infrastructure provider's instructions to rename the host.

For hosts provisioned through IBM Cloud:



1.  Log in to the IBM Cloud console and navigate to Classic infrastructure>Device list.
2.  In the Devices table, locate the host you want to rename. Click the Action menu and select Rename.
3.  Rename the host.



2.  Follow your host infrastructure provider's instructions to reload the host.

For hosts provisioned through IBM Cloud:



*  In the IBM Cloud console:



1.  Navigate to Classic infrastructure>Device list.
2.  In the Devices table, click on the host you want to reload.
3.  From the Actions drop down menu, select OS Reload.
4.  Click Reload. Note that it may take several minutes for the reload process to complete.



*  In the CLI:



1.  List your hosts and note the ID of the host you want to reload.

ibmcloud sl vs list
2.  Run the ibmcloud sl vs reload command. Note that it may take several minutes for the reload process to complete.

ibmcloud sl vs reload <host_id>
3.  Check if the host is ready.

ibmcloud sl vs ready <host_id>





3.  [Attach the host to the location](https://cloud.ibm.com/docs/satellite?topic=satellite-attach-hosts).
Registration.addIndexFactory(STIndexFactory)
Registration.addMetadataFilterFactory(STMetaDataFilterFactory)
Registration.addClauseTranslator(STParquetMetadatastoreClauseTranslator)
Registration.addMetaDataTranslator(STParquetMetaDataTranslator)
* For Python:

from xskipper import Xskipper
from xskipper import Registration

Registration.addMetadataFilterFactory(spark, 'com.ibm.xskipper.stmetaindex.filter.STMetaDataFilterFactory')
Registration.addIndexFactory(spark, 'com.ibm.xskipper.stmetaindex.index.STIndexFactory')
Registration.addMetaDataTranslator(spark, 'com.ibm.xskipper.stmetaindex.translation.parquet.STParquetMetaDataTranslator')
Registration.addClauseTranslator(spark, 'com.ibm.xskipper.stmetaindex.translation.parquet.STParquetMetadatastoreClauseTranslator')







 Index building 

To build an index, you can use the addCustomIndex API:



* For Scala:

import com.ibm.xskipper.stmetaindex.implicits._

// index the dataset
val xskipper = new Xskipper(spark, dataset_path)

xskipper
.indexBuilder()
// using the implicit method defined in the plugin implicits
.addSTBoundingBoxLocationIndex("location")
// equivalent
//.addCustomIndex(STBoundingBoxLocationIndex("location"))
.build(reader).show(false)
* For Python:

xskipper = Xskipper(spark, dataset_path)

 adding the index using the custom index API
xskipper.indexBuilder()
Working with firewalls and gateway appliances 

Ensure that a firewall or gateway appliance (such as a [Juniper vSRX](https://cloud.ibm.com/docs/vsrx?topic=vsrx-getting-startedgetting-started)) is not blocking your traffic.

For instance, the Juniper vSRX is a firewall appliance that sits between IBM Cloud classic infrastructure and the BCR. If you perform a trace route and cannot get to any address at all, then it could be that the vSRX that is blocking the traffic.
ibmcloud analytics-engine-v3 instance current-resource-consumption --id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --output json





 Example output 

The output is as follows:

{
"cores" : "2",
"memory" : "8G"
}







 ibmcloud analytics-engine-v3 instance resource-consumption-limits 

Returns the maximum total memory and virtual processor cores allocated across all the applications running in the service instance at any point in time.

ibmcloud analytics-engine-v3 instance resource-consumption-limits [--id ID]



 Command options 

-i, --id (string)
: ID of the Analytics Engine instance. When specified, it overrides the target instance ID if it was set. Required, if the target instance was not set.

The value must match the regular expression /u0008[0-9a-f]{8}u0008-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-u0008[0-9a-f]{12}u0008/.





 Examples 

ibmcloud analytics-engine-v3 instance resource-consumption-limits --output json

ibmcloud analytics-engine-v3 instance resource-consumption-limits --id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --output json





 Example output 

The output is as follows:

{
"max_cores" : "150",
"max_memory" : "600G"
}









 Spark application commands 

With the Spark application CLI commands, you can perform the following operations:
Tagging IBM Cloud resources and managing access with tags 

All IBM Cloud resources in your account should be tagged with security attributes that you define. Apply user tags to organize your resources and easily find them later, to help you with identifying specific team usage or cost allocation, and to control access to your resources without requiring updates to your IAM policies.

For more information, see:



* [Working with tags](https://cloud.ibm.com/docs/account?topic=account-tag) (through the Global Search and Tagging platform service)
* [Controlling access to resources by using tags](https://cloud.ibm.com/docs/account?topic=account-access-tags-tutorial)





 Example tagging scheme 

The IBM Cloud Framework for Financial Services doesn't require a specific set of tags that must be used, but they should be meaningful for your application. For illustration, you might use a scheme such as the one outlined.

In this example, it is suggested that you use one the following impact levels from [Federal Information Processing Standards Publication 199](https://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.199.pdf):



* fips199-high: The loss of confidentiality, integrity, or availability could be expected to have a severe or catastrophic adverse effect on organizational operations, organizational assets, or individuals.
* fips199-moderate: The loss of confidentiality, integrity, or availability could be expected to have a serious adverse effect on organizational operations, organizational assets, or individuals.
* fips199-low: The loss of confidentiality, integrity, or availability could be expected to have a limited adverse effect on organizational operations, organizational assets, or individuals.



In addition, for services that store data, it is suggested that you use one or more of these tags corresponding to different kinds of data:



* audit: Audit logs or archives
* public: Publicly accessible, such as Cloud Object Storage buckets that hold public documentation or samples
* consumer-owned-data: Data owned by the consumer
* consumer-metadata: Metadata owned by the consumer
* regulated-data: Data that is regulated







 Related controls in IBM Cloud Framework for Financial Services
Table 10. Lists of metastore user events

 Action Description 

 lakehouse.metastoreusers.create An event is generated when you create a metastore user. 
 lakehouse.metastoreusers.update An event is generated when you update a metastore user. 
 lakehouse.metastoreusers.delete An event is generated when you delete a metastore user. 







 Events for data policy 



Table 11. Lists of data policy events

 Action Description 

 lakehouse.datapolicy.create An event is generated when you create a data policy. 
 lakehouse.datapolicy.update An event is generated when you update a data policy. 
 lakehouse.datapolicy.delete An event is generated when you delete a data policy.
lakehouse.authtoken.create An event is generated when a user logs in. 
 lakehouse.authtoken.delete An event is generated when a user logs out. 







 Events for engine users 



Table 6. Lists of engine user events

 Action Description 

 lakehouse.engineusers.create An event is generated when you create an engine user. 
 lakehouse.engineusers.update An event is generated when you update an engine user. 
 lakehouse.engineusers.delete An event is generated when you delete an engine user. 







 Events for bucket users 



Table 7. Lists of bucket user events

 Action Description 

 lakehouse.bucketusers.create An event is generated when you create a bucket user. 
 lakehouse.bucketusers.update An event is generated when you update a bucket user. 
 lakehouse.bucketusers.delete An event is generated when you delete a bucket user. 







 Events for catalog users 



Table 8. Lists of catalog user events

 Action Description 

 lakehouse.catalogusers.create An event is generated when you create a catalog user. 
 lakehouse.catalogusers.update An event is generated when you update a catalog user. 
 lakehouse.catalogusers.delete An event is generated when you delete a catalog user. 







 Events for database connection users 



Table 9. Lists of database connection user events

 Action Description 

 lakehouse.dbconnusers.create An event is generated when you create a database connection user. 
 lakehouse.dbconnusers.update An event is generated when you update a database connection user. 
 lakehouse.dbconnusers.delete An event is generated when you delete a database connection user. 







 Events for metastore users 



Table 10. Lists of metastore user events

 Action Description 

 lakehouse.metastoreusers.create An event is generated when you create a metastore user. 
 lakehouse.metastoreusers.update An event is generated when you update a metastore user. 
 lakehouse.metastoreusers.delete An event is generated when you delete a metastore user. 







 Events for data policy 



Table 11.
Security and compliance 



  General Data Protection Regulation (GDPR) 

If you have an account with IBM Cloud®, your personal data is held by IBM Cloud. The [IBM Data Processing Addendum (IBM DPA)](https://www.ibm.com/support/customer/csol/terms/?cat=dpa) applies to the processing of clients' personal data by IBM on behalf of the client to provide IBM standard services.

IBM Cloud® Data Engine processes limited client Personal Information (PI) in the course of running the service and optimizing the user experience.

IBM Cloud Data Engine provides a [Data Sheet Addendum (DSA)](https://www.ibm.com/software/reports/compatibility/clarity-reports/report/html/softwareReqsForProduct?deliverableId=AC17FFB0B52911E7A9EB066095601ABB) with its policies as a data processor regarding content and data protection.





  HIPAA 

IBM Cloud Data Engine with IBM® Key Protect for managing encryption keys meets the required IBM controls that are commensurate with the Health Insurance Portability and Accountability Act of 1996 (HIPAA) Security and Privacy Rule requirements. These requirements include the appropriate administrative, physical, and technical safeguards required of Business Associates in 45 CFR Part 160 and Subparts A and C of Part 164. HIPAA must be requested at the time of provisioning and requires a representative to sign a Business Associate Addendum (BAA) agreement with IBM.

Note: If you are a HIPAA user, you must not use catalog management (Hive Metastore).





  ISO 27001/27017/27018 and ISO 27701 

IBM Cloud Data Engine is ISO 27001, 27017, 27018, and ISO 27701 certified. The certificates and the certified cloud products listing can be found in the [IBM Trust Center](https://www.ibm.com/trust) in section [IBM Cloud Compliance Programs - Global](https://www.ibm.com/cloud/complianceGlobal%20programs).
Lakehouse Storage Capacity com.ibm.cloud.lakehouse.storage_capacity storage_capacity_error An event is generated when the storage capacity reaches error levels High 
 Lakehouse Presto Group com.ibm.cloud.lakehouse.presto_group presto_group_unavailable An event is generated when the Presto group is unavailable due to an unexpected outage High 
 Lakehouse Presto Group com.ibm.cloud.lakehouse.presto_group presto_group_available An event is generated when the Presto group is available Low 
 Lakehouse Metastore com.ibm.cloud.lakehouse.metastore metastore_unavailable An event is generated when the metastore is unavailable due to an unexpected outage High 
 Lakehouse Metastore com.ibm.cloud.lakehouse.metastore metastore_available An event is generated when the metastore is available Low 







 Enabling notifications 

Events that are generated by an instance of the watsonx.data service can be forwarded to an Event Notifications service instance that is available in the same account. You can configure only one watsonx.data instance to one Event Notifications service instance.





 Delivering notifications to select destinations 

After you enable notifications for watsonx.data, create topics and subscriptions in Event Notifications so that alerts can be forwarded and delivered to your selected destinations.

For a complete list of supported destinations, see the [Event Notifications documentation](https://cloud.ibm.com/docs/event-notifications?topic=event-notifications-en-destination).



 Email notifications 

You can use the [IBM Cloud email service](https://cloud.ibm.com/docs/event-notifications?topic=event-notifications-en-destinations-email) as a delivery channel for watsonx.data event notifications. [Create an Event Notifications subscription](https://cloud.ibm.com/docs/event-notifications?topic=event-notifications-en-create-en-subscription) between an existing topic and the IBM Cloud email service to forward your alerts to various recipients by email.
Auditing events for watsonx.data 

As a security officer, auditor, or manager, you can use the IBM Cloud® Activity Tracker service to track how users and applications interact with the IBM® watsonx.data service in IBM Cloud.

IBM Cloud Activity Tracker records user-initiated activities that change the state of a service in IBM Cloud. You can use this service to investigate abnormal activity and critical actions and to comply with regulatory audit requirements. In addition, you can be alerted about actions as they happen. The events that are collected comply with the Cloud Auditing Data Federation (CADF) standard. For more information, see the [getting started tutorial for IBM Cloud Activity Tracker](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-getting-started).



 List of platform events 

The following table lists the actions that generate an event:



Table 1. Actions that generate platform events

 Action Description 

 lakehouse.lakehouse.create An event is generated when you provision a service instance. 
 lakehouse.presto.create An event is generated when you provision a Presto group. 
 lakehouse.hms.create An event is generated when you provision a metastore. 
 lakehouse.lakehouse.delete An event is generated when a service instance is deleted. 
 lakehouse.presto.delete An event is generated when a when a Presto group is deleted. 
 lakehouse.hms.delete An event is generated when a metastore is deleted. 







 Events for engines 



Table 2. Lists of engine events

 Action Description 

 lakehouse.engines.create An event is generated when you create an engine. 
 lakehouse.engines.get An event is generated when you get engines. 
 lakehouse.engines.update An event is generated when you update an engine. 
 lakehouse.engines.delete An event is generated when you delete an engine. 
 lakehouse.engines.pause An event is generated when you pause an engine. 
 lakehouse.engines.resume An event is generated when you resume an engine. 







 Events for buckets 



Table 3. Lists of bucket events

 Action Description 

 lakehouse.buckets.create An event is generated when you create a bucket.
x-amz-object-lock-mode string Valid value is COMPLIANCE - required if x-amz-object-lock-retain-until-date is present. 
 x-amz-object-lock-retain-until-date ISO8601 Date and Time Required if x-amz-object-lock-mode is present. 
 x-amz-object-lock-legal-hold string Valid values are ON or OFF. 



Example request

PUT /apiary/queen-bee HTTP/1.1
Authorization: Bearer {token}
Content-Type: text/plain; charset=utf-8
Host: s3.us.cloud-object-storage.appdomain.cloud

Content-Length: 533

The 'queen' bee is developed from larvae selected by worker bees and fed a
substance referred to as 'royal jelly' to accelerate sexual maturity. After a
short while the 'queen' is the mother of nearly every bee in the hive, and
the colony will fight fiercely to protect her.

Example request

PUT /apiary/queen-bee HTTP/1.1
Authorization: 'AWS4-HMAC-SHA256 Credential={access-key}/{date}/{region}/s3/aws4_request,SignedHeaders=host;x-amz-date;,Signature={signature}'
x-amz-date: {timestamp}
x-amz-content-sha256: {payload_hash}
Content-Type: text/plain; charset=utf-8
Host: s3.us.cloud-object-storage.appdomain.cloud

Content-Length: 533

The 'queen' bee is developed from larvae selected by worker bees and fed a
substance referred to as 'royal jelly' to accelerate sexual maturity. After a
short while the 'queen' is the mother of nearly every bee in the hive, and
the colony will fight fiercely to protect her.

Example response

HTTP/1.1 200 OK
Date: Thu, 25 Aug 2016 18:30:02 GMT
Why doesn't the secondary vCenter Server display in the vSphere Web Client inventory? 

  What’s happening 

In a multisite configuration, after you add a secondary instance, its VMware vCenter Server® system is not visible when logged in to the VMware vSphere® Web Client of a previously ordered instance.

  Why it’s happening 

This is a known VMware® issue.

  How to fix it 

To resolve the problem, you must restart the vSphere Web Client:



1.  Using the root account, connect over ssh to the vCenter VM (virtual machine) of the previously ordered instance.
2.  Type shell to enter the bash shell.
3.  Enter service-control --stop vsphere-client to stop the client.
4.  Enter service-control --start vsphere-client to restart the client.
5.  After the vSphere Web Client of the previously ordered instance is restarted, confirm that the vCenter Server system for the newly added secondary instance is visible in the vSphere Web Client.