vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Auditing events for IBM Analytics Engine serverless instances 

As a security officer, auditor, or manager, you can use the Activity Tracker service to track how users and applications interact with IBM Analytics Engine in IBM Cloud®.

IBM Cloud Activity Tracker records user-initiated activities that change the state of a service in IBM Cloud. You can use this service to investigate abnormal activity and critical actions and to comply with regulatory audit requirements. In addition, you can be alerted about actions as they happen. The events that are collected comply with the Cloud Auditing Data Federation (CADF) standard. For more information, see the documentation about the [IBM Cloud Activity Tracker](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-getting-started).

The following table lists the actions that generate an event:



 List of management events 



Table 1. Actions that generate management events

 Action Description 

 ibmanalyticsengine.instance.read Retrieve the details of an instance 
 ibmanalyticsengine.instance.update Update the configuration of the instance after instance creation 
 ibmanalyticsengine.instance-logging.create Create the logging configuration for an instance 
 ibmanalyticsengine.instance-logging.update Update the logging configuration of an instance 
 ibmanalyticsengine.instance-logging.read Retrieve the logging configuration details of an instance 
 ibmanalyticsengine.instance-logging.delete Delete the logging configuration of an instance 
 ibmanalyticsengine.application.create Deploy a Spark application 
 ibmanalyticsengine.application.read Retrieve the details of the submitted Spark application including the state and configuration settings 
 ibmanalyticsengine.application.delete Delete a Spark application request or a running Spark application 
 ibmanalyticsengine.livybatch.create Create a Spark application through the Livy batch interface 
 ibmanalyticsengine.livybatch.read Retrieve the details of a Livy batch application, including the state 
 ibmanalyticsengine.livybatch.delete Delete a Livy application request or a running Livy batch application 







 Viewing events
(https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/tut-convo-action-3.png)

Figure 9. First example user question

The editor closes. We want to add a few more examples.
4. Click the Customer starts with tile to continue adding examples.

Zoom

![Shows the actions page with the customer prompt tile selected.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/tut-convo-action-4.png)

Figure 10. User question examples
5. Add the following questions:

Are there any working papers on the shipping industry?

Are there any papers that focus on inflation?

Are there papers about how trade policy affects pricing?

What's the latest research on municipal bond markets?

Zoom

![Shows the list of example prompt questions.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/tut-convo-action-5.png)

Figure 11. User examples list
6. Click the first step in the Conversation steps section.

Zoom

![Shows the first step selected.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/tut-convo-action-6.png)

Figure 12. First step
7.
The addition of more hosts increases CPU, RAM, and storage to your cluster. For more information about vSAN technology, see [Expanding a vSAN cluster](https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.virtualsan.doc/GUID-666D9839-2726-4936-8C0F-94476ECE0606.html). 
 Remove vSAN storage For more information about removing storage from a vSAN cluster, see [Removing ESXi servers from vCenter Server instances](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-vc_removingservers). The removal of hosts decreases CPU, RAM, and storage in your cluster. 
 Using default vSAN alarms The default vSAN alarms can be used to monitor the cluster, hosts, and existing vSAN licenses. These alarms are automatically triggered when the events corresponding to the alarms are activated or if one or all the conditions that are specified in the alarms are met. You cannot edit the conditions or delete the default alarms. To configure alarms that are specific to your requirements, create custom alarms for vSAN. For more information about monitoring alarms, events, editing existing alarm settings, and using the default vSAN alarms to monitor your cluster, hosts, analyze any new events, and assess the overall cluster health, see [Using the vSAN default alarms](https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.vsan-monitoring.doc/GUID-E7885CDE-654D-4732-A5FE-31D0AB2B2F57.html). 
 Enable SIOC By default, Storage IO Control (SIOC) is disabled. If you are experiencing poor performance for VMs in a datastore, you can enable SIOC to help with prioritization to storage resources. SIOC is only activated when there is storage contention to ensure that every VM gets its share of storage resources.
Troubleshooting virtual server instances for VPC 

If you experience difficulties with your IBM Cloud® Virtual Servers for Virtual Private Cloud instances, review the following possible causes.



 Permissions not set up in IBM Cloud 

Before you can create Virtual Servers for VPC, you need to set up the correct permissions in IBM Cloud console. If your permissions are not correct, you can create a server and it shows Pending status, which quickly turns into Failed status. Make sure that the administrator of the account assigns you the correct [permissions](https://cloud.ibm.com/docs/vpc?topic=vpc-managing-user-permissions-for-vpc-resourcesmanaging-user-permissions-for-vpc-resources).





 Servers are all in Unknown status 

Most likely you do not have adequate permissions to view the server status. Make sure that you have the correct [permissions](https://cloud.ibm.com/docs/vpc?topic=vpc-managing-user-permissions-for-vpc-resourcesmanaging-user-permissions-for-vpc-resources).

The Unknown status also might be caused by an expired IMS token. Run bx sl init again and rebuild the ims_subject with the new token. Make sure that you are passing in the X-Subject-Token:$ims_subject parameter in the request header.





 Error: 409 Conflict when you create an instance action 

You can't create certain instance actions if the status of your instance is in conflict with another action. For example, if your instance status is stopped, and you try to create a reboot action, the system returns a 409 error.



Table 1.Instance status conflicts

 Status Action Conflict 

 Running start yes 
 Stopped Any action other than start yes 
 Not running pause yes 
 Not running reboot yes 
 Not paused resume yes 
 Paused Any action other than resume yes 







 Why can't I add my SSH key? 

If you try to add an SSH key to your account and get an error that the key can't be parsed, ensure there are no line breaks in the string.
Some services, such as Watson services, require an upgrade of the service plan if you want to receive them.





 Data events 

Data events report on actions where the initiator's request modifies, or operates on your data.

You identify data events by the value of the field dataEvent. When this field is set to true, the event is a data event.

Data events are enabled in all the IBM Cloud accounts. However, there are some exceptions:



* Services that generates high volumes of data events require that you opt-in:

IBM Cloud Object Storage

IBM Cloud App ID

IBM Cloudant
* Services might require a service plan to opt-in to data events, such as some Watson services.







 Global events 

Global events report on activity in your account that relate to data and resources that are generally synchronized across all regions.

Global events are generated by integrated platform services such as IAM.

The following services generate global events in the IBM Cloud:



* Integrated platform services fo managing account settings and resources such as service instances, users, billing, and tags
* IBM Cloud® Identity and Access Management (IAM)
* IBM Cloud Object Storage
* IBM Db2 Warehouse on Cloud
* IBM Cloud® Transit Gateway
* IBM Cloud® DNS Services
* IBM Cloud® Internet Services (CIS)
* Content Delivery Network
* Provisioning
* Global Search Service
* Catalog Management
* Software instances
* Context-based restrictions
* Mass Data Migration
* IBM Cloud for VMware Solutions
* IBM Cloud® Direct Link solution



For more information on services generating global events, see [IBM Cloud services that generate Activity Tracker events](https://cloud.ibm.com/docs/atracker?topic=atracker-atracker_cloud_services).

For Activity Tracker Event Routing, you can choose the region where events are collected. [Learn more](https://cloud.ibm.com/docs/atracker?topic=atracker-getting-started-routinggetting-started-routing-setp5).

For example, when account administrators invite users to join the account, they can do it from any IBM Cloud location.
To create, view, update, or delete a VPN connection by using CLI, see the CLI reference for [VPN connections](https://cloud.ibm.com/docs/power-iaas-cli-plugin?topic=power-iaas-cli-plugin-power-iaas-cli-referencevpn-connections).

When you delete a virtual server instance, it deletes the private networks (subnets) and other resources. It would help if you considered the following:



* You must delete VPN connections before deleting the virtual server instance.
* When there are more than one virtual server instances that use the Cloud Connection, and you delete one, you can delete the Cloud Connection from the existing server instances.



IKE policy version 2 is not compatible with policy-based VPN connections. If you attempt to add an IKE policy version 2 to a policy-based VPN connection, an error is displayed.

The display of an example VPN connection is as follows:

ID 1471c65163dd44daa969cf3edddd20a8
Name rs-vpc-vpn01
Status active
Mode route
Local Gateway Address 169.48.225.198
Peer Gateway Address 130.198.12.241
VPN Gateway Address 169.48.225.198
IKE Policy ID: a757fb8d0a324e4abe0589bc17fbad7c, Name: rs-ike-1
IPSec Policy ID: befd77bd25a04c388c43ccb3973966be, Name: rs-ipsec-1
Peer Subnets 10.245.0.0/27
Networks cb36a4e8-23d1-4ddc-b6c0-cf640ae0456d
Dead Peer Detection Action: restart, Interval: 10, Threshold: 5





 Attaching subnets to VPN connections 

If you create a Power Systems Virtual Servers workspace that contains VPN connections, you must also have Local subnets and Peer subnets that are connected to the VPN connection.
The Spark history server REST API is available as a service endpoint in a service key (also known as service credential). See [Retrieving service endpoints](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-retrieve-endpoints-serverless).

When you invoke the Spark history server REST API, you must specify your IAM token as a bearer token in the Authorization header. E.g.

curl --location --request GET 'https://spark-console.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance id>/spark_history_api/v1/applications?status=completed' --header 'Authorization: Bearer <iam token>'

See the [Spark history server REST API documentation](https://spark.apache.org/docs/latest/monitoring.htmlrest-api) for more details.





 Customizing the Spark history server 

By default, the Spark history server consumes 1 CPU core and 4 GiB of memory while it is running. If you want your Spark history server to use more resources, you can set the following properties in your Analytics Engine instance default configurations:



* ae.spark.history-server.cores for the number of CPU cores
* ae.spark.history-server.memory for the amount of memory





 Updating the CPU cores and memory settings using the REST API 

You can update the CPU cores and memory settings using the Analytics Engine REST API as follows:

curl --location --request PATCH "https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_id>/default_configs" --header "Authorization: bearer <iam_token>" --header 'Content-Type: application/json' --data-raw '{
"ae.spark.history-server.cores": "2",
"ae.spark.history-server.memory": "8G"
}'

Only a pre-defined list of Spark driver and executor vCPU, and memory size combinations are supported.
You can use the following configuration settings to further control autoscaling the number of executors.



Table 1. Configuration settings for autoscaling

 Configuration settings Description Default 

 ae.spark.autoscale.enable Signals that the application will autoscale based on the application's demand and any other autoscaling configurations that are set. If specified at instance level, all applications in the instance will autoscale. false 
 spark.dynamicAllocation.initialExecutors Specifies the initial number of executors to be created, irrespective of any demand made by the Spark application 0 
 spark.dynamicAllocation.minExecutors Specifies the minimum number of executors to be maintained, irrespective of any demand by the Spark application 0 
 spark.dynamicAllocation.maxExecutors Specifies the maximum number of executors to be created irrespective of any demand by the Spark Application 2 
 ae.spark.autoscale.scaleupDemandPercentage Specifies the percentage of the executors demanded by Spark Application that should be fulfilled by the application auto-scaler. For example, if at a certain stage a Spark application requests 10 executors and the value for this configuration is set to 50 percent, the application auto-scaler will scale up the number of executors by 5 executors. The default value of this configuration is 100 percent which essentially means that any number of executors requested by an application's demand can be added by the application auto-scaler. 100 
 ae.spark.autoscale.scaledownExcessPercentage Specifies the percentage of the idle executors held up by the Spark application that are to be removed by the application auto-scaler. For example, if after completing a certain stage in a Spark application, there are 20 idle executors held up by the application and the value for this configuration is set to 25 percent, the application auto-scaler will scale down the executors in the application by 4 executors. The default value for this configuration is 100 percent which means that the application auto-scaler can scale down all idle executors. 100
So if different applications need different library sets, it is better for you to use separate library sets, so that they can be specified individually at the time the application is submitted.<br> * Use customization only to bring in files that cannot be brought in by the application details parameters. See [Parameters for submitting Spark applications](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-spark-app-rest-apispark-submit-parms). You must use the standard spark-submit equivalent parameter options such as the files, jars, packages and pyFiles options if that fits your use case. Only if you need files that don't fit into any of these categories, for example a self signed certificate, a JAAS configuration file, or a .so file, you should use the "customization for file download" option.<br><br><br> <br><br> * [Customization options](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-cust-instance)<br><br><br> 
 Apply filters when retrieving list of applications When you need to retrieve list of applications either in UI or using the API or CLI, it is better to apply the appropriate filters and retrieve the set that you need. <br><br> * [Spark application commands](https://cloud.ibm.com/docs/AnalyticsEngine?topic=analytics-engine-cli-plugin-CLI_analytics_enginespark-application-commands)<br><br><br> 
 Use other services or tools for supporting functions Apart from using an IBM Log Analysis and IBM Cloud Object Storage instance and depending on your use case, you might want to use other supporting tools and services. For instance, you can use Apache Airflow (managed by you) for orchestrating, scheduling and automating your applications. You can also make use of IBM Secrets Manager to store the secrets required for your applications and use your automation scripts to read the secrets from the Secrets Manager before submitting your applications. You can also get creative with your application arguments, passing a token required to read the required secrets from the Secrets Manager directly from within your application.
ibmcloud analytics-engine-v3 instance current-resource-consumption --id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --output json





 Example output 

The output is as follows:

{
"cores" : "2",
"memory" : "8G"
}







 ibmcloud analytics-engine-v3 instance resource-consumption-limits 

Returns the maximum total memory and virtual processor cores allocated across all the applications running in the service instance at any point in time.

ibmcloud analytics-engine-v3 instance resource-consumption-limits [--id ID]



 Command options 

-i, --id (string)
: ID of the Analytics Engine instance. When specified, it overrides the target instance ID if it was set. Required, if the target instance was not set.

The value must match the regular expression /u0008[0-9a-f]{8}u0008-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-u0008[0-9a-f]{12}u0008/.





 Examples 

ibmcloud analytics-engine-v3 instance resource-consumption-limits --output json

ibmcloud analytics-engine-v3 instance resource-consumption-limits --id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --output json





 Example output 

The output is as follows:

{
"max_cores" : "150",
"max_memory" : "600G"
}









 Spark application commands 

With the Spark application CLI commands, you can perform the following operations:



* [Submit a Spark application](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-CLI_analytics_engineanalytics-engine-v3-cli-spark-app-submit-command)
ibmcloud analytics-engine-v3 instance current-resource-consumption --id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --output json





 Example output 

The output is as follows:

{
"cores" : "2",
"memory" : "8G"
}







 ibmcloud analytics-engine-v3 instance resource-consumption-limits 

Returns the maximum total memory and virtual processor cores allocated across all the applications running in the service instance at any point in time.

ibmcloud analytics-engine-v3 instance resource-consumption-limits [--id ID]



 Command options 

-i, --id (string)
: ID of the Analytics Engine instance. When specified, it overrides the target instance ID if it was set. Required, if the target instance was not set.

The value must match the regular expression /u0008[0-9a-f]{8}u0008-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-u0008[0-9a-f]{12}u0008/.





 Examples 

ibmcloud analytics-engine-v3 instance resource-consumption-limits --output json

ibmcloud analytics-engine-v3 instance resource-consumption-limits --id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --output json





 Example output 

The output is as follows:

{
"max_cores" : "150",
"max_memory" : "600G"
}









 Spark application commands 

With the Spark application CLI commands, you can perform the following operations:



* [Submit a Spark application](https://cloud.ibm.com/docs/cli?topic=cli-CLI_analytics_engineanalytics-engine-v3-cli-spark-app-submit-command)
Spark batch applications 

You can submit Spark batch applications by using the:



*  [IBM Analytics Engine serverless Spark application REST API](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-spark-app-rest-api)
*  [Livy batch APIs](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-livy-api-serverless)
For more information, see [Building actions from a template](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-actions-templates).





 26 January 2023 

Display formats for variables
: In Global settings for actions, Display formats lets you specify the display formats for variables that use date, time, numbers, currency, or percentages. You can also choose a default locale to use if one isn't provided by the client application. This lets you make sure that the format of a variable that's displayed in the web chat is what you want for your assistant. For example, you can choose to have the output of a time variable appear in HH:MM format instead of HH:MM:SS. For more information, see [Display formats](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-actions-global-settingsactions-global-settings-display-formats)





 18 January 2023 

Algorithm version stability improvement
: As of this date, the Latest (01 Jun 2022) and Beta algorithm versions now have more stable behavior across retrained models, in the presence of overlapping entities (the same entity value belonging to more than one entity type). Previously, when there were overlapping entities definitions, confidences could differ across different retraining. With this improvement, you can expect to see similar confidences. For more information, see [Algorithm version and training](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-algorithm-version).





 12 January 2023 

Autocorrection setting for actions
: The Global settings for actions now include an Autocorrection setting. Autocorrection fixes misspellings that users make in their requests. The corrected words are used to match to an action.

While the setting is new, the autocorrection feature was already used automatically by all English-language assistants. Autocorrection is also available in French-language assistants, but is disabled by default. The autocorrection setting isn't available for any other languages. The new setting lets you disable or enable autocorrection if necessary.
Getting started with IBM Cloud Pak for Data 

Collect, organize, and analyze your data to generate meaningful insight with an extensible, end-to-end platform for governance, analytics, and AI that runs on Red Hat OpenShift on IBM Cloud. With IBM Cloud Pak® for Data, it's easy to find and access trusted data so that you can put your data to work quickly and efficiently. Make data-driven decisions and operationalize AI with trust and transparency throughout your business.

See the [IBM Cloud Pak® for Data readme file](https://cloud.ibm.com/catalog/content/ibm-cp-datacore-6825cc5d-dbf8-4ba2-ad98-690e6f221701-global/readme/roks/cloud-pak/ed370810-180a-455c-b949-d23d5b8b4ff6-global) for detailed information about getting started with Cloud Pak for Data on IBM Cloud.

Learn more about IBM Cloud Pak® for Data by reviewing the [product documentation](https://www.ibm.com/docs/SSQNUZ_4.6.x/cpd/overview/overview.html).



 What's inside this Cloud Pak 

You can choose which services to install when you install IBM Cloud Pak® for Data.



* [Apache Spark](https://www.ibm.com/docs/SSQNUZ_4.6.x/svc-welcome/spark.html): Use the Analytics Engine powered by Apache Spark as a compute engine to run analytical and machine learning jobs.
* [Cognos Dashboards](https://www.ibm.com/docs/SSQNUZ_4.6.x/svc-welcome/cde.html): Use sophisticated visualizations in an analytics project to identify patterns in your data so that you can make timely and effective decisions.