vania-janet/multiturn-rag-retrieval-data huggingface dataset:
* Data skipping sometimes does not work if type casting is used in the WHERE clause. For example, given a MinMax index on a column with a short data type, the following query does not benefit from data skipping:

select * from table where shortType > 1

Apache Spark evaluates the query as (cast(shortType3 as int) > 1) because the constant 1 is of type integer.

In some cases, Apache Spark automatically casts the literal to the right type. For example, the previous query works for all other numerical types, except for the byte type, as it requires casting, as well. To benefit from data skipping in such cases, ensure that the literal has the same type as the column type, as in the following example:

select * from table where shortType > cast(1 as short)
* Concurrent CREATE/REFRESH operations are not supported.
* Indexing nested geospatial field is not supported.
* With the Lite plan, data skipping features, such as CREATE METAINDEX, are not allowed.







 References 



* [Data skipping for Data Engine](https://www.ibm.com/cloud/blog/data-skipping-for-ibm-cloud-sql-query).
* [Data skipping demo at Think 2019](https://www.ibm.com/cloud/blog/ibm-cloud-sql-query-at-think-2019) for the [Danaos use case](https://www.danaos.com/home/default.aspx) of [BigDataStack](https://bigdatastack.eu/?utm_source=IBM-Ta-Shma).
* [How to lay out Big Data in IBM Cloud Object Storage for Spark SQL](https://www.ibm.com/cloud/blog/big-data-layout).
* [Querying Geospatial Data by using Data Engine](https://www.ibm.com/cloud/blog/querying-geospatial-data-using-ibm-sql-query).
[Edit panel.](https://cloud.ibm.com/docs-content/v1/content/6f3c07c34058f637b1c86044ca4f9b24214330a7/activity-tracker/images/streaming-topic-metric.png)

Then, customize the metric to see data per topic.

![Customize metric.](https://cloud.ibm.com/docs-content/v1/content/6f3c07c34058f637b1c86044ca4f9b24214330a7/activity-tracker/images/streaming-topic-metric-1.png)

Check that the resolution is set to 10M.







 Monitoring streaming by using IBM Cloud Activity Tracker 

Streaming generates Activity Tracker events with the action logdnaat.streaming-logs.send to notify about failures that occur when data is streamed to Event Streams.

There are different reasons for failure, for example:



* unknown_topic or unknown_topic_or_partition: This notification reports that the topic that is configured for streaming is not valid.

If you get this notification, check the topic that is configured for streaming is defined in Event Streams.
* broker_handle_destroyed: This notification reports a change in the Kafka cluster configuration.

If you get this notification, check the SASL URLs and update the streaming configuration.
* authentication_failure: This notification reports a failure to authenticate with Event Streams.

If you get this notification, check the credentials that are configured for streaming.
* broker_transport_failure: This notification reports connectivity problems with Event Streams. There are different reasons why you might get this notification such as that credentials were deleted after streaming was activated and started.
* message_timed_out or timed_out: This notification reports generic timeout errors when streaming messages to Event Streams.



The Activity Tracker event includes different fields that you can use to monitor streaming data failures:



* reason.reasonCode = 500 and severity = critical notify of a failure streaming data.
* reason.reasonType indicates the type of failure.
ae.spark.autoscale.scaledownExcessPercentage Specifies the percentage of the idle executors held up by the Spark application that are to be removed by the application auto-scaler. For example, if after completing a certain stage in a Spark application, there are 20 idle executors held up by the application and the value for this configuration is set to 25 percent, the application auto-scaler will scale down the executors in the application by 4 executors. The default value for this configuration is 100 percent which means that the application auto-scaler can scale down all idle executors. 100 
 ae.spark.autoscale.frequency The frequency in seconds at which the application auto-scaler scales up executors or scales down executors in an application. 10s 



Example of an auto-scaling application payload showing the lower and upper bounds on the number of executors that the application can scale up or scale down to.

{
"application_details": {
"application": "cos://<application-bucket-name>.<cos-reference-name>/my_spark_application.py",
"conf": {
"ae.spark.autoscale.enable": "true",
"spark.dynamicAllocation.initialExecutors": "0",
"spark.dynamicAllocation.minExecutors": "1",
"spark.dynamicAllocation.maxExecutors": "10",
"spark.hadoop.fs.cos.<cos-reference-name>.endpoint": "s3.direct.us-south.cloud-object-storage.appdomain.cloud",
"spark.hadoop.fs.cos.<cos-reference-name>.iam.api.key": "<iam-api-key-of-application-bucket>"
}
}
}





 Enabling autoscaling on an instance 

By default, application autoscaling is disabled when you create an instance of the Analytics Engine Standard serverless plan.
space(n)
: Returns a string that consists of n spaces.
: Example of an SQL function usage fragment

> SELECT concat(space(2), '1')
: Result value

1





 spark_partition_id 

spark_partition_id()
: Returns the current partition ID.





 split 

split(str, regex)
: Splits str around occurrences that match regex.
: Example of an SQL function usage fragment

> SELECT split('oneAtwoBthreeC', '[ABC]')
: Result value

["one","two","three",""]





 sqrt 

sqrt(expr)
: Returns the square root of expr.
: Example of an SQL function usage fragment

> SELECT sqrt(4)
: Result value

2.0





 stack 

stack(n, expr1, ..., exprk)
: Separates expr1, ..., exprk into n rows.
: Example of an SQL function usage fragment

> SELECT stack(2, 1, 2, 3)
: Result value

1 2 3 NULL





 std 

std(expr)
: Returns the sample standard deviation that is calculated from values of a group.





 stddev 

stddev(expr)
: Returns the sample standard deviation that is calculated from values of a group.





 stddev_pop 

stddev_pop(expr)
: Returns the population standard deviation that is calculated from values of a group.





 stddev_samp 

stddev_samp(expr)
: Returns the sample standard deviation that is calculated from values of a group.





 str_to_map 

str_to_map(text[, pairDelim[, keyValueDelim]])
: Creates a map after splitting the text into key/value pairs by using delimiters. Default delimiters are ',' for pairDelim and ':' for keyValueDelim.
: Example of an SQL function usage fragment
To create a service instance using the Resource Controller REST API:



1. Get the resource group ID by logging into the IBM Cloud® CLI and running the following command:

ibmcloud resource groups

Sample result:

Retrieving all resource groups under account <Account details..>
OK
Name ID Default Group State
Default XXXXX true ACTIVE
2. Use the following resource plan ID for the Standard Serverless for Apache Spark plan:

8afde05e-5fd8-4359-a597-946d8432dd45
3. Get the IAM token by performing the following [steps](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-retrieve-iam-token-serverless).
4. Create an instance using the Resource Controller REST API:

curl -X POST https://resource-controller.cloud.ibm.com/v2/resource_instances/
--header "Authorization: Bearer $token" -H 'Content-Type: application/json' -d @provision.json

The provision.json file contains the provisioning parameters for the instance you want to create. See [Architecture and concepts in serverless instances](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-serverless-architecture-concepts) for a description of the provisioning parameters in the payload.

Note that both Spark 3.1 and Spark 3.3 are supported. If you don't specify a default Spark runtime version when you create a service instance, Spark 3.1 is taken by default.

This is a sample of what the provision.json file can look like:

{
"name": "your-service-instance-name",
"resource_plan_id": "8afde05e-5fd8-4359-a597-946d8432dd45",
"resource_group": "resource-group-id",
"target": "us-south",
"parameters": {
"default_runtime": {
Once the logging service is created, you can log in to [IBM Cloud®](https://cloud.ibm.com/docs/cloud.ibm.com), search for the logging service instance, and click on the monitoring dashboard. There you can view the driver and executor logs, as well as all application logs for your Spark application.

Search using the application_id or instance_id.
2. Enable platform logging:

To view IBM Analytics Engine platform logs, you must use the Observability dashboard in IBM Cloud to configure platform logging. See [Configuring platform logs through the Observability dashboard](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-config_svc_logsconfig_svc_logs_ui) for the steps you need to follow to enable logging through the Observability dashboard.
3. Enable logging for Analytics Engine:

Action
: Enter:

ibmcloud analytics-engine-v3 log-config COMMAND [arguments...] [command options]

Parameter values:



* analytics-engine-v3: Use ae-v3 to use the v3 CLI commands
* COMMAND: Use the update command to enable logging



Example
: Enter:

ibmcloud ae-v3 log-config update --instance-id 181ea9ee01b --enable --output json







 Step 5: Delete Analytics Engine instance 

You can use the CLI to delete an instance, for example if you need an instance with a completely different configuration to handle greater workloads.

You can retain an Analytics Engine instance as long as you want and submit your Spark applications against the same instance on an as-needed basis.

If you want to delete an Analytics Engine instance:

Action
: Enter:

ibmcloud resource service-instance-delete NAME|ID [-g RESOURCE_GROUP] -f

Parameter values:



* NAME|ID: The value of Name or GUID from the response of the Analytics Engine instance creation call
* RESOURCE_GROUP: Optional parameter.
For details on how to access Object Storage, see [Using IBM Object Storage as the instance home volume](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-cos-serverless).
* The default Spark configuration options to override configuration settings.

For a list of the default Spark configurations set for serverless instances, see [Default Spark configurations](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-serverless-architecture-conceptsdefault-spark-config).









 Console applications tab 

You can monitor the status of submitted applications in your IBM Analytics Engine serverless instance from the IBM Cloud® Resource list.



1. Access your [IBM Cloud® Resource list](https://test.cloud.ibm.com/resources).
2. Click Services and software, find the IBM Analytics Engine serverless instance and click the instance to see its details.
3. Click Manage > Applications to list the submitted applications. You can:



* Click the Filter icon on the action bar of the result list to narrow your search results to find applications by their status.
* Click the Settings icon to customize the result list by adjusting the row height or the display columns.
* Refresh the list



By clicking the arrow to the left of an application ID in the result list, you are shown additional properties that are not included in any list columns, for example, the ID assigned to the application by Spark.

By clicking the Action icon to the right of a selected application, you can:



* Stop the application
* [Instance management](https://cloud.ibm.com/docs/analytics-engine-cli-plugin?topic=analytics-engine-cli-plugin-CLI_analytics_engineanalytics-engine-v3-instance-cli)
* [Spark application management](https://cloud.ibm.com/docs/analytics-engine-cli-plugin?topic=analytics-engine-cli-plugin-CLI_analytics_engineanalytics-engine-v3-spark-app-cli)
* [Log forwarding configuration](https://cloud.ibm.com/docs/analytics-engine-cli-plugin?topic=analytics-engine-cli-plugin-CLI_analytics_engineanalytics-engine-v3-log-forwarding-config-cli)
* [Spark history server management](https://cloud.ibm.com/docs/analytics-engine-cli-plugin?topic=analytics-engine-cli-plugin-CLI_analytics_engineanalytics-engine-v3-history-server-cli)



$ ibmcloud ae-v3 --help

NAME:
analytics-engine-v3 - Manage serverless Spark instances and run applications.

USAGE:
ibmcloud analytics-engine-v3 [command] [options]

ALIASES:
analytics-engine-v3, ae-v3

COMMANDS:
history-server Commands for HistoryServer resource.
instance Commands for Instance resource.
log-forwarding-config Commands for LogForwardingConfig resource.
spark-app Commands for SparkApp resource.
target All commands run after setting a target will be run on the target instance

OPTIONS:
-h, --help Show help
-j, --jmes-query string Provide a JMESPath query to customize output.
--output string Choose an output format - can be 'json', 'yaml', or 'table'. (default "table")
-q, --quiet Suppresses verbose messages.
Enabling application autoscaling 

It is not always possible to estimate the resource requirements (number of Spark executors) of a Spark application upfront, because these requirements might vary based on the size of the input data set.

To assist you in this situation, you can submit a Spark application with auto-scaling, which will automatically determine the number of executors required by an application based on the application's demand. You can enable basic autoscaling for a Spark application by adding the configuration setting ae.spark.autoscale.enable=true to the existing application configuration.



 Submitting an application with basic autoscaling 

The steps to submit an application with autoscaling enabled is the same as the steps to submit an application without autoscaling. The only difference is that you need to add the configuration setting ae.spark.autoscale.enable=true to the application payload.



1. [Submit a Spark application](https://test.cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-spark-app-rest-apispark-submit-app).
2. Use the following sample JSON payload as an example to enable basic autoscaling:

{
"application_details": {
"application": "/opt/ibm/spark/examples/src/main/python/wordcount.py",
"arguments": ["/opt/ibm/spark/examples/src/main/resources/people.txt"]
},
"conf": {
"ae.spark.autoscale.enable":"true"
}
}







 Autoscaling application configurations 

You can use the following configuration settings to further control autoscaling the number of executors.



Table 1. Configuration settings for autoscaling

 Configuration settings Description Default 

 ae.spark.autoscale.enable Signals that the application will autoscale based on the application's demand and any other autoscaling configurations that are set. If specified at instance level, all applications in the instance will autoscale. false
See [Spark environment variables](https://spark.apache.org/docs/latest/configuration.htmlenvironment-variables) for a list of the supported variables.





 Examples 

ibmcloud analytics-engine-v3 spark-app submit --arg "/opt/ibm/spark/examples/src/main/resources/people.txt" --app "/opt/ibm/spark/examples/src/main/python/wordcount.py" --runtime='{"spark_version": "3.1"}' --output json

ibmcloud analytics-engine-v3 spark-app submit --instance-id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --arg "/opt/ibm/spark/examples/src/main/resources/people.txt" --app "/opt/ibm/spark/examples/src/main/python/wordcount.py" --runtime='{"spark_version": "3.1"}' --output json





 Example output 

The output is as follows:

{
"id" : "87e63712-a823-4aa1-9f6e-7291d4e5a113",
"state" : "accepted"
}







 ibmcloud analytics-engine-v3 spark-app list 

Returns a list of all Spark application submitted to the specified Analytics Engine instance. The result can be filtered by specifying query parameters. The number of applications returned can be limited by specifying the limit query parameter. Use start together with limit to fetch the next or previous page of results.

ibmcloud analytics-engine-v3 spark-app list [--instance-id INSTANCE-ID] [--state STATE] [--limit LIMIT] [--start START]



 Command options 

-i, --instance-id (string)
: The identifier of the Analytics Engine instance associated with the Spark application(s). Required.
* bootstrap_logging - Writes connector framework startup log; useful for advanced debugging only. Possible values are true or false. Log file is written to crawler_temp_dir
* read-timeout - Sets the time in seconds that the crawler waits for a response from the connector framework. The default value is 5 seconds.







 Output adapter 



* class - Defines the Data Crawler output adapter class.
* config - Defines which configuration key to pass to the output adapter. The string must correspond to a key within this configuration object. In the following code example:

class - "com.ibm.watson.crawler.discoveryserviceoutputadapter.DiscoveryServiceOutputAdapter"

config - "discovery_service"

discovery_service {
include "discovery/discovery_service.conf"
}

the configuration key is discovery_service.



You must select an output adapter by specifying its class parameter and config key.



* Discovery Service Output Adapter - Uploads crawled documents to the IBM Watson™ Discovery Service. Select this adapter by setting the class parameter and config key as follows.



class - "com.ibm.watson.crawler.discoveryserviceoutputadapter.DiscoveryServiceOutputAdapter"

config - "discovery_service"

discovery_service {
include "discovery/discovery_service.conf"
}



* Test Output Adapter - The Test Output Adapter writes a representation of the crawled files to disk in a specified location. Select this adapter by setting the class parameter and config key as follows.

An additional parameter, output_directory, selects the directory to which the representation of the crawled data must be written.



class - "com.ibm.watson.crawler.testoutputadapter.TestOutputAdapter"

config - "test"

output_directory - "/tmp/crawler-test-output"



* retry - Specifies the options for retry in case of failed attempts to push to the output adapter.
[<metadata> element](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-elementsmm_element) None <br><br> * [interpret-as attribute](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-elementssay-as-interpret-as)<br><br><br> Partial 
 [<paragraph> element](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-elementsps_element) Full [<sentence> element](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-elementsps_element) Full 
 [<phoneme> element](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-elementsphoneme_element) Full [<speak> element](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-elementsspeak_element) Full 
 [<sub> element](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-elementssub_element) Full 
 [<voice> element](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-elementsvoice_element) None 







 The <audio> element 

This <audio> element inserts recorded elements into the service-generated audio. It is not supported.





 The <break> element 

The <break> element inserts a pause into the spoken text. It has the following optional attributes:



* strength specifies the length of the pause in terms of varying strength values:



* none suppresses a break that might otherwise be produced during processing.
* x-weak, weak, medium, strong, or x-strong insert increasingly stronger breaks.



* time specifies the length of the pause in terms of seconds or milliseconds.
.config("fs.cos.impl", "com.ibm.stocator.fs.ObjectStoreFileSystem") 
.config("fs.stocator.scheme.list", "cos") 
.config("fs.stocator.cos.impl", "com.ibm.stocator.fs.cos.COSAPIClient") 
.config("fs.stocator.cos.scheme", "cos") 
 register the required Cloud Object Storage path used in our application, add endpoints for all buckets
.config("spark.hadoop.fs.cos.us-geo.endpoint", "https://s3.us.cloud-object-storage.appdomain.cloud") 
.config("spark.hadoop.fs.cos.us-geo.iam.endpoint", "https://iam.cloud.ibm.com/identity/token") 
.config("spark.hadoop.fs.cos.us-geo.iam.api.key", '<YourAPIkey>') 
.config("spark.sql.hive.metastore.version", "3.0") 
 directory where the Hive client has been placed
.config("spark.sql.hive.metastore.jars", "/tmp/dataengine/") 
.config("spark.hive.metastore.uris", "thrift://catalog.<region>.sql-query.cloud.ibm.com:9083") 
.config("spark.hive.metastore.use.SSL", "true") 
.config("spark.hive.metastore.truststore.password", "changeit") 
.config("spark.hive.metastore.client.auth.mode", "PLAIN")
* [Spark application management](https://cloud.ibm.com/docs/cli?topic=cli-CLI_analytics_engineanalytics-engine-v3-spark-app-cli)
* [Log forwarding configuration](https://cloud.ibm.com/docs/cli?topic=cli-CLI_analytics_engineanalytics-engine-v3-log-forwarding-config-cli)
* [Spark history server management](https://cloud.ibm.com/docs/cli?topic=cli-CLI_analytics_engineanalytics-engine-v3-history-server-cli)



$ ibmcloud ae-v3 --help

NAME:
analytics-engine-v3 - Manage serverless Spark instances and run applications.

USAGE:
ibmcloud analytics-engine-v3 [command] [options]

ALIASES:
analytics-engine-v3, ae-v3

COMMANDS:
history-server Commands for HistoryServer resource.
instance Commands for Instance resource.
log-forwarding-config Commands for LogForwardingConfig resource.
spark-app Commands for SparkApp resource.
target All commands run after setting a target will be run on the target instance

OPTIONS:
-h, --help Show help
-j, --jmes-query string Provide a JMESPath query to customize output.
--output string Choose an output format - can be 'json', 'yaml', or 'table'. (default "table")
-q, --quiet Suppresses verbose messages.
-v, --version version for analytics-engine-v3

Use "ibmcloud analytics-engine-v3 service-command --help" for more information about a command.
Show more





 Set a target instance 

The following command allows you to set a target instance against which all subsequent CLI commands are run.

Note: If the instance ID is passed explicitly in any v3 CLI commands, it will override the instance ID that is set as the target.
: Returns the logarithm of expr with base 2.
: Example of an SQL function usage fragment

> SELECT log2(2)
: Result value

1.0





 lower 

lower(str)
: Returns str with all characters that are changed to lowercase.
: Example of an SQL function usage fragment

> SELECT lower('SparkSql')
: Result value

sparksql





 lpad 

lpad(str, len, pad)
: Returns str, left-padded with pad to a length of len. If str is longer than len, the return value is shortened to len characters.
: Example of an SQL function usage fragment

> SELECT lpad('hi', 5, '??')
: Result value

???hi
: Example of an SQL function usage fragment

> SELECT lpad('hi', 1, '??')
: Result value

h





 ltrim 

ltrim(str)
: Removes the leading space characters from str.

ltrim(trimStr, str)
: Removes the leading string contains the characters from the trim string.
: Arguments



* str - A string expression.
* trimStr - The trim string characters to trim, the default value is a single space.



: Example of an SQL function usage fragment

> SELECT ltrim(' SparkSQL ')
: Result value

SparkSQL
: Example of an SQL function usage fragment

> SELECT ltrim('Sp', 'SSparkSQLS')
: Result value

arkSQLS





 map 

map(key0, value0, key1, value1, ...)
: Creates a map with the indicated key/value pairs.
: Example of an SQL function usage fragment

> SELECT map(1.0, '2', 3.0, '4')
: Result value

{1.0:"2",3.0:"4"}





 map_keys 

map_keys(map)