vania-janet/multiturn-rag-retrieval-data huggingface dataset:
See [Use the Spark history server](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-spark-history-serverless).





 September 2022 



 21 September 2022 

Support for Spark 3.3
: You can now provision IBM® Analytics Engine severless plan instances with the default Spark runtime set to Spark 3.3, which enables you to run Spark applications on Spark 3.3.





 09 September 2022 

You can now use Hive metastore to manage the metadata related to your applications tables, columns, and partition information when working with Spark SQL.
: You could choose to externalize this metastore database to an external data store, like to an IBM Cloud Data Engine (previously SQL Query) or an IBM Cloud Databases for PostgreSQL instance. For details, see [Working with Spark SQL and an external metastore](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-external-metastore).







 July 2022 



 12 July 2022 

You can now provision IBM® Analytics Engine serverless instances in a new region.
: In addition to the IBM Cloud® us-south (Dallas) region, you can now also provision serverless instances in the eu-de (Frankurt) region.





 08 July 2022 

New API for platform logging
: Start using the log_forwarding_config API to forward platform logs from an IBM Analytics Engine instance to IBM Log Analysis. Although you can still use the logging API, it is deprecated and will be removed in the near future. For details on how to use the log_forwarding_config API, see [Configuring and viewing logs](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-viewing-logs).







 13 May 2022 

Support for Python 3.9
: You can now run Spark applications using Python 3.9. on your IBM Analytics Engine serverless instances.





 04 April 2022 

Limitation on how long Spark applications can run
: Spark applications can run for a maximum period of 3 days (72 hours).
If isIgnoreNull is true, returns only nonnull values.





 lcase 

lcase(str)
: Returns str with all characters that are changed to lowercase.
: Example of an SQL function usage fragment

> SELECT lcase('SparkSql')
: Result value

sparksql





 lead 

lead(input[, offset[, default]])
: Returns the value of input at the offsetth row after the current row in the window. The default value of offset is 1 and the default value of default is null. If the value of input at the offsetth row is null, null is returned. If no such offset row exists (for example, when the offset is 1, the last row of the window does not have any subsequent row), default is returned.





 least 

least(expr, ...)
: Returns the least value of all parameters, skipping null values.
: Example of an SQL function usage fragment

> SELECT least(10, 9, 2, 4, 3)
: Result value

2





 left 

left(str, len)
: Returns the leftmost len(len can be string type) characters from the string str, if len is less or equal than 0 the result is an empty string.
: Example of an SQL function usage fragment

> SELECT left('Spark SQL', 3)
: Result value

Spa





 length 

length(expr)
: Returns the character length of string data or number of bytes of binary data. The length of string data includes the trailing spaces. The length of binary data includes binary zeros.
: Example of an SQL function usage fragment

> SELECT length('Spark SQL ')
: Result value

10
: Example of an SQL function usage fragment

> SELECT CHAR_LENGTH('Spark SQL ')
: Result value

10
: Example of an SQL function usage fragment

> SELECT CHARACTER_LENGTH('Spark SQL ')
: Result value

10





 levenshtein 

levenshtein(str1, str2)
"start_time" : "2021-04-21T04:22:01Z",
"end_time" : "2021-04-21T04:23:18Z",
"finish_time" : "2021-04-21T04:23:18Z",
"auto_termination_time" : "2021-04-24T04:22:01Z"
} ],
"first" : {
"href" : "https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/e64c907a-e82f-46fd-addc-ccfafbd28b09/spark_applications?limit=25"
},
"next" : {
"href" : "https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/e64c907a-e82f-46fd-addc-ccfafbd28b09/spark_applications?limit=25&start=QiwyMDIyLTA5LTI2IDA4OjE4OjU2LjE4MiwxMDAyYTVlZC0xZWU4LTQwZWItOWUyNC00OTMyNTcxZjgzYzE",
"start" : "QiwyMDIyLTA5LTI2IDA4OjE4OjU2LjE4MiwxMDAyYTVlZC0xZWU4LTQwZWItOWUyNC00OTMyNTcxZjgzYzE"
},
"limit": 25
}







 ibmcloud analytics-engine-v3 spark-app show 

Gets the details of a given Spark application.

ibmcloud analytics-engine-v3 spark-app show [--instance-id INSTANCE-ID] --app-id APP-ID



 Command options 

-i, --instance-id (string)
: Identifier of the instance to which the application belongs. When specified, it overrides the target instance ID if it was set. Required, if the target instance was not set.
The value must match regular expression /u0008[0-9a-f]{8}u0008-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-u0008[0-9a-f]{12}u0008/.

--state ([]string)
: List of Spark application states that will be used to filter the response.

Allowable list items are: finished, running, failed, accepted, stopped, auto_terminated, ops_terminated.

--limit (int64)
: Number of application entries to be included in the response.

The maximum value is 1000. The minimum value is 1.

--start (string)
: Token used to fetch the next or the previous page of the applications list.

--all-pages (bool)
: Invoke multiple requests to display all pages of the collection for spark-app-list.





 Examples 

ibmcloud analytics-engine-v3 spark-app list --state=finished --limit=25

ibmcloud analytics-engine-v3 spark-app list --instance-id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --state=finished --limit=25





 Example output 

The output is as follows:

{
"applications" : [ {
"id" : "db933645-0b68-4dcb-80d8-7b71a6c8e542",
"spark_application_id" : "app-20211009103247-0000",
"spark_application_name" : "PythonWordCount",
"state" : "finished",
"submission_time" : "2021-04-21T04:23:50Z",
"start_time" : "2021-04-21T04:24:01Z",
"end_time" : "2021-04-21T04:25:18Z",
The value must match the regular expression /u0008[0-9a-f]{8}u0008-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-u0008[0-9a-f]{12}u0008/.

--enabled (bool)
: Enable or disable log forwarding. Required.

--sources ([]string)
: List of sources of logs that will be forwarded. By default, only 'spark-driver' logs are forwarded.

--tags ([]string)
: List of tags to be applied to the logs being forwarded. They can be used to filter the logs in the IBM Log Analysis server.





 Examples 

ibmcloud analytics-engine-v3 log-forwarding-config replace --enabled=true --sources=spark-driver,spark-executor --tags=<tag_1>,<tag_2>,<tag_n> --output json

ibmcloud analytics-engine-v3 log-forwarding-config replace --instance-id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --enabled=true --sources=spark-driver,spark-executor --tags=<tag_1>,<tag_2>,<tag_n> --output json





 Example output 

The output is as follows:

{
"sources" : [ "spark-driver", "spark-executor" ],
"log_server" : {
"type" : "ibm-log-analysis"
},
"enabled" : true
}







 ibmcloud analytics-engine-v3 log-forwarding-config show 

Retrieve the log forwarding configuration of the Analytics Engine instance.

ibmcloud analytics-engine-v3 log-forwarding-config show [--instance-id INSTANCE-ID]



 Command options
About Visual Explain 

The visual explain feature in IBM® watsonx.data shows the execution plan for a specified SQL query. This feature also validates the SQL query. The output results can be visualized in different formats, which can be rendered into a graph or a flow chart. Data exchange happens between single or multiple nodes within a fragment. Each fragment has a set of data that is distributed between the nodes.

With this visual explain feature, you can run the query and show the output in a distributed environment. You can output the results in different formats. When queries are run, they scan through the database. The queries retrieve table metadata to fetch the correct output.

With this visual explain feature, you can visualize the query in a graphical representation. When a query is run in the SQL editor and selects the Explain option, watsonx.data uses an EXPLAIN SQL statement on the query to create a corresponding graph. This graph can be used to analyze, fix, and improve the efficiency of your queries by saving time and cost.

To view the execution plans for a query that is run in watsonx.data SQL editor, follow the steps:



1.  In the Query workspace page, enter the query and click Run on starter to get the results.
2.  Click Explain on the screen to visualize the graphical representation of the query.



On the Explain window, click a stage to view the details on the right window. The details of each stage that is displayed are the estimate values for rows, CPU, memory, and network.
The following additional data is also displayed:

_account
: A field for internal support use only.

_id
: The line ID from the original line in the source account.

_instanceId
: The logging instance ID for the account.

_region
: Contains the region where the logging data originated. Additional information is included for internal support use only. The region within this field is region name where the data originated.

For example, a2.ca-tor.prod.ibm indicates the region originating the log is ca-tor.



If you do not see data, do 1 of the following:



* Resend sample logs to verify that sample data is streaming.
* Check my setup to go back to the configuration panel. After you make your changes, retry the verification steps.



If you see data, click YES.





 Step 3. Start streaming 

After you verify that sample data is reaching the Log Analysis instance, click Start stream.

Streaming may take up to 15 minutes to begin.
The SparkUiUrl property in response will have a non-null value when the UI is available for serverless Spark instance.





 Delete a job 

To delete a submitted batch job enter:

curl -H 'Authorization: Bearer <TOKEN>' -H 'Content-Type: application/json' -X DELETE https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance-id>/livy/batches/<batch-id>

For example:

curl -i -X DELETE https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/43f79a18-768c-44c9-b9c2-b19ec78771bf/livy/batches/14 -H 'content-type: application/json' -H "Authorization: Bearer $TOKEN"

Sample response:

{
"msg": "deleted"
}
ID: crn:v1:staging:public:ibmanalyticsengine:us-south:a/d628eae2cc7e4373bb0c9d2229f2ece5:1e32e-afd9-483a-b1-724ba5cf4::
GUID: 1e32e-afd9-483a-b1-724ba5cf4
Location: us-south
State: provisioning
Type: service_instance
Sub Type:
Service Endpoints: public
Allow Cleanup: false
Locked: false
Created at: 2021-11-29T07:20:40Z
Updated at: 2021-11-29T07:20:42Z
Last Operation:
Status create in progress
Message Started create instance operation
Show more

Make a note of the instance ID from the output. You will need the instance ID when you call instance management or Spark application management APIs. See [Spark application REST API](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-spark-app-rest-api).
5. [Track instance readiness](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-provisioning-serverlessinstance-readiness).







 Creating a service instance using the Resource Controller REST API 

An IBM Analytics Engine serverless instance must reside in an IBM Cloud® resource group. As a first step towards creating an IBM Analytics Engine serverless instance through the Resource Controller REST API, you need to have the resource group ID and serverless plan ID close at hand.

To create a service instance using the Resource Controller REST API:



1. Get the resource group ID by logging into the IBM Cloud® CLI and running the following command:

ibmcloud resource groups

Sample result:

Retrieving all resource groups under account <Account details..>
OK
Name ID Default Group State
Default XXXXX true ACTIVE
2. Use the following resource plan ID for the Standard Serverless for Apache Spark plan:
ibmcloud analytics-engine-v3 spark-app submit --instance-id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --arg "/opt/ibm/spark/examples/src/main/resources/people.txt" --app "/opt/ibm/spark/examples/src/main/python/wordcount.py" --runtime='{"spark_version": "3.1"}' --output json





 Example output 

The output is as follows:

{
"id" : "87e63712-a823-4aa1-9f6e-7291d4e5a113",
"state" : "accepted"
}







 ibmcloud analytics-engine-v3 spark-app list 

Returns a list of all Spark application submitted to the specified Analytics Engine instance. The result can be filtered by specifying query parameters. The number of applications returned can be limited by specifying the limit query parameter. Use start together with limit to fetch the next or previous page of results.

ibmcloud analytics-engine-v3 spark-app list [--instance-id INSTANCE-ID] [--state STATE] [--limit LIMIT] [--start START]



 Command options 

-i, --instance-id (string)
: The identifier of the Analytics Engine instance associated with the Spark application(s). Required.

The value must match regular expression /u0008[0-9a-f]{8}u0008-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-u0008[0-9a-f]{12}u0008/.

--state ([]string)
: List of Spark application states that will be used to filter the response.

Allowable list items are: finished, running, failed, accepted, stopped, auto_terminated, ops_terminated.

--limit (int64)
The following additional data is also displayed:

_account
: A field for internal support use only.

_id
: The line ID from the original line in the source account.

_instanceId
: The logging instance ID for the account.

_region
: Contains the region where the logging data originated. Additional information is included for internal support use only. The region within this field is region name where the data originated.

For example, a2.ca-tor.prod.ibm indicates the region originating the log is ca-tor.



If you do not see data, do 1 of the following:



* Resend sample logs to verify that sample data is streaming.
* Check my setup to go back to the configuration panel. After you make your changes, retry the verification steps.



If you see data, click YES.





 Step 3. Start streaming 

After you verify that sample data is reaching the Activity Tracker instance, click Start stream.

Streaming may take up to 15 minutes to begin.
* The [Analytics Engine instance UI](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-spark-history-serverlessiae-ui)





 Analytics Engine REST API 

You can use the Analytics Engine REST API:



1. To view the status of the Spark history server

curl "https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_id>/spark_history_server" --header "Authorization: bearer <iam token>"
2. To start the Spark history server

curl --location --request POST "https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_id>/spark_history_server" --header "Authorization: bearer <iam token>"
3. To stop the Spark history server

curl --location --request DELETE "https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_id>/spark_history_server" --header "Authorization: bearer <iam token>"







 Analytics Engine instance UI 

You can use the Analytics Engine instance UI:



1. To view the Spark history server status:



1. Open your resource list on [IBM Cloud](https://cloud.ibm.com/resources).
2. Click Services and software and select your instance to open the details page.
3. Select the Spark history tab. The current status of the server is shown on this page.



If the status of the Spark history server is set to Started, you can also click View Spark history to launch the Web UI of the Spark history server in a new browser tab.
2. To start the Spark history server:



1. On the Spark history page, click Start history server.
2. Choose the server configuration and click Start to start the Spark history server.



3. To stop the Spark history server:



1. On the Spark history page, click Stop history server.











 Opening the Spark history server Web UI
Data skipping for Spark SQL 

Data skipping can significantly boost the performance of SQL queries by skipping over irrelevant data objects or files based on a summary metadata associated with each object.

Data skipping uses the open source Xskipper library for creating, managing and deploying data skipping indexes with Apache Spark. See [Xskipper - An Extensible Data Skipping Framework](https://xskipper.io).

For more details on how to work with Xskipper see:



* [Quick Start Guide](https://xskipper.io/getting-started/quick-start-guide/)
* [Demo Notebooks](https://xskipper.io/getting-started/sample-notebooks/)



In addition to the open source features in Xskipper, the following features are also available:



* [Geospatial data skipping](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-data-skippinggeospatial-skipping)
* [Encrypting indexes](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-data-skippingencrypting-indexes)
* [Samples showing these features](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-data-skippingsamples)





 Geospatial data skipping 

You can also use data skipping when querying geospatial data sets using [geospatial functions](https://www.ibm.com/support/knowledgecenter/en/SSCJDQ/com.ibm.swg.im.dashdb.analytics.doc/doc/geo_functions.html) from the [spatio-temporal library](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-geospatial-geotemporal-lib).



* To benefit from data skipping in data sets with latitude and longitude columns, you can collect the min/max indexes on the latitude and longitude columns.
ibmcloud analytics-engine-v3 spark-app list --instance-id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --state=finished --limit=25





 Example output 

The output is as follows:

{
"applications" : [ {
"id" : "db933645-0b68-4dcb-80d8-7b71a6c8e542",
"spark_application_id" : "app-20211009103247-0000",
"spark_application_name" : "PythonWordCount",
"state" : "finished",
"submission_time" : "2021-04-21T04:23:50Z",
"start_time" : "2021-04-21T04:24:01Z",
"end_time" : "2021-04-21T04:25:18Z",
"finish_time" : "2021-04-21T04:25:18Z",
"auto_termination_time" : "2021-04-24T04:24:01Z"
}, {
"id" : "a2a2b23f-0929-4c49-9cc0-bd4c2bf953d9",
"spark_application_id" : "app-20211009103147-0000",
"spark_application_name" : "PythonWordCount",
"state" : "finished",
"submission_time" : "2021-04-21T04:21:50Z",
"start_time" : "2021-04-21T04:22:01Z",
"end_time" : "2021-04-21T04:23:18Z",
"finish_time" : "2021-04-21T04:23:18Z",
"auto_termination_time" : "2021-04-24T04:22:01Z"
} ],
"auto_termination_time" : "2022-02-24T07:37:47Z"
}







 ibmcloud analytics-engine-v3 history-server show 

Get the details of the Spark history server of the given Analytics Engine instance.

ibmcloud analytics-engine-v3 history-server show [--instance-id INSTANCE-ID]



 Command options 

-i, --instance-id (string)
: The ID of the Analytics Engine instance to which the Spark history server belongs. When specified, it overrides the target instance ID if it was set. Required, if the target instance was not set.

The value must match the regular expression /u0008[0-9a-f]{8}u0008-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-u0008[0-9a-f]{12}u0008/.





 Examples 

ibmcloud analytics-engine-v3 history-server show --output json

ibmcloud analytics-engine-v3 history-server show --instance-id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --output json





 Example output 

The output is as follows:

{
"state" : "started",
"cores" : "1",
"memory" : "4G",
"start_time" : "2022-02-21T07:37:47Z",
"auto_termination_time" : "2022-02-24T07:37:47Z"
}







 ibmcloud analytics-engine-v3 history-server stop 

Stop the Spark history server of the given Analytics Engine instance.

ibmcloud analytics-engine-v3 history-server stop [--instance-id INSTANCE-ID]



 Command options 

-i, --instance-id (string)
: The ID of the Analytics Engine instance to which the Spark history server belongs.
Service Endpoints: public
Allow Cleanup: false
Locked: false
Created at: 2022-01-03T08:40:25Z
Updated at: 2022-01-03T08:40:26Z
Last Operation:
Status create in progress
Message Started create instance operation
Show more
2. Check the status of the Analytics Engine service:

Action
: Enter:

ibmcloud ae-v3 instance show –id INSTANCE_ID

Parameter values:



* INSTANCE_ID: The value of GUID from the response the of Analytics Engine instance creation call



Example
: Enter:

ibmcloud ae-v3 instance show –id 181ea9ee01b

Response
: The example returns:

{
"default_runtime": {
"spark_version": "3.1" },
"id": "181ea9ee01b ",
"instance_home": {
"bucket": "do-not-delete-ae-bucket-e965d-b7a82",
"endpoint": "https://s3.direct.us-south.cloud-object-storage.appdomain.cloud",
"hmac_access_key": "",
"hmac_secret_key": "",
"provider": "ibm-cos",
"region": "us-south",
"type": "objectstore" },
"state": "active",
"state_change_time": ""
}

Only submit your Spark application when the state of the Analytics Engine service is active.







 Step 3: Upload and submit a Spark application 

Upload an application file to Cloud Object Storage and submit a Spark application.

This tutorial shows you how to add the Spark application to the Cloud Object Storage instance bucket that is used as instance home by the Analytics Engine instance.
ibmcloud fn trigger create MyMessageHubTrigger -f myMessageHub/messageHubFeed -p topic mytopic -p isJSONData true







 Listening for messages 

After a trigger is created, the system monitors the specified topic in your messaging service. When new messages are posted, the trigger is fired.

The payload of that trigger contains a messages field, which is an array of messages that are posted from the last time the trigger is fired. Each message object in the array contains the following fields:



* topic
* partition
* offset
* key
* value



In Kafka terms, the fields are self-evident. However, key has a feature that is called isBinaryKey that allows the key to transmit binary data. Additionally, the value requires special consideration. Fields isJSONData and isBinaryValue are available to handle JSON and binary messages. These fields, isJSONData, and isBinaryValue, cannot be used together.

As an example, if isBinaryKey was set to true when the trigger was created, the key is encoded as a Base64 string when returned from they payload of a fired trigger.

If a key of Some key is posted with isBinaryKey set to true, the trigger payload resembles the following example.

{
"messages": [
{
"partition": 0,
"key": "U29tZSBrZXk=",
"offset": 421760,
"topic": "mytopic",
"value": "Some value"
}
]
}

If the isJSONData parameter was set to false (or not set at all) when the trigger was created, the value field is the raw value of the posted message. However, if isJSONData was set to true when the trigger was created, the system attempts to parse this value as a JSON object, on a best-effort basis. If parsing is successful, then the value in the trigger payload is the resulting JSON object.