vania-janet/multiturn-rag-retrieval-data huggingface dataset:
[the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) 
 Invoke application management REST API to retrieve the state of a Spark application ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) 
 Invoke instance management REST API to enable or disable logging ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) 
 Invoke instance management REST API to view logging configuration !
"max_cores" : "150",
"max_memory" : "600G"
}









 Spark application commands 

With the Spark application CLI commands, you can perform the following operations:



* [Submit a Spark application](https://cloud.ibm.com/docs/cli?topic=cli-CLI_analytics_engineanalytics-engine-v3-cli-spark-app-submit-command)
* [List all applications submitted](https://cloud.ibm.com/docs/cli?topic=cli-CLI_analytics_engineanalytics-engine-v3-cli-spark-app-list-command)
* [Retrieve the details of a Spark application](https://cloud.ibm.com/docs/cli?topic=cli-CLI_analytics_engineanalytics-engine-v3-cli-spark-app-show-command)
* [Show the status of a submitted application](https://cloud.ibm.com/docs/cli?topic=cli-CLI_analytics_engineanalytics-engine-v3-cli-spark-app-status-command)
* [Stop a submitted Spark application](https://cloud.ibm.com/docs/cli?topic=cli-CLI_analytics_engineanalytics-engine-v3-cli-spark-app-stop-command)



ibmcloud analytics-engine-v3 spark-app --help



 ibmcloud analytics-engine-v3 spark-app submit 

Deploys a Spark application on a given serverless Spark instance.

ibmcloud analytics-engine-v3 spark-app submit [--instance-id INSTANCE-ID] [--app APP] [--runtime RUNTIME] [--jars JARS] [--packages PACKAGES] [--repositories REPOSITORIES] [--files FILES] [--archives ARCHIVES] [--name NAME] [--class CLASS] [--arg ARG] [--conf CONF] [--env ENV]



 Command options
"state": "active",
"state_change_time": ""
}

Only submit your Spark application when the state of the Analytics Engine service is active.







 Step 3: Upload and submit a Spark application 

Upload an application file to Cloud Object Storage and submit a Spark application.

This tutorial shows you how to add the Spark application to the Cloud Object Storage instance bucket that is used as instance home by the Analytics Engine instance. If you want to separate the instance related files from the files you use to run your applications, for example the applications files themselves, data files, and any results of your analysis, you can use a different bucket in the same Cloud Object Storage instance or use a different Cloud Object Storage instance.



1. Upload the Spark application file:

Action
: Enter:

ibmcloud cos upload --bucket BUCKET_NAME --key KEY --file PATH [--concurrency VALUE] [--max-upload-parts PARTS] [--part-size SIZE] [--leave-parts-on-errors] [--cache-control CACHING_DIRECTIVES] [--content-disposition DIRECTIVES] [--content-encoding CONTENT_ENCODING] [--content-language LANGUAGE] [--content-length SIZE] [--content-md5 MD5] [--content-type MIME] [--metadata STRUCTURE] [--region REGION] [--output FORMAT] [--json]

Parameter values:



* BUCKET_NAME: Name of bucket used when bucket was created
* KEY: Application file name
* PATH: file name and path to the Spark application file



Example
: Enter:

ibmcloud cos upload --bucket test-cos-storage-bucket --key test-math.py --file test-math.py

Sample application file
: Sample of test-math.py:

from pyspark.sql import SparkSession
import time
import random
import cmath

def init_spark():
spark = SparkSession.builder.appName("test-math").getOrCreate()
sc = spark.sparkContext
Currently, you can monitor Spark applications in the following ways:



* By tracking the state of the Spark application. For details, see [Getting the state of a submitted application](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-spark-app-rest-apispark-app-status).
* By viewing the Spark history events that are forwarded to the IBM Cloud Object Storage instance that you specified as instance home. You can download these events and view them in the Spark history server UI installed on your desktop. At a later stage, you will be able to launch Spark history server UI from within the IBM Analytics Engine service instance details UI page.



* How do I set up autoscaling policies for my serverless instance?

You can enable autoscaling for all applications at instance level at the time you create an instance of the Analytics Engine Standard serverless plan for Apache Spark or per application at the time you submit the application. For details, see [Enabling application autoscaling](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-appl-auto-scaling).
* Can I connect to a serverless instance with the Apache Livy API?

Yes, the IBM Analytics Engine Standard serverless plan for Apache Spark provides an API interface similar to Livy batch API. For details, see [Livy API](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-livy-api).
* Where can I find the logs for my Spark applications?

You can aggregate the logs from your Spark applications to Log Analysis. For details, see [Configuring and viewing logs](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-viewing-logs).
* How can I track actions performed by users on a serverless Spark instance?

You can use the Activity Tracker service to track how users and applications interact with IBM Analytics Engine in IBM Cloud®. You can use this service to investigate abnormal activity and critical actions and to comply with regulatory audit requirements. In addition, you can be alerted about actions as they happen.
"spark.hive.metastore.client.auth.mode":"PLAIN",
            "spark.hive.metastore.use.SSL":"true",
            "spark.hive.stats.autogather":"false",
            "spark.hive.metastore.client.plain.username":"<CHANGEME-CRN-DATA-ENGINE-INSTANCE>",
 for spark 3.3
"spark.hive.metastore.truststore.path":"/opt/ibm/jdk/lib/security/cacerts",
 for spark 3.1, spark 3.2
"spark.hive.metastore.truststore.path":"file:///opt/ibm/jdk/jre/lib/security/cacerts",
            "spark.sql.catalogImplementation":"hive",
            "spark.sql.hive.metastore.jars":"/opt/ibm/connectors/data-engine/hms-client/",
            "spark.sql.hive.metastore.version":"3.0",
            "spark.sql.warehouse.dir":"file:///tmp",
            "spark.sql.catalogImplementation":"hive",
            "spark.hadoop.metastore.catalog.default":"spark"
},
        "application": "cos://mybucket.ALIAS NAME/select_query_data_engine.py"
    }
}
Show more

Parameter values:



* ALIAS NAME: specify the data engine endpoint for your region. For more information on the currently supported data engine endpoints, see [Data engine endpoints](https://cloud.ibm.com/docs/sql-query?topic=sql-query-overviewendpoints).
* THRIFT URL: specify the region-specific thrift URL. For example, thrift://catalog.us.dataengine.cloud.ibm.com:9083.
"crn": "crn:v1:staging:public:ibmanalyticsengine:us-south:a/d628eae2cxxxxx229f2xxxx:79c757e9-xxxx-xxxx-xxxx-dff7f01xxxx:resource-key:2278xxxx-xxxx-xxxx-xxxx-59a6da6xxxxe",
"state": "active",
"account_id": "d628eae2cxxxxx229f2xxxx",
"resource_group_id": "65828fxxxx594594816e872xxxx",
"resource_id": "18dexxxx-xxxx-xxxx-xxxx-0ed2f93fxxxx",
"credentials": {
"apikey": "xxxxx",
"endpoints": {
"applications_api": "https://api-staging.us-south.ae.cloud.ibm.com/v3/analytics_engines/79c757e9-xxxx-xxxx-xxxx-dff7f01xxxx/spark_applications",
"instance_api": "https://api-staging.us-south.ae.cloud.ibm.com/v3/analytics_engines/79c757e9-xxxx-xxxx-xxxx-dff7f01xxxx",
"history_server_ui": "https://spark-console.us-south.ae.cloud.ibm.com/v3/analytics_engines/79c757e9-xxxx-xxxx-xxxx-dff7f01xxxx/spark_history_ui",
"history_server_api":"https://spark-console.us-south.ae.cloud.ibm.com/v3/analytics_engines/79c757e9-xxxx-xxxx-xxxx-dff7f01xxxx/spark_history_api/v1"
},
If the status of the Spark history server is set to Started, you can also click View Spark history to launch the Web UI of the Spark history server in a new browser tab.
2. To start the Spark history server:



1. On the Spark history page, click Start history server.
2. Choose the server configuration and click Start to start the Spark history server.



3. To stop the Spark history server:



1. On the Spark history page, click Stop history server.











 Opening the Spark history server Web UI 

You can open the Spark history Web UI by opening the instance details page of your Analytics Engine service instance, switching to the Spark history tab and clicking View Spark history.

Alternatively, the Spark history server Web UI URL can be obtained through a service endpoint that is made available to you as a service key (also known as a service credential). See [Retrieving service endpoints](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-retrieve-endpoints-serverless).

Ensure that the Spark history server is running before you open the Web UI.

Log links under the Stages and Executors tabs of the Spark history server UI will not work as logs are not preserved with the Spark events. To review the task and executor logs, enable platform logging. For details, see [Configuring and viewing logs](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-viewing-logs).





 Accessing the Spark history server REST API 

In addition to the web UI, the Spark history server also provides a REST API which can be queried to view the Spark events generated by your Spark applications. The Spark history server REST API is available as a service endpoint in a service key (also known as service credential). See [Retrieving service endpoints](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-retrieve-endpoints-serverless).

When you invoke the Spark history server REST API, you must specify your IAM token as a bearer token in the Authorization header. E.g.
: You can now limit the number of applications returned by the Analytics Engine serverless REST API endpoint, SDK method and CLI command for [listing applications](https://cloud.ibm.com/apidocs/ibm-analytics-engine/ibm-analytics-engine-v3list-applications). Use the query parameter limit to specify the number of applications to be returned and specify the value of next.start or previous.start from the API response as the value of the start query parameter to fetch the next or previous page of the results. The applications are listed in descending order based on the submission time, with the newest application being the first.

The pagination is an optional feature in this release. From the next release of the service, the results will be paginated by default.







 05 January 2023 

Analyze application runs on the Spark history server
: You can now run a Spark history server on your IBM Analytics Engine serverless instance.

The Spark history server provides a Web UI to view Spark events that were forwarded to the Object Storage bucket that was defined as the instance home. The Web UI helps you analyze how your Spark applications ran by displaying useful information like:



* A list of the stages that the application goes through when it is run
* The number of tasks in each stage
* The configuration details such as the running executors and memory usage



You are charged for the CPU cores and memory consumed by the Spark history server while it is running. The rate is $0.1475 USD per virtual processor core hour and $0.014 USD per gigabyte hour.

See [Use the Spark history server](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-spark-history-serverless).





 September 2022 



 21 September 2022 

Support for Spark 3.3
: You can now provision IBM® Analytics Engine severless plan instances with the default Spark runtime set to Spark 3.3, which enables you to run Spark applications on Spark 3.3.





 09 September 2022 

You can now use Hive metastore to manage the metadata related to your applications tables, columns, and partition information when working with Spark SQL.
3. Submit the Spark application:

Action
: Enter:

ibmcloud ae-v3 spark-app submit --instance-id INSTANCE_ID -–app APPLICATION_PATH

Parameter values:



* INSTANCE_ID: The value of GUID from the response the of Analytics Engine instance creation call
* APPLICATION_PATH: The file name and path to the Spark application file



Example for IOS and Linux
: Enter:

ibmcloud ae-v3 spark-app submit --instance-id 181ea9ee01b --app "cos://test-cos-storage-bucket.mycos/test-math.py" --conf '{"spark.hadoop.fs.cos.mycos.endpoint": "https://s3.direct.us-south.cloud-object-storage.appdomain.cloud", "spark.hadoop.fs.cos.mycos.access.key": "21bf1f4", "spark.hadoop.fs.cos.mycos.secret.key": "c5ad3e0a6c"}'

Example for Windows (Not Powershell). Note that on Windows, the quotes needs to be escaped.
: Enter:

ibmcloud ae-v3 spark-app submit --instance-id myinstanceid --app "cos://matrix.mycos/test-math.py" --conf "{"spark.hadoop.fs.cos.mycos.endpoint": "https://s3.direct.us-south.cloud-object-storage.appdomain.cloud", "spark.hadoop.fs.cos.mycos.access.key": "mykey", "spark.hadoop.fs.cos.mycos.secret.key": "mysecret"}"

Response
: The example returns:

id 7f7096d2-5c44-4d9a-ac01-b904c7611b7b
state accepted
4. Check the details or status of the application that you submitted:
Currently, you can monitor Spark applications in the following ways:



* By tracking the state of the Spark application. For details, see [Getting the state of a submitted application](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-spark-app-rest-apispark-app-status).
* By viewing the Spark history events that are forwarded to the IBM Cloud Object Storage instance that you specified as instance home. You can download these events and view them in the Spark history server UI installed on your desktop. At a later stage, you will be able to launch Spark history server UI from within the IBM Analytics Engine service instance details UI page.







 How do I set up autoscaling policies for my serverless instance? 

You can enable autoscaling for all applications at instance level at the time you create an instance of the Analytics Engine Standard serverless plan for Apache Spark or per application at the time you submit the application. For details, see [Enabling application autoscaling](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-appl-auto-scaling).





 Can I connect to a serverless instance with the Apache Livy API? 

Yes, the IBM Analytics Engine Standard serverless plan for Apache Spark provides an API interface similar to Livy batch API. For details, see [Livy API](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-livy-api).





 Where can I find the logs for my Spark applications? 

You can aggregate the logs from your Spark applications to Log Analysis. For details, see [Configuring and viewing logs](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-viewing-logs).





 How can I track actions performed by users on a serverless Spark instance? 

You can use the Activity Tracker service to track how users and applications interact with IBM Analytics Engine in IBM Cloud®. You can use this service to investigate abnormal activity and critical actions and to comply with regulatory audit requirements. In addition, you can be alerted about actions as they happen.
* By using the logging API endpoint (deprecated):

curl -X PATCH https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_guid>/logging -H "Authorization: Bearer $TOKEN" -H "content-type: application/json" -d '{"enable":false}'







 Viewing the log forwarding configuration of an IBM Analytics Engine instance 

View the current logging configuration of an IBM Analytics Engine instance, in one of the following ways:



* By using the log_forwarding_config API endpoint:

curl -X GET https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_guid>/log_forwarding_config -H "Authorization: Bearer $TOKEN"
* By using the logging API endpoint (deprecated):

curl -X GET https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_guid>/logging -H "Authorization: Bearer $TOKEN"







 Examples of IBM Analytics Engine log searches in the Log Analysis dashboard 

You can narrow down the search for the platform logs related to IBM Analytics Engine as follows:



1. Click Sources on Log Analysis dashboard and choose ibmanalyticsengine to get the logs from IBM Analytics Engine.
2. To further narrow down your search, you can enter and search for any keywords related to your Spark application in the search window.





 Examples of searches 

This section lists examples of how you can search for logs. The examples all use the following sample values:



* IBM Analytics Engine instance ID: 4308f7a6-2cfd-4d7e-b7a7-678e71a27957
* Spark application ID: 5438585f-6413-4bb2-9251-8162d34f3dc3
* Sources for which log forwarding is enabled: spark-driver and spark-executor
* Associated tags: prod and us-south



The following image shows you an example of the returned search results in the Log Analysis dashboard.
"max_cores" : "150",
"max_memory" : "600G"
}









 Spark application commands 

With the Spark application CLI commands, you can perform the following operations:



* [Submit a Spark application](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-CLI_analytics_engineanalytics-engine-v3-cli-spark-app-submit-command)
* [List all applications submitted](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-CLI_analytics_engineanalytics-engine-v3-cli-spark-app-list-command)
* [Retrieve the details of a Spark application](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-CLI_analytics_engineanalytics-engine-v3-cli-spark-app-show-command)
* [Show the status of a submitted application](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-CLI_analytics_engineanalytics-engine-v3-cli-spark-app-status-command)
* [Stop a submitted Spark application](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-CLI_analytics_engineanalytics-engine-v3-cli-spark-app-stop-command)



ibmcloud analytics-engine-v3 spark-app --help



 ibmcloud analytics-engine-v3 spark-app submit 

Deploys a Spark application on a given serverless Spark instance.

ibmcloud analytics-engine-v3 spark-app submit [--instance-id INSTANCE-ID] [--app APP] [--runtime RUNTIME] [--jars JARS] [--packages PACKAGES] [--repositories REPOSITORIES] [--files FILES] [--archives ARCHIVES] [--name NAME] [--class CLASS] [--arg ARG] [--conf CONF] [--env ENV]
ibmcloud ae-v3 instance show –id 181ea9ee01b

Response
: The example returns:

{
"default_runtime": {
"spark_version": "3.1" },
"id": "181ea9ee01b ",
"instance_home": {
"bucket": "do-not-delete-ae-bucket-e965d-b7a82",
"endpoint": "https://s3.direct.us-south.cloud-object-storage.appdomain.cloud",
"hmac_access_key": "",
"hmac_secret_key": "",
"provider": "ibm-cos",
"region": "us-south",
"type": "objectstore" },
"state": "active",
"state_change_time": ""
}

Only submit your Spark application when the state of the Analytics Engine service is active.
3. Submit the Spark application:

Action
: Enter:

ibmcloud ae-v3 spark-app submit --instance-id INSTANCE_ID -–app APPLICATION_PATH

Parameter values:



* INSTANCE_ID: The value of GUID from the response the of Analytics Engine instance creation call
* APPLICATION_PATH: The file name and path to the Spark application file



Example for IOS and Linux
: Enter:
ibmcloud ae-v3 spark-app submit --instance-id myinstanceid --app "cos://matrix.mycos/test-math.py" --conf "{"spark.hadoop.fs.cos.mycos.endpoint": "https://s3.direct.us-south.cloud-object-storage.appdomain.cloud", "spark.hadoop.fs.cos.mycos.access.key": "mykey", "spark.hadoop.fs.cos.mycos.secret.key": "mysecret"}"

Response
: The example returns:

id 7f7096d2-5c44-4d9a-ac01-b904c7611b7b
state accepted
4. Check the details or status of the application that you submitted:

Action
: Enter:

ibmcloud ae-v3 spark-app show --instance-id INSTANCE_ID --app-id APPLICATION_ID

Parameter values:



* INSTANCE_ID: The value of GUID from the response the of Analytics Engine creation call
* APPLICATION_ID: The value of id from the response of the spark-app submit call



Example
: Enter:

ibmcloud ae-v3 spark-app show --instance-id 181ea9ee01b --app-id 7f7096d2-5c44-4d9a-ac01-b904c7611b7b

Response
: The example returns:

application_details <Nested Object>
id 7f7096d2-5c44-4d9a-ac01-b904c7611b7b
state finished
start_time 2022-03-01T12:58:54.000Z
finish_time 2022-03-01T13:09:14.000Z

The application might take between 2 to 5 minutes to complete.







 Step 4: Create logging service to see logs 

You can use use the Analytics Engine CLI to enable logging to help you troubleshoot issues in IBM Analytics Engine.
The SparkUiUrl property in response will have a non-null value when the UI is available for serverless Spark instance.





 List the details of a job 

To list the job details for a particular Spark batch job enter:

curl -H 'Authorization: Bearer <TOKEN>' -H 'Content-Type: application/json' -X GET https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance-id>/livy/batches/<batch-id>

The response body for listing the job details:



Table 3. Response body for listing job details

 Name Description Type 

 id The batch ID int 
 appId The Spark application ID string 
 appInfo Detailed application information map of key=val 
 state State of submitted batch job string 



An example:

curl -i -X GET https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/43f79a18-768c-44c9-b9c2-b19ec78771bf/livy/batches/14 -H 'content-type: application/json' -H "Authorization: Bearer $TOKEN"

Sample response:

{
"id": 14,
"appId": "app-20201213175030-0000",
"appInfo": {
"sparkUiUrl": null
},
"state": "success"
}

The SparkUiUrl property in response will have a non-null value when the UI is available for serverless Spark instance.





 Get job state 

To get the state of your submitted job enter:

curl -H 'Authorization: Bearer <TOKEN>' -H 'Content-Type: application/json' -X GET https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance-id>/livy/batches/<batch-id>/state

The response body for getting the state of the batch job:



Table 4. Response body for getting state of batch job

 Name Description Type