vania-janet/multiturn-rag-retrieval-data huggingface dataset:
You use IAM access policies to invite users to collaborate on your instance and grant them the necessary privileges. See [Granting permissions to users](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-grant-permissions-serverless).
* How do I define the size of the cluster to run my Spark application?

You can specify the size of the cluster either at the time the instance is created or when submitting Spark applications. You can choose the CPU and memory requirements of your Spark driver and executor, as well the number of executors if you know those requirements up-front. Alternatively, you can choose to let the IBM Analytics Engine service autoscale the Spark cluster based on the application's demand. To override default Spark configuration settings at instance creation or when submitting an application, see [Default Spark configuration](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-serverless-architecture-conceptsdefault-spark-config). For details on autoscaling, [Enabling application autoscaling](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-appl-auto-scaling).
* How do I install custom libraries to my serverless instance?

You can use custom libraries in Python, R, Scala or Java and make them available to your Spark application by creating a library set and referencing it in your application at the time you submit the Spark application. See [Creating a library set](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-create-lib-set).
* How can the serverless instance be monitored?

Currently, you can monitor Spark applications in the following ways:



* By tracking the state of the Spark application. For details, see [Getting the state of a submitted application](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-spark-app-rest-apispark-app-status).
* By viewing the Spark history events that are forwarded to the IBM Cloud Object Storage instance that you specified as instance home.
Use start together with limit to fetch the next or previous page of results.

ibmcloud analytics-engine-v3 spark-app list [--instance-id INSTANCE-ID] [--state STATE] [--limit LIMIT] [--start START]



 Command options 

-i, --instance-id (string)
: The identifier of the Analytics Engine instance associated with the Spark application(s). Required.

The value must match regular expression /u0008[0-9a-f]{8}u0008-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-u0008[0-9a-f]{12}u0008/.

--state ([]string)
: List of Spark application states that will be used to filter the response.

Allowable list items are: finished, running, failed, accepted, stopped, auto_terminated, ops_terminated.

--limit (int64)
: Number of application entries to be included in the response.

The maximum value is 1000. The minimum value is 1.

--start (string)
: Token used to fetch the next or the previous page of the applications list.

--all-pages (bool)
: Invoke multiple requests to display all pages of the collection for spark-app-list.





 Examples 

ibmcloud analytics-engine-v3 spark-app list --state=finished --limit=25

ibmcloud analytics-engine-v3 spark-app list --instance-id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --state=finished --limit=25





 Example output 

The output is as follows:

{
"applications" : [ {
"id" : "db933645-0b68-4dcb-80d8-7b71a6c8e542",
"spark_application_id" : "app-20211009103247-0000",
"log_server" : {
"type" : "ibm-log-analysis"
},
"enabled" : true
}







 ibmcloud analytics-engine-v3 log-forwarding-config show 

Retrieve the log forwarding configuration of the Analytics Engine instance.

ibmcloud analytics-engine-v3 log-forwarding-config show [--instance-id INSTANCE-ID]



 Command options 

-i, --instance-id (string)
: ID of the Analytics Engine instance. When specified, it overrides the target instance ID if it was set. Required, if the target instance was not set.

The value must match the regular expression /u0008[0-9a-f]{8}u0008-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-u0008[0-9a-f]{12}u0008/.





 Examples 

ibmcloud analytics-engine-v3 log-forwarding-config show --output json

ibmcloud analytics-engine-v3 log-forwarding-config show --instance-id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --output json





 Example output 

The output is as follows:

{
"sources" : [ "spark-driver", "spark-executor" ],
"log_server" : {
"type" : "ibm-log-analysis"
},
"enabled" : true
}









 Spark history server 

Use these commands to start or stop the Spark history server of your instance.

ibmcloud analytics-engine-v3 history-server --help



 ibmcloud analytics-engine-v3 history-server start 

Start the Spark history server for the given Analytics Engine instance.

ibmcloud analytics-engine-v3 history-server start [--instance-id INSTANCE-ID]



 Command options
Smart Document Understanding 

Smart Document Understanding (SDU) trains IBM Watson™ Discovery to extract custom fields in your documents. Customizing how your documents are indexed into Discovery improves the answers that your application returns.

With SDU, you annotate fields within your documents to train custom conversion models. As you annotate, Watson is learning and starts to predict annotations. SDU models can be exported and used on other collections.

This documentation applies to Discovery service instances that you create with a Lite or an Advanced plan, or that you created with a Premium plan before 16 July 2020. For more information about features in Premium plan instances created on or after 16 July 2020, and in Plus (including Plus Trial) plan instances, see [these docs](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-configuring-fields).



 Supported document types and browsers 

Supported document types for Smart Document Understanding:



* Lite plans: PDF, Word, PowerPoint, Excel, JSON*, HTML*
* Advanced plans: PDF, Word, PowerPoint, Excel, PNG**, TIFF**, JPG**, JSON*, HTML*



* JSON and HTML documents are supported by IBM Watson™ Discovery, but can not be edited using the SDU editor. To change the configuration of HTML and JSON docs, you need to use the API. For more information, see the [API reference](https://cloud.ibm.com/apidocs/discovery/).

** Individual image files (PNG, TIFF, JPG) are scanned and the text (if any) is extracted. PNG, TIFF, and JPEG images embedded in PDF, Word, PowerPoint, and Excel files are also scanned and the text, if any, is extracted.





 Using the Smart Document Understanding editor 

The SDU editor is only available for collections that contain supported document types and do not have the Element Classification enrichment applied. If you do not want to use the SDU editor, you can set up your configuration using the API, see the [API reference](https://cloud.ibm.com/apidocs/discovery/).
authenticator = IAMAuthenticator(apikey=IAM_API_KEY, url=API_AUTH_URL)

 Construct the service client.
ibm_analytics_engine_api_service = IbmAnalyticsEngineApiV3(authenticator=authenticator)

 Set our custom service URL
ibm_analytics_engine_api_service.set_service_url(IAE_ENDPOINT_URL)

 Service operations can now be invoked using the "ibm_analytics_engine_api_service" variable.

Show more
* Retrieve the details of a single instance:

get_instance(self,
instance_id: str,
kwargs
) -> DetailedResponse

Example request:

instance = ibm_analytics_engine_api_service.get_instance(
instance_id='dc0e9889-eab2-4t9e-9441-566209499546'
).get_result()
print(json.dumps(instance, indent=2))
* Deploy a Spark application on a given serverless Spark instance:

create_application(self,
instance_id: str,
,
application_details: 'ApplicationRequestApplicationDetails' = None,
kwargs
) -> DetailedResponse

Example request:

application_request_application_details_model = {
'application': '/opt/ibm/spark/examples/src/main/python/wordcount.py',
'arguments': ['/opt/ibm/spark/examples/src/main/resources/people.txt']
}
create_application_response = ibm_analytics_engine_api_service.create_application(
instance_id='dc0e9889-eab2-4t9e-9441-566209499546',
application_details=application_request_application_details_model
).get_result()
"document": true,
"targets":
"IBM",
"Watson"
]

},
"emotion": {
"document": true,
"targets":
"IBM",
"Watson"
]
},
"categories": {},
"concepts": {
"limit": 8
},
"semantic_roles": {
"entities": true,
"keywords": true,
"limit": 50
},
"relations": {
"model": "WKS-model-id"
}
}
}
}
]
Show more

Discovery supports adding Natural Language Understanding and Element Classification enrichments. Each field that you want to enrich is defined by an object in the enrichments array. Each enrichment object requires a source_field, a destination_field and enrichments to specified.

The Element Classification enrichment is deprecated and will no longer be available, effective 10 July 2020.



* "enrichment" : string - required - The type of enrichment to use on this field. To extract Natural Language Understanding enrichments use natural_language_understanding, to perform Element Classification use elements.

When you use the elements enrichment, it is important to follow the guidelines specified in [Element Classification](https://cloud.ibm.com/docs/discovery?topic=discovery-element-classification) documentation. Specifically, only PDF files can be ingested when this enrichment is specified.
* "source_field" : string - required - The source field to be enriched. This field must exist in your source after the json_normalizations operation completes.
* "destination_field" : string - required - The name of the container object where enrichments are created.

Field names defined in your configuration must meet the restrictions defined in [Field Name Requirements](https://cloud.ibm.com/docs/discovery?topic=discovery-configreffield_reqs).





 Element Classification enrichments 

The Element Classification enrichment is deprecated and will no longer be available, effective 10 July 2020.
Before you can submit a Spark application, you need to get authentication credentials and set the correct permissions on the Analytics Engine serverless instance.



1. You need the GUID of the service instance you noted down when you provisioned the instance. If you didn't make a note of the GUID, see [Retrieving the GUID of a serverless instance](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-retrieve-instance-details).
2. You must have the correct permissions to perform the required operations. See [User permissions](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-grant-permissions-serverless).
3. The Spark application REST APIs use IAM based authentication and authorization.







 Submitting a Spark application 

Analytics Engine Serverless provides you with a REST interface to submit Spark applications. The payload passed to the REST API maps to various command-line arguments supported by the spark-submit command. See [Parameters for submitting Spark applications](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-spark-app-rest-apispark-submit-parms) for more details.

When you submit a Spark application, you need to reference the application file. To help you to get started quickly and learn how to use the AE serverless Spark APIs, this section begins with an example that uses pre-bundled Spark application files that are referenced in the submit application API payload. The subsequent section shows you how to run applications that are stored in an Object Storage bucket.



 Referencing pre-bundled files 

The provided sample application show you how to reference a .py word count application file and a data file in a job payload.

To learn how to quickly get started using pre-bundled sample application files:



1. Generate an IAM token if you haven’t already done so. See [Retrieving IAM access tokens](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-retrieve-iam-token-serverless).
2.
[the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) 
 Invoke instance management REST API to view Spark history server state ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) 
 View Spark History UI ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) !
You use IAM access policies to invite users to collaborate on your instance and grant them the necessary privileges. See [Granting permissions to users](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-grant-permissions-serverless).





 How do I define the size of the cluster to run my Spark application? 

You can specify the size of the cluster either at the time the instance is created or when submitting Spark applications. You can choose the CPU and memory requirements of your Spark driver and executor, as well the number of executors if you know those requirements up-front. Alternatively, you can choose to let the IBM Analytics Engine service autoscale the Spark cluster based on the application's demand. To override default Spark configuration settings at instance creation or when submitting an application, see [Default Spark configuration](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-serverless-architecture-conceptsdefault-spark-config). For details on autoscaling, [Enabling application autoscaling](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-appl-auto-scaling).





 How do I install custom libraries to my serverless instance? 

You can use custom libraries in Python, R, Scala or Java and make them available to your Spark application by creating a library set and referencing it in your application at the time you submit the Spark application. See [Creating a library set](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-create-lib-set).





 How can the serverless instance be monitored? 

Currently, you can monitor Spark applications in the following ways:



* By tracking the state of the Spark application. For details, see [Getting the state of a submitted application](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-spark-app-rest-apispark-app-status).
* By viewing the Spark history events that are forwarded to the IBM Cloud Object Storage instance that you specified as instance home.
There are different reasons why you might get this notification such as that credentials were deleted after streaming was activated and started.
* message_timed_out or timed_out: This notification reports generic timeout errors when streaming messages to Event Streams.



The Log Analysis event includes different fields that you can use to monitor streaming data failures:



* reason.reasonCode = 500 and severity = critical notify of a failure streaming data.
* reason.reasonType indicates the type of failure.
* target.typeURI = logdna/account/streaming-notification indicates that the event is a notification event.



The following sections include samples of some fields in the notification event that you can use to monitor and be alerted when a failure happens:

"requestData": {
"notificationType": "unknown_topic_or_partition",
"start": "2021-06-29T21:33:52.333Z",
"end": "2021-06-29T21:35:13.772Z",
"topic": "kafka-java-console-sample-topic",
"error": "Error: Broker: Unknown topic or partition"
},

"reason": {
"reasonCode": 500,
"reasonType": "unknown_topic_or_partition",
"reasonForFailure": "Error: Broker: Unknown topic or partition"
},

"target": {
"id": "crn:v1:staging:public:logdna:us-south:a/9bb3ea01633c4d828080de16ce34ea70:d8ba830f-b0e0-4c42-9345-8981d4a94b31:streaming-logs:",
"typeURI": "logdna/account/streaming-notification"



 Query to monitor failures 

In Log Analysis, you can create a view with the following query to filter for event stream failure notifications:

action:logdnaat.streaming-logs.send reason.reasonCode:500 severity:critical target.typeURI:"logdna/account/streaming-notification"
* "parquet.crypto.factory.class": the class implementing EncryptionPropertiesFactory. Set to "com.ibm.parquet.key.management.IBMKeyToolsFactory".

sc.hadoopConfiguration.set("parquet.crypto.factory.class","com.ibm.parquet.key.management.IBMKeyToolsFactory")
* "parquet.encryption.kms.instance.id": The ID of your KeyProtect instance, for example:

sc.hadoopConfiguration.set("parquet.encryption.kms.instance.id" , "27861a9a-6779-4026-bca4-01e59acf0767")
* "parquet.encryption.kms.instance.url": The URL of your KeyProtect instance, for example:

sc.hadoopConfiguration.set("parquet.encryption.kms.instance.url" , "https://<region>.kms.cloud.ibm.com")
* "parquet.encryption.key.access.token": A valid IAM token with access rights to the required keys in your KeyProtect instance, for example:

sc.hadoopConfiguration.set("parquet.encryption.key.access.token" , "<token string>")

If you keep the token in a local file, you can load it.

val token = scala.io.Source.fromFile("<token file>").mkString
sc.hadoopConfiguration.set("parquet.encryption.key.access.token" , token)



2. Specify which columns need to be encrypted, and with which master keys. You must also specify the footer key. In key management by Key Protect, the master key IDs are the IDs of the Key Protect CRKs (customer root keys), that you can find on the IBM Cloud service window. For example:
The Spark UI endpoint of the running Spark application can also be obtained by invoking the following IBM Analytics Engine REST API endpoints or corresponding SDK methods:



* [Retrieve the details of a given Spark application](https://cloud.ibm.com/apidocs/ibm-analytics-engine-v3get-application)
* [List Spark applications](https://cloud.ibm.com/apidocs/ibm-analytics-engine-v3list-applications)



A sample url template for the Spark UI endpoint is as below:

https://spark-console.{REGION}.ae.cloud.ibm.com/v3/analytics_engines/{INSTANCE_ID}/spark_applications/{APPLICATION_ID}/spark_ui

Parameter values:



* REGION: For applications submitted on an instance in the Dallas region, substitute the REGION with us-south and for applications submitted on an instance in the Frankfurt region, substitute the REGION in the endpoint with eu-de.
* INSTANCE_ID: Identifier of the IBM Analytics Engine instance under which the application is running.
* APPLICATION_ID: Identifier of the application for which Spark UI is accessed.







 Accessing the Spark UI REST API 

In addition to the Web UI, Spark UI APIs are also exposed for programmatic consumption.

For example, enter:

curl "https://spark-console.{REGION}.ae.cloud.ibm.com/v3/analytics_engines/{INSTANCE_ID}/spark_applications/{APPLICATION_ID}/spark_ui_api/v1/applications" --header "Authorization: Bearer {IAM TOKEN}"

Parameter values:



* REGION: For applications submitted on an instance in the Dallas region, substitute the REGION with us-south and for applications submitted on an instance in the Frankfurt region, substitute the REGION in the endpoint with eu-de.
* INSTANCE_ID: Identifier of the IBM Analytics Engine instance under which the application is running.
* APPLICATION_ID: Identifier of the application for which Spark UI is accessed.
* IAM TOKEN: Ensure that you have the necessary privileges to access the service instance.
.config("spark.hive.metastore.uris", "thrift://catalog.<region>.sql-query.cloud.ibm.com:9083") 
.config("spark.hive.metastore.use.SSL", "true") 
.config("spark.hive.metastore.truststore.password", "changeit") 
.config("spark.hive.metastore.client.auth.mode", "PLAIN") 
.config("spark.hive.metastore.client.plain.username", '<YourDataengineCRN>') 
.config("spark.hive.metastore.client.plain.password", '<YourAPIkey>') 
.config("spark.hive.execution.engine", "spark") 
.config("spark.hive.stats.autogather", "false") 
.config("spark.sql.warehouse.dir", "file:///tmp") 
 only spark is allowed as the default catalog
.config("metastore.catalog.default", "spark") 
.enableHiveSupport() 
.getOrCreate()
Show more





 Apache Hive metastore version 3.1.2 compatible client 

Download the [Hive-compatible client](https://us.sql-query.cloud.ibm.com/download/catalog/hive-metastore-standalone-client-3.1.2-sqlquery-1.0.13.jar) and place it in a directory of your Apache Spark cluster that is not on the classpath. This step is necessary, as the client is loaded into an isolated classloader to avoid version conflicts. Note that in the examples the files are placed in /tmp/dataengine, when you use a different folder, adjust the example accordingly.

The client differs from the Hive version 3.1.2 release by more enhancements that add support for TLS and authentication through IBM Cloud® Identity and Access Management.
"auto_termination_time" : "2021-04-24T04:24:01Z"
}







 ibmcloud analytics-engine-v3 spark-app stop 

Stops a running application identified by the app_id identifier. This is an idempotent operation. Performs no action if the requested application is already stopped or completed.

ibmcloud analytics-engine-v3 spark-app stop [--instance-id INSTANCE-ID] --app-id APP-ID



 Command options 

-i, --instance-id (string)
: Identifier of the instance to which the application belongs. When specified, it overrides the target instance ID if it was set. Required, if the target instance was not set.

The value must match the regular expression /u0008[0-9a-f]{8}u0008-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-u0008[0-9a-f]{12}u0008/.

--app-id (string)
: Identifier of the application that needs to be stopped. Required.

The value must match the regular expression /u0008[0-9a-f]{8}u0008-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-u0008[0-9a-f]{12}u0008/.





 Examples 

ibmcloud analytics-engine-v3 spark-app stop --app-id=ff48cc19-0e7e-4627-aac6-0b4ad080397b
Sample data in parquet format is inserted from source COS bucket source-bucket into the watsonx.data table yellow_taxi_2022. It also shows ingesting data in CSV format from COS bucket source-bucket into the table zipcode in the database demodb.

You can download the sample formats from:



* [Sample parquet file](https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page).
* [Sample CSV file](https://raw.githubusercontent.com/spark-examples/spark-scala-examples/3ea16e4c6c1614609c2bd7ebdffcee01c0fe6017/src/main/resources/zipcodes.csv)



* Modifying schema in watsonx.data

The Schema evolution section from the [sample python file](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-run_samp_filepython_file) allows you to modify data in watsonx.data.
* Performing table maintenance activities in watsonx.data

Table maintenance helps in keeping the watsonx.data table performant. Iceberg provides table maintenance procedures out of the box that allows performing powerful table optimizations in a declarative fashion. The sample below demonstrates how to do some table maintenance operations by using Spark. For more information about the Iceberg Spark table maintenance operations, see [Table Operations](https://iceberg.apache.org/docs/1.2.1/spark-procedures/).







 Running the sample use case 

Follow the steps to run the Spark sample python file.



 Spark sample python file 

from pyspark.sql import SparkSession
import os

def init_spark():