vania-janet/multiturn-rag-retrieval-data huggingface dataset:
* By using the log_forwarding_config API endpoint:

curl -X PUT https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_guid>/log_forwarding_config -H "Authorization: Bearer $TOKEN" -H "content-type: application/json" -d '{"enabled":true}'
* By using the logging API endpoint (deprecated):

curl -X PUT https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_guid>/logging -H "Authorization: Bearer $TOKEN" -H "content-type: application/json" -d '{"enable":true}



* Customized: With this option, you get more control over the log forwarding configuration. For example, you can specify the sources of the logs you want to get forwarded. You can also choose to associate some tags with the logs, which can later be used in IBM Log Analysis as search keywords to narrow down the search on the applicable logs. You can do this in one of the following ways:



* By using the log_forwarding_config API endpoint:

curl -X PUT https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_guid>/log_forwarding_config -H "Authorization: Bearer $TOKEN" -H "content-type: application/json" -d ' {"enabled":true,"sources": ["spark-driver", "spark-executor"],"tags": ["<tag_1>", "<tag_2>",...]} '
* By using the logging API endpoint (deprecated):
If no IBM Cloud Object Storage instances are found in your account, you can use the REST APIs to update instance home after instance creation.

You can't change instance home after instance creation. You can only edit the access keys.





 Default Spark configuration 

You can specify default Spark configurations at the instance level and let that be inherited by Spark applications created on the instance. This is an optional section that you can specify at the time of instance creation. Values specified as instance level defaults can be overridden at the time of submitting Spark applications.

Currently, the following Apache Spark configurations are supported:



* spark.driver.memory
* spark.driver.cores
* spark.executor.memory
* spark.executor.cores
* spark.driver.defaultJavaOptions
* spark.driver.extraLibaryPathe
* spark.driver.extraClassPath



See [Spark Configuration](https://spark.apache.org/docs/latest/configuration.html) to understand more about the Apache Spark properties.

Additionally, you can also specify default values for autoscaling Spark applications at the instance level. See [Autoscaling application configurations](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-appl-auto-scaling) for details.

The following list shows the default values for Apache Spark configuration settings adapted for IBM Analytics Engine instances:



* "spark.driver.memory": "4G"
* "spark.driver.cores": "1"
* "spark.executor.memory": "4G"
* "spark.executor.cores": "1"
* "spark.eventLog.enabled": "true"



For the default limits and quotas for Analytics Engine instances and the supported Spark driver and executor vCPU and memory combinations, see [Limits and quotas for Analytics Engine instances](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-limits).

When you create an instance you can override the open source default Apache Spark configuration settings.
Direct endpoints provide better performance than public endpoints and do not incur charges for any outgoing or incoming bandwidth.

To submit a Spark application:



1. Create a bucket for your application file. See [Bucket operations](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-compatibility-api-bucket-operations) for details on creating buckets.
2. Add the application file to the newly created bucket. See [Upload an object](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-object-operationsobject-operations-put) for adding your application file to the bucket.
3. Generate an IAM token if you haven’t already done so. See [Retrieving IAM access tokens](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-retrieve-iam-token-serverless).
4. Export the token into a variable:

export token=<token generated>
5. Prepare the payload JSON file. For example, submit-spark-app.json:

{
"application_details": {
"application": "cos://<application-bucket-name>.<cos-reference-name>/my_spark_application.py",
"arguments": ["arg1", "arg2"],
"conf": {
"spark.hadoop.fs.cos.<cos-reference-name>.endpoint": "https://s3.direct.us-south.cloud-object-storage.appdomain.cloud",
"spark.hadoop.fs.cos.<cos-reference-name>.access.key": "<access_key>",
"spark.hadoop.fs.cos.<cos-reference-name>.secret.key": "<secret_key>",
"spark.app.name": "MySparkApp"
}
Go to [Manage > Access (IAM) > Authorizations](https://cloud.ibm.com/iam/authorizations) in the IBM Cloud console.



1. Click the Create button.
2. In the Source service menu, select Data Engine (previously SQL Query).
3. Switch to Resources based on selected attributes, check Source service instance and select the log-analysis-sql Data Engine service instance just created.
4. In the Target service menu, select Key Protect.
5. Switch to Resources based on selected attributes, check Instance ID, select the log-analysis-kp Key Protect service instance created earlier.
6. Enable the Reader role.
7. Click the Authorize button.









 Inspect log data with Analytics Engine 



1. Create an instance of [Analytics Engine](https://cloud.ibm.com/catalog/services/analytics-engine).



1. Select a region.
2. Select the Standard Serverless for Apache Spark plan.
3. Set the Service name to log-analysis-iae.
4. Select the same Resource group as the above service.
5. Select the Spark runtime version.
6. Select the Cloud Object Storage instance created earlier
7. Click Create service credentials
8. Click Create to create the Analytics Engine instance.











 Step 2: Streams landing from Event Streams to Cloud Object Storage 



 Configure stream landing 

In this section, you will learn how to run a fully-managed stream data ingestion from Event Streams into Parquet on IBM Cloud Object Storage. IBM Cloud Data Engine is the key component in the Stream Landing approach. It is the service that connects to Event Streams and copies the data to IBM Cloud Object Storage.

[Parquet](https://parquet.apache.org/docs/) is an open source file format for nested data structures in a flat columnar format. Compared to the traditional approach where data is stored in rows, Parquet is more efficient in terms of storage and performance.



1.
* [Stop a submitted Spark application](https://cloud.ibm.com/docs/analytics-engine-cli-plugin?topic=analytics-engine-cli-plugin-CLI_analytics_engineanalytics-engine-v3-cli-spark-app-stop-command)



ibmcloud analytics-engine-v3 spark-app --help



 ibmcloud analytics-engine-v3 spark-app submit 

Deploys a Spark application on a given serverless Spark instance.

ibmcloud analytics-engine-v3 spark-app submit [--instance-id INSTANCE-ID] [--app APP] [--runtime RUNTIME] [--jars JARS] [--packages PACKAGES] [--repositories REPOSITORIES] [--files FILES] [--archives ARCHIVES] [--name NAME] [--class CLASS] [--arg ARG] [--conf CONF] [--env ENV]



 Command options 

-i, --instance-id (string)
: The identifier of the Analytics Engine instance associated with the Spark application(s). When specified, it overrides the target instance ID if it was set. Required, if the target instance was not set.

The value must match the regular expression /u0008[0-9a-f]{8}u0008-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-u0008[0-9a-f]{12}u0008/.

--app (string)
: Path of the application to run. Required.

--runtime ([Runtime](https://cloud.ibm.com/docs/analytics-engine-cli-plugin?topic=analytics-engine-cli-plugin-CLI_analytics_enginecli-runtime-example-schema))
: Runtime enviroment for applications and other workloads.

--jars (string)
: Path of the jar files containing the application.
ibmcloud analytics-engine-v3 instance show 

Retrieve the details of a single Analytics Engine instance.

ibmcloud analytics-engine-v3 instance show [--id ID]



 Command options 

-i, --id (string)
: GUID of the Analytics Engine service instance to retrieve. When specified, it overrides the target instance ID if it was set. Required if target instance was not set.

The value must match the regular expression /u0008[0-9a-f]{8}u0008-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-u0008[0-9a-f]{12}u0008/.





 Examples 

The following example shows how to use the show command without an instance ID. The requested operation is performed on the target instance.

ibmcloud analytics-engine-v3 instance show --output json

The following example shows how to use the show command with an instance ID that overrides the target instance ID if defined:

ibmcloud analytics-engine-v3 instance show --id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --output json





 Example output 

The output is as follows:

{
"id" : "dc0e9889-eab2-4t9e-9441-566209499546",
"state" : "active",
"state_change_time" : "2021-04-21T04:24:01Z",
"default_runtime" : {
"spark_version" : "3.1"
},
"instance_home" : {
"guid" : "30d5c4a7-9fb7-4712-9039-a79417dec87b",
"provider" : "ibm-cos",
"type" : "objectstore",
"region" : "us-south",
sc.hadoopConfiguration.set("parquet.encryption.key.access.token" , token)



2. Specify which columns need to be encrypted, and with which master keys. You must also specify the footer key. In key management by Key Protect, the master key IDs are the IDs of the Key Protect CRKs (customer root keys), that you can find on the IBM Cloud service window. For example:

val k1 = "d1ae3fc2-6b7d-4a03-abb6-644e02933734"
val k2 = "c4a21521-2a78-4968-a7c2-57c481f58d5c"
val k3 = "a4ae4bc2-9d78-8748-f8a2-17f584d48c5b"

dataFrame.write
.option("parquet.encryption.footer.key" , k1)
.option("parquet.encryption.column.keys" , k2+":SSN,Address;"+k3+":CreditCard")
.parquet("<path to encrypted files>")

Note: If either the "parquet.encryption.column.keys" parameter or the "parquet.encryption.footer.key" parameter is not set, an exception will be thrown.







 Reading encrypted data 

The required metadata, including the ID and URL of the KeyProtect instance, is stored in the encrypted Parquet files.

To read the encrypted metadata:



1. Set the class implementing EncryptionPropertiesFactory:

sc.hadoopConfiguration.set("parquet.crypto.factory.class","com.ibm.parquet.key.management.IBMKeyToolsFactory")
2. Provide the IAM access token for the relevant keys:

sc.hadoopConfiguration.set("parquet.encryption.key.access.token" , "<token string>")
3.
For details, see [Configuring and viewing logs](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-viewing-logs).





 How can I track actions performed by users on a serverless Spark instance? 

You can use the Activity Tracker service to track how users and applications interact with IBM Analytics Engine in IBM Cloud®. You can use this service to investigate abnormal activity and critical actions and to comply with regulatory audit requirements. In addition, you can be alerted about actions as they happen. The events that are collected comply with the Cloud Auditing Data Federation (CADF) standard. See [Auditing events for IBM Analytics Engine serverless instances](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-at_events-serverless).
: Returns the sample standard deviation that is calculated from values of a group.





 str_to_map 

str_to_map(text[, pairDelim[, keyValueDelim]])
: Creates a map after splitting the text into key/value pairs by using delimiters. Default delimiters are ',' for pairDelim and ':' for keyValueDelim.
: Example of an SQL function usage fragment

> SELECT str_to_map('a:1,b:2,c:3', ',', ':')
: Result value

map("a":"1","b":"2","c":"3")
: Example of an SQL function usage fragment

> SELECT str_to_map('a')
: Result value

map("a":null)





 string 

string(expr)
: Casts the value expr to the target data type string.





 struct 

struct(col1, col2, col3, ...)
: Creates a struct with the indicated field values.





 substr 

substr(str, pos[, len])
: Returns the substring of str that starts at pos and is of length len, or the slice of byte array that starts at pos and is of length len.
: Example of an SQL function usage fragment

> SELECT substr('Spark SQL', 5)
: Result value

k SQL
: Example of an SQL function usage fragment

> SELECT substr('Spark SQL', -3)
: Result value

SQL
: Example of an SQL function usage fragment

> SELECT substr('Spark SQL', 5, 1)
: Result value

k





 substring 

substring(str, pos[, len])
: Returns the substring of str that starts at pos and is of length len, or the slice of byte array that starts at pos and is of length len.
: Example of an SQL function usage fragment
"auto_termination_time" : "2022-02-24T07:37:47Z"
}







 ibmcloud analytics-engine-v3 history-server stop 

Stop the Spark history server of the given Analytics Engine instance.

ibmcloud analytics-engine-v3 history-server stop [--instance-id INSTANCE-ID]



 Command options 

-i, --instance-id (string)
: The ID of the Analytics Engine instance to which the Spark history server belongs. When specified, it overrides the target instance ID if it was set. Required, if the target instance was not set.

The value must match the regular expression /u0008[0-9a-f]{8}u0008-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-u0008[0-9a-f]{12}u0008/.





 Examples 

ibmcloud analytics-engine-v3 history-server stop --output json

ibmcloud analytics-engine-v3 history-server stop --instance-id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --output json









 Log configuration 

Note: Deprecated. Use equivalent commands from the [log-forwarding-config](https://cloud.ibm.com/docs/cli?topic=cli-CLI_analytics_engineanalytics-engine-v3-log-forwarding-config-cli) command group instead.

ibmcloud analytics-engine-v3 log-config --help



 ibmcloud analytics-engine-v3 log-config update 

Enable or disable log forwarding from IBM Analytics Engine to IBM Log Analysis server.

Note: Deprecated. Use replace from the log-forwarding-config command group instead.

ibmcloud analytics-engine-v3 log-config update [--instance-id INSTANCE-ID] [--enable ENABLE]



 Command options
ibmcloud analytics-engine-v3 spark-app submit [--instance-id INSTANCE-ID] [--app APP] [--runtime RUNTIME] [--jars JARS] [--packages PACKAGES] [--repositories REPOSITORIES] [--files FILES] [--archives ARCHIVES] [--name NAME] [--class CLASS] [--arg ARG] [--conf CONF] [--env ENV]



 Command options 

-i, --instance-id (string)
: The identifier of the Analytics Engine instance associated with the Spark application(s). When specified, it overrides the target instance ID if it was set. Required, if the target instance was not set.

The value must match the regular expression /u0008[0-9a-f]{8}u0008-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-u0008[0-9a-f]{12}u0008/.

--app (string)
: Path of the application to run. Required.

--runtime ([Runtime](https://cloud.ibm.com/docs/cli?topic=cli-CLI_analytics_enginecli-runtime-example-schema))
: Runtime enviroment for applications and other workloads.

--jars (string)
: Path of the jar files containing the application.

--packages (string)
: Package names.

--repositories (string)
: Repositories names.

--files (string)
: File names.

--archives (string)
: Archive Names.

--name (string)
: Name of the application.

--class (string)
: Entry point for a Spark application bundled as a '.jar' file. This is applicable only for Java or Scala applications.
To compensate this, you can switch off the data integrity verification support and write the encrypted files with the alternative algorithm AES-GCM-CTR, which verifies the integrity of the metadata parts only and not that of the data parts, and has a lower throughput overhead compared to the AES-GCM algorithm.

parameter name: "parquet.encryption.algorithm"
parameter value: "AES_GCM_CTR_V1"
* Plain text footer mode for legacy readers

By default, the main Parquet metadata module (the file footer) is encrypted to hide the file schema and list of sensitive columns. However, you can decide not to encrypt the file footers in order to enable other Spark and Parquet readers (that don't yet support Parquet encryption) to read the unencrypted columns in the encrypted files. To switch off footer encryption, set the following parameter:

parameter name: "parquet.encryption.plaintext.footer"
parameter value: "true"

Important: The "parquet.encryption.footer.key" parameter must also be specified in the plain text footer mode. Although the footer is not encrypted, the key is used to sign the footer content, which means that new readers could verify its integrity. Legacy readers are not affected by the addition of the footer signature.









 Usage examples 

The following sample code snippets for Python and Scala show how to create data frames, written to encrypted parquet files, and read from encrypted parquet files.



* Python: Writing encrypted data

from pyspark.sql import

RowsquaresDF = spark.createDataFrame(
sc.parallelize(range(1, 6))
.map(lambda i: Row(int_column=i, square_int_column=i 2)))

sc._jsc.hadoopConfiguration().set("parquet.crypto.factory.class",
"com.ibm.parquet.key.management.IBMKeyToolsFactory")
ibmcloud analytics-engine-v3 log-config update --instance-id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --enable=true --output json





 Example output 

The output is as follows:

{
"components" : [ "spark-driver", "spark-executor" ],
"log_server" : {
"type" : "ibm-log-analysis"
},
"enable" : true
}







 ibmcloud analytics-engine-v3 log-config show 

Retrieve the logging configuration of a given Analytics Engine instance.

Note: Deprecated. Use show from the log-forwarding-config command group instead.

ibmcloud analytics-engine-v3 log-config show [--instance-id INSTANCE-ID]



 Command options 

-i, --instance-id (string)
: GUID of the Analytics Engine service instance to retrieve log configuration. When specified, it overrides the target instance ID if it was set. Required, if the target instance was not set.

The value must match the regular expression /u0008[0-9a-f]{8}u0008-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-u0008[0-9a-f]{12}u0008/.





 Examples 

ibmcloud analytics-engine-v3 log-config show --output json

ibmcloud analytics-engine-v3 log-config show --instance-id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --output json





 Example output 

The output is as follows:

{
"components" : [ "spark-driver", "spark-executor" ],
"log_server" : {
"type" : "ibm-log-analysis"
},
}









 Spark history server 

Use these commands to start or stop the Spark history server of your instance.

ibmcloud analytics-engine-v3 history-server --help



 ibmcloud analytics-engine-v3 history-server start 

Start the Spark history server for the given Analytics Engine instance.

ibmcloud analytics-engine-v3 history-server start [--instance-id INSTANCE-ID]



 Command options 

-i, --instance-id (string)
: The ID of the Analytics Engine instance to which the Spark history server belongs. When specified, it overrides the target instance ID if it was set. Required, if the target instance was not set.

The value must match the regular expression /u0008[0-9a-f]{8}u0008-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-u0008[0-9a-f]{12}u0008/.





 Examples 

ibmcloud analytics-engine-v3 history-server start --output json

ibmcloud analytics-engine-v3 history-server start --instance-id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --output json





 Example output 

The output is as follows:

{
"state" : "started",
"cores" : "1",
"memory" : "4G",
"start_time" : "2022-02-21T07:37:47Z",
"auto_termination_time" : "2022-02-24T07:37:47Z"
}







 ibmcloud analytics-engine-v3 history-server show 

Get the details of the Spark history server of the given Analytics Engine instance.

ibmcloud analytics-engine-v3 history-server show [--instance-id INSTANCE-ID]



 Command options 

-i, --instance-id (string)
"auto_termination_time" : "2022-02-24T07:37:47Z"
}







 ibmcloud analytics-engine-v3 history-server stop 

Stop the Spark history server of the given Analytics Engine instance.

ibmcloud analytics-engine-v3 history-server stop [--instance-id INSTANCE-ID]



 Command options 

-i, --instance-id (string)
: The ID of the Analytics Engine instance to which the Spark history server belongs. When specified, it overrides the target instance ID if it was set. Required, if the target instance was not set.

The value must match the regular expression /u0008[0-9a-f]{8}u0008-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-u0008[0-9a-f]{12}u0008/.





 Examples 

ibmcloud analytics-engine-v3 history-server stop --output json

ibmcloud analytics-engine-v3 history-server stop --instance-id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --output json









 Log configuration 

Note: Deprecated. Use equivalent commands from the [log-forwarding-config](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-CLI_analytics_engineanalytics-engine-v3-log-forwarding-config-cli) command group instead.

ibmcloud analytics-engine-v3 log-config --help



 ibmcloud analytics-engine-v3 log-config update 

Enable or disable log forwarding from IBM Analytics Engine to IBM Log Analysis server.

Note: Deprecated. Use replace from the log-forwarding-config command group instead.

ibmcloud analytics-engine-v3 log-config update [--instance-id INSTANCE-ID] [--enable ENABLE]
This object contains the following fields:<br><br>id: The ID of the tool integration instance.<br><br>tool_type_id: The ID of the [tool type](https://cloud.ibm.com/docs/ContinuousDelivery?topic=ContinuousDelivery-integrations&interface=apitool_integrations).<br><br>href: The public API endpoint for the tool integration instance.<br><br>state: The state of the tool integration instance.<br><br>referent: An object that contains information about the tool that is represented by the tool integration instance. For example, ui_href which is the UI endpoint for the tool integration that is represented by the tool integration instance.<br><br>name: Optional. The name of the tool integration instance. 
 toolchain.pipeline-run The object that represents the Tekton pipeline run or Classic pipeline stage run where the event originated. This object is applicable only to pipeline events and contains the following fields:<br><br>id: The ID of the Tekton pipeline run or Classic pipeline stage.<br><br>ui_href: The UI endpoint of the Tekton pipeline run or Classic pipeline stage.<br><br>run_number: Optional. The run number of the Tekton pipeline run or the Classic pipeline stage.<br><br>start_time: Optional. The time that the run started, in ISO 8601 format.<br><br>finish_time: Optional. The time that the run finished, in ISO 8601 format.<br><br>duration: Optional. The duration of the run, in ISO 8601 format.
pulse","parameters":{"amp":-0.09850270935496451,-0.0019986417971878563],"beta":-0.7783748646463993,"duration":160,"sigma":40},"pulse_shape":"drag","t0":160},{"ch":"d15","name":"fc","phase":"-(P1)","t0":320},{"ch":"u26","name":"fc","phase":"-(P2)","t0":0},{"ch":"u26","name":"fc","phase":"-(P0)","t0":160},{"ch":"u26","name":"fc","phase":"-(P1)","t0":320},{"ch":"u37","name":"fc","phase":"-(P2)","t0":0},{"ch":"u37","name":"fc","phase":"-(P0)","t0":160},{"ch":"u37","name":"fc","phase":"-(P1)","t0":320}]},{"name":"u3","qubits":16],"sequence":{"ch":"d16","name":"fc","phase":"-(P2)","t0":0},{"