vania-janet/multiturn-rag-retrieval-data huggingface dataset:
'arguments': ['/opt/ibm/spark/examples/src/main/resources/people.txt']
}
create_application_response = ibm_analytics_engine_api_service.create_application(
instance_id='dc0e9889-eab2-4t9e-9441-566209499546',
application_details=application_request_application_details_model
).get_result()
print(json.dumps(create_application_response, indent=2))
* Retrieve all Spark applications run on a given instance:

list_applications(self,
instance_id: str,
kwargs
) -> DetailedResponse

Example request:

list_applications_response = ibm_analytics_engine_api_service.list_applications(
instance_id='dc0e9889-eab2-4t9e-9441-566209499546'
).get_result()
print(json.dumps(list_applications_response, indent=2))
* Retrieve the details of a given Spark application:

get_application(self,
instance_id: str,
application_id: str,
kwargs
) -> DetailedResponse

Example request:

get_application_response = ibm_analytics_engine_api_service.get_application(
instance_id='dc0e9889-eab2-4t9e-9441-566209499546',
application_id='db933645-0b68-4dcb-80d8-7b71a6c8e542'
).get_result()
print(json.dumps(get_application_response, indent=2))
* Stop a running application identified by the app_id identifier. This is an idempotent operation. Performs no action if the requested application is already stopped or completed.
$service'.secret.key": "'$secret_access_key'"
}'

Again verify the results in the platform log.

The final step is to submit the spark application that accesses the data in the same bucket. Create a file, solution.py, with the following contents and upload it to the bucket. Notice the COS_PARQUET environment variable that will be initialized in the next step.

solution.py:

from pyspark.sql import SparkSession
from pyspark.sql import SQLContext
import os
import sys

def main():
 COS_PARQUET in format: cos://<bucket>.<service>/landing_folder/topic=<topic>/jobid=<jobid>/
 like cos://ABC-log-analysis.solution/logs-stream-landing/topic=webserver/jobid=48914a16-1d33-4d3e-93e3-7efb855b662e/
print("solution v1.0")
if "COS_PARQUET" not in os.environ:
print("COS_PARQUET must be in environment")
return 1
cos_parquet = os.environ["COS_PARQUET"]
print(f"cos_parquet: {cos_parquet}")

spark = SparkSession.builder.appName("solution").getOrCreate()
sc = spark.sparkContext
sqlContext = SQLContext(sc)

df = spark.read.parquet(cos_parquet)
sqlContext.registerDataFrameAsTable(df, "Table")
df_query = sqlContext.sql("SELECT * FROM Table LIMIT 10")
df_query.show(10)
return 0

if __name__ == '__main__':
sys.exit(main())
Show more

Now submit the spark application that accesses the data in the same bucket.
encryptedParquetPath = "squares.parquet.encrypted"
parquetFile = spark.read.parquet(encryptedParquetPath)
parquetFile.show()
* Scala: Writing encrypted data

case class SquareItem(int_column: Int, square_int_column: Double)
val dataRange = (1 to 6).toList
val squares = sc.parallelize(dataRange.map(i => new SquareItem(i, scala.math.pow(i,2))))
sc.hadoopConfiguration.set("parquet.crypto.factory.class","com.ibm.parquet.key.management.IBMKeyToolsFactory")
sc.hadoopConfiguration.set("parquet.encryption.key.list","key1: AAECAwQFBgcICQoLDA0ODw==, key2: AAECAAECAAECAAECAAECAA==")
val encryptedParquetPath = "squares.parquet.encrypted"
squares.toDS().write
.option("parquet.encryption.column.keys", "key1:square_int_column")
.option("parquet.encryption.footer.key", "key2")
.parquet(encryptedParquetPath)
* Scala: Reading encrypted data

sc.hadoopConfiguration.set("parquet.crypto.factory.class","com.ibm.parquet.key.management.IBMKeyToolsFactory")
sc.hadoopConfiguration.set("parquet.encryption.key.list","key1: AAECAwQFBgcICQoLDA0ODw==, key2: AAECAAECAAECAAECAAECAA==")
val encryptedParquetPath = "squares.parquet.encrypted"
val parquetFile = spark.read.parquet(encryptedParquetPath)
df2=spark.sql("SELECT * from MYPARQUETBBSPEED")
  df2.show()
def main():
  spark,sc = init_spark()
  generate_and_store_data(spark,sc)
  create_table_from_data(spark,sc)
  time.sleep(30)
if __name__ == '__main__':
  main()
Show more
7. Run the following PySpark script called postgres-parquet-table-select.py to access this Parquet table with metadata from another Spark workload:

from pyspark.sql import SparkSession
import time
def init_spark():
  spark = SparkSession.builder.appName("postgres-select-parquet-table-test").getOrCreate()
  sc = spark.sparkContext
  return spark,sc
def select_data_from_table(spark,sc):
  df=spark.sql("SELECT * from MYPARQUETBBSPEED")
  df.show()
def main():
  spark,sc = init_spark()
  select_data_from_table(spark,sc)
  time.sleep(60)
if __name__ == '__main__':
 main()
Working with Spark SQL and an external metastore 

Spark SQL uses Hive metastore to manage the metadata of a user's applications tables, columns, partition information.

By default, the database that powers this metastore is an embedded Derby instance that comes with the Spark cluster. You could choose to externalize this metastore database to an external data store, like to an IBM Cloud Databases for PostgreSQL or an IBM Cloud Data Engine (previously SQL Query) instance.

Placing your metadata outside of the Spark cluster will enable you to reference the tables in different applications across your IBM Analytics Engine instances. This, in combination with storing your data in IBM Cloud Object Storage, helps persisting data and metadata and allows you to work with this data seamlessly across different Spark workloads.



  Enabling and testing an external metastore with IBM Analytics Engine 

To enable and test an external metastore with IBM Analytics Engine, you need to perform the following steps:



1.  Create a metastore to store the metadata. You can choose to provision either an IBM Cloud Databases for PostgreSQL or an IBM Cloud Data Engine (previously SQL Query) instance.
2.  Configure IBM Analytics Engine to work with the database instance.
3.  Create a table in one Spark application and then access this table from another Spark application.
RowsquaresDF = spark.createDataFrame(
sc.parallelize(range(1, 6))
.map(lambda i: Row(int_column=i, square_int_column=i 2)))

sc._jsc.hadoopConfiguration().set("parquet.crypto.factory.class",
"com.ibm.parquet.key.management.IBMKeyToolsFactory")
sc._jsc.hadoopConfiguration().set("encryption.key.list",
"key1: AAECAwQFBgcICQoLDA0ODw==, key2: AAECAAECAAECAAECAAECAA==")

encryptedParquetPath = "squares.parquet.encrypted"squaresDF.write
.option("parquet.encryption.column.keys", "key1:square_int_column")
.option("parquet.encryption.footer.key", "key2")
.parquet(encryptedParquetPath)
* Python: Reading encrypted data

sc._jsc.hadoopConfiguration().set("parquet.crypto.factory.class",
"com.ibm.parquet.key.management.IBMKeyToolsFactory")
sc._jsc.hadoopConfiguration().set("parquet.encryption.key.list",
"key1: AAECAwQFBgcICQoLDA0ODw==, key2: AAECAAECAAECAAECAAECAA==")

encryptedParquetPath = "squares.parquet.encrypted"
parquetFile = spark.read.parquet(encryptedParquetPath)
parquetFile.show()
* Scala: Writing encrypted data

case class SquareItem(int_column: Int, square_int_column: Double)
val dataRange = (1 to 6).toList
[EnvironmentVariableName]" configuration (application_details > env) also. They will, however, be accessible only to the tasks running on the executor and not the driver.

Example of pyspark application that accesses the environment variables that are passed using the "os.getenv" call.

from pyspark.sql.types import IntegerType
import os

def init_spark():
spark = SparkSession.builder.appName("spark-env-test").getOrCreate()
sc = spark.sparkContext
return spark,sc

def returnExecutorEnv(x):
 Attempt to access environment variable from a task running on executor
return os.getenv("TESTENV1")

def main():
spark,sc = init_spark()

 dummy dataframe
df=spark.createDataFrame([("1","one")])
df.show()
df.rdd.map(lambda x: (x[0],returnExecutorEnv(x[0]))).toDF().show()
 Attempt to access environment variable on driver
print (os.getenv("TESTENV1"))
spark.stop()

if __name__ == '__main__':
main()
Show more





 Run a Spark application with non-default language version 

The Spark runtime support Spark application written in the following languages:



* Scala
* Python
* R



A Spark runtime version comes with default runtime language version. IBM extend support for new language versions and remove the existing language version to keep the runtime free from any security vulnerabilities. The system also provides settling time to transition your workloads when ever there is a new language versions. You can test your workload with a language version by passing an environment variable that points to the language version of the application.

Example:

"application_details": {
ibmcloud analytics-engine-v3 log-config show --instance-id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --output json





 Example output 

The output is as follows:

{
"components" : [ "spark-driver", "spark-executor" ],
"log_server" : {
"type" : "ibm-log-analysis"
},
"enable" : true
}









 Schema examples 

The following schema example represents the data required to specify command option. These example model the data structure and include placeholder values for the expected value type. When you run a command, replace these values with the values that apply to your environment as appropriate.



 Runtime 

The following example shows the format of the Runtime object.

{
"spark_version" : "3.3"
}
"components" : [ "spark-driver", "spark-executor" ],
"log_server" : {
"type" : "ibm-log-analysis"
},
"enable" : true
}







 ibmcloud analytics-engine-v3 log-config show 

Retrieve the logging configuration of a given Analytics Engine instance.

Note: Deprecated. Use show from the log-forwarding-config command group instead.

ibmcloud analytics-engine-v3 log-config show [--instance-id INSTANCE-ID]



 Command options 

-i, --instance-id (string)
: GUID of the Analytics Engine service instance to retrieve log configuration. When specified, it overrides the target instance ID if it was set. Required, if the target instance was not set.

The value must match the regular expression /u0008[0-9a-f]{8}u0008-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-u0008[0-9a-f]{12}u0008/.





 Examples 

ibmcloud analytics-engine-v3 log-config show --output json

ibmcloud analytics-engine-v3 log-config show --instance-id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --output json





 Example output 

The output is as follows:

{
"components" : [ "spark-driver", "spark-executor" ],
"log_server" : {
"type" : "ibm-log-analysis"
},
"enable" : true
}









 Schema examples 

The following schema example represents the data required to specify command option. These example model the data structure and include placeholder values for the expected value type. When you run a command, replace these values with the values that apply to your environment as appropriate.



 Runtime
num-executors application_details -> conf -> ae.spark.executor.count 
 pyFiles application_details -> conf -> spark.submit.pyFiles 
 <environment-variables> application_details -> env -> {"key1" : "value1", "key2" : "value2", ..... "} 







 Getting the state of a submitted application 

To get the state of a submitted application, enter:

curl -X GET https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_id>/spark_applications/<application_id>/state --header "Authorization: Bearer $token"

Sample response:

{
"id": "a9a6f328-56d8-4923-8042-97652fff2af3",
"state": "finished",
"start_time": "2020-11-25T14:14:31.311+0000",
"finish_time": "2020-11-25T14:30:43.625+0000"
}





 Getting the details of a submitted application 

To get the details of a submitted application, enter:

curl -X GET https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_id>/spark_applications/<application_id> --header "Authorization: Bearer $token"

Sample response:

{
"id": "ecd608d5-xxxx-xxxx-xxxx-08e27456xxxx",
"spark_application_id": "null",
"application_details": {
"application": "cos://sbn-test-bucket-serverless-1.mycosservice/my_spark_application.py",
"conf": {
Spark user interface 

The Spark user interface (Spark UI) helps you to keep track of various aspects of a running Spark application.

The list below includes a few examples:



* current running stage
* number of tasks in a stage
* reason for a longer running stage
* strangler task in a stage
* whether the executors in the application are used optimally
* inspect into memory and disk consumption of the driver and executors



For details, see the [Spark-UI documentation](https://spark.apache.org/docs/latest/monitoring.htmlweb-interfaces).

IBM Analytics Engine displays the Spark UI only for the Spark applications that are currently running. You cannot access Spark UI for a completed application.

Use the Spark history server to inspect the run of a completed Spark application. To access the Spark history server, see the [Access Spark history server](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-spark-history-serverless).



 Accessing the Spark UI 

The Spark UI endpoint of a running Spark application is accessible from the service details page of the IBM Analytics Engine instance.

The following image shows you an example of the Application tab with the link to the Spark UI of a running application.

Zoom

![Shows the ink to Spark-ui of a running application](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/spark_ui.png)

Figure 1. Application tab of IBM Analytics Engine service details page

The Spark UI endpoint of the running Spark application can also be obtained by invoking the following IBM Analytics Engine REST API endpoints or corresponding SDK methods:



* [Retrieve the details of a given Spark application](https://cloud.ibm.com/apidocs/ibm-analytics-engine-v3get-application)
* [List Spark applications](https://cloud.ibm.com/apidocs/ibm-analytics-engine-v3list-applications)
"spark_application_id" : "app-20211009103247-0000",
"spark_application_name" : "PythonWordCount",
"state" : "finished",
"submission_time" : "2021-04-21T04:23:50Z",
"start_time" : "2021-04-21T04:24:01Z",
"end_time" : "2021-04-21T04:25:18Z",
"finish_time" : "2021-04-21T04:25:18Z",
"auto_termination_time" : "2021-04-24T04:24:01Z"
}, {
"id" : "a2a2b23f-0929-4c49-9cc0-bd4c2bf953d9",
"spark_application_id" : "app-20211009103147-0000",
"spark_application_name" : "PythonWordCount",
"state" : "finished",
"submission_time" : "2021-04-21T04:21:50Z",
"start_time" : "2021-04-21T04:22:01Z",
"end_time" : "2021-04-21T04:23:18Z",
"finish_time" : "2021-04-21T04:23:18Z",
"auto_termination_time" : "2021-04-24T04:22:01Z"
} ],
"first" : {
"href" : "https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/e64c907a-e82f-46fd-addc-ccfafbd28b09/spark_applications?limit=25"
},
"next" : {
ibmcloud cos upload --bucket test-cos-storage-bucket --key test-math.py --file test-math.py

Sample application file
: Sample of test-math.py:

from pyspark.sql import SparkSession
import time
import random
import cmath

def init_spark():
spark = SparkSession.builder.appName("test-math").getOrCreate()
sc = spark.sparkContext
return spark,sc

def transformFunc(x):
return cmath.sqrt(x)+cmath.log(x)+cmath.log10(x)

def main():
spark,sc = init_spark()
partitions=[10,5]
for i in range (0,2):
data=range(1,20000000)
v0 = sc.parallelize(data, partitions[i])
v1 = v0.map(transformFunc)
print(f"v1.count is {v1.count()}. Done")
time.sleep(60)

if __name__ == '__main__':
main()
Show more
2. Check the status of the Analytics Engine service:

Action
: Enter:

ibmcloud ae-v3 instance show –id INSTANCE ID

Parameter values:



* INSTANCE_ID: The value of GUID from the response the of Analytics Engine instance creation call



Example
: Enter:

ibmcloud ae-v3 instance show –id 181ea9ee01b

Response
: The example returns:

{
"default_runtime": {
"spark_version": "3.1" },
"id": "181ea9ee01b ",
"instance_home": {
"bucket": "do-not-delete-ae-bucket-e965d-b7a82",
When submitting an application, you can use the "env" section under "application_details" in the payload JSON script to pass environment specific information, which determines the outcome of the application, for example the data sets to use or any secret values.

Example of the "env" section in the payload:

{
"application_details": {
"application": "cos://<application-bucket-name>.<cos-reference-name>/my_spark_application.py",
"arguments": ["arg1", "arg2"],
"conf": {
"spark.hadoop.fs.cos.<cos-reference-name>.endpoint": "https://s3.direct.us-south.cloud-object-storage.appdomain.cloud",
"spark.hadoop.fs.cos.<cos-reference-name>.access.key": "<access_key>",
"spark.hadoop.fs.cos.<cos-reference-name>.secret.key": "<secret_key>",
"spark.app.name": "MySparkApp"
},
"env": {
"key1": "value1",
"key2": "value2",
"key3": "value3"
}
}
}
Show more

The environment variables set using "application_details" > "env" as described here, will be accessible to both executor and driver code.

The environment variables can be set using "spark.executorEnv.[EnvironmentVariableName]" configuration (application_details > env) also. They will, however, be accessible only to the tasks running on the executor and not the driver.

Example of pyspark application that accesses the environment variables that are passed using the "os.getenv" call.

from pyspark.sql.types import IntegerType
import os

def init_spark():
Using a library set 

After you have created a library set, you can reference and consume it in your Spark applications. When you run your Spark application in the IBM Analytics Engine instance, the library set is loaded from the instance home and is made available to the Spark application.

A library set is referenced in a Spark application using the "ae.spark.librarysets" parameter in the "conf" section of the Spark application submission payload.

To reference a library set when submitting a Spark application:



1. Get the IAM token. See [Retrieving IAM access tokens](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-retrieve-iam-token-serverless).
2. Issue the following cURL command:

curl -X POST https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_id>/spark_applications --header "Authorization: Bearer <IAM token>" -H "content-type: application/json" -d @submit-spark-app.json

Example for submit-spark-app.json:

{
"application_details": {
"application": "cos://<bucket-name>.<cos-name>/my_spark_application.py",
"arguments": ["arg1", "arg2"],
"conf": {
"spark.hadoop.fs.cos.<cos-name>.endpoint":"https://s3.us-south.cloud-object-storage.appdomain.cloud",
"spark.hadoop.fs.cos.<cos-name>.access.key":"<access_key>",
"spark.hadoop.fs.cos.<cos-name>.secret.key":"<secret_key>",
"ae.spark.librarysets":"my_library_set"
}
}
}
"spark_version" : "3.1"
},
"instance_home" : {
"guid" : "30d5c4a7-9fb7-4712-9039-a79417dec87b",
"provider" : "ibm-cos",
"type" : "objectstore",
"region" : "us-south",
"endpoint" : "s3.direct.us-south.cloud-object-storage.appdomain.cloud",
"bucket" : "ae-bucket-do-not-delete-dc0e9889-eab2-4t9e-9441-566209499546",
"hmac_access_key" : "eHg=",
"hmac_secret_key" : "4d76"
},
"default_config" : {
"spark.driver.memory" : "4g",
"spark.driver.cores" : 1
}
}







 ibmcloud analytics-engine-v3 instance state 

Retrieve the state of a single Analytics Engine instance.

ibmcloud analytics-engine-v3 instance state [--id ID]



 Command options 

-i, --id (string)
: GUID of the Analytics Engine service instance to retrieve state. When specified, it overrides the target instance ID if it was set. Required, if the target instance was not set.

The value must match the regular expression /u0008[0-9a-f]{8}u0008-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-u0008[0-9a-f]{12}u0008/.





 Examples 

ibmcloud analytics-engine-v3 instance state --output json
employeesDF = spark.read.option("header",True).csv("cos://cosbucketname.cosservicename/employees.csv")
print("Hello2" , employeesDF)
employeesDF.createOrReplaceTempView("empTable")
juniors = spark.sql("SELECT empTable.NAME FROM empTable WHERE empTable.BAND < 6")
print("Hello3", juniors)
juniors.show()

def main():
spark,sc = init_spark()
read_employees(spark,sc)

if __name__ == '__main__':
main()
Show more

Then POST the following payload JSON script called read-employees-iam-key-cos-submit.json with your access key and password. Insert the Object Storage bucket name where the CSV file is located and the modify the endpoint path.

{
"application_details": {
"application": "cos://cosbucketname.cosservicename/read-employees-iam-key-cos.py",
"conf": {
"spark.hadoop.fs.cos.cosservicename.endpoint": "https://s3.direct.us-south.cloud-object-storage.appdomain.cloud/CHANGME-according-to-instance",
"spark.hadoop.fs.cos.cosservicename.access.key": "<CHANGEME>",
"spark.hadoop.fs.cos.cosservicename.secret.key": "<CHANGEME>"
}
}
}





 Reading and writing to Object Storage using IAM in Scala 



1. Create an Eclipse project of the following format:



Zoom

![Shows the Eclipse project format to use in which to develop your Scala jar.]
4 14 13.994738 12.336533 15.652943 0.643756
Show more





 Time series SQL 

The time series library is tightly integrated with Apache Spark. By using new data types in Spark Catalyst, you are able to perform time series SQL operations that scale out horizontally using Apache Spark. This enables you to easily use time series extensions in IBM Analytics Engine or in solutions that include IBM Analytics Engine functionality like the Watson Studio Spark environments.

SQL extensions cover most aspects of the time series functions, including segmentation, transformations, reducers, forecasting, and I/O. See [Analyzing time series data](https://cloud.ibm.com/docs/sql-query?topic=sql-query-ts_intro).





 Learn more 

To use the tspy Python SDK, see the [tspy Python SDK documentation](https://ibm-cloud.github.io/tspy-docs/).
"spark.hadoop.fs.cos.mycos.endpoint": "https://s3.direct.us-south.cloud-object-storage.appdomain.cloud",
"spark.hadoop.fs.cos.mycos.access.key": "XXXX",
"spark.hadoop.fs.cos.mycos.secret.key": "XXXX",
"spark.app.name": "MySparkApp"
}
}

Sample response:

{"id":13,"app_info":{},"state":"not_started"}





 Submit batch job with job file on local disk 

To submit a batch job where the job file is located on a local disk enter:

curl -i -X POST https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance-id>/livy/batches -H 'content-type: application/json' -H "Authorization: Bearer $TOKEN" -d @livypayload.json

Sample payload:

{
"file": "/opt/ibm/spark/examples/src/main/python/wordcount.py",
"args": ["/opt/ibm/spark/examples/src/main/resources/people.txt"],
"className": "org.apache.spark.deploy.SparkSubmit"
}

Sample response:

{"id":15,"app_info":{},"state":"not_started"}

The SparkUiUrl property in response will have a non-null value when the UI is available for serverless Spark instance.





 List the details of a job 

To list the job details for a particular Spark batch job enter:

curl -H 'Authorization: Bearer <TOKEN>' -H 'Content-Type: application/json' -X GET https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance-id>/livy/batches/<batch-id>