vania-janet/multiturn-rag-retrieval-data huggingface dataset:
spark.sql("SELECT file_path, file_size_in_bytes FROM lakehouse.demodb.yellow_taxi_2022.files").show()

 Expire earlier snapshots only latest one with comacted data is required
 list all the snapshots
spark.sql("SELECT committed_at, snapshot_id, operation FROM lakehouse.demodb.yellow_taxi_2022.snapshots").show()
retain only the latest one
latest_snapshot_committed_at=spark.sql("SELECT committed_at, snapshot_id, operation FROM lakehouse.demodb.yellow_taxi_2022.snapshots").tail(1)[0].committed_at
print (latest_snapshot_committed_at)
spark.sql(f"CALL lakehouse.system.expire_snapshots(table => 'demodb.yellow_taxi_2022',older_than => TIMESTAMP '{latest_snapshot_committed_at}',retain_last => 1)").show()

 Removing Orphan data files
spark.sql(f"CALL lakehouse.system.remove_orphan_files(table => 'demodb.yellow_taxi_2022')").show(truncate=False)

 Rewriting Manifest Files
spark.sql(f"CALL lakehouse.system.rewrite_manifests('demodb.yellow_taxi_2022')").show()

 demonstration: Schema evolution
 Add column fare_per_mile to the table
spark.sql('ALTER TABLE lakehouse.demodb.yellow_taxi_2022 ADD COLUMN(fare_per_mile double)')
 describe the table
spark.sql('describe table lakehouse.demodb.yellow_taxi_2022').show(25)
finally:
 clean-up the demo database
spark.sql('drop table if exists lakehouse.demodb.testTable purge')
spark.sql('drop table if exists lakehouse.demodb.zipcodes purge')
.appName("Spark DataEngine integration")
.enableDataengine(args(0), args(1), "public")
.config("fs.cos.impl", "com.ibm.stocator.fs.ObjectStoreFileSystem")
.config("fs.stocator.scheme.list", "cos")
.config("fs.stocator.cos.impl", "com.ibm.stocator.fs.cos.COSAPIClient")
.config("fs.stocator.cos.scheme", "cos")
.getOrCreate()
println("Got a spark session, listing all tables")
val sqlDf = spark.sql("SHOW TABLES")
sqlDf.show()
}
}
Show more







 Troubleshooting error message 

If you receive the following error message when you run the SQL statement, check which of the possible causes exist.

AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

The following possible causes might exist.



* The CRN is invalid or the service does not exist.
* THE APIKEY is invalid.
* The Data Engine service has a Lite plan.



Py4JJavaError: An error occurred while calling o8820.sql.
: java.util.concurrent.ExecutionException: com.ibm.stocator.fs.common.exception.ConfigurationParseException: Configuration parse exception: Access KEY is empty. Please provide valid access key

The following possible cause might exist.



* Tables that are based on sample data that is provided by Data Engine do not work for a SELECT statement. Therefore, use data stored in your Cloud Object Storage bucket.
* There is no limit on the number of Spark applications that can be run in an IBM Analytics Engine service instance.







 Default Spark runtime 

You can select which Spark version to use when the instance is provisioned. Currently, you can choose between Spark 3.1 and Spark 3.3. If you don't select a Spark runtime version, Spark 3.1 is taken by default.

The runtime contains only open source Spark binaries and is configured to help you to quickly get started to create and run Spark applications in the instance. In addition to the Spark binaries, the runtime also includes the geospatial, data skipping, and Parquet modular encryption libraries.

On a Spark 3.1 or Spark 3.3 runtime, you can submit Spark applications written in the following languages: Scala 2.12, Python 3.9, and R 3.6.3.

Note that the language versions are upgraded periodically to keep the runtime free from any security vulnerabilities.

You can always override the Spark runtime version when you submit an application. For details on what to add to the payload, see [Passing the runtime Spark version when submitting an application](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-spark-app-rest-apipass-spark-version).





 Instance home 

Instance home is the storage attached to the instance for instance related data only, such as custom application libraries and Spark history events. Currently, only IBM Cloud Object Storage is accepted for instance home. This instance can be an instance in your IBM Cloud® account or an instance from a different account.

When you provision an instance using the IBM Cloud console, the IBM Cloud Object Storage instances in your IBM Cloud® account are auto discovered and displayed in a list for you to select from. If no IBM Cloud Object Storage instances are found in your account, you can use the REST APIs to update instance home after instance creation.

You can't change instance home after instance creation. You can only edit the access keys.





 Default Spark configuration 

You can specify default Spark configurations at the instance level and let that be inherited by Spark applications created on the instance. This is an optional section that you can specify at the time of instance creation.
ServiceCall<Instance> getInstance(GetInstanceOptions getInstanceOptions)

Example request:

// Construct an instance of the GetInstanceOptions model
GetInstanceOptions getInstanceOptionsModel = new GetInstanceOptions.Builder()
.instanceId("dc0e9889-eab2-4t9e-9441-566209499546")
.build();

// Invoke operation with valid options model (positive test)
Response<Instance> response = ibmAnalyticsEngineApiService.getInstance(getInstanceOptionsModel).execute();
Instance responseObj = response.getResult();
System.out.println(String.valueOf(responseObj));
* Deploy a Spark application on a given serverless Spark instancet:

ServiceCall<ApplicationResponse> createApplication(CreateApplicationOptions createApplicationOptions)

Example request:

// Construct an instance of the ApplicationRequestApplicationDetails model
ApplicationRequestApplicationDetails applicationRequestApplicationDetailsModel = new ApplicationRequestApplicationDetails.Builder()
.application("cos://ae-bucket-do-not-delete-dc0e9889-eab2-4t9e-9441-566209499546.s3.us-south.cloud-object-storage.appdomain.cloud/my_spark_application.py")
.xClass("IbmAnalyticsEngineApi")
.arguments(new java.util.ArrayList<String>(java.util.Arrays.asList("/opt/ibm/spark/examples/src/main/resources/people.txt")))
.conf(new java.util.HashMap<String, Object>() { { put("spark.app.name", "MySparkApp"); } })
"spark.hadoop.fs.cos.cosservicename.secret.key": "<CHANGEME>"
}
}
}





 Reading a CSV file from Object Storage using IAM API Key or different HMAC credentials 

The following code samples show you how to create a Python script that reads data from a CSV file to a Python DataFrame. Both the Python script and the CSV file are located in Object Storage.

This example shows you how to access IBM Cloud Object Storage using the IAM API key.

Example of the application called read-employees-iam-key-cos.py. Insert the Object Storage bucket name where the CSV file is located and the modify the endpoint path.

from pyspark.sql import SparkSession

def init_spark():
spark = SparkSession.builder.appName("read-write-cos-test").getOrCreate()
sc = spark.sparkContext
hconf=sc._jsc.hadoopConfiguration()
hconf.set("fs.cos.testcos.endpoint", "s3.direct.us-south.cloud-object-storage.appdomain.cloud/CHANGEME-according-to-instance="
("fs.cos.testcos.iam.api.key","<CHANGEME>")
return spark,sc

def read_employees(spark,sc):
print("Hello1 " , spark )
employeesDF = spark.read.option("header",True).csv("cos://cosbucketname.cosservicename/employees.csv")
print("Hello2" , employeesDF)
employeesDF.createOrReplaceTempView("empTable")
juniors = spark.sql("SELECT empTable.NAME FROM empTable WHERE empTable.BAND < 6")
print("Hello3", juniors)
juniors.show()

def main():
spark,sc = init_spark()
"id": "ecd608d5-xxxx-xxxx-xxxx-08e27456xxxx",
"spark_application_id": "null",
"application_details": {
"application": "cos://sbn-test-bucket-serverless-1.mycosservice/my_spark_application.py",
"conf": {
"spark.hadoop.fs.cos.mycosservice.endpoint": "https://s3.direct.us-south.cloud-object-storage.appdomain.cloud",
"spark.hadoop.fs.cos.mycosservice.access.key": "xxxx",
"spark.app.name": "MySparkApp",
"spark.hadoop.fs.cos.mycosservice.secret.key": "xxxx"
},
"arguments": [
"arg1",
"arg2"
]
},
"state": "failed",
"submission_time": "2021-11-30T18:29:21+0000"
}
Show more





 Stopping a submitted application 

To stop a submitted application, run the following:

curl -X DELETE https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_id>/spark_applications/<application_id> --header "Authorization: Bearer $token"

Returns 204 – No Content, if the deletion is successful. The state of the application is set to STOPPED.

This API is idempotent. If you attempt to stop an already completed or stopped application, it will still return 204.

You can use this API to stop an application in the following states: accepted, waiting, submitted, and running.





 Passing the runtime Spark version when submitting an application 

You can use the "runtime" section under "application_details" in the payload JSON script to pass the Spark runtime version when submitting an application.
sqlContext.registerDataFrameAsTable(df, "Table")
df_query = sqlContext.sql("SELECT * FROM Table LIMIT 10")
df_query.show(10)
return 0

if __name__ == '__main__':
sys.exit(main())
Show more

Now submit the spark application that accesses the data in the same bucket. Note that you will need to substitute in your jobid instead of 01234567-0123-0123-0123-012345678901. Double check the full path to the jobid in the COS bucket

jobid=01234567-0123-0123-0123-012345678901
service=solution
endpoint=s3.direct.us-south.cloud-object-storage.appdomain.cloud
ibmcloud ae-v3 spark-app submit --instance-id $GUID --app "cos://$bucket.$service/solution.py" --conf '{
"spark.hadoop.fs.cos.'$service'.endpoint": "https://'$endpoint'",
"spark.hadoop.fs.cos.'$service'.access.key": "'$access_key_id'",
"spark.hadoop.fs.cos.'$service'.secret.key": "'$secret_access_key'"
}' --env '{
"COS_PARQUET": "cos://'$bucket'.'$service'/logs-stream-landing/topic=webserver/jobid='$jobid'/"
}'

Check out the platform log and look for something that looks like this:
[the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) 
 Invoke application management REST API to view a specific Spark application, by using the application ID ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) 
 Invoke application management REST API to retrieve the state of a Spark application ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) !
"auto_termination_time" : "2021-04-24T04:22:01Z"
} ],
"first" : {
"href" : "https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/e64c907a-e82f-46fd-addc-ccfafbd28b09/spark_applications?limit=25"
},
"next" : {
"href" : "https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/e64c907a-e82f-46fd-addc-ccfafbd28b09/spark_applications?limit=25&start=QiwyMDIyLTA5LTI2IDA4OjE4OjU2LjE4MiwxMDAyYTVlZC0xZWU4LTQwZWItOWUyNC00OTMyNTcxZjgzYzE",
"start" : "QiwyMDIyLTA5LTI2IDA4OjE4OjU2LjE4MiwxMDAyYTVlZC0xZWU4LTQwZWItOWUyNC00OTMyNTcxZjgzYzE"
},
"limit": 25
}







 ibmcloud analytics-engine-v3 spark-app show 

Gets the details of a given Spark application.

ibmcloud analytics-engine-v3 spark-app show [--instance-id INSTANCE-ID] --app-id APP-ID



 Command options 

-i, --instance-id (string)
: Identifier of the instance to which the application belongs. When specified, it overrides the target instance ID if it was set. Required, if the target instance was not set.
df_may = spark.read.option("header",True).parquet("s3a://source-bucket/nyc-taxi/yellow_tripdata_2022-05.parquet")
df_june = spark.read.option("header",True).parquet("s3a://source-bucket/nyc-taxi/yellow_tripdata_2022-06.parquet")

df_q1_q2 = df_feb.union(df_march).union(df_april).union(df_may).union(df_june)
df_q1_q2.write.insertInto("lakehouse.demodb.yellow_taxi_2022")
 Query the metadata files table to list underlying data files
spark.sql("SELECT file_path, file_size_in_bytes FROM lakehouse.demodb.yellow_taxi_2022.files").show()

 There are many smaller files compact them into files of 200MB each using the
 rewrite_data_files Iceberg Spark procedure
spark.sql(f"CALL lakehouse.system.rewrite_data_files(table => 'demodb.yellow_taxi_2022', options => map('target-file-size-bytes','209715200'))").show()

 Query the metadata files table to list underlying data files, 6 files are compacted
 to 3 files
spark.sql("SELECT file_path, file_size_in_bytes FROM lakehouse.demodb.yellow_taxi_2022.files").show()

 Expire earlier snapshots only latest one with comacted data is required
 list all the snapshots
spark.sql("SELECT committed_at, snapshot_id, operation FROM lakehouse.demodb.yellow_taxi_2022.snapshots").show()
retain only the latest one
Connecting Apache Spark with Data Engine 

IBM Cloud® Data Engine catalog provides an interface that is compatible with Apache Hive metastore. This unified metadata repository enables any Big Data engine, such as Apache Spark, to use Data Engine as metastore. The same definition for tables and views can be created once and used from any connected engine. Each instance of Data Engine exports its catalog as a database called default.



 Catalog usage within Data Engine 

The Catalog can be used in Data Engine in read and write mode. Seamless access is configured without any configuration steps needed.





 Connecting Apache Spark with Data Engine 

When you use the Hive metastore compatible interface, access is limited to read only operations. Thus, existing tables and views can be used, but not modified.

Depending from where you want to connect to your catalog, the steps may vary.



 Usage within Watson Studio Notebooks 

Watson Studio has the compatible Hive metastore client already included. It also includes a convenience library to configure the connection to the Hive metastore. Set the required variables (CRN and apikey) and call the helper function to connect to the Hive metastore:

 change the CRN and the APIkey according to your instance
crn='yourDataengineCRN'
apikey='yourAPIkey'

from dataengine import SparkSessionWithDataengine

 call the helper function to create a session builder equipped with the correct config
session_builder = SparkSessionWithDataengine.enableDataengine(crn, apikey, "public")
spark = session_builder.appName("Spark DataEngine integration test").getOrCreate()

Display your tables and run an SQL statement:

spark.sql('show tables').show(truncate=False)

 replace yourTable with a valid table. Do not use the sample tables as you do not have access to the data!
spark.sql('select * from yourTable').show()





 Usage with IBM Analytics Engine 

The Hive metastore client is also already included in IBM® Analytics Engine.
4308f7a6-2cfd-4d7e-b7a7-678e71a27957 5438585f-6413-4bb2-9251-8162d34f3dc3
* Only the application ID:

5438585f-6413-4bb2-9251-8162d34f3dc3



* To search the Spark master logs of an application, you can pass:



* A combination of the IBM Analytics Engine instance ID, the Spark application ID and the keyword spark-driver:

4308f7a6-2cfd-4d7e-b7a7-678e71a27957 5438585f-6413-4bb2-9251-8162d34f3dc3 spark-driver
* A combination of the application ID and the keyword spark-driver:

5438585f-6413-4bb2-9251-8162d34f3dc3 spark-driver



* To search by multiple log source names, you can pass:



* A combination of the IBM Analytics Engine instance ID, the Spark application ID and the source names:

4308f7a6-2cfd-4d7e-b7a7-678e71a27957 5438585f-6413-4bb2-9251-8162d34f3dc3 (spark-driver OR spark-executor-2)
* A combination of the application ID and the log source names:

5438585f-6413-4bb2-9251-8162d34f3dc3 (spark-driver OR spark-executor-2)



* To get the logs related to all the applications running in a particular instance within the us-south region and the prod environment, you can enter the following in the search field:

4308f7a6-2cfd-4d7e-b7a7-678e71a27957 "prod us-south"
For details on how to use the log_forwarding_config API, see [Configuring and viewing logs](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-viewing-logs).







 13 May 2022 

Support for Python 3.9
: You can now run Spark applications using Python 3.9. on your IBM Analytics Engine serverless instances.





 04 April 2022 

Limitation on how long Spark applications can run
: Spark applications can run for a maximum period of 3 days (72 hours). Any applications that run beyond this period will be auto-cleaned in order to adhere to the security and compliance patch management processes for applications in Analytics Engine.





 30 March 2022 

Start using the Analytics Engine serverless CLI
: Use this tutorial to help you get started quickly and simply with provisioning an Analytics Engine serverless instance, and submitting and monitoring Spark applications. See [Create service instances and submit applications using the CLI](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-using-cli).





 9 September 2021 

Introducing IBM Analytics Engine Standard serverless plan for Apache Spark
: The IBM Analytics Engine Standard serverless plan for Apache Spark offers the ability to spin up IBM Analytics Engine serverless instances within seconds, customize them with library packages of your choice, and run your Spark workloads.

New: The IBM Analytics Engine Standard serverless plan for Apache Spark is now GA in the Dallas IBM Cloud service region.
: This plan offers a new consumption model using Apache Spark whereby resources are allocated and consumed only when Spark workloads are running.

Capabilities available in the IBM Analytics Engine Standard serverless plan for Apache Spark include:



* Running Spark batch and streaming applications
* Creating and working with Jupyter kernels for interactive use cases
* Running Spark batch applications through an Apache Livy like interface
* Customizing instance with your own libraries
* Autoscaling Spark workloads
* Aggregating logs of your Spark workloads to the Log Analysis server
The data stream landed to cos can be also queried using Apache Spark that is part of the Analytics Engine service serverless plan. Programs will need to be loaded into a bucket and executed from the command line. The pyspark python environment will be used in this tutorial.

Open the analytics engine service:



1. Access the Analytics Engine service instance from the [Resource List](https://cloud.ibm.com/resources) under Analytics.
2. Click the Applications tab.



Try the following:

If you are not logged in, use ibmcloud login or ibmcloud login --sso to log in interactively. Target the region and resource group you have been using (edit for your specifics):

ibmcloud target -r <region> -g <resource group>

Record the guid of the Analytics Engine and set a shell variable:

ibmcloud resource service-instance log-analysis-iae

GUID=<YOUR GUID>

Enable output to the platform logs that you created earlier in the region:

ibmcloud ae-v3 log-config update --instance-id $GUID --enable=true

Run a program that is built into the IBM spark runtime to print some output:

ibmcloud ae-v3 spark-app submit --instance-id $GUID --app "/opt/ibm/spark/examples/src/main/python/wordcount.py" --arg '["/opt/ibm/spark/examples/src/main/resources/people.txt"]'

Back in the Analytics Engine instance on the Applications click refresh to see the status of the application.

Open the platform logs for the region:



1. Navigate to the [Observability](https://cloud.ibm.com/observe) page and click Logging, look for the existing log analysis service in the region with Platform logs enabled.
2. Click Open dashboard.
3. In a few minutes you should see the logs associated with the program



Search for host:ibmanalyticsengine. There will be a lot of output. Look for:
ibmcloud analytics-engine-v3 spark-app show --instance-id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --app-id=ff48cc19-0e7e-4627-aac6-0b4ad080397b --output json





 Example output 

The output is as follows:

{
"application_details" : {
"application" : "/opt/ibm/spark/examples/src/main/python/wordcount.py",
"arguments" : [ "/opt/ibm/spark/examples/src/main/resources/people.txt" ],
"runtime": {
"spark_version": "3.1"
}
},
"id" : "a9a6f328-56d8-4923-8042-97652fff2af3",
"spark_application_id" : "app-20210908112356-0000",
"spark_application_name" : "PythonWordCount",
"state" : "finished",
"submission_time" : "2021-04-21T04:23:55Z",
"start_time" : "2021-04-21T04:24:01Z",
"end_time" : "2021-04-21T04:28:15Z",
"finish_time" : "2021-04-21T04:28:15Z",
"auto_termination_time" : "2021-04-24T04:24:01Z"
}







 ibmcloud analytics-engine-v3 spark-app stop 

Stops a running application identified by the app_id identifier. This is an idempotent operation. Performs no action if the requested application is already stopped or completed.

ibmcloud analytics-engine-v3 spark-app stop [--instance-id INSTANCE-ID] --app-id APP-ID



 Command options
"default_runtime": {
"spark_version": "3.3",
"instance_home": {
"provider": "ibm-cos",
"type": "objectstore",
"region": "us-south",
"endpoint": "https://s3.direct.us-south.cloud-object-storage.appdomain.cloud",
"bucket": "ae-bucket-do-not-delete-dc0e-eab2-4t-9441-566209499546",
"hmac_access_key": "eHg=",
"hmac_secret_key": "4d76"
},
"default_config": {
"spark.driver.memory": "4g",
"spark.driver.cores": 1
}
}
}
Show more
2. Check the value of the state attribute. It must be active before you can start running applications in the instance.







 Learn more 

When provisioning serverless instances, follow the recommended [Best practices](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-best-practices-serverless).
Spark history server 

The Spark applications that are submitted on an IBM Analytics Engine instance forward their Spark events to the Object Storage bucket that was defined as the instance home. The Spark history server provides a Web UI to view these Spark events. The Web UI helps you to analyze how your Spark applications ran by displaying useful information like:



* A list of the stages that the application goes through when it is run
* The number of tasks in each stage
* The configuration details such as the running executors and memory usage



See the [Spark History server documentation](https://spark.apache.org/docs/latest/monitoring.htmlviewing-after-the-fact) for more details.

You can disable forwarding Spark events from a Spark application by setting the property spark.eventLog.enabled to false in the Spark application configuration.



 Starting and stopping the Spark history server 

Before accessing the Spark history server, you need to start the server. When you no longer need it, you should stop the server. You will be charged for the CPU cores and memory consumed by the Spark history server while it is running.

The Spark history server can be started and stopped by using:



* The [Analytics Engine REST API](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-spark-history-serverlessrest-api)
* The [Analytics Engine instance UI](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-spark-history-serverlessiae-ui)





 Analytics Engine REST API 

You can use the Analytics Engine REST API:



1. To view the status of the Spark history server

curl "https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_id>/spark_history_server" --header "Authorization: bearer <iam token>"
2. To start the Spark history server