vania-janet/multiturn-rag-retrieval-data huggingface dataset:
If you want to allow multiple virtual instances to write data to the same data volume, you must click On under Shareable. <br>Attached Volume: You can select an existing data volume from the Attached volumes list. If a previously used data volume does not appear, it might exist under a different account or resource instance. 
 Public Networks Select this option to use an IBM-provided public network. There is a cost that is associated with selecting this option. <br>[Learn more](https://cloud.ibm.com/docs/power-iaas?topic=power-iaas-about-virtual-serverpublic-private-networks) 
 Private Networks Click Add to identify a new private network for the virtual server. If you already added a private network, you can select it from the list. For more information, see [Configure private network](https://cloud.ibm.com/docs/power-iaas?topic=power-iaas-configuring-subnet). 







 Reusing Volume names or VM names in Power System Virtual Servers 

You can deploy a Power System Virtual Server VM by specifying any name. If you want to delete that VM and deploy a new VM with the same name, you must allow 1 hour between deleting the original instance and creating a VM by using the same name again.

For example, you create a VM with the name TEST-VM and you delete this VM later. The name "TEST-VM" is not immediately available for reuse. Before you attempt to use the name TEST-VM again, you must allow 1 hour to pass after the VM was deleted.





 Implementing SAP NetWeaver and SAP HANA in the Power Systems Virtual Server environment 

You can deploy SAP NetWeaver on an AIX or Linux® operating system, and SAP HANA on Linux operating system, in your Power Systems Virtual Server environment. You must consider several SAP-specific infrastructure requirements to run SAP applications on Power Systems Virtual Servers.
GET /bare_metal_servers/{bare_metal_server_id}/network_interfaces/{network_interface_id}/floating_ips/{id} .target 
 PUT /bare_metal_servers/{bare_metal_server_id}/network_interfaces/{network_interface_id}/floating_ips/{id} .target 
 GET /floating_ips .floating_ips[].target 
 POST /floating_ips .target 
 GET /floating_ips/{id} . 
 PATCH /floating_ips/{id} . 
 GET /instance/templates .templates[].network_interfaces[], .templates[].primary_network_interface 
 POST /instance/templates .network_interfaces[], primary_network_interface .network_interfaces[], .primary_network_interface 
 GET /instance/templates/{id} .network_interfaces[], .primary_network_interface 
 PATCH /instance/templates/{id} .network_interfaces[], .primary_network_interface 
 GET /instances .instances[].network_interfaces[], .instances[].primary_network_interface 
 POST /instances .network_interfaces[], primary_network_interface .network_interfaces[], .primary_network_interface 
 GET /instances/{id} .network_interfaces[], .primary_network_interface 
 PATCH /instances/{id} .network_interfaces[], .primary_network_interface 
 GET /instances/{id}/network_interfaces .network_interfaces[] 
 POST /instances/{id}/network_interfaces . . 
 GET /instances/{instance_id}/network_interfaces/{id} . 
 GET /instances/{instance_id}/network_interfaces/{id}/floating_ips .floating_ips[].target 
 GET /instances/{instance_id}/network_interfaces/{network_interface_id}/floating_ips/{id} .target 
 PUT /instances/{instance_id}/network_interfaces/{network_interface_id}/floating_ips/{id} .target
Learn more about the service variable in the [stocator](https://github.com/CODAIT/stocatorstocator-and-ibm-cloud-object-storage-ibm-cos) project.

Click Buckets on the left, and select your bucket. Open the Configuration tag and scroll down to the Endpoints section and notice the Direct endpoint. Fill in these shell variables. Something like the following:

 GUID= set earlier
service=hello
access_key_id="0012345678901234567899550cfa9a60"
secret_access_key="00f02d12345678948bc82602c123456789963849c68c0f19"
endpoint=s3.direct.us-south.cloud-object-storage.appdomain.cloud
bucket=ABC-log-analysis

Submit the hello application. Notice the single and double quotes, it is a little tricky:

ibmcloud ae-v3 spark-app submit --instance-id $GUID --app "cos://$bucket.$service/hello.py" --conf '{
"spark.hadoop.fs.cos.'$service'.endpoint": "https://'$endpoint'",
"spark.hadoop.fs.cos.'$service'.access.key": "'$access_key_id'",
"spark.hadoop.fs.cos.'$service'.secret.key": "'$secret_access_key'"
}'

Again verify the results in the platform log.

The final step is to submit the spark application that accesses the data in the same bucket. Create a file, solution.py, with the following contents and upload it to the bucket. Notice the COS_PARQUET environment variable that will be initialized in the next step.

solution.py:

from pyspark.sql import SparkSession
* [Apache Spark](https://www.ibm.com/docs/SSQNUZ_4.6.x/svc-welcome/spark.html): Use the Analytics Engine powered by Apache Spark as a compute engine to run analytical and machine learning jobs.
* [Cognos Dashboards](https://www.ibm.com/docs/SSQNUZ_4.6.x/svc-welcome/cde.html): Use sophisticated visualizations in an analytics project to identify patterns in your data so that you can make timely and effective decisions.
* [Db2 Data Gate](https://www.ibm.com/docs/SSQNUZ_4.6.x/svc-welcome/dg.html): Extract, load, and synchronize your mission-critical data from Db2 for z/OS to Cloud Pak for Data for quick access by your new, high volume, read-only transactional, and analytic applications.
* [Db2 Data Management Console](https://www.ibm.com/docs/SSQNUZ_4.6.x/svc-welcome/dmc.html): Administer, monitor, manage and optimize the performance of IBM Db2 for Linux, UNIX and Windows databases.
* [Db2 Warehouse](https://www.ibm.com/docs/SSQNUZ_4.6.x/svc-welcome/db2wh.html): Take advantage of in-memory data processing and in-database analytics in an analytics data warehouse that supports automated scaling.
* [RStudio Server](https://www.ibm.com/docs/SSQNUZ_4.6.x/svc-welcome/rstudio.html): Use an integrated development environment for working with R in Watson Studio to create R Shiny applications.
* [Scheduling](https://www.ibm.com/docs/SSQNUZ_4.6.x/fixlist/sched-fixlist.html): The scheduling service offers enhancements over the default Kubernetes scheduler including Quota enforcement, Co-scheduling of pods and GPU sharing.
Replace the default Spark configuration properties that will be applied to all applications of the instance.

ibmcloud analytics-engine-v3 instance default-configs-replace [--id ID] --body BODY



 Command options 

-i, --id (string)
: The ID of the Analytics Engine instance. When specified, it overrides the target instance ID if it was set. Required, if the target instance was not set.

The value must match the regular expression /u0008[0-9a-f]{8}u0008-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-u0008[0-9a-f]{12}u0008/.

--body (map[string]string)
: Spark configuration properties to replace existing instance default Spark configurations. Required.





 Examples 

ibmcloud analytics-engine-v3 instance default-configs-replace --body='{"spark.driver.memory" : "8G", "spark.driver.cores" : "2"}' --output json

ibmcloud analytics-engine-v3 instance default-configs-replace --id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --body='{"spark.driver.memory" : "8G", "spark.driver.cores" : "2"}' --output json





 Example output 

The output is as follows:

{
"spark.driver.memory" : "8G",
"spark.driver.cores" : "2"
}







 ibmcloud analytics-engine-v3 instance default-configs-update 

Update the default Spark configuration properties that will be applied to all applications of the instance.

ibmcloud analytics-engine-v3 instance default-configs-update [--id ID] --body BODY



 Command options 

-i, --id (string)
: The ID of the Analytics Engine instance.
SHA-224, SHA-256, SHA-384, and SHA-512 are supported. Bit length of 0 is equivalent to 256.
: Example of an SQL function usage fragment

> SELECT sha2('Spark', 256)
: Result value

529bc3b07127ecb7e53a4dcf1991d9152c24537d919178022b2c42657f79a26b





 shiftleft 

shiftleft(base, expr)
: Bitwise left shift.
: Example of an SQL function usage fragment

> SELECT shiftleft(2, 1)
: Result value

4





 shiftright 

shiftright(base, expr)
: Bitwise (signed) right shift.
: Example of an SQL function usage fragment

> SELECT shiftright(4, 1)
: Result value

2





 shiftrightunsigned 

shiftrightunsigned(base, expr)
: Bitwise unsigned right shift.
: Example of an SQL function usage fragment

> SELECT shiftrightunsigned(4, 1)
: Result value

2





 sign 

sign(expr)
: Returns -1.0, 0.0 or 1.0 as expr is negative, 0 or positive.
: Example of an SQL function usage fragment

> SELECT sign(40)
: Result value

1.0





 signum 

signum(expr)
: Returns -1.0, 0.0 or 1.0 as expr is negative, 0 or positive.
: Example of an SQL function usage fragment

> SELECT signum(40)
: Result value

1.0





 sin 

sin(expr)
: Returns the sine of expr.
: Example of an SQL function usage fragment

> SELECT sin(0)
: Result value

0.0





 sinh 

sinh(expr)
: Returns the hyperbolic sine of expr.
: Example of an SQL function usage fragment

> SELECT sinh(0)
: Result value
ibmcloud analytics-engine-v3 instance default-configs 

Get the default Spark configuration properties that will be applied to all applications of the instance.

ibmcloud analytics-engine-v3 instance default-configs [--id ID]



 Command options 

-i, --id (string)
: The ID of the Analytics Engine instance. When specified, it overrides the target instance ID if it was set. Required, if the target instance was not set.

The value must match the regular expression /u0008[0-9a-f]{8}u0008-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-u0008[0-9a-f]{12}u0008/.





 Examples 

ibmcloud analytics-engine-v3 instance default-configs --output json

ibmcloud analytics-engine-v3 instance default-configs --id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --output json





 Example output 

The output is as follows:

{
"spark.driver.memory" : "4G",
"spark.driver.cores" : "1"
}







 ibmcloud analytics-engine-v3 instance default-configs-replace 

Replace the default Spark configuration properties that will be applied to all applications of the instance.

ibmcloud analytics-engine-v3 instance default-configs-replace [--id ID] --body BODY



 Command options 

-i, --id (string)
: The ID of the Analytics Engine instance. When specified, it overrides the target instance ID if it was set. Required, if the target instance was not set.
ibmcloud analytics-engine-v3 instance default-runtime-replace --id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --runtime-spark-version=3.3 --output json





 Example output 

The output is as follows:

{
"spark_version" : "3.3"
}







 ibmcloud analytics-engine-v3 instance current-resource-consumption 

Provides the total memory and virtual processor cores allocated to all the applications running in the service instance at this point in time. When auto-scaled applications are running, the resources allotted changes over time, based on the applications's demands.

Note: The consumption is not an indication of actual resource consumption by Spark processes. It is the sum of resources allocated to the currently running applications at the time of application submission.

ibmcloud analytics-engine-v3 instance current-resource-consumption [--id ID]



 Command options 

-i, --id (string)
: ID of the Analytics Engine instance. When specified, it overrides the target instance ID if it was set. Required, if the target instance was not set.

The value must match the regular expression /u0008[0-9a-f]{8}u0008-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-u0008[0-9a-f]{12}u0008/.





 Examples 

ibmcloud analytics-engine-v3 instance current-resource-consumption --output json

ibmcloud analytics-engine-v3 instance current-resource-consumption --id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --output json





 Example output 

The output is as follows:

{
"cores" : "2",
"memory" : "8G"
}







 ibmcloud analytics-engine-v3 instance resource-consumption-limits
The value must match the regular expression /u0008[0-9a-f]{8}u0008-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-u0008[0-9a-f]{12}u0008/.





 Examples 

ibmcloud analytics-engine-v3 spark-app stop --app-id=ff48cc19-0e7e-4627-aac6-0b4ad080397b

ibmcloud analytics-engine-v3 spark-app stop --instance-id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --app-id=ff48cc19-0e7e-4627-aac6-0b4ad080397b







 ibmcloud analytics-engine-v3 spark-app status 

Returns the status of the given Spark application.

ibmcloud analytics-engine-v3 spark-app status [--instance-id INSTANCE-ID] --app-id APP-ID



 Command options 

-i, --instance-id (string)
: Identifier of the instance to which the applications belongs. When specified, it overrides the target instance ID if it was set. Required, if the target instance was not set.

The value must match the regular expression /u0008[0-9a-f]{8}u0008-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-u0008[0-9a-f]{12}u0008/.

--app-id (string)
: Identifier of the application for which details are requested. Required.
ibmcloud analytics-engine-v3 instance default-configs-update --id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --body='{"ae.spark.history-server.cores" : "1", "ae.spark.history-server.memory" : "4G"}' --output json





 Example output 

The output is as follows:

{
"ae.spark.history-server.cores" : "1",
"ae.spark.history-server.memory" : "4G",
"spark.driver.memory" : "8G",
"spark.driver.cores" : "2"
}







 ibmcloud analytics-engine-v3 instance default-runtime 

Get the default runtime environment on which all workloads of the instance will run.

ibmcloud analytics-engine-v3 instance default-runtime [--id ID]



 Command options 

-i, --id (string)
: The ID of the Analytics Engine instance. When specified, it overrides the target instance ID if it was set.

The value must match the regular expression /u0008[0-9a-f]{8}u0008-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-u0008[0-9a-f]{12}u0008/.





 Examples 

ibmcloud analytics-engine-v3 instance default-runtime --output json

ibmcloud analytics-engine-v3 instance default-runtime --id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --output json





 Example output 

The output is as follows:

{
"spark_version" : "3.1"
}







 ibmcloud analytics-engine-v3 instance default-runtime-replace 

Replace the default runtime environment on which all workloads of the instance will run.
: Casts the value expr to the target data type binary.





 bit_length 

bit_length(expr)
: Returns the bit length of string data or number of bits of binary data.
: Example of an SQL function usage fragment

> SELECT bit_length('Spark SQL')
: Result value

72





 boolean 

boolean(expr)
: Casts the value expr to the target data type boolean.





 bround 

bround(expr, d): Returns expr rounded to d decimal places by using HALF_EVEN rounding mode.
: Example of an SQL function usage fragment

> SELECT bround(2.5, 0)
: Result value

2.0





 cast 

cast(expr AS type)
: Casts the value expr to the target data type type.
: Example of an SQL function usage fragment

> SELECT cast('10' as int)
: Result value

10





 cbrt 

cbrt(expr)
: Returns the cube root of expr.
: Example of an SQL function usage fragment

> SELECT cbrt(27.0)
: Result value

3.0





 ceil 

ceil(expr)
: Returns the smallest integer not smaller than expr.
: Example of an SQL function usage fragment

> SELECT ceil(-0.1)
: Result value

0
: Example of an SQL function usage fragment

> SELECT ceil(5)
: Result value

5





 ceiling 

ceiling(expr)
: Returns the smallest integer not smaller than expr.
: Example of an SQL function usage fragment

> SELECT ceiling(-0.1)
: Result value

0
: Example of an SQL function usage fragment

> SELECT ceiling(5)
: Result value

5





 char 

char(expr)
: Returns the ASCII character that has the binary equivalent to expr. If n is larger than 256, the result is equivalent to chr(n % 256).
: Example of an SQL function usage fragment

> SELECT char(65)
Then POST the following payload JSON script called read-write-cos-scala-submit.json with your access key and password. Insert the Object Storage bucket name where the CSV file is located and the modify the endpoint path.

{
"application_details": {
"application": "cos://cosbucketname.cosservicename/scalareadwriteiamcosexample_2.12-1.0.jar",
"class":"com.ibm.analyticsengine.ScalaReadWriteIAMCOSExample",
"conf": {
"spark.hadoop.fs.cos.cosservicename.endpoint": "https://s3.direct.us-south.cloud-object-storage.appdomain.cloud-CHANGME-according-to-instance",
"spark.hadoop.fs.cos.cosservicename.access.key": "<CHANGEME>",
"spark.hadoop.fs.cos.cosservicename.secret.key": "<CHANGEME>"
}
}
}







 Submitting application details with IAM API key 

The following example show the application details for the application called read-employees.py that reads data from employees.csv in Object Storage using the IAM API key. Insert the Object Storage bucket and service name where the CSV file is located and the API key.

{
"application_details": {
"application": "cos://cosbucketname.cosservicename/read-employees.py",
"conf": {
"spark.hadoop.fs.cos.cosservicename.endpoint": "https://s3.direct.us-south.cloud-object-storage.appdomain.cloud-CHANGME-according-to-instance",
"spark.hadoop.fs.cos.cosservicename.iam.api.key": "CHANGME"
}
}
}
* Optical Proximity Correction (OPC) in EDA
* Full chip Integrated Circuit Validator (ICV) in EDA
* Any Hadoop map/reduce or Spark workload
* MPI workloads that cannot fit on a single virtual machine



The recommendation for such workloads is to start with BX2-16x64 as it gives the best performance in tests so far. Based on your results, you might get insight into alternative options that might better suit your specific workload. Some workloads might be network-latency sensitive, and IBM Cloud configurations might not appear promising; yet even for these cases, engage with the offering owner and your sales team so the HPC experts can evaluate your specific requirements and provide assistance.
ibmcloud analytics-engine-v3 instance default-runtime --id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --output json





 Example output 

The output is as follows:

{
"spark_version" : "3.1"
}







 ibmcloud analytics-engine-v3 instance default-runtime-replace 

Replace the default runtime environment on which all workloads of the instance will run.

ibmcloud analytics-engine-v3 instance default-runtime-replace [--id ID] --runtime-spark-version RUNTIME-SPARK-VERSION



 Command options 

-i, --id (string)
: The ID of the Analytics Engine instance. When specified, it overrides the target instance ID if it was set. Required, if the target instance was not set.

The value must match the regular expression /u0008[0-9a-f]{8}u0008-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-u0008[0-9a-f]{12}u0008/.

--runtime-spark-version (string)
: Spark version of the runtime environment. Required.

Allowable values are: 3.1, 3.3.





 Examples 

ibmcloud analytics-engine-v3 instance default-runtime-replace --runtime-spark-version=3.3 --output json

ibmcloud analytics-engine-v3 instance default-runtime-replace --id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --runtime-spark-version=3.3 --output json





 Example output 

The output is as follows:

{
"spark_version" : "3.3"
}







 ibmcloud analytics-engine-v3 instance current-resource-consumption
Use separate buckets and service credentials for application files, data files, and home instance Use the "separation of concerns" principle to distinguish the access between different resources.<br><br><br><br> * Do not store data or application files in the home instance bucket.<br> * Use separate buckets for data and application files.<br> * Use separate access credentials (IAM Key based) with restricted access to the bucket for application files and the bucket that contains your data.<br><br><br> <br><br> * [Assigning access to an individual bucket](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-iam-bucket-permissions&interface=ui)<br><br><br> 
 Applications must run within 72 hours There is a limit on the number of hours an application or kernel can run. For security and compliance patching, all runtimes that run for more than 72 hours are stopped. If you do have a large application, break your application into smaller chunks that will run within 72 hours. If you are running Spark streaming applications, make sure that you configure checkpoints and have monitoring in place to restart your applications if they are stopped <br><br> * [Application limits](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-limitslimits-application)<br><br><br>
func main() {
// Construct an instance of the CreateApplicationOptions model
createApplicationOptionsModel := new(ibmanalyticsengineapiv3.CreateApplicationOptions)
createApplicationOptionsModel.InstanceID = core.StringPtr("dc0e9889-eab2-4t9e-9441-566209499546")
createApplicationOptionsModel.Application = core.StringPtr("cos://bucket_name.my_cos/my_spark_app.py")
createApplicationOptionsModel.Class = core.StringPtr("IbmAnalyticsEngineApi")
createApplicationOptionsModel.Arguments = []string{"/opt/ibm/spark/examples/src/main/resources/people.txt"}
createApplicationOptionsModel.Conf = make(map[string]interface{})
createApplicationOptionsModel.Env = make(map[string]interface{})

_, applicationCreationResponse, _ := ibmAnalyticsEngineApiService.CreateApplication(createApplicationOptionsModel)
fmt.Println(applicationCreationResponse)
}
Show more
* Retrieve all Spark applications:

(ibmAnalyticsEngineApi IbmAnalyticsEngineApiV3) ListApplications(listApplicationsOptions ListApplicationsOptions) (result ApplicationCollection, response core.DetailedResponse, err error)

Example request:

func main() {
// Construct an instance of the ListApplicationsOptions model
listApplicationsOptionsModel := new(ibmanalyticsengineapiv3.ListApplicationsOptions)
listApplicationsOptionsModel.InstanceID = core.StringPtr("dc0e9889-eab2-4t9e-9441-566209499546")
.addSTBoundingBoxLocationIndex("location")
// equivalent
//.addCustomIndex(STBoundingBoxLocationIndex("location"))
.build(reader).show(false)
* For Python:

xskipper = Xskipper(spark, dataset_path)

 adding the index using the custom index API
xskipper.indexBuilder() 
.addCustomIndex("com.ibm.xskipper.stmetaindex.index.STBoundingBoxLocationIndex", ['location'], dict()) 
.build(reader) 
.show(10, False)







 Supported functions 

The list of supported geospatial functions includes the following:



* ST_Distance
* ST_Intersects
* ST_Contains
* ST_Equals
* ST_Crosses
* ST_Touches
* ST_Within
* ST_Overlaps
* ST_EnvelopesIntersect
* ST_IntersectsInterior









 Encrypting indexes 

If you use a Parquet metadata store, the metadata can optionally be encrypted using Parquet Modular Encryption (PME). This is achieved by storing the metadata itself as a Parquet data set, and thus PME can be used to encrypt it. This feature applies to all input formats, for example, a data set stored in CSV format can have its metadata encrypted using PME.

In the following section, unless specified otherwise, when referring to footers, columns, and so on, these are with respect to metadata objects, and not to objects in the indexed data set.

Index encryption is modular and granular in the following way:



* Each index can either be encrypted (with a per-index key granularity) or left plaintext
* Footer + object name column:



* Footer column of the metadata object which in itself is a Parquet file contains, among other things:



* Schema of the metadata object, which reveals the types, parameters and column names for all indexes collected.