vania-janet/multiturn-rag-retrieval-data huggingface dataset:
TimeStamp: 12 Value: {value=11.996821082897318, lower_bound=10.704895525154571, upper_bound=13.288746640640065, error=0.5015571318964149}
TimeStamp: 13 Value: {value=12.995779355240911, lower_bound=11.50957896664928, upper_bound=14.481979743832543, error=0.5769793776877866}
TimeStamp: 14 Value: {value=13.994737627584504, lower_bound=12.33653268707341, upper_bound=15.652942568095598, error=0.6437557559526337}

print(predictions.to_time_series().to_df())

timestamp value lower_bound upper_bound error
0 10 10.000000 10.000000 10.000000 0.000000
1 11 10.997863 9.934621 12.061104 0.412776
2 12 11.996821 10.704896 13.288747 0.501557
3 13 12.995779 11.509579 14.481980 0.576979
4 14 13.994738 12.336533 15.652943 0.643756
Show more





 Time series SQL 

The time series library is tightly integrated with Apache Spark. By using new data types in Spark Catalyst, you are able to perform time series SQL operations that scale out horizontally using Apache Spark. This enables you to easily use time series extensions in IBM Analytics Engine or in solutions that include IBM Analytics Engine functionality like the Watson Studio Spark environments.
Streaming generates Activity Tracker events with the action logdna.streaming-logs.send to notify of failures sending data. There are different reasons for failure such as invalid credentials and topic deleted.

For more information, see [Monitoring streaming by using Activity Tracker](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-streaming-monitorstreaming-monitor-2).







 Conditional streaming 

You can configure exclusion rules to filter out data from streaming. For more information, see [Configuring conditional streaming](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-streaming-conditional).



* You configure streaming exclusion rules through Settings > Streaming > Exclusion rules.
* The exclusion rules that you define for streaming are different from the exclusion rules that you can define at the instance level through Settings > Usage > Exclusion rules.



When you define exclusion rules, either at the instance level, or for streaming, they are applied as follows:



* Exclusion rules that you define at the instance level are applied first.
* Only the data that is retained and available for search is in scope of the exclusion rules that you define for streaming.
* After a streaming exclusion rule is active, data that matches the filter criteria is not streamed.
* Conditions that are applied by a query are enforced.







 Activity Tracker events 

The following Activity Tracker events are generated when you configure streaming:



Table 1. Events generated when configuring streaming

 Action Description 

 logdna.streaming-configuration.validate This event is generated when you configure the connection in Log Analysis to Event Streams. 
 logdna.streaming-samples.send This event is generated when sample data is sent to verify the connection. 
 logdna.account-streaming-setting.configure This event is generated when you start streaming. 
 logdna.streaming-configuration.deactivate This event is generated when you stop streaming. 
 logdna.streaming-logs.send This event is generated when there is a failure streaming data. 
 logdna.exclusion-rule.create This event is generated when an streaming exclusion rule is configured.
"spark.hadoop.fs.cos.<cos-reference-name>.endpoint": "s3.direct.us-south.cloud-object-storage.appdomain.cloud",
"spark.hadoop.fs.cos.<cos-reference-name>.iam.api.key": "<iam-api-key-of-application-bucket>"
}
}
}





 Enabling autoscaling on an instance 

By default, application autoscaling is disabled when you create an instance of the Analytics Engine Standard serverless plan. If you want to enable automatic scaling for all applications at the instance level, you can explicitly set "ae.spark.autoscale.enable": "true" as a default Spark configuration when you create the instance. See [Default Spark configuration](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-serverless-architecture-conceptsdefault-spark-config).
"history_server_ui": "https://spark-console.us-south.ae.cloud.ibm.com/v3/analytics_engines/79c757e9-xxxx-xxxx-xxxx-dff7f01xxxx/spark_history_ui",
"history_server_api":"https://spark-console.us-south.ae.cloud.ibm.com/v3/analytics_engines/79c757e9-xxxx-xxxx-xxxx-dff7f01xxxx/spark_history_api/v1"
},
"iam_apikey_description": "Auto-generated for key 2278e09d-b815-4c4b-b824-xxxx",
"iam_apikey_name": "service_key_02",
"iam_role_crn": "crn:v1:bluemix:public:iam::::serviceRole:Writer",
"iam_serviceid_crn": "crn:v1:staging:public:iam-identity::a/d628eae2cxxxxx229f2xxxx::serviceid:ServiceId-209exxxx-xxxx-xxxx-xxxx-4ab9ce16xxxx"
},
"iam_compatible": true,
"migrated": false,
"resource_instance_url": "/v1/resource_instances/crn%3Av1%3Astaging%3Apublic%3Aibmanalyticsengine%3Aus-south%3Aa%2Fd628eae2cxxxxx229f2xxxx%3A79c757e9-xxxx-xxxx-xxxx-dff7f01xxxx%3A%3A",
"resource_alias_url": null
}
Show more





 Obtaining the service endpoints from the IBM Cloud console
ibmcloud analytics-engine-v3 instance resource-consumption-limits --id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --output json





 Example output 

The output is as follows:

{
"max_cores" : "150",
"max_memory" : "600G"
}









 Spark application commands 

With the Spark application CLI commands, you can perform the following operations:



* [Submit a Spark application](https://cloud.ibm.com/docs/analytics-engine-cli-plugin?topic=analytics-engine-cli-plugin-CLI_analytics_engineanalytics-engine-v3-cli-spark-app-submit-command)
* [List all applications submitted](https://cloud.ibm.com/docs/analytics-engine-cli-plugin?topic=analytics-engine-cli-plugin-CLI_analytics_engineanalytics-engine-v3-cli-spark-app-list-command)
* [Retrieve the details of a Spark application](https://cloud.ibm.com/docs/analytics-engine-cli-plugin?topic=analytics-engine-cli-plugin-CLI_analytics_engineanalytics-engine-v3-cli-spark-app-show-command)
* [Show the status of a submitted application](https://cloud.ibm.com/docs/analytics-engine-cli-plugin?topic=analytics-engine-cli-plugin-CLI_analytics_engineanalytics-engine-v3-cli-spark-app-status-command)
* [Stop a submitted Spark application](https://cloud.ibm.com/docs/analytics-engine-cli-plugin?topic=analytics-engine-cli-plugin-CLI_analytics_engineanalytics-engine-v3-cli-spark-app-stop-command)



ibmcloud analytics-engine-v3 spark-app --help



 ibmcloud analytics-engine-v3 spark-app submit
Although you can change the Hive configuration on IBM Analytics Engine clusters, it is your responsibility that the cluster functions correctly after you have made any such changes.
* How much time does it take for the cluster to get started?

When using the Spark software pack, a cluster takes about 7 to 9 minutes to be started and be ready to run applications. When using the Hadoop and Spark software pack, a cluster takes about 15 to 20 minutes to be started and be ready to run applications.
* How can I access or interact with my cluster?

There are several interfaces which you can use to access the cluster.



* SSH
* Ambari console
* REST APIs
* Cloud Foundry CLI



* How do I get data into the cluster?

The recommended way to read data to a cluster for processing is from IBM Cloud Object Storage. Upload your data to IBM Cloud Object Storage (COS) and use COS, Hadoop or Spark APIs to read the data. If your use-case requires data to be processed directly on the cluster, you can use one of the following ways to ingest the data:



* SFTP
* WebHDFS
* Spark
* Spark-streaming
* Sqoop



See [Uploading files to HDFS](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-upload-files-hdfs).
* How do I configure my cluster?

You can configure a cluster by using customization scripts or by directly modifying configuration parameters in the Ambari console. Customization scripts are a convenient way to define different sets of configurations through a script, to spin up different types of clusters, or to use the same configuration repeatedly for repetitive jobs. See [Customizing a cluster](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-cust-cluster).
* Can I stop or shutdown my IBM Analytics Engine clusters to be charged on a per-use basis?

You are charged as long as the cluster is active and not on a per-use basis. For this reason, you should delete the instance after your job has completed and create a new instance before you start another job.
"spark.sql.catalog.lakehouse.type": "hive",
"spark.sql.catalog.lakehouse.uri": "<hms-thrift-endpoint-from-watsonx.data> for example (thrift://81823aaf-8a88-4bee-a0a1-6e76a42dc833.cfjag3sf0s5o87astjo0.databases.appdomain.cloud:32683) ",
"spark.hive.metastore.client.auth.mode": "PLAIN",
"spark.hive.metastore.client.plain.username": "<hms-user-from-watsonx.data> (for example, ibmlhapikey)",
"spark.hive.metastore.client.plain.password": "<hms-password-from-watsonx.data>",
"spark.hive.metastore.use.SSL": "true",
"spark.hive.metastore.truststore.type": "JKS",
"spark.hive.metastore.truststore.path": "file:///opt/ibm/jdk/lib/security/cacerts",
"spark.hive.metastore.truststore.password": "changeit"
}
Show more
* BASE_URL: The Analytics Engine URL for the region where you provisioned the instance. For example, api.region.ae.ibmcloud.com.
* INSTANCE_ID: The Analytics Engine instance ID. For more information about how to retrieve an instance ID, see [Obtaining the service endpoints](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-retrieve-endpoints-serverlessendpoints-cli).
* hms-thrift-endpoint-from-watsonx.data: Specify the credentials for watsonx.data.
* hms-user-from-watsonx.data: The watsonx.data username.
* hms-password-from-watsonx.data: The watsonx.data password.
--runtime ([Runtime](https://cloud.ibm.com/docs/analytics-engine-cli-plugin?topic=analytics-engine-cli-plugin-CLI_analytics_enginecli-runtime-example-schema))
: Runtime enviroment for applications and other workloads.

--jars (string)
: Path of the jar files containing the application.

--packages (string)
: Package names.

--repositories (string)
: Repositories names.

--files (string)
: File names.

--archives (string)
: Archive Names.

--name (string)
: Name of the application.

--class (string)
: Entry point for a Spark application bundled as a '.jar' file. This is applicable only for Java or Scala applications.

The value must match the regular expression /([\p{L}_$]\.)[\p{L}_$]/.

--arg ([]string)
: An array of arguments to be passed to the application.

--conf (generic map)
: Application configurations to override the value specified at instance level. See [Spark environment variables](https://spark.apache.org/docs/latest/configuration.htmlavailable-properties) for a list of the supported variables.

--env (generic map)
: Application environment configurations to use. See [Spark environment variables](https://spark.apache.org/docs/latest/configuration.htmlenvironment-variables) for a list of the supported variables.





 Examples 

ibmcloud analytics-engine-v3 spark-app submit --arg "/opt/ibm/spark/examples/src/main/resources/people.txt" --app "/opt/ibm/spark/examples/src/main/python/wordcount.py" --runtime='{"spark_version": "3.1"}' --output json
"spark.driver.cores" : "2"
}







 ibmcloud analytics-engine-v3 instance default-configs-update 

Update the default Spark configuration properties that will be applied to all applications of the instance.

ibmcloud analytics-engine-v3 instance default-configs-update [--id ID] --body BODY



 Command options 

-i, --id (string)
: The ID of the Analytics Engine instance. When specified, it overrides the target instance ID if it was set. Required, if the target instance was not set.

The value must match the regular expression /u0008[0-9a-f]{8}u0008-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-u0008[0-9a-f]{12}u0008/.

--body (generic map)
: Spark configuration properties to be updated. Properties will be merged with existing configuration properties. Set a property value to null in order to unset it. Required.





 Examples 

ibmcloud analytics-engine-v3 instance default-configs-update --body='{"ae.spark.history-server.cores" : "1", "ae.spark.history-server.memory" : "4G"}' --output json

ibmcloud analytics-engine-v3 instance default-configs-update --id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --body='{"ae.spark.history-server.cores" : "1", "ae.spark.history-server.memory" : "4G"}' --output json





 Example output 

The output is as follows:

{
"ae.spark.history-server.cores" : "1",
"ae.spark.history-server.memory" : "4G",
"spark.driver.memory" : "8G",
ibmcloud analytics-engine-v3 instance show --id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --output json





 Example output 

The output is as follows:

{
"id" : "dc0e9889-eab2-4t9e-9441-566209499546",
"state" : "active",
"state_change_time" : "2021-04-21T04:24:01Z",
"default_runtime" : {
"spark_version" : "3.1"
},
"instance_home" : {
"guid" : "30d5c4a7-9fb7-4712-9039-a79417dec87b",
"provider" : "ibm-cos",
"type" : "objectstore",
"region" : "us-south",
"endpoint" : "s3.direct.us-south.cloud-object-storage.appdomain.cloud",
"bucket" : "ae-bucket-do-not-delete-dc0e9889-eab2-4t9e-9441-566209499546",
"hmac_access_key" : "eHg=",
"hmac_secret_key" : "4d76"
},
"default_config" : {
"spark.driver.memory" : "4g",
"spark.driver.cores" : 1
}
}







 ibmcloud analytics-engine-v3 instance state 

Retrieve the state of a single Analytics Engine instance.

ibmcloud analytics-engine-v3 instance state [--id ID]



 Command options 

-i, --id (string)
: GUID of the Analytics Engine service instance to retrieve state. When specified, it overrides the target instance ID if it was set. Required, if the target instance was not set.
("io.xskipper.parquet.mdlocation.type", "EXPLICIT_BASE_PATH_LOCATION")])
Xskipper.setConf(spark, jvmConf)
 configure footer key
conf = dict([("io.xskipper.parquet.encryption.footer.key", "k1")])
xskipper.setConf(conf)
 adding the indexes
xskipper.indexBuilder() 
.addMinMaxIndex("temp", "k1") 
.addValueListIndex("city") 
.build(reader) 
.show(10, False)



If you want the footer + object name to be left in plain text mode (as mentioned above), you need to add the configuration parameter:



* For Scala:

// index the dataset
val xskipper = new Xskipper(spark, dataset_path)
// Configuring the JVM wide parameters
val jvmComf = Map(
"io.xskipper.parquet.mdlocation" -> md_base_location,
"io.xskipper.parquet.mdlocation.type" -> "EXPLICIT_BASE_PATH_LOCATION")
Xskipper.setConf(jvmConf)
// set the footer key
val conf = Map(
"io.xskipper.parquet.encryption.footer.key" -> "k1",
"io.xskipper.parquet.encryption.plaintext.footer" -> "true")
xskipper.setConf(conf)
xskipper
.indexBuilder()
// Add an encrypted MinMax index for temp
.addMinMaxIndex("temp", "k2")
// Add a plaintext ValueList index for city
.addValueListIndex("city")
.build(reader).show(false)


Show more
"pulse_shape":"drag","t0":160},{"ch":"d5","name":"fc","phase":"-(P1)","t0":320},{"ch":"u16","name":"fc","phase":"-(P2)","t0":0},{"ch":"u16","name":"fc","phase":"-(P0)","t0":160},{"ch":"u16","name":"fc","phase":"-(P1)","t0":320},{"ch":"u7","name":"fc","phase":"-(P2)","t0":0},{"ch":"u7","name":"fc","phase":"-(P0)","t0":160},{"ch":"u7","name":"fc","phase":"-(P1)","t0":320}]},{"name":"u3","qubits":6],"sequence":{"ch":"d6","name":"fc","phase":"-(P2)","t0":0},{"ch":"d6","label":"X90p_d6","name":"parametric_pulse","parameters":{"amp"
Element Classification 

The Element Classification enrichment is deprecated and will no longer be available, effective 10 July 2020.

Element Classification makes it possible to rapidly parse through governing documents to convert, identify, & classify elements of importance. Using state of the art Natural Language Processing, party (who it refers to), nature (type of element), and category (specific class) are extracted from elements of a document.

Element Classification is designed to provide:



* Natural Language Understanding of Contracts with emphasis on Software Procurement Contracts
* The ability to convert programmatic PDF to annotated JSON
* Identification of legal Entities and Categories that align with subject matter expertise



Element Classification brings together a functionally rich set of integrated, automated Watson APIs to input a programmatic PDF to identify: sections, lists (numbered and bulleted), footnotes, and tables converting these items into a structured HTML format. Furthermore, classification of this structured format is annotated and output as JSON with labeled elements, types, and categories.

Element Classification securely transmits your data performing encryption in-flight and at rest. For information about IBM Cloud security, see the [IBM Cloud Service Description](https://www.ibm.com/software/sla/sladb.nsf/searchsaas/?searchview&searchorder=4&searchmax=0&query=%28IBM+Cloud+Service+description%29)

Element Classification returns a JSON object that contains:



* A documents object that includes the input document's title, an HTML version of the input document, and the MD5 hash of the input document.
* Information about the model used to classify the input document.
* An elements array that details semantic elements identified in the input document.
* A tables array that breaks down the tables identified in the input document.
* A document_structure object that lists section titles and leading sentences identified in the input document.
* A parties array that lists the parties, roles, addresses, and contacts of parties identified in the input document.
* Arrays defining effective_dates, contract_amounts, termination_dates, contract_types, contract_terms, payment_terms, and contract_currencies.



This feature is currently supported in English only.
[Edit panel.](https://cloud.ibm.com/docs-content/v1/content/f61dbc5d84bf0c89e477a05cab04c72f429e1b7a/log-analysis/images/streaming-topic-metric.png)

Then, customize the metric to see data per topic.

![Customize metric.](https://cloud.ibm.com/docs-content/v1/content/f61dbc5d84bf0c89e477a05cab04c72f429e1b7a/log-analysis/images/streaming-topic-metric-1.png)

Check that the resolution is set to 10M.







 Monitoring streaming by using IBM Cloud Activity Tracker 

Streaming generates Activity Tracker events with the action logdna.streaming-logs.send to notify about failures when data is streamed to Event Streams.

There are different reasons for failure, for example:



* unknown_topic or unknown_topic_or_partition: This notification reports that the topic that is configured for streaming is not valid.

If you get this notification, check the topic that is configured for streaming is defined in Event Streams.
* broker_handle_destroyed: This notification reports a change in the Kafka cluster configuration.

If you get this notification, check the SASL URLs and update the streaming configuration.
* authentication_failure: This notification reports a failure to authenticate with Event Streams.

If you get this notification, check the credentials that are configured for streaming.
* broker_transport_failure: This notification reports connectivity problems with Event Streams. There are different reasons why you might get this notification such as that credentials were deleted after streaming was activated and started.
* message_timed_out or timed_out: This notification reports generic timeout errors when streaming messages to Event Streams.



The Log Analysis event includes different fields that you can use to monitor streaming data failures:



* reason.reasonCode = 500 and severity = critical notify of a failure streaming data.
* reason.reasonType indicates the type of failure.
! ! ,",
"2019-06-24T16:46:10.428693Z stdout: dbname: 'test',",
"2019-06-24T16:46:10.428697Z stdout: id: '6ca436c44074c4c2aa6a40c9a188b348',",
"2019-06-24T16:46:10.428700Z stdout: seq: '103-g1AAAAeLeJy91M9NwzAUBnCrrVQqDvTKCa4gpcT5YycnugFsAH5' }"
<-- </ol> --><-- <section "id="section-cloudant_struct" "> --> Data structure of a trigger activation The content of the generated event has the following parameters.<-- <table> -->Table 2. Parameters for the generated event| Parameter | Description || --------- | ---------------------------------------------------------------------------------------------------- || id | The document ID. || seq | The sequence identifier that is generated by IBM Cloudant. || changes | An array of objects, each of which has a rev field that contains the revision ID of the document. |<-- </table ""> -->The following example is a JSON representation of the trigger activation. {
"dbname": "test",
"id": "6ca436c44074c4c2aa6a40c9a188b348",
What is the difference between 'Class A' and 'Class B' requests? 

'Class A' requests are operations that involve modification or listing. This includes creating buckets, uploading or copying objects, creating or changing configurations, listing buckets, and listing the contents of buckets.'Class B' requests are those related to retrieving objects or their associated metadata/configurations from the system. There is no charge for deleting buckets or objects from the system.





 What is the best way to structure your data by using Object Storage so you can 'look' at it and find what you are looking for? Without a directory structure, having thousands of files at one level is difficult to view. 

You can use metadata that is associated with each object to find the objects you are looking for. The biggest advantage of Object Storage is the metadata that is associated with each object. Each object can have up to 4 MB of metadata in Object Storage. When offloaded to a database, metadata provides excellent search capabilities. Many (key, value) pairs can be stored in 4 MB. You can also use Prefix searching to find what you are looking for. For example, if you use buckets to separate each customer data, you can use prefixes within buckets for organization. For example: /bucket1/folder/object where 'folder/' is the prefix.





 Can you confirm that Object Storage is ‘immediately consistent’, as opposed to ‘eventually consistent’? 

Object Storage is ‘immediately consistent’ for data and ‘eventually consistent’ for usage accounting.





 Can Object Storage partition the data automatically using HDFS, so I can read the partitions in parallel, for example, with Spark? 

Object Storage supports a ranged GET on the object, so an application can do a distributed striped-read-type operation. Doing the striping is managed by the application.





 Can I restore a bucket from a specific back-up file? 

It is possible to overwrite an existing bucket. Restore options depend on the capabilities provided by the back-up tool you use; check with your back-up provider.
curl --location --request PATCH "https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_id>/default_configs" --header "Authorization: bearer <iam_token>" --header 'Content-Type: application/json' --data-raw '{
"ae.spark.history-server.cores": "2",
"ae.spark.history-server.memory": "8G"
}'

Only a pre-defined list of Spark driver and executor vCPU, and memory size combinations are supported. See [Supported Spark driver and executor vCPU and memory combinations](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-limitscpu-mem-combination).





 Additional customisations 

You can customize the Spark history server further by adding properties to the default Spark configuration of your Analytics Engine instance. See standard [Spark history configuration options](https://spark.apache.org/docs/latest/monitoring.htmlspark-history-server-configuration-options).

As this is a managed offering, you can't customize all of the standard Spark configuration options.

For a list of the supported configurations and default values for settings, see [Default Spark configuration](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-serverless-architecture-conceptsdefault-spark-config).







 Best practices 

Always stop the Spark history server when you no longer need to use it. Bear in mind that the Spark history server consumes CPU amd memory resources continuously while its state is Started.