vania-janet/multiturn-rag-retrieval-data huggingface dataset:
* "spark.eventLog.enabled": "true"



For the default limits and quotas for Analytics Engine instances and the supported Spark driver and executor vCPU and memory combinations, see [Limits and quotas for Analytics Engine instances](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-limits).

When you create an instance you can override the open source default Apache Spark configuration settings. Note that you can specify or change configuration options after the instance was created. You can override or add settings by using the REST API. See [Spark application REST API](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-spark-app-rest-api).







 Serverless instance features and execution methods 

The following table shows the supported serverless instance features by access role and execution methods.



Table 1. Supported serverless instance features by access role and execution methods

 Operation Access roles IBM Console API CLI 

 Provision instances Administrator ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) 
 Delete instances Administrator !
curl https://api.<region>.ae.cloud.ibm.com/v3/analytics_engines/<iae-instance-guid>/spark_applications -H "Authorization: Bearer <iam-bearer-token>" -X POST -d '{
"application_details": {
"application": "s3a://<application_bucket>/lakehouse-hms-test-cloud-doc-sample.py",
"conf": {
"spark.hadoop.fs.s3a.bucket.<application-bucket>.endpoint": "https://s3.direct.us-south.cloud-object-storage.appdomain.cloud",
"spark.hadoop.fs.s3a.bucket.<application-bucket>.access.key": "<hmac_access_key_for_application-bucket>",
"spark.hadoop.fs.s3a.bucket.<application-bucket>.secret.key": "<hmac_secret_key_for_application-bucket>"
}
}
}'



This sample is tested on the Cloud Object Storage buckets in the us-south region. Change the region in the Cloud Object Storage endpoint configuration as per the region where your Cloud Object Storage buckets reside. It is recommended to provision the COS buckets in the region where Analytics Engine instance is provisioned.
audit-log.bare-metal.power-off An event is generated when an initiator powers off a bare metal server. 
 audit-log.bare-metal.reboot An event is generated when an initiator reboots a bare metal server. 
 audit-log.bare-metal.soft-reboot An event is generated when an initiator does a soft reboot on a bare metal server. 
 audit-log.bare-metal.hard-reboot An event is generated when an initiator does a hard reboot on a bare metal server. 
 audit-log.bare-metal.power-on An event is generated when an initiator powers on a bare metal server. 
 audit-log.bare-metal.rename An event is generated when an initiator renames a bare metal server. 
 audit-log.bare-metal.rescue An event is generated when an initiator rescues a bare metal server. 
 audit-log.bare-metal.reload An event is generated when an initiator performs an operating system (OS) reload for a bare metal server. 







 Viewing events 

The Activity Tracker events are available in the Activity Tracker account domain that is available in the IBM Cloud region where the events are generated. For more information, see [Viewing account events](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-view_events).

Activity Tracker events are automatically forwarded to the Activity Tracker service in the same region where the action happens.
SELECT version, collector_crn, attached_endpoint_type, network_interface_id, instance_crn, capture_start_time, capture_end_time, vnic-id, record-type, year, month, day, hour, stream-id, flow.
FROM EXPLODED_FLOW



You can verify that flow log data is being collected by using IBM Cloud Monitoring. For more information, see [Monitoring flow logs for VPC metrics](https://cloud.ibm.com/docs/vpc?topic=vpc-fl-monitoring-metrics).



 Optimizing flow logs layout with IBM Cloud® Data Engine 

When large amounts of flow logs are analyzed, it is recommended to convert flow logs to a layout optimal for queries. This conversion improves query execution time by at least one order of magnitude. For more information about data optimization on Object Storage, see [How to Layout Big Data in IBM Cloud Object Storage for Spark SQL](https://www.ibm.com/cloud/blog/big-data-layout).

The following SQL statement is an ETL job addresses two aspects that contribute significantly to query execution time:



* object size - Flow logs have a maximum size of 100 KB. An optimal object size for a query is approximately 128 MB.
* data format - Flow logs are stored as compressed JSON. The optimal format for queries is parquet, which allows to read only the columns that a query needs, not the whole object.



Note the INTO clause, which defines the target location and partitioning layout of the data that is produced by this ETL job.

Important: Replace these variables in the following steps:



* bucket - The bucket where you want to store the optimized flow logs.
* region - The [region alias](https://cloud.ibm.com/docs/sql-queryendpoints) of the bucket that holds your flow logs.



To optimize flow logs layout with IBM Cloud® Data Engine, follow these steps:



1. Convert all flow logs for a specific date to parquet, writing one parquet object per hour:

SELECT * FROM FLOW_FLAT
WHERE
day = day(DATE("2020-07-24")) AND
* URL: org_id.messaging.internetofthings.ibmcloud.com

Where org_id is the ID of your Watson IoT Platform organization.
* Port:



* 1883 (disabled by default)
* 8883 (encrypted)
* 443 (websockets)



* Device ID: org_id:device_type:device_id
* User name: use-token-auth
* Password: Authentication token
* Event topic format: iot-2/evt/event_id/fmt/format_string

Where event_id specifies the event name that's shown in Watson IoT Platform, and format_string is the format of the event, such as JSON.
* Message format: JSON
* Example payload: { "d": {"key1": "value1"} }



For more information, see [MQTT connectivity for devices ![External link icon](https://cloud.ibm.com/docs-content/v1/content/icons/launch-glyph.svg)](https://www.ibm.com/support/knowledgecenter/SSQP8H/iot/platform/devices/mqtt.html).







 What's next 

Extend the data analytics features by creating and connecting your own apps to consume device data.



* For more information on how to connect specific device types to Watson IoT Platform, see [developerWorks recipes ![External link icon](https://cloud.ibm.com/docs-content/v1/content/icons/launch-glyph.svg)](https://developer.ibm.com/recipes/tutorials/category/internet-of-things-iot/).
* Check out the [client libraries !
About Watson Discovery 

IBM Watson® Discovery is an intelligent document processing engine that helps you to gain insights from complex business documents.

Use Discovery to visually train AI for deep understanding of your content, including tables and images, to help you find business value that is hidden in your enterprise data. Use natural language or structured queries to find relevant answers, surface insights, and build AI-enhanced business processes anywhere.

Start by connecting your data to Discovery. Next, teach Discovery to understand the language and concepts that are unique to your business and industry. Enrich your data with award-winning Watson Natural Language Processing (NLP) technologies so you can identify key information and patterns. Finally, build search solutions that find answers to queries, explore your data to uncover patterns and insights, and leverage search results in automated workflows.

Zoom

![Use Watson Discovery to Connect and ingest data from existing content repositories. Teach Discovery to interpret your data. Enrich data with custom NLU to help it fit your domain. Next, choose whether to 1. Use the query API to get search Results by retrieving relevant data in your applications. 2. Explore your data in Watson Assistant, with the Discovery API, or from the Content Mining application. 3. Submit queries from the product user interface to ask a question in a natural language.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/aboutdiscovery.png)

Figure 1. How to use Watson Discovery

Find out how Discovery is transforming data into artist insights at the 2023 GRAMMYs®. Read the [IBM Business Operations blog post](https://www.ibm.com/blogs/internet-of-things/watson-discovery-grammys) to learn more.



 Overview video 

Watch a video about how Discovery uses AI-powered search, retrieval, and content mining. This overview covers the key basics of projects, collections, fields, and enrichments.
: Result value

0
: Example of an SQL function usage fragment

> SELECT ceiling(5)
: Result value

5





 char 

char(expr)
: Returns the ASCII character that has the binary equivalent to expr. If n is larger than 256, the result is equivalent to chr(n % 256).
: Example of an SQL function usage fragment

> SELECT char(65)
: Result value

A





 char_length 

char_length(expr)
: Returns the character length of string data or number of bytes of binary data. The length of string data includes the trailing spaces. The length of binary data includes binary zeros.
: Example of an SQL function usage fragment

> SELECT char_length('Spark SQL ')
: Result value

10
: Example of an SQL function usage fragment

> SELECT CHAR_LENGTH('Spark SQL ')
: Result value

10
: Example of an SQL function usage fragment

> SELECT CHARACTER_LENGTH('Spark SQL ')
: Result value

10





 character_length 

character_length(expr)
: Returns the character length of string data or number of bytes of binary data. The length of string data includes the trailing spaces. The length of binary data includes binary zeros.
: Example of an SQL function usage fragment

> SELECT character_length('Spark SQL ')
: Result value

10
: Example of an SQL function usage fragment

> SELECT CHAR_LENGTH('Spark SQL ')
: Result value

10
: Example of an SQL function usage fragment

> SELECT CHARACTER_LENGTH('Spark SQL ')
: Result value

10





 chr 

chr(expr)
: Returns the ASCII character that has the binary equivalent to expr. If n is larger than 256, the result is equivalent to chr(n % 256).
: Example of an SQL function usage fragment

> SELECT chr(65)
: Result value

A





 coalesce 

coalesce(expr1, expr2, ...)
spark.sql('ALTER TABLE lakehouse.demodb.yellow_taxi_2022 ADD COLUMN(fare_per_mile double)')
 describe the table
spark.sql('describe table lakehouse.demodb.yellow_taxi_2022').show(25)
finally:
 clean-up the demo database
spark.sql('drop table if exists lakehouse.demodb.testTable purge')
spark.sql('drop table if exists lakehouse.demodb.zipcodes purge')
spark.sql('drop table if exists lakehouse.demodb.yellow_taxi_2022 purge')
spark.sql('drop database if exists lakehouse.demodb cascade')
spark.stop()

if __name__ == '__main__':
main()
Show more



1. Save the following sample python file.
2. Upload the Python file to the Cloud Object Storage bucket. You must maintain the Spark applications and their dependencies in a Cloud Object Storage bucket and not mix them with data buckets.
3. Generate the IAM token for the IBM Analytics Engine token. For more information about how to generate an IAM token, see [IAM token](https://test.cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-con-presto-thru-cliget-api-iam-token)
4. Run the following curl command to submit the Spark application:

curl https://api.<region>.ae.cloud.ibm.com/v3/analytics_engines/<iae-instance-guid>/spark_applications -H "Authorization: Bearer <iam-bearer-token>" -X POST -d '{
"application_details": {
"application": "s3a://<application_bucket>/lakehouse-hms-test-cloud-doc-sample.py",
"conf": {
If the isJSONData parameter was set to false (or not set at all) when the trigger was created, the value field is the raw value of the posted message. However, if isJSONData was set to true when the trigger was created, the system attempts to parse this value as a JSON object, on a best-effort basis. If parsing is successful, then the value in the trigger payload is the resulting JSON object.

If a message of {"title": "Some string", "amount": 5, "isAwesome": true} is posted with isJSONData set to true, the trigger payload might look something like the following example.

{
"messages": [
{
"partition": 0,
"key": null,
"offset": 421760,
"topic": "mytopic",
"value": {
"amount": 5,
"isAwesome": true,
"title": "Some string"
}
}
]
}

However, if the same message content is posted with isJSONData set to false, the trigger payload would look like the following example.

{
"messages": [
{
"partition": 0,
"key": null,
"offset": 421761,
"topic": "mytopic",
"value": "{"title": "Some string", "amount": 5, "isAwesome": true}"
}
]
}

Similar to isJSONData, if isBinaryValue was set to true during trigger creation, the resultant value in the trigger payload is encoded as a Base64 string.

If a value of Some data is posted with isBinaryValue set to true, the trigger payload might look something like the following example.

{
"messages": [
{
"partition": 0,
"key": null,
"offset": 421760,
"topic": "mytopic",
"value": "U29tZSBkYXRh"
}
[the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) 
 Invoke the instance management REST API to set instance home ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) 
 Invoke application management REST API to submit a Spark application ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) 
 Invoke application management REST API to stop a submitted application ![the confirm icon](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/confirm.png) !
"spark.sql.catalog.lakehouse.type": "hive",
"spark.sql.catalog.lakehouse.uri": "<hms-thrift-endpoint-from-watsonx.data> for example (thrift://81823aaf-8a88-4bee-a0a1-6e76a42dc833.cfjag3sf0s5o87astjo0.databases.appdomain.cloud:32683) ",
"spark.hive.metastore.client.auth.mode": "PLAIN",
"spark.hive.metastore.client.plain.username": "<hms-user-from-watsonx.data> (for example, ibmlhapikey)",
"spark.hive.metastore.client.plain.password": "<hms-password-from-watsonx.data>",
"spark.hive.metastore.use.SSL": "true",
"spark.hive.metastore.truststore.type": "JKS",
"spark.hive.metastore.truststore.path": "file:///opt/ibm/jdk/lib/security/cacerts",
"spark.hive.metastore.truststore.password": "changeit"
}
Show more
* INSTANCE_ID: The Analytics Engine instance ID. For more information about how to retrieve an instance ID, see [Obtaining the service endpoints](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-retrieve-endpoints-serverlessendpoints-cli)
* Hms-thrift-endpoint-from-watsonx.Data: Specify the credentials for watsonx.data. For more information on getting the HMS credentials, see [Getting (Hive metastore) HMS Credentials](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-hms).
* Hms-user-from-watsonx.Data: The watsonx.data username.
logdna.streaming-samples.send This event is generated when sample data is sent to verify the connection. 
 logdna.account-streaming-setting.configure This event is generated when you start streaming. 
 logdna.streaming-configuration.deactivate This event is generated when you stop streaming. 
 logdna.streaming-logs.send This event is generated when there is a failure streaming data. 
 logdna.exclusion-rule.create This event is generated when an streaming exclusion rule is configured. 
 logdna.exclusion-rule.delete This event is generated when an streaming exclusion rule is deleted.
Spark application REST API 

The IBM Analytics Engine serverless plan provides REST APIs to submit and manage Spark applications. The following operations are supported:



1. [Get the required credentials and set permissions](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-spark-app-rest-apispark-app-creds).
2. [Submit the Spark application](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-spark-app-rest-apispark-submit-app).
3. [Retrieve the state of a submitted Spark application](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-spark-app-rest-apispark-app-status).
4. [Retrieve the details of a submitted Spark application](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-spark-app-rest-apispark-app-details).
5. [Stop a running Spark application](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-spark-app-rest-apispark-app-stop).



For a description of the available APIs, see the [IBM Analytics Engine REST APIs for the serverless plan](https://test.cloud.ibm.com/apidocs/ibm-analytics-engine-v3introduction).

The following sections in this topic show samples for each of the Spark application management APIs.



 Required credentials and permissions 

Before you can submit a Spark application, you need to get authentication credentials and set the correct permissions on the Analytics Engine serverless instance.



1. You need the GUID of the service instance you noted down when you provisioned the instance. If you didn't make a note of the GUID, see [Retrieving the GUID of a serverless instance](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-retrieve-instance-details).
2.
* 20 minutes
* 2023-03-31



[Power your assistant with answers from web resources](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-tutorial-assistant-fred)Power your assistant with answers from web resources

In this tutorial, you will use the Watson Discovery and Watson Assistant services to create a virtual assistant that can answer questions about the latest research from the US Federal Reserve. The assistant will answer questions by using up-to-date, existing research publications from the Federal Reserve Economic Data (FRED) website.

Discovery v2



* 2 hours
* 2023-03-24



[Use Smart Document Understanding (SDU) to improve search results](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-tutorial-sdu)Use Smart Document Understanding (SDU) to improve search results

In this tutorial, you use the Smart Document Understanding feature of the Discovery service to create a user-trained Smart Document Understanding (SDU) model. You then split a single document into many smaller documents so that some types of answers are easier to find.

Discovery v2



* 3 hours
* 2023-03-16



[Text analysis with Code Engine](https://cloud.ibm.com/docs/solution-tutorials?topic=solution-tutorials-text-analysis-code-engine)Text analysis with Code Engine

In this tutorial, you will learn about IBM Cloud® Code Engine by deploying a text analysis with Natural Language Understanding application. You will create a Code Engine project, select the project and deploy Code Engine entities - applications and jobs - to the project. You will learn how to bind IBM Cloud services to your Code Engine entities. You will also understand the auto-scaling capability of Code Engine where instances are scaled up or down (to zero) based on incoming workload.

Code Engine Kubernetes service

+2

Object Storage,Natural Language Understanding



* 2 hours
* 2023-05-05
Input adapter 



* class - Internal use only; defines the Data Crawler input adapter class. The default value is: com.ibm.watson.crawler.connectorframeworkinputadapter.Crawl
* config - Internal use only; defines the connector framework configuration. The default configuration key within this block to pass to the chosen input adapter is: connector_framework

The connector framework is what allows you to talk to your data. It could be internal data within the enterprise, or it could be external data on the web or in the cloud. The connectors allow access to a number of different data sources, while connecting is actually controlled by the crawling process.

Data retrieved by the Connector Framework Input Adapter is cached locally. It is not stored encrypted. By default, the data is cached to a temporary directory that you must clear after performing a reboot and it must be readable only by the user who executed the crawler command.

There is a chance that this directory can outlive the crawler if the connector framework is removed before it can clean up after itself. Consider the location for your cached data. You can put data on an encrypted filesystem, but that might have performance implications. Pick the appropriate balance between speed and security for your crawls.
* crawl_config_file - The configuration file to use for the crawl. Default value is: connectors/filesystem.conf
* crawl_seed_file - The crawl seed file to use for the crawl. Default value is: seeds/filesystem-seed.conf
* id_vcrypt_file - Keyfile used for data encryption by the Crawler; the default key included with the crawler is id_vcrypt. Use the vcrypt script in the bin folder if you need to generate a new id_vcrypt file.
* crawler_temp_dir - The Crawler temporary folder for connector logs. Default value, tmp, is provided. If it doesn't already exist, the tmp folder is created in the current working directory.
* extra_jars_dir - Adds a directory of extra JARs to the connector framework classpath.
Release notes for IBM Analytics Engine serverless instances 

Use these release notes to learn about the latest updates to IBM Analytics Engine serverless instances that are grouped by date.



 July 2023 



 07 July 2023 

Spark maintenance release version update for 3.3
: Spark applications with runtime set to Spark 3.3 will run internally using Spark 3.3.2 from now on. The patch version is now upgraded from 3.3.0 to 3.3.2.







 May 2023 



 29 May 2023 

Removal of Python v3.9 support from Spark 3.1 and Spark 3.3 runtimes
: The IBM Analytics Engine - Serverless Spark application plans to discontinue Python v3.9 support from Spark 3.1 and Spark 3.3 runtimes by June15, 2023. Support for Python v3.10 is already deployed for Spark 3.1 and Spark 3.3 runtime. Based on the workload, make sure that you test your spark application with the new version of Python, v 3.10 for any failures before June 15, 2023. Contact IBM Support for any issues. See the procedure for testing the Spark application [Run a Spark application with nondefault language version](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-spark-app-rest-apirun-a-spark-application-with-non-default-language-version).





 25 May 2023 

Pagination of application list in the REST API and CLI
: You can now limit the number of applications returned by the Analytics Engine serverless REST API endpoint, SDK method and CLI command for [listing applications](https://cloud.ibm.com/apidocs/ibm-analytics-engine/ibm-analytics-engine-v3list-applications). Use the query parameter limit to specify the number of applications to be returned and specify the value of next.start or previous.start from the API response as the value of the start query parameter to fetch the next or previous page of the results. The applications are listed in descending order based on the submission time, with the newest application being the first.

The pagination is an optional feature in this release.