vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Edit and push code through the rich web based IDE. Create toolchains to enable tool integrations that support your development, deployment, and operation tasks. The Continuous Delivery service includes Delivery Pipeline, Eclipse Orion Web IDE, and Git Repos and Issue Tracking. For more information,see [Getting started with Continuous Delivery](https://cloud.ibm.com/docs/ContinuousDelivery?topic=ContinuousDelivery-getting-started).





 Adding metrics for IBM Cloud® Monitoring 

IBM® Cloud Foundry apps can configure and expose metrics that can be monitored using IBM Cloud Monitoring. For more information, see [Monitoring Cloud Foundry metrics](https://cloud.ibm.com/docs/Monitoring-with-Sysdig?topic=Monitoring-with-Sysdig-monitor-cf-sysdig).







 Tips 

Use the following tips while developing your web apps.



 Persistence 

Don't specify any local storage for your apps. Each instance of your app, even if only one instance is running, can be restarted or moved to a different virtual machine at any time, typically for load balancing. Anything stored in local storage is erased when the app is moved or deleted. Use one of the IBM Cloud data store services for persistence.





 Resource limits 

Be aware of limits on the quantities of resources that a trial account can use. The limits are as follows.



Table 1. IBM Cloud resource limits for a trial account

 Resource type Quantity limit 

 Number of services that are used across all apps 10 
 Memory used across all apps 2 G 
 Number of routes 500
But don't forget that there are many CLI and GUI tools readily available for use as part of your migration.



* [COS CLI](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-cli-plugin-ic-cos-cli) can be used for many operations. For example, you may wish to use the CLI to configure your IBM Cloud Object Storage instances, and to create and configure buckets.
* [AWS CLI](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-aws-cli) can be used to list your current bucket's contents to prepare for migrating from AWS, among other operations:



aws s3 ls --recursive s3://<BUCKET_NAME> --summarize > bucket-contents-source.txt



* [rclone](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-rclone) has many uses, and we'll look at it specifically, next.





 Migrate your data 

Based on the process and tools you've chosen, you will want to choose a strategy for migrating your data. We can take a look at a simplified process using the command line and the Go-based rclone executable as an example.



1. Install rclone from [either a package manager or precompiled binary](https://rclone.org/install/). There are more configuration options available with explanations at the IBM Cloud Object Storage [documentation](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-rclone).

curl https://rclone.org/install.sh | sudo bash





 Configure rclone with your AWS credentials 

You may start by creating 'profiles' for your source and destination of the migration in rclone. A profile contains the configuration and credentials needed for working with your date. You will be migrating from AWS, so you will need those credentials to continue.
It is used by the language understanding pipeline microservices (clu-controller, clu-serving, clu-training, tf-mm, ed-mm, and dragonfly) to store and load trained models for intent and entity classification. It is used by the authoring and runtime service (store) to store versioned data of the skills and also used as a temporary storage for asynchronous uploads for the authoring experience. In Watson Assistant, Multicloud Object Gateway is often referred to as Cloud Object Storage. For more information, see [Multicloud Object Gateway](https://www.ibm.com/docs/en/cloud-paks/cp-data/4.7.x?topic=cluster-installing-multicloud-object-gateway).
* Redis: An in-memory data store, often used for caching or sharing session state. Redis is used by the Store microservice for storing current conversation state for assistants. The UI microservice stores session state in Redis. The Recommends and Dialog microservices use a Redis instance as a cache.



1.5.0: The following data source is no longer used, starting with the 1.5.0 release:



* MongoDB: A document-oriented database. Mongo is used in read-only mode for storing embeddings and other data that is used by the Recommends microservice for synonym recommendations. For more information, see [MongoDB database](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-architecturearchitecture-mongodb).

MongoDB is a document-based, distributed database. The MongoDB database has three pods. It runs in replicaSet mode, which means that one pod runs in the coordinator role in read/write mode, and the rest of the pods run in the secondary role and are read-only. Changes from the coordinator pod are replicated to the secondary pods. During the service installation, data is loaded into the Mongo database by a Kubernetes job. No data is written to the Mongo database after the service is installed.
* SITE-POLICIES [golang/tools (go)] (Copyright (c) 2009 The Go Authors)([https://github.com/golang/tools/archive/4735f53fb37db012e63d1a62914417980fb3c023.zip](https://github.com/golang/tools/archive/4735f53fb37db012e63d1a62914417980fb3c023.zip))







 Creative Commons Attribution Share Alike 2.0 generic 

The Program includes some or all of the following works licensed under the Creative Commons. The URL to the license is [https://creativecommons.org/licenses/by-sa/2.0/legalcode](https://creativecommons.org/licenses/by-sa/2.0/legalcode).

DISPUTES [npm] (Copyright (c) npm, Inc. and Contributors)([http://registry.npmjs.org/npm/-/npm-6.14.6.tgz](http://registry.npmjs.org/npm/-/npm-6.14.6.tgz))





 SIL Open Font License 1.1 

The Program includes some or all of the following works licensed under the SIL Open Font License 1.1.

PREAMBLE The goals of the Open Font License (OFL) are to stimulate worldwide development of collaborative font projects, to support the font creation efforts of academic and linguistic communities, and to provide a free and open framework in which fonts may be shared and improved in partnership with others.

The OFL allows the licensed fonts to be used, studied, modified, and redistributed freely as long as they are not sold by themselves. The fonts, including any derivative works, can be bundled, embedded, redistributed and/or sold with any software provided that any reserved names are not used by derivative works. The fonts and derivatives, however, cannot be released under any other type of license. The requirement for fonts to remain under this license does not apply to any document created using the fonts or their derivatives.
Developer tools 



 Style checkers 

Recommended style checkers are



* Go: [golint](https://github.com/golang/lint)
* Java: [checkstyle](http://checkstyle.sourceforge.net/)
* Node: [eslint](https://eslint.org/)
* Python: [pylint](https://www.pylint.org/)
* Swift: [SwiftLint](https://github.com/realm/SwiftLint)







 Code coverage 

Recommended test coverage tools are:



* Go: [cover](https://golang.org/cmd/cover/)
* Java: [jacoco](http://www.jacoco.org/jacoco/trunk/index.html)
* Node: [jest](https://jestjs.io/)
* Python: [coverage](https://pypi.python.org/pypi/coverage)
* Swift: [Codecov](https://codecov.io/)







 Package management / distribution 

It is highly recommended that individual services belonging to an IBM Cloud service category are delivered together within a single SDK project (one project per language). For example, the services within the Platform Services category are contained within the platform-services-go-sdk project.

Recommended package management / distribution systems:



 Go 

Your Go SDK should be packaged as a [Go Module](https://blog.golang.org/using-go-modules), using the GitHub repository URL as a package name.





 Java 

Your Java SDK should publish its artifacts on [Maven Central](https://search.maven.org/). It is recommended that artifacts use a Maven groupId of com.ibm.cloud.





 Node
ibmanalyticsengine.application.delete Delete a Spark application request or a running Spark application 
 ibmanalyticsengine.livybatch.create Create a Spark application through the Livy batch interface 
 ibmanalyticsengine.livybatch.read Retrieve the details of a Livy batch application, including the state 
 ibmanalyticsengine.livybatch.delete Delete a Livy application request or a running Livy batch application 







 Viewing events 

Events that are generated by an instance of the IBM Analytics Engine service are automatically forwarded to the IBM Cloud Activity Tracker service instance that is available in the same location.

IBM Cloud Activity Tracker can have only one instance per location. To view events, you must access the web UI of the IBM Cloud Activity Tracker service in the same location where your service instance is available. For more information, see [Launching the web UI through the IBM Cloud UI](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-launch).
* [Where to find support](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ansibleansible-support)



[Building an IBM Blockchain Platform network using Ansible playbooks](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ansible-buildansible-build)



* [Prerequisites](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ansible-buildansible-build-prereqs)
* [Step one: Gather console connection information](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ansible-buildansible-build-console)
* [Step two: Clone the Ansible collection](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ansible-buildansible-build-clone)
* [Step three: Get started with using the playbooks](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ansible-buildansible-build-get-start)
* [Step four: Run a playbook](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ansible-buildansible-build-run-playbook)
* [Step five: Run a script](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ansible-buildansible-build-run-script)
* [Hints and Tips](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ansible-buildansible-build-hints)
* [Next steps](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ansible-buildansible-build-next-steps)
Tombstones are used for more consistent deletion of documents from databases. This purpose is especially important for mobile devices: without tombstone documents, a deletion might not replicate correctly to a mobile device, with the result that documents might never be deleted from the device.

If you re-create a database, for example, a new target for a replication. Any clients that use the target database as a server must work through all the changes again because the database sequence numbers are likely to be different.

If you're using a validate_doc_update function, avoid replicating that function to clients. This rule is to prevent the possibility of unwanted side effects that result from having the function present on the client.

[IBM Cloudant sync](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-client-librariesmobile-supported) libraries don't replicate design documents, so replication of validate_doc_update functions is not normally a problem for IBM Cloudant. However, other clients might replicate the design documents or validate_doc_update functions, potentially resulting in unwanted side effects.
[Use an approved reference architecture](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-best-practicesbest-practices-reference-architecture) <br><br> * Use the [Satellite reference architecture](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-satellite-architecture-about) described here.<br><br><br> <br><br> * Use the [Satellite reference architecture](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-satellite-architecture-about) described here.<br><br><br> 
 [Use only services that are IBM Cloud for Financial Services Validated](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-best-practicesbest-practices-financial-services-validated-services) <br><br> * Use the [Financial Services Validated services](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-satellite-architecture-aboutcomponents-ibm-cloud-services) that are part of the Satellite reference architecture.<br><br><br> <br><br> * No infrastructure (beyond what is provided for your Satellite location) is needed to use Financial Services Validated services that are part of the reference architecture.<br><br><br> 
 [Ensure that all deployed software meets all control requirements](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-best-practicesbest-practices-self-installed-software) <br><br> * Consider the software that you plan to install into your workload clusters in your Satellite location. Ensure that software is developed, deployed, configured, and managed in a way that is consistent with the control requirements.<br><br><br> <br><br> * Consider the pieces of software that you plan to install into any part of your underlying infrastructure in your Satellite location or your supporting on-premises environment outside of Satellite.
Sample instructions are provided for using a client application or command line once your CLI client has been configured and is operational.



 rClone example 

The rclone tool is typically used to keep directories synchronized and for migrating data between storage platforms. You can learn more from the documentation on [!using rclone](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-rclone).

rclone purge {remote}:{path} [flags]





 Minio example 

The open source Minio client allows you to use UNIX-like commands (ls, cp, cat, etc.) with IBM Cloud® Object Storage. For more information, check out [using Minio](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-minio).

mc rm --recursive --force {instance-alias}/{bucket-name}





 AWS example 

The official command-line interface for AWS is compatible with the IBM Cloud Object Storage S3 API and you can find out more on how to [use the AWS CLI](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-aws-cli).

aws s3 rm s3://{bucket-name} --recursive







 Code Example 

Deleting an entire directory or removing all the contents of a bucket can be time consuming deleting each object, one at a time. The ability to delete one item at a time can be leveraged to save time and effort by collecting a list of all the items before deletion.

The topic itself points out the danger of deletion: data will be lost. Of course, when that is the goal, caution should be exercised that only the targeted deletions should occur. Check—and double-check—instances, bucket names, and any prefixes or paths that need to be specified.



 Overview 

The code pattern in this exercise configures a client before creating one for the purpose of gathering a list of items for the purpose of deleting each object.
Entity type systems 

The entity type system that Discovery uses differs depending on which version date and which language you are using. This page describes, for each version date, which type system is used for each language.

For example, analyzing entities in French text with version date set to 2018-09-21 uses the [Version 2 entity type system](https://cloud.ibm.com/docs/discovery?topic=discovery-entity-types-version-2). Analyzing entities in French text with a 2017-02-27 version date uses the [Version 1 entity type system](https://cloud.ibm.com/docs/discovery?topic=discovery-entity-types-version-1). For information about available enrichments, see [Adding enrichments](https://cloud.ibm.com/docs/discovery?topic=discovery-configserviceadding-enrichments).



 Entity type systems for version 2018-11-16 

The following entity type systems are used when you set the version parameter to 2018-11-16.



 Language Entity type system 

 English [Version 1](https://cloud.ibm.com/docs/discovery?topic=discovery-entity-types-version-1) 
 French [Version 2](https://cloud.ibm.com/docs/discovery?topic=discovery-entity-types-version-2) 
 German [Version 2](https://cloud.ibm.com/docs/discovery?topic=discovery-entity-types-version-2) 
 Italian [Version 2](https://cloud.ibm.com/docs/discovery?topic=discovery-entity-types-version-2) 
 Japanese [Version 2](https://cloud.ibm.com/docs/discovery?topic=discovery-entity-types-version-2) 
 Korean [Version 1](https://cloud.ibm.com/docs/discovery?topic=discovery-entity-types-version-1) 
 Brazilian Portuguese [Version 2](https://cloud.ibm.com/docs/discovery?topic=discovery-entity-types-version-2)
"spark.hadoop.fs.cos.cosservicename.access.key": "<CHANGEME>",
"spark.hadoop.fs.cos.cosservicename.secret.key": "<CHANGEME>"
}
}
}





 Reading and writing to Object Storage using IAM in Scala 



1. Create an Eclipse project of the following format:



Zoom

![Shows the Eclipse project format to use in which to develop your Scala jar.](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/eclipse-proj.png)

Figure 1. Eclipse project folder structure



1. Add the following application called ScalaReadWriteIAMCOSExample.scala to the analyticsengine folder. Insert your IAM API key and the Object Storage bucket name.

package com.ibm.analyticsengine
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.SparkContext._

object ScalaReadWriteIAMCOSExample {
def main(args: Array[String]) {
val sparkConf = new SparkConf().setAppName("Spark Scala Example")
val sc = new SparkContext(sparkConf)
val prefix="fs.cos.cosservicename"
sc.hadoopConfiguration.set(prefix + ".endpoint", "s3.direct.us-south.cloud-object-storage.appdomain.cloud")
sc.hadoopConfiguration.set(prefix + ".iam.api.key","<CHANGEME>")
val data = Array("Sweden", "England", "France", "Tokyo")
val myData = sc.parallelize(data)
Creating an ingestion job by using commands 

After installing the ibm-lh tool, you can run the tool to ingest data into IBM® watsonx.data through the CLI. The commands to run the ingestion job and examples are listed in this topic.

To access IBM Cloud Object Storage (COS) and MinIO object storage, specify the ENDPOINT_URL in --source-s3-creds and --staging-s3-creds to pass the corresponding url to the tool. For more information about IBM COS, see [https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-endpoints](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-endpoints).

Replace the absolute values inside the angular brackets of command examples or config file examples with the values applicable to your environment. See [Options and variables supported in ibm-lh tool](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-cli_commands).

Following are the details of the command line option to ingest data files from S3 or local location to watsonx.data Iceberg table:



 Ingest a single CSV/Parquet file from S3 location by using a command 

To ingest a single CSV/Parquet file from a S3 location, run the following command:
Read more about using [Prometheus in IBM Cloud](https://cloud.ibm.com/docs/compare-and-comply?topic=compare-and-comply-alerts).



Storage



* Utilize IBM Cloud [File](https://cloud.ibm.com/docs/FileStorage?topic=FileStorage-getting-started) or [Block](https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-getting-started) storage when blockchain nodes are provisioned. See the topic on [Persistent storage considerations](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-deploy-iksibp-console-storage) to learn more about how blockchain integrates with IBM Cloud storage options.
* Set up [Portworx](https://cloud.ibm.com/docs/containers?topic=containers-portworxportworx) to manage local persistent storage across your containerized databases, or share data between pods across multiple zones.



For more information about available IBM Cloud services and other third-party integrations, see this list of [Supported IBM Cloud and third-party integrations](https://cloud.ibm.com/docs/containers?topic=containers-supported_integrationssupported_integrations).





 Compliance 

For a list of the current security certifications that IBM Blockchain Platform adheres to, see the [Software Compatibility Reports](https://www.ibm.com/software/reports/compatibility/clarity-reports/report/html/softwareReqsForProduct?deliverableId=58A3B7D0BEE911E5BA010CF56D8211B6).





 Getting support 

For more information about how to get support on IBM Blockchain Platform for IBM Cloud, as well as free blockchain developer resources and support forums that you can use to troubleshoot problems, see [Getting support](https://cloud.ibm.com/docs/blockchain?topic=blockchain-blockchain-supportblockchain-support).
Use [applications](https://cloud.ibm.com/docs/codeengine?topic=codeengine-application-workloads).

Do you need to process a large amount of data in a short time in response to events or requests?
: If your workload requires a fast response to unpredicted requests or events, then applications are typically a better fit because applications are scaled dynamically, even from zero. Use [applications](https://cloud.ibm.com/docs/codeengine?topic=codeengine-application-workloads).

Combining apps and jobs
: You can even combine apps and jobs, where an application can start a job to outsource specific computations. Jobs can also query an application. A typical example of a combination of both jobs and apps is training and serving of machine learning models. Jobs are typically used to train the models and applications are used to serve the models. Use [applications](https://cloud.ibm.com/docs/codeengine?topic=codeengine-application-workloads) and [jobs](https://cloud.ibm.com/docs/codeengine?topic=codeengine-job-plan).
See [Serverless instance architecture and concepts](https://cloud.ibm.com/docs/analyticsengine?-serverless-architecture-concepts).
2. [Provision a serverless instance](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-provisioning-serverless)







 Run applications 

To run Spark applications in a serverless IBM Analytics Engine instance:



1. Optionally, give users access to the provisioned instance to enable collaboration. See [Managing user access to share instances](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-grant-permissions-serverless).
2. Optionally, customize the instance to fit the requirements of your applications. See [Customizing the instance](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-cust-instance).
3. Submit your Spark application by using the Spark application REST API. See [Running Spark batch applications](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-spark-batch-serverless).
4. Submit your Spark application by using the Livy batch API. See [Running Spark batch applications using the Livy API](https://cloud.ibm.com/docs/analyticsengine?topic=AnalyticsEngine-livy-api-serverless).







 End-to-end scenario using the Analytics Engine serverless CLI 

To help you get started quickly and simply with provisioning an Analytics Engine instance and submitting Spark applications, you can use the Analytics Engine serverless CLI.

For an end-to-end scenario of the steps you need to take, from creating the services that are required, to submitting and managing your Spark applications by using the Analytics Engine CLI, see [Create service instances and submit applications using the CLI](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-using-cli).