vania-janet/multiturn-rag-retrieval-data huggingface dataset:
For more information, see [Crawl schedule options](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionscrawlschedule).
7. Complete the following fields in the Enter your credentials section:

Content Engine Web Service URL
: The Content Engine web service URL of the IBM FileNet P8 server.

When you enter the URL, use the format: <protocol>://<server>:<port>/wsi/FNCEWS40MTOM. You can use the HTTP or HTTPS protocol. The <server> is the hostname of the server where the Content Platform Engine is deployed and the <port> is the HTTP port that the application server uses, or where the Content Platform Engine is deployed.

User
: The username to use to crawl the FileNet P8 server. You can obtain your username from your FileNet administrator.

Password
: The password that is associated with the user.
8. In the Specify what you want to crawl section, enter the display name of the object store that you want to use to create, search, retrieve, and store documents in the ObjectStore Name field.
9. In Crawler Space Type, select either Folder or Class.
10. Complete the following field:

Folder subpath or Subclass name
: The subfolder path that you can specify under RootFolder that crawls all documents that belong to the specified folder or the custom subclass of the Document class that crawls all documents that belong to the specified class. Before you specify anything in this field, keep in mind the following items:



* You can specify multiple crawler spaces by using both the Class and Folder types and crawl the documents belonging to the folder name and class name.
* You cannot specify a class outside the object store that you defined.
* No support is available for specifying a class that is a subclass of a Custom Object and Folder.



11. After you enter one or more paths, click Add.
12. Optional: In the Security section, if you want to enable document-level security, set the Enable Document Level Security switch to On.
Dates from metadata date fields, such as extracted_metadata.publicationdate, are stored in the index as dates as long as the date format matches one of the supported date data type formats. You can't see nested fields from the Manage fields page. And when you view a search result as JSON, date field values are displayed as string values because the JSON editor shows the date as a string. However, values from date fields behave like dates. You can use greater than (>) or less than (<) operators with such fields in Discovery Query Language queries, for example.

Structured files
: Structure files that you import, such as CSV or JSON files, might contain date fields that you want to store as date data types. Discovery can recognize many date formats. However, you might need to add a format to the list. For more information, see Date format settings.



 Date format settings 

If your documents have a root-level field with date information in it, you can set the field to be a Date data type field in the index.

Discovery recognizes the following date formats automatically:

yyyy-MM-dd'T'HH:mm:ssZ
yyyy-MM-dd'T'HH:mm:ssXXX
yyyy-MM-dd'T'HH:mm:ss.SSSZ
yyyy-MM-dd'T'HH:mm:ss.SSSX
yyyy-MM-dd
M/d/yy
yyyyMMdd
yyyy/MM/dd

If you store dates in other formats, you can add the format to the list of supported formats.

To add more date formats, complete the following steps:



1. From the Manage fields page for the collection, add a format as a new line in the Date formats field.

Specify a date format that is supported by the Java [SimpleDateFormat](https://docs.oracle.com/javase/8/docs/api/java/text/SimpleDateFormat.html) class.
The resource impacted by a PUT request MUST be located at the URI used in the request.

[8]For example, if PUT /users/bob updates a user, GET /users/bob MUST retrieve the same user.

When a PUT request is successfully and synchronously used to replace an existing resource, a status of 200 OK MUST be returned.

[9]

Fields which are not mutable or not recognized MUST result in a 400 status code and appropriate error response model.

In a scenario where the URI of a new resource can be inferred in advance by the client, a PUT request MAY create a new resource. In this case, if the creation is performed synchronously, the response MUST return a 201 Created status.

[10]

The response to a successful, synchronous PUT request SHOULD include the created or replaced resource in the response body. If doing so is expensive (in terms of bandwidth or computation), [preference headers](https://cloud.ibm.com/docs/api-handbook?topic=api-handbook-headerspreference-headers) SHOULD be supported to give the client control over whether the resource is included.





 PATCH 

A PATCH request SHOULD specify new values for one or more fields on a resource and SHOULD be considered successful if each field was updated to, or already contained, the value specified in the request. The resource updated by a PATCH request MUST be located at the URI used in the request.

[11]Unlike with a PUT request, services SHOULD NOT require a PATCH request to reference all mutable fields.

Services MAY support either or both of the below formats for PATCH requests:



* [JSON merge patch format (RFC 7396)](https://datatracker.ietf.org/doc/html/rfc7396)
* [JSON patch format (RFC 6902)](https://datatracker.ietf.org/doc/html/rfc6902)



The JSON merge patch format (RFC 7396) MAY be supported for requests with a content type of application/json or without an explicit content type.
1 Backing store Backing stores are s3 compatible object storage buckets. In the Multicloud Object Gateway, your backing stores can be in any cloud environment, regardless of provider. You can connect several backing stores to the Multicloud Object Gateway. When you deploy ODF, the default backing store uses the raw block storage devices that you specify for your ODF storage cluster. However, you can optionally set up IBM Cloud Object Storage as your default backing store. 
 2 Bucket class A bucket class consists of one or more backing stores (buckets) and a placement policy. You can configure backing stores and placement policies to manage your objects across locations and clouds. 
 3 Namespace resource A namespace resource is an s3 compatible bucket. After you add namespace resources (buckets) to your Multicloud Object Gateway, you can reference those buckets by creating a Namespace bucket which can be used to read and write from one or more namespace resource. 
 4 Namespace bucket A namespace bucket abstracts one or more namespace resources in the NooBaa namespace. When you create a namespace bucket, you can specify read or write policies to the namespace resources that you have configured in your Multicloud Object Gateway. For example, you can read from two buckets across different cloud providers and write to a third bucket in another, separate cloud environment. 
 5 Secret ref A secret ref is a reference to a Kubernetes secret in your cluster. When you create an Object Bucket Claim, NooBaa creates a corresponding secret and config map. Then, you can reference the secret and config map in your apps without needing to directly include the credentials in your apps. 
 6 Placement policy When you create a bucket class, you must specify a placement policy. Placement policies define how your data is written to your backing stores. A Mirror placement policy mirrors objects across the backing stores in your bucket class and a Spread placement policy distributes the objects across the backing stores in your bucket class. 
 7 Config map ref A config map ref is a reference to a Kubernetes config map in your cluster. When you create an Object Bucket Claim, NooBaa dynamically creates a corresponding secret and config map.
The default_field value specifies how the $text operator can be used with the index.

The default_field includes two keys:



Table 5. Default_field field keys

 Key Description 

 enabled Enable or disable the default_field index. The default value is true. 



The analyzer key in the default_field specifies how the index analyzes text. Later, the index can be queried by using the $text operator. For more information, see the [Search documentation](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-search-analyzers) for alternative analyzers. You might choose an alternative analyzer when documents are indexed in languages other than English, or when you have other special requirements for the analyzer, such as matching email addresses.

If the default_field isn't specified, or is supplied with an empty object, it defaults to true and the standard analyzer is used.





 The fields array 

The fields array includes a list of fields that must be indexed for each document. If you know an index queries only on specific fields, then this field can be used to limit the size of the index. Each field must also specify a type to be indexed. The acceptable types are shown in the following list:



* "boolean"
* "string"
* "number"







 The index_array_lengths field 

IBM Cloudant Query text indexes have a property that is called index_array_lengths. If the property isn't explicitly set, the default value is true.

If the field is set to true, the index requires extra work. This work contains a scan of every document for any arrays, and creating a field to hold the length for each array found.

You might prefer to set the index_array_lengths field to false in the following situations:



* You don't need to know the length of an array.
* You don't use the $size operator.
* The documents in your database are complex, or not completely under your control. As a result, it's difficult to estimate the impact of the extra processing that is needed to determine and store the array lengths.
The index can be used to efficiently lookup known keys and ranges of keys without having to rescan the entire database.

Another trick that you can employ at index time is the partial filter. You can optionally supply a partial filter in your index definition. This IBM Cloudant Query selector is executed at index time to decide which documents' data makes it to the index and which are ignored.

In this example, a selector is employed that allows only dates that fall on a weekend to make it to the index. Smaller indexes are faster and more efficient. If you have a use-case that needs only a subset of the data to be indexed, then a partial filter selector at index-time can help to make the index smaller and more efficient. For example, you might want to index only completed orders, or only expired accounts, or only published blog posts.

To summarize, the _index endpoint is used to define the index, and an optional partial filter can be applied at query time to make smaller, sparse indexes.

That's the end of this part. The next part is called MapReduce.







 MapReduce video 

Learn about MapReduce, which is another way to configure secondary indexes.



* MapReduce video script

Welcome to the Introduction to IBM Cloudant course, an 17-part video series that gives you an overview of the IBM Cloudant database-as-a-service.

This video is part 12 - MapReduce.

We saw how a combination of the _find and _index endpoints allows queries to be performed on the contents of JSON documents. They are backed by secondary indexes to make queries scale as your application grows.

In this part, we introduce another way of configuring secondary indexes that are called MapReduce.

MapReduce used to be the only way to configure secondary indexes in CouchDB and is still a popular way of querying data from within the document body.

To create a MapReduce index, you need to supply a JavaScript function that is wrapped in a special document that is called a design document to IBM Cloudant. Design documents' _id fields begin with _design/ for example, _design/mydesigndoc.
Run until all the data from the source reaches the target and then stop.
2. The same as one, but the replication runs continuously forever, transferring new data from the source to the target as it arrives.



Replication can also be resumed from where it last stopped. IBM Cloudant keeps a note of checkpoints between replicating parties to allow the resumption of a pre-existing replication from its last known position.

The IBM Cloudant Dashboard includes a replication tab. A replication is started by specifying the source and target databases that include authentication credentials, and whether this replication is a one time or continuous operation.

A replication can also be given a name, which is handy for tracking which replication job is which.

Now it's time for a practical exercise.



1. Go to the IBM Cloudant Dashboard.
2. Create a database called books2.
3. Start a continuous replication from books to books2.
4. Visit the books2 database to check that documents from books are now in books2.
5. Add a document to the books database.
6. Verify that the change makes it to the books2 database.



Replication can be used to move data from an IBM Cloudant database to an on-premises CouchDB instance. The replication can be controlled by IBM Cloudant or CouchDB. For example, you can ask IBM Cloudant to send its changes to CouchDB, or you can ask CouchDB to pull the changes from IBM Cloudant. The replication controller must have network visibility of both HTTP APIs.

PouchDB also speaks the same replication protocol, so it can be used to transfer data to and from PouchDB and IBM Cloudant. It's most likely that PouchDB would be the replication controller in this case.

PouchDB is commonly used to create offline first apps. These apps collect data even when not connected to the internet, and then write data to IBM Cloudant when they come back online, giving their users an always on service.

Replication cannot always be required. If your application needs to store data and then write it to IBM Cloudant later, then replication isn't strictly speaking required.
Required.--available-fields: List of all available fields.--ray-id value: Lookup logs by specific Ray ID.--fields value: Define field set in return. This must be specified as a comma separated list without any spaces, and all fields must exist. The order in which fields are specified doesn't matter, and the order of fields in the response is not specified. Note that fields are expected to be case sensitive.--start value: The (inclusive) beginning of the requested time frame. This can be a unix timestamp (in seconds or nanoseconds), or an absolute timestamp that conforms to RFC 3339. At this point in time, it cannot exceed a time in the past greater than 7 days. Default is 65 minutes earlier.--end value: The (exclusive) end of the requested time frame. This can be a unix timestamp (in seconds or nanoseconds), or an absolute timestamp that conforms to RFC 3339. The end must be at least 5 minutes earlier than now and must be later than start. Difference between start and end must be not greater than 1h. Default is 5 minutes earlier.--count value: Number of logs to retrieve. The default value is -1.--sample value: Percentage of sampling. When sample is provided, a sample of matching records is returned. If sample=0.1 then 10% of records will be returned. Sampling is random: repeated calls will not only return different records, but likely will also vary slightly in number of returned records. When count is also specified, count is applied to the number of returned records, not the sampled records. So, with sample=0.05 and count=7, when there is a total of 100 records available, approximately 5 will be returned. When there are 1000 records, 7 will be returned. When there are 10,000 records, 7 will be returned. The default value is 1.--timestamps value: Set the format in which response timestamps are returned. Valid values: unix, unixnano, rfc3339.-i, --instance: Instance name or ID.
IBM Cloudant Operators 

Operators are identified by the use of a dollar sign ($) prefix in the name field.

The selector syntax has two core types of operators:



* Combination operators
* Condition operators



In general, combination operators are applied at the topmost level of selection. They're used to combine conditions, or to create combinations of conditions, into one selector.

Every explicit operator has the form:

{
"$operator": "argument"
}

A selector without an explicit operator is considered to have an implicit operator. The exact implicit operator is determined by the structure of the selector expression.



 Implicit operators 

The two implicit operators are shown in the following list:



* Equality
* And



In a selector, any field that contains a JSON value, but that has no operators in it, is considered to be an equality condition. The implicit equality test also applies for fields and subfields.

Any JSON object that isn't the argument to a condition operator is an implicit $and operator on each field.

See the following example selector that uses an operator to match any document, where the year field has a value greater than 2010:

{
"selector": {
"year": {
"$gt": 2010
}
}
}

In the following example, a matching document must have a field that is called director, and the field must have a value exactly equal to Lars von Trier.

See the following example of the implicit equality operator:

{
"director": "Lars von Trier"
}

You can also make the equality operator explicit, as shown in the following example.

See the following example of an explicit equality operator:

{
"director": {
"$eq": "Lars von Trier"
}
}

In the following example that uses subfields, the field imdb in a matching document must also have a subfield rating, and the subfield must have a value equal to 8.

See the following example of implicit operator that is applied to a subfield test:

{
"imdb": {
"rating": 8
}
}
For example, instead of specifying a database password in your source code, you can store it in an environment variable that you reference in your source code.





 Step 6: Use the database 

We're now going to update your local code to point to this database. We'll store the credentials for the services in a properties file. This file will get used ONLY when the app is running locally. When running in IBM Cloud, the credentials will be read from the VCAP_SERVICES environment variable.



1. Find your app in the IBM Cloud [resource list](https://cloud.ibm.com/resources). On the Service Details page for your app, click Connections in the sidebar. Click the IBM Cloudant menu icon (…) and select View credentials.
2. Copy and paste just the url from the credentials to the url field of the cloudant.properties file, and save the changes.

cloudant_url=https://123456789 ... bluemix.cloudant.com
3. Restart the server



Refresh your browser view at: http://localhost:8080/GetStartedTomcat/. Any names you enter into the app will now get added to the database.

Your local app and the IBM Cloud app are sharing the database. Names you add from either app will appear in both when you refresh the browsers.

Remember, if you don't need your app live on IBM Cloud, stop the app so you don't incur any unexpected charges.





 Next steps 



* [Samples](https://ibm-cloud.github.io)
* [Architecture Center](https://www.ibm.com/cloud/architecture/architectures)
oc exec <pod_name> -it bash
4. Navigate to your volume mount path that you defined in your app deployment.
5. Create a text file.

echo "This is a test" > test.txt
6. From the IBM Cloud® dashboard, navigate to your IBM Cloud Object Storage service instance.
7. From the menu, select Buckets.
8. Open your bucket, and verify that you can see the test.txt that you created.







 Using object storage in a stateful set 

If you have a stateful app such as a database, you can create stateful sets that use IBM Cloud Object Storage to store your app's data. Alternatively, you can use an IBM Cloud database-as-a-service, such as IBM Cloudant and store your data in the cloud.

Before you begin:



* [Create and prepare your IBM Cloud Object Storage service instance](https://cloud.ibm.com/docs/openshift?topic=openshift-storage-cos-understandcreate_cos_service).
* [Create a secret to store your IBM Cloud Object Storage service credentials](https://cloud.ibm.com/docs/openshift?topic=openshift-storage-cos-understandcreate_cos_secret).
* [Decide on the configuration for your IBM Cloud Object Storage](https://cloud.ibm.com/docs/openshift?topic=openshift-storage_cos_installconfigure_cos).



To deploy a stateful set that uses object storage:



1. Create a configuration file for your stateful set and the service that you use to expose the stateful set. The following examples show how to deploy NGINX as a stateful set with three replicas, each replica with a separate bucket or sharing the same bucket.

Example to create a stateful set with three replicas, with each replica using a separate bucket.

apiVersion: v1
kind: Service
metadata:
name: nginx-v01
namespace: default
labels:
Toggle the Cloud Object Storage bucket slider to also store all the evidence in a Cloud Object Storage bucket, which you can configure on the next page.





 Cloud Object Storage bucket 

To use this feature, you must have a Cloud Object Storage instance and a bucket. Click [Using IBM Cloud Object Storage buckets as an evidence locker](https://cloud.ibm.com/docs/devsecops?topic=devsecops-cd-devsecops-cos-bucket-evidence) to create a Cloud Object Storage instance. See [Configuring IBM Cloud Object Storage for storing evidence](https://cloud.ibm.com/docs/devsecops?topic=devsecops-cd-devsecops-cos-config) to configure a bucket that can act as a compliance evidence locker.



* Cloud Object Storage instance, Bucket name, and Cloud Object Storage endpoint fields are automatically populated.
* Enter the Service ID API key:



* Preferred: An existing key can be imported from a secrets vault by clicking the key icon.
* You can also copy and paste an existing key.





The endpoint field is optional. Select or provide the endpoint during the setup of the toolchain or during the pipeline run.





 Deploy 



* IBM Cloud API Key: The API key interacts with the IBM Cloud CLI tool in several tasks, it also clones the Git repository.



* Preferred: Import an existing key from a secrets vault by clicking the key icon.
* You can also copy and paste an existing key.
* Create a key by clicking New +.





Save the newly generated API key to a secrets vault.

If the API key is valid and has sufficient access, the schematic workspace, Git repos, and resource group are automatically populated. You can change any of these fields to match your configuration if needed.

Configure the target schematic workspace where the terraform states are maintained:



* Schematic workspace name: The name of the schematic workspace.



* Default: toolchainName-toolchainId-pipelineID









 Signing