vania-janet/multiturn-rag-retrieval-data huggingface dataset:
You can apply enrichments to the text and html fields, and to custom fields that were added from uploaded JSON or CSV files or from the Smart Document Understanding (SDU) tool.

If the field that you choose comes from a JSON file, after you apply the enrichment, the field data type is converted to an array. The field is converted to an array even if it contains a single value. For example, "field1": "Discovery" becomes "field1": ["Discovery"]. Only the first 50,000 characters of a custom field from a JSON file are enriched.
* To remove an enrichment, clear the checkboxes for fields that you want to remove the enrichment from.



6. Click Apply changes and reprocess to apply your changes to the collection.





 Deleting an enrichment 

You can delete a custom enrichment that you built to teach Discovery about your service. Custom enrichments include dictionaries, regular expressions, patterns, and so on. For more information, see [Adding domain-specific resources](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain).

You cannot delete a prebuilt enrichment. Prebuilt enrichments include the Natural Language Understanding enrichments, such as the Entities enrichment, that are built into the product. To determine which enrichments are built in, check the Type column of the Enrichments page for a collection. Prebuilt enrichments have the System type.

To delete a custom enrichment, complete the following steps:



1. Open a project that uses the custom enrichment.
2. Click Manage collections, and then open a collection where the enrichment is in use.
3. From the Enrichments page of a collection, remove the enrichment from any fields where it is applied.
4. Click Apply changes and reprocess, and then wait for the system to process the documents in your collection without the enrichment.
5. Repeat the previous step for every collection in every project where the enrichment is used.

Remember, a custom enrichment can be used by any collection in any project within a single Discovery service instance.
6.
[HashiCorp Vault Tool Integration form with required fields and example values](https://cloud.ibm.com/docs-content/v1/content/5eb88a2575e30e316dbb83829c7f4c8b1ddbc67c/devsecops/images/hc-tool-int.png)

Figure 2. HashiCorp Vault Tool Integration

To use HashiCorp Vault, you must provide the following information:



* Name: A name for this tool integration. This name is displayed in the toolchain.
* Server URL: The server URL for your HashiCorp Vault Instance. For example, https://<vault-service>.<org>.com:8200.
* Integration URL: The URL that you want to navigate to when you click the HashiCorp Vault Integration tile.
* Secrets Path: The mount path where your secrets are stored in your HashiCorp Vault Instance.
* Authentication Method: The Authentication method for your HashiCorp Vault Instance. Use AppRole.
* Role ID: Identifier that selects the AppRole against which the other credentials are evaluated.
* Secret ID: Credential that is required by default for any login (with secret_id) and is intended to always be secret.



The templates also come with an IBM® Key Protect for IBM Cloud® tool integration:

Zoom

![Key Protect tool integration form with required fields and example values](https://cloud.ibm.com/docs-content/v1/content/5eb88a2575e30e316dbb83829c7f4c8b1ddbc67c/devsecops/images/kp-int.png)

Figure 3. IBM Key Protect tool integration

If you stored the role id and secret id in Key Protect in advance, then you can select the Key Protect instance that contains those secrets in the tool card as shown in Figure 2. After that is done, then you can click the key icons on the role id and secret id fields in the HashiCorp Vault tool card, and use the picker to apply the secrets to those fields.
The default behavior of the toolchain is to Create new evidence locker repository that creates a new repository as IBM-hosted Git Repos and Issue Tracking Repository. If you ant to link an existing Evidence Locker for the toolchain, choose Use existing evidence locker repository option and provide it as input to Repository URL field. As noted earlier, the toolchain currently supports linking only to existing Git Repos and Issue Tracking repositories.

However, you must collect and store all the evidence in a Cloud Object Storage bucket that can be configured as described in the following image.

Zoom

![DevSecOps Evidence Storage](https://cloud.ibm.com/docs-content/v1/content/5eb88a2575e30e316dbb83829c7f4c8b1ddbc67c/devsecops/images/devsecops-cd-evidence-storage.png)

Figure 11. DevSecOps Evidence Storage





 Cloud Object Storage Bucket 

Zoom

![Cloud Object Storage bucket toggle](https://cloud.ibm.com/docs-content/v1/content/5eb88a2575e30e316dbb83829c7f4c8b1ddbc67c/devsecops/images/cos-evidence.png)

Figure 12. COS bucket toggle

[Cloud Object Storage](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-about-cloud-object-storage) is used to store the evidence and artifacts that are generated by the DevSecOps Pipelines. If you want to use this feature, you must have a Cloud Object Storage instance and a Bucket. Read the recommendation for configuring a Bucket that can act as a Compliance Evidence Locker.

You can optionally set any kind of Cloud Object Storage bucket as a locker, even without a retention policy. The pipeline doesn't check or enforce settings at the moment.
Managing the Multi-Cloud Object Gateway 

You can use NooBaa to manage your s3 compatible object storage resources, like IBM Object Storage, AWS s3, or Azure Blob storage. With NooBaa, you can also create and manage your object storage consistently across clusters and zones.



 Setting up backing stores by using the NooBaa CLI 

Backing stores are NooBaa resources for managing your s3 compatible services and buckets. When you create a backing store, you provide your s3 service credentials like your access key ID, secret access key, and endpoint for your object storage service. After you add backing stores to your cluster, you can [create bucket classes](https://cloud.ibm.com/docs/openshift?topic=openshift-odf-manage-noobaaodf-bucketclass) which allow you to configure data federation policies for your s3 compatible storage services.

After you deploy ODF, you can configure more backing stores in your storage cluster. You can create a backing store by using any s3 compatible object store such as AWS or IBM Cloud Object Storage.

You can also create and manage your backing stores in the Red Hat OpenShift web console.

To add a backing store to your storage cluster by using the NooBaa CLI:

[Access your Red Hat OpenShift cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-access_cluster).



1. Install the [NooBaa Operator CLI](https://github.com/noobaa/noobaa-operator).
2. Run the noobaa backingstore create command to see a list of supported backing store types.

noobaa backingstore create

Example output

Create backing store

Available Commands:
aws-s3 Create aws-s3 backing store
azure-blob Create azure-blob backing store
google-cloud-storage Create google-cloud-storage backing store
ibm-cos Create ibm-cos backing store
pv-pool Create pv-pool backing store
s3-compatible Create s3-compatible backing store
When specified in combination with PARTITIONED BY, it sorts the rows within each partition by the sort order that is specified in the SORT BY clause. When specified in combination with PARTITIONED INTO, the same is done, which is often referred to as clustering the rows by the specified columns into the fixed number of partitions specified by PARTITIONED INTO. When specified without the PARTITIONED clause, it is equivalent to an ORDER BY clause specified at the top level of the SQL SELECT statement. If PARTITIONED INTO is specified, the ORDER BY clause is ignored.



 Partition by columns 

When you use the PARTITIONED BY (column-list) clause without specifying INTO x BUCKETS/OBJECTS, you can store the query result by using Hive-style partitioning, which is to create partitions that contain only rows that have certain values for one or more columns. Choose this physical layout if the stored object is further analyzed by using SQL queries that specify predicates on the partition columns.

For example, a result object that contains worldwide order data has a column country to represent the country that the order is initiated from. Partitioning the result object by the column PARTITIONED BY (country), would create a result object with a partition for each country present in the query result.

When the result object is stored this way on Cloud Object Storage, each SQL query that contains a predicate, such as country = 'USA' or country in ('MALTA', 'ITALY', 'VATICAN CITY'), benefits from this physical layout. The reason is that during SQL query execution partitions must be read only if they contain data for the countries of interest. This layout tremendously cuts down the I/O traffic of the SQL query.

See the following extra remarks on Hive-style partitioning.



* Hive-style partitions have an eye-catching naming scheme because the column names that are used for partitioning are part of the partition object prefix, for example, /order/COUNTRY=USA/part-m-00000.snappy.parquet.
* Hive-style partitions do not contain any values for partition columns since their values are stored in the object prefix of the partition. Thus, if you copy a HIVE-style partition and rename the object prefix by removing the partition column values, you lose data.
Format 

APIs MUST support JSON by default for sending and receiving data in request and response bodies. Other formats SHOULD NOT be supported except to meet a compelling industry-specific need.



  Object encapsulation 

All JSON data MUST be structured in an object at the top level; arrays MUST NOT be returned as the top-level structure in a response body.

[1]





  Request body constraints 

Services MUST have a documented and enforced maximum payload size for request bodies. Requests with bodies larger than the limit MUST be rejected with a 413 Payload Too Large status code and appropriate error response model.

If the request body size can be determined from the request headers, the request MUST be rejected prior to any processing of the payload

[2]. For chunked transfer or other cases where the request body size cannot be predetermined, the request SHOULD be rejected as soon as it has been determined to exceed the limit.





  Content-Type behavior 

If a request Accept header is either not provided or matches

[3]application/json, the response MUST be JSON and its Content-Type header MUST be application/json.





  JSON processing 



  Case insensitivity 

Field names in JSON objects in a request payload SHOULD NOT be case-normalized to support case insensitivity; a field that does not match the case of a defined field but otherwise matches its name SHOULD be treated as any other extraneous input

[4].

However, field name case normalization MAY be supported for backward compatibility with existing clients.





  Ambiguous JSON data 

Duplicate field names MUST NOT exist in any one JSON object

[5]contained in a JSON response. If duplicate field names exist in an object within a request payload, the request MUST be rejected [6] with a 400 status and appropriate error response model.

Be aware that the JSON specification itself does not strictly prohibit duplicate field names in a single object and many unmarshalling libraries may silently choose from among duplicated fields.

If a service supports field name [case insensitivity](https://cloud.ibm.com/docs/api-handbook?topic=api-handbook-formatcase-insensitivity), field names MUST be normalized prior to validating uniqueness.







  Cross-origin support 

If a service has a need to support cross-origin requests, CORS headers SHOULD be supported to enable this. The JSONP format SHOULD NOT be supported.
Migrate selective data from ECC to the upgraded shell system Used in Business split scenarios (e.g. Divestitures), and Transformation/Conversion projects with SAP S/4HANA Yes minimal, the target can be built and tested in advance (repeatedly testing the conversion steps and remediation). Data can be replicated to target in advance, so downtime is only for the delta data synchronization and replacing the old system with target. Significant preparation is required. SAP Landscape Transformation Replication Server, handled through a direct SAP engagement 
 Selective Data Transition, with Mix & Match Merge of two or more system configurations to create a new SAP System with required configuration; then upgrade / conversion to SAP S/4HANA Used in Business merge scenarios (e.g. Acquisitions) or multiple SAP system consolidation (e.g. ERPs for each Geographic Region or Business Units), and Transformation/Conversion projects with SAP S/4HANA Yes minimal, the target can be built and tested in advance (repeatedly testing the conversion steps and remediation). Data can be replicated to target in advance, so downtime is only for the delta data synchronization and replacing the old system with target. Significant preparation is required. SAP Landscape Transformation Replication Server, handled through a direct SAP engagement
"{field_name_N}": {
"css_selector": "{CSS_selector_expression_N}",
"type": "{field_type}"
}
}
...
}
}
}
Show more

Specify values for the new fields as follows:



* field_name — The name of the field that you want to add to the JSON output.
* CSS_selector_expression — The CSS selector that is to be run against the input HTML to extract the fields. The expression can have one or more matches.

Valid CSS selectors are those specified by the [JSoup parser](https://jsoup.org/apidocs/org/jsoup/select/Selector.html) and its [selector syntax](https://jsoup.org/cookbook/extracting-data/selector-syntax). A short list is provided at [Common selectors](https://cloud.ibm.com/docs/discovery?topic=discovery-configservicecommon-selectors).
* field_type — Either array or string. If the field type is not specified, it defaults to array. Both the string and array types can be enriched.



If a CSS selector matches both a parent node and one or more of its children, the text content of the nodes are duplicated in the JSON output.

Field names must meet the restrictions defined in [Field name requirements](https://cloud.ibm.com/docs/discovery?topic=discovery-configreffield_reqs).

The following JSON passage shows the relevant section of the configuration to which you add CSS selector information.

{
"name": "Default Configuration",
"description": "The configuration used by default when creating a new collection without specifying a configuration_id.",
"conversions": {
...
"html": {
"exclude_tags_completely": [
"script",
"sup"
],
[Cloud Object Storage](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-about-cloud-object-storage) is used to store the evidence and artifacts that are generated by the DevSecOps Pipelines. If you want to use this feature, you must have a Cloud Object Storage instance and a Bucket. Read the recommendation for configuring a Bucket that can act as a Compliance Evidence Locker.

You can optionally set any kind of Cloud Object Storage bucket as a locker, even without a retention policy. The pipeline doesn't check or enforce settings at the moment. For help, see the [Cloud Object Storage documentation](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-getting-started-cloud-object-storage).

You need to provide the following information for the Pipelines to reach the bucket:



* Cloud Object Storage endpoint
* Bucket name
* Service API key



You can set up the Cloud Object Storage locker later, by providing the necessary cos-bucket-name and cos-endpoint. To remove the Cloud Object Storage bucket, use the Cloud Object Storage bucket toggle in the previous step.

To get the Cloud Object Storage Endpoint, refer to your Cloud Object Storage instance's page, and select the 'Endpoints' section in the menu. You need to copy the public endpoint that matches the bucket's region and resiliency.

Zoom

![DevSecOps Cloud Object Storage Endpoint menu](https://cloud.ibm.com/docs-content/v1/content/5eb88a2575e30e316dbb83829c7f4c8b1ddbc67c/devsecops/images/devsecops-cos-endpoint-menu.png)

Figure 13. DevSecOps Cloud Object Storage Endpoint menu

If you decide not to use Cloud Object Storage as an evidence locker, you can also set the required values after the creation of the toolchain by setting the cos-bucket-name, cos-endpoint environment variables in the CI Pipeline.





 Deploy
* Preferred: An existing key can be imported from a secrets vault by clicking the key icon.
* An existing key can be copy and pasted.
* A new key can be created from here by clicking the New +.





The newly generated API key can be immediately saved to a secrets vault.

If the API key is valid and has sufficient access, the Container registry region and namespace Cluster region, name, namespace, and resource group are automatically populated. You can change any of these fields to match your configuration if needed.





 Image Signing 

Any images that are built by this toolchain and recorded in the inventory must be signed before they can be deployed to production.

To enable image signing, you need to provide a GPG key. You can select an existing key from a secret store or input the value directly. Alternatively you can create a new GPG key by clicking NEW.

Ensure that the key follows the appropriate encoding as required by the chosen tool to store secrets.





 DevOps Insights 

[IBM Cloud DevOps Insights](https://cloud.ibm.com/docs/ContinuousDelivery?topic=ContinuousDelivery-di_working) is included in the created toolchain. View your pipeline test results for every build, from every deployment and environment.

You do not need to provide any configuration steps for DevOps Insights (DOI). Select a DOI instance for your toolchain:



* Current (to be created) CI toolchain
* Another toolchain instance: select an existing toolchain (that has a DOI instance)







 SonarQube 

SonarQube provides an overview of the overall health and quality of your source code and highlights issues that are found in new code. The static code analyzers detect tricky bugs, such as null-pointer dereferences, logic errors, and resource leaks for multiple programming languages.

Default configuration
: Run a SonarQube scan in an isolated Docker-in-Docker container without persisting the SonarQube instance.

Cluster configuration
: A new SonarQube instance is provisioned in the Kubernetes cluster (chosen at the Deploy step) during the first CI pipeline run.

Custom configuration
: Link an existing SonarQube instance with the toolchain.
1 root root 10 Feb 11 03:14 scsi-3600605b00d87b43027b3bc310a64c6c9-part1 -> ../../sdc1
lrwxrwxrwx. 1 root root 10 Feb 11 03:15 scsi-3600605b00d87b43027b3bc310a64c6c9-part2 -> ../../sdc2
7. Repeat the previous steps for each worker node that you want to use for your OpenShift Data Foundation deployment.
8. [Install ODF in your cluster](https://cloud.ibm.com/docs/openshift?topic=openshift-deploy-odf-classicinstall-odf-cli-classic).









 Installing the add-on from the CLI 

You can install the add-on by using the [ibmcloud oc cluster addon enable command](https://cloud.ibm.com/docs/openshift?topic=openshift-kubernetes-service-clics_cluster_addon_enable).

If you want to use an IBM Cloud Object Storage service instance as your default backing store, make sure that you [created the service instance](https://cloud.ibm.com/docs/openshift?topic=openshift-deploy-odf-classicodf-create-cos-classic), and created the Kubernetes secret in your cluster. When you create the ODF CRD in your cluster, ODF looks for a secret named ibm-cloud-cos-creds to set up the default backing store that uses your Object Storage HMAC credentials.



1. Review the [parameter reference](https://cloud.ibm.com/docs/openshift?topic=openshift-deploy-odf-classicodf-classic-param-ref). When you enable the add-on, you can override the default values by specifying the --param "key=value" option for each parameter that you want to override.
2.
Manage enrichments 

Apply enrichments to fields in your documents to make meaningful information easier to find and return in searches.

You typically apply enrichments to fields at the time that you create the enrichment. However, you can apply enrichments to fields later. You might want to apply an existing enrichment to a new custom field that you create by using the SDU tool, for example. You can remove enrichments that you applied to fields also.

For more information about available enrichments, see the following topics:



* [Adding domain-specific resources](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain)
* [Applying prebuilt enrichments](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-nlu)



To manage enrichments, complete the following steps:



1. From the navigation pane, open the Manage collections page, and then click a collection to open it.
2. Click the Enrichments tab.

A list of available enrichments is displayed.

You can identify built-in enrichments because they are categorized as type System. The list also includes any custom enrichments that were created in any of the projects in your service instance.
3. Find the enrichment that you want to apply or remove.
4. Click the twistie in the associated field to expand a list of fields.
5. Do one of the following things:



* To apply an enrichment to documents, select the field or fields that you want to enrich. You can apply enrichments to the text and html fields, and to custom fields that were added from uploaded JSON or CSV files or from the Smart Document Understanding (SDU) tool.

If the field that you choose comes from a JSON file, after you apply the enrichment, the field data type is converted to an array. The field is converted to an array even if it contains a single value. For example, "field1": "Discovery" becomes "field1": ["Discovery"]. Only the first 50,000 characters of a custom field from a JSON file are enriched.
In Satellite clusters, you can use local volumes on your worker nodes, or dynamically provision block storage devices by using a block storage driver that supports dynamic provisioning. In Classic clusters, the OSD block devices are local disks on your worker nodes. When you deploy ODF, each device is mounted by an OSD pod. The total storage that is available to your applications is equal to the osdSize multiplied by the numOfOsd. 
 3 Object Storage Daemon (OSD) pods The OSD pods manage data placement and replication across your storage devices. 
 4 Monitor (Mon) pods The Monitor pods keep a map of your OpenShift Data Foundation storage cluster and monitor storage cluster health. 
 5 Monitor (Mon) block storage device The monitor storage devices are the underlying storage devices for the monitor pods. Each monitor device is a raw block storage device that can be a local disk on your worker node or dynamically provisioned when you deploy ODF. Each device provides storage to a monitor pod. 







 Multicloud Object Gateway overview 

The Multicloud Object Gateway consists of the open source tool [NooBaa](https://www.noobaa.io/) and is a component of OpenShift Data Foundation. With the Multicloud Object Gateway, you can manage objects and buckets across cloud providers.

Zoom

![Multicloud Object Gateway overview](https://cloud.ibm.com/docs-content/v1/content/bce0f0917a9eea684d1b4b704ac6343a1f65f446/openshift/images/noobaa.svg)

Figure 2. Multicloud Object Gateway overview



NooBaa overview

 Number Multicloud Object Gateway component Description 

 1 Backing store Backing stores are s3 compatible object storage buckets. In the Multicloud Object Gateway, your backing stores can be in any cloud environment, regardless of provider. You can connect several backing stores to the Multicloud Object Gateway. When you deploy ODF, the default backing store uses the raw block storage devices that you specify for your ODF storage cluster. However, you can optionally set up IBM Cloud Object Storage as your default backing store.