vania-janet/multiturn-rag-retrieval-data huggingface dataset:
image: k8s.gcr.io/nginx-slim:0.8
ports:
- containerPort: 80
name: web
volumeMounts:
- name: mypvc
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: mypvc
annotations:
ibm.io/auto-create-bucket: "true"
ibm.io/auto-delete-bucket: "true"
ibm.io/bucket: ""
ibm.io/secret-name: mysecret
volume.beta.kubernetes.io/storage-class: ibmc-s3fs-standard-perf-cross-region
volume.beta.kubernetes.io/storage-provisioner: ibm.io/ibmc-s3fs
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "ibmc-s3fs-standard-perf-cross-region"
resources:
requests:
storage: 1Gi
Show more

Example to create a stateful set with three replicas that share the same bucket mybucket.

apiVersion: v1
kind: Service
metadata:
name: nginx-v01
namespace: default
labels:
app: nginx-v01 must match spec.template.metadata.labels and spec.selector.matchLabels in stateful set YAML
spec:
ports:
- port: 80
name: web
clusterIP: None
selector:
app: nginx-v01 must match spec.template.metadata.labels and spec.selector.matchLabels in stateful set YAML
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: web-v01
namespace: default
spec:
selector:
matchLabels:
app: nginx-v01 must match spec.template.metadata.labels in stateful set YAML and metadata.labels in service YAML
For more information about available command options, see [ibmcloud is volume](https://cloud.ibm.com/docs/cli?topic=cli-vpc-referencevolume).





 Viewing all Block Storage for VPC volumes from the CLI 

Run this command to list summary information about all volumes:

ibmcloud is volumes [--json]

Specifying the resource group ID or name filters the list to volumes that belong to a resource group. When you omit this argument, it defaults to all resource groups. You can also view all volumes in a particular availability zone.

By default, the first 25 volumes are displayed per page.

The following example shows all volumes for all resource groups in your availability zone.

$ ibmcloud is volumes
Listing volumes in all resource groups and region us-east under account Test Account as user test.user@ibm.com...
ID Name Status Capacity IOPS Profile Attachment state Attachment type Zone Resource group
r014-0a7c28f3-3612-46e6-b874-51136c7f1def concurrent-vol-09afy4vz700 unusable 20 3000 general-purpose unattached - us-east-1 defaults
r014-faefcc1d-f899-4688-ae97-67e5079da702 concurrent-vol-1s26tgtqg70 unusable 20 3000 general-purpose unattached - us-east-1 defaults
r014-f0e809bf-9afb-4006-b2a8-274f81f0f34e concurrent-vol-8xif5f1tid0 unusable 20 3000 general-purpose unattached - us-east-1 defaults
r014-b8e23307-e93e-4f7b-918f-7b2c2b14b132 concurrent-vol-gpwucqfpni0 available 20 3000 general-purpose attached data us-east-1 defaults
[Create the rule-based model](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks_rule_train).
2. [Deploy the rule-based model](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks_rule_publish).







 Restoring incomplete annotation tasks 

If you had any annotation tasks that were created but not completed in the previous workspace, complete the following steps to re-create the incomplete annotation tasks:



1. [Upload any documents](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-documents-for-annotationwks_projadd) that have not been annotated yet but that you want to add to the ground truth to continue to improve the model.
2. From the newly imported and unannotated documents, [create annotation sets](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks_tutml_introwks_tutless_ml2).
3. [Re-create the annotation tasks](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-annotate-documentswks_hatask). Give the task the same name and an appropriate due date, and assign annotation sets to the appropriate human annotators.
* [Deleting a document that is associated with an annotation task and human annotation has begun](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-documents-for-annotationdeleteanno)





 Deleting a document that has not been associated with an annotation task 

If the document you want to delete is not associated with an annotation task, complete these steps to delete the document.



 Procedure 

Log in as a Knowledge Studio administrator and select your workspace.



1. Select the Assets> Documents > Document sets tab.
2. Select the document set that contains the document you want to delete. The document set opens.
3. Find the document that you want to remove, and then click Delete.









 Deleting a document that is associated with an annotation task and human annotation has not begun 

If the document you want to delete is associated with an annotation task and human annotation has not yet begun, complete these steps to delete the document.



 Procedure 



1. Log in as a Knowledge Studio administrator and select your workspace.
2. Delete the annotation task:



1. Open the Machine Learning Model > Annotations page. Click the Annotation Tasks tab.
2. Find the annotation task that the document is associated with, click the Show menu icon on the task, and then click Delete.



3. Delete the document as described in [Deleting a document that has not been associated with an annotation task](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-documents-for-annotationdeletenotask).
4. After you delete the document, re-create the annotation task and associate the same annotation set, which now has one less document in it.









 Deleting a document that is associated with an annotation task and human annotation has begun
[Restore the workspace data](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-backup-restorerestoredata)
6. [Restore the models](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-backup-restorerestoremodels)
7. [Restore any incomplete annotation tasks](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-backup-restorerestoretasks)





 Data that can be backed up 

The following artifacts can be backed up and migrated manually:



* Editable dictionaries
* Type system
* Approved ground truth document sets



The following types of artifacts cannot be backed up and migrated manually:



* In-progress human annotation documents
* Annotation tasks
* Models and snapshots
* Read-only dictionaries







 Preparing for backup 

To prepare for backing up and restoring your data, complete the following steps:



1. Complete any work that is in progress in your workspace.



* > Important: Finish any in-progress annotation tasks. Only documents that have been annotated, adjudicated, and approved and promoted to ground truth can be backed up. If you do not finish the annotation work, you will lose any annotation effort that is in progress, but not completely done.
* If you created annotation tasks to track work that you want to do, but none of the annotation work has begun and will not take place until after the workspace is restored, then make a list of the outstanding annotation tasks. Be sure to note the document sets that you imported, but that have not been added to the ground truth yet. Also, make a note of who you assigned to annotate each document set. You will need to reupload these document sets and recreate the annotation tasks after the workspace is restored.



2. Understand tokenizer use.
For more information about the options, see [Tokenizers](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-create-projectwks_tokenizer).

To use a dictionary-based tokenizer, expand the Advanced Options section of the "Create Workspace" window (in IBM Marketplace, the "Create Workspace" window) and change the tokenizer setting.







 Restoring workspace data 

After re-creating the workspaces, upload the previously downloaded artifacts:



1. Upload the type system from the previously created type system backup. For more information, see [Uploading resources from another workspace](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-exportimport).

You must upload the type system before you can upload any other artifacts that you are moving from the backed up workspace.
2. Upload the dictionaries from the previously created dictionary backup. For more information, see [Uploading resources from another workspace](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-exportimport).

If you used any read-only dictionaries in the previous version of the workspace, re-upload them into this workspace from their original source.
3. For dictionary pre-annotators, associate the dictionaries with an entity type. Dictionaries that don't have mappings for entity types will not apply annotations when you pre-annotate documents.
4. Upload the documents that you downloaded from the previous version of the workspace into this version of the workspace. For more information, see [Uploading resources from another workspace](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-exportimport).







 Restoring models 

At this point, all the artifacts that were used to train the model in the previous (backed up) version of the workspace are now available in this new instance.
Using the rule-based model 

Leverage a rule-based model that you created with Knowledge Studio by making it available to other Watson applications.

You can deploy a rule-based model to make it available for use in these services as an [experimental](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks-faqsexperimental) feature.

Before a model can be deployed for use by a service, you must have a subscription to the service. IBM Watson services are hosted on IBM Cloud, which is the cloud platform for IBM. See [What is IBM Cloud?](https://cloud.ibm.com/docs/overview) for more information about the platform. To subscribe to one of the IBM Watson services, create an account from the [IBM Cloud](https://cloud.ibm.com/) website.

For some of the services, you must know details about the service instance that you plan to deploy to, such as the IBM Cloud space name and service instance name. The space and instance name information is available from the IBM Cloud services page.

You can also pre-annotate new documents with the rule-based model. See [Pre-annotating documents with the rule-based model](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-preannotationwks_preannotrule) for details.



 Deploying a rule-based model to IBM Watson Discovery 

Deploy the model to enable an application that uses the Discovery service to use the rule-based model to find and extract entities during document enrichment.

This is currently an [experimental](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks-faqsexperimental) feature of the service.



 Before you begin 

You must have administrative access to a Watson Discovery service instance, and know the IBM Cloud space and instance names that are associated with it.





 Procedure 

To deploy a rule-based model to Watson Discovery, complete the following steps:



1.
Create a [Calico preDNAT network policy](https://cloud.ibm.com/docs/containers?topic=containers-network_policiesblock_ingress) to block traffic to the public NodePorts.

Ingress ALB
: Load-balancing method: HTTPS load balancing that exposes the app with a subdomain and uses custom routing rules.
: Use case: Implement custom routing rules and SSL termination for multiple apps.
: Implementation: 1. [Disable the public ALB](https://cloud.ibm.com/docs/containers?topic=containers-kubernetes-service-clics_alb_configure). 2. [Enable the private ALB and create an Ingress resource](https://cloud.ibm.com/docs/containers?topic=containers-managed-ingress-setup). 3. Customize ALB routing rules with [annotations](https://cloud.ibm.com/docs/containers?topic=containers-comm-ingress-annotations). 4. An NLB with a portable private IP address still has a public node port open on every worker node. Create a [Calico preDNAT network policy](https://cloud.ibm.com/docs/containers?topic=containers-network_policiesblock_ingress) to block traffic to the public NodePorts.





 Setting up private load balancing for a private VLAN only setup 

When your worker nodes are connected to a private VLAN only, you can make your app externally accessible from a private network only by creating private NodePort, LoadBalancer, or Ingress services.

If your cluster is connected to a private VLAN only and you enable the master and worker nodes to communicate through a private-only service endpoint, you can't automatically expose your apps to a private network.
Avoid field names that meet the following conditions. Field names with these restricted characters are not queried.



* Start with the characters _, +, and -. For example, +extracted-content.
* Contain the characters ., ,, , ?, (, ), or : or spaces. For example, extracted content or new:extracted-content.
* End with numbers, for example, extracted-content2.





 HTML fields 

The html field in the document index stores structural information about the document.



* If you use the Smart Document Understanding tool to annotate a collection, the document representation is indexed in the html field.
* If you use the Smart Document Understanding tool to apply a pretrained model to a collection, the document representation is indexed both in the html field and text field.
* The html field has a size limit. For more information, see [Field limits](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionscollections-field-limits).



Note about enhancing data:



* If you want to apply an enrichment that can understand the tables in a document, the document must contain an html field.









 How dates are handled 

Dates are captured in different ways by different file types.

Unstructured files
: The best way to capture date information from the body of a document with unstructured data is to use a natural language processing model enrichment. For example, the prebuilt Entities enrichment recognizes dates and annotates them in the text field (or other body fields with the String data type). In a document where the enrichment is applied, you can find dates by looking for fields that are labeled as enriched_{fieldname}.entities.type=Date.

Dates from metadata date fields, such as extracted_metadata.publicationdate, are stored in the index as dates as long as the date format matches one of the supported date data type formats. You can't see nested fields from the Manage fields page. And when you view a search result as JSON, date field values are displayed as string values because the JSON editor shows the date as a string. However, values from date fields behave like dates.
For more information, see [Upgrading Watson Discovery to a newer 4.0 refresh](https://www.ibm.com/docs/SSQNUZ_4.0/svc-discovery/discovery-upgrade-v4.html)





 IBM Cloud Private End Of Support 

Effective 30 September 2021, IBM withdrew support for the following programs:



* IBM Watson Assistant Discovery Extension for IBM Cloud Private 2.1.0–2.1.4
* IBM Watson Discovery for ICP for Data 2.1.0–2.1.4
* IBM Watson Discovery for ICP for Data Add-on 2.1.0–2.1.4



For more information, see announcements [ENUS921-005.PDF](https://www.ibm.com/common/ssi/cgi-bin/ssialias?subtype=ca&infotype=an&appname=iSource&supplier=897<=tternum=ENUS921-005) and [ENUSLP21-0099.PDF](https://www.ibm.com/common/ssi/ShowDoc.wss?docURL=/common/ssi/rep_ca/9/899/ENUSLP21-0099/index.html&request_locale=en).





 4 release, 13 July 2021 

New version now available
: Discovery for Cloud Pak for Data 4 is available
: This release is supported on IBM Cloud Pak® for Data 4.0.0.

Change to service name
: The new name is IBM Watson® Discovery Cartridge for IBM Cloud Pak® for Data.

New Smart Document Understanding (SDU) predefined model
: When you identify fields, instead of annotating documents with the SDU tool, you can choose to use a pretrained model. The pretrained model applies a non-customizable model that automatically extracts text and identifies tables, lists, and sections.

Improved contract analysis
: To enable the Contracts enrichment that recognizes and tags contract-related concepts in your data, you can choose to create a Document Retrieval project type, and then select Apply contracts enrichment.
Note that if you want to use different domains for the apps within the same project, you can create one resource per domain.
* If the apps in your cluster are in different projects, you must create one Route resource for each project to define the app's routing rules.



For more information, see [Planning networking for single or multiple projects](https://cloud.ibm.com/docs/openshift?topic=openshift-ingress-roks4multiple_projects).

If you want to customize routing rules for your app, you can use [route-specific HAProxy annotations](https://cloud.ibm.com/docs/openshift?topic=openshift-ingress-configureingress-configure-annotations) that manages traffic for your app. These supported annotations are in the format haproxy.router.openshift.io/<annotation> or router.openshift.io/<annotation>. Note that IBM Cloud Kubernetes Service annotations (ingress.bluemix.net/<annotation>) and NGINX annotations (nginx.ingress.kubernetes.io/<annotation>) are not supported for the Ingress controller or the Route resource in Red Hat OpenShift version 4.







 How does a request get to my app in a classic cluster? 



 Single-zone cluster 

The following diagram shows how Ingress directs communication from the internet to an app in a classic single-zone cluster.

Zoom

![Expose an app in a single-zone cluster by using Ingress](https://cloud.ibm.com/docs-content/v1/content/bce0f0917a9eea684d1b4b704ac6343a1f65f446/openshift//images/roks_router_ingress_single.png)

Figure 1. Expose an app in a single-zone cluster by using Ingress



1. A user sends a request to your app by accessing your app's URL.
: Convert num from from_base to to_base.
: Example of an SQL function usage fragment

> SELECT conv('100', 2, 10)
: Result value

4
: Example of an SQL function usage fragment

> SELECT conv(-10, 16, -10)
: Result value

-16





 corr 

corr(expr1, expr2)
: Returns Pearson coefficient of correlation between a set of number pairs.





 cos 

cos(expr)
: Returns the cosine of expr.
: Example of an SQL function usage fragment

> SELECT cos(0)
: Result value

1.0





 cosh 

cosh(expr)
: Returns the hyperbolic cosine of expr.
: Example of an SQL function usage fragment

> SELECT cosh(0)
: Result value

1.0





 cot 

cot(expr)
: Returns the cotangent of expr.
: Example of an SQL function usage fragment

> SELECT cot(1)
: Result value

0.6420926159343306





 count 

count(*)
: Returns the total number of retrieved rows, including rows that contain null.

count(expr)
: Returns the number of rows for which the supplied expression is nonnull.

count(DISTINCT expr[, expr...])
: Returns the number of rows for which the supplied expressions are unique and nonnull.





 count_min_sketch 

count_min_sketch(col, eps, confidence, seed)
: Returns a count-min sketch of a column with the indicated esp, confidence, and seed. The result is an array of bytes, which can be de-serialized to a CountMinSketch before usage. Count-min sketch is a probabilistic data structure that is used for cardinality estimation by using sublinear space.





 covar_pop 

covar_pop(expr1, expr2)
: Returns the population covariance of a set of number pairs.
chr 

chr(expr)
: Returns the ASCII character that has the binary equivalent to expr. If n is larger than 256, the result is equivalent to chr(n % 256).
: Example of an SQL function usage fragment

> SELECT chr(65)
: Result value

A





 coalesce 

coalesce(expr1, expr2, ...)
: Returns the first nonnull argument if exists. Otherwise, null.
: Example of an SQL function usage fragment

> SELECT coalesce(NULL, 1, NULL)
: Result value

1





 collect_list 

collect_list(expr)
: Collects and returns a list of nonunique elements.





 collect_set 

collect_set(expr)
: Collects and returns a set of unique elements.





 concat 

concat(str1, str2, ..., strN)
: Returns the concatenation of str1, str2, ..., strN.
: Example of an SQL function usage fragment

> SELECT concat('Spark', 'SQL')
: Result value

SparkSQL





 concat_ws 

concat_ws(sep, [str | array(str)]+)
: Returns the concatenation of the strings that are separated by sep.
: Example of an SQL function usage fragment

> SELECT concat_ws(' ', 'Spark', 'SQL')
: Result value

Spark SQL





 conv 

conv(num, from_base, to_base)
: Convert num from from_base to to_base.
: Example of an SQL function usage fragment

> SELECT conv('100', 2, 10)
: Result value

4
: Example of an SQL function usage fragment

> SELECT conv(-10, 16, -10)
: Result value

-16





 corr 

corr(expr1, expr2)
: Returns Pearson coefficient of correlation between a set of number pairs.
$ curl -X GET "https://<region>.kms.cloud.ibm.com/api/v2/key_rings?totalCount=<show_total>&offset=<offset_value>&limit=<offset_limit>" -H "accept: application/vnd.ibm.kms.key_ring+json" -H "authorization: Bearer <IAM_token>" -H "bluemix-instance: <instance_ID>" -H "correlation-id: <correlation_ID>"

Query parameters following the question mark ? are optional, but included here to document their use. (: :note)

Replace the variables in the example request according to the following table.





Table 3. Describes the variables that are needed to view key rings with the Key Protect API.

 Variable Description 

 region Required. The region abbreviation, such as us-south or eu-gb, that represents the geographic area where your Key Protect instance resides. <br> <br>For more information, see [Regional service endpoints](https://cloud.ibm.com/docs/key-protect?topic=key-protect-regionsservice-endpoints). 
 IAM_token Required. Your IBM Cloud access token. Include the full contents of the IAM token, including the Bearer value, in the curl request. <br> <br>For more information, see [Retrieving an access token](https://cloud.ibm.com/docs/key-protect?topic=key-protect-retrieve-access-token). 
 instance_ID Required. The unique identifier that is assigned to your Key Protect instance. <br> <br>For more information, see [Retrieving an instance ID](https://cloud.ibm.com/docs/key-protect?topic=key-protect-retrieve-instance-ID). 
 correlation_ID Optional. The unique identifier that is used to track and correlate transactions. 
 offset_limit Optional. By default, GET /key_rings returns a sequence of 51 keyRings including the default keyRing.
>>> ts2 = ts.transform(transformers.awgn(sd=.3))
>>> dtw_distance = ts.reduce(ts2,reducers.dtw(lambda obs1, obs2: abs(obs1.value - obs2.value)))
>>> print(dtw_distance)
1.8557981638880405
* Math reducers

Several convenient math reducers for numeric time series are provided. These include basic ones such as average, sum, standard deviation, and moments. Entropy, kurtosis, FFT and variants of it, various correlations, and histogram are also included. A convenient basic summarization reducer is the describe function that provides basic information about the time series.

>>> from tspy.functions import 
>>> ts = tspy.time_series([1.0, 2.0, 3.0, 4.0, 5.0, 6.0])
>>> ts2 = ts.transform(transformers.awgn(sd=.3))
>>> corr = ts.reduce(ts2, reducers.correlation())
>>> print(corr)
0.9938941942380525

>>> adf = ts.reduce(reducers.adf())
>>> print(adf)
pValue: -3.45
satisfies test: false

>>> ts2 = ts.transform(transformers.awgn(sd=.3))
>>> granger = ts.reduce(ts2, reducers.granger(1))
>>> print(granger) f_stat, p_value, R2
-1.7123613937876463,-3.874412217575385,1.0
Show more
* Another basic reducer that is very useful for getting a first order understanding of the time series is the describe reducer. The following illustrates this reducer:

>>> desc = ts.describe()