vania-janet/multiturn-rag-retrieval-data huggingface dataset:
A mention type indicates whether the mention is a name, nominal, or pronoun, and a mention class indicates whether the mention is specific, generic, or negated.
4. Open the Assets> Relation Types page and specify how two mentions can relate to each other.

Order between the first and second entities in a relation type is usually relevant. For example, a PERSON entity can be an employee of an ORGANIZATION entity or a geo-political entity (GPE), such as MaryemployedByIBM, but organizations and geo-political entities cannot be employed by a person. When a human annotator clicks an entity in the ground truth editor , the list of available relation types is controlled by what is defined in the type system.

Do not define relation attributes. They are not used by the machine learning model. The model uses only the relation type and order, and ignores the relation attributes.
5. Use the Edit and Delete icons to modify entity types and their associated relation types, or to delete an entity type or relation type from the type system.

If you delete an entity that is used in a relation type definition, the relation type definition is also deleted.







Related tasks:

[Modifying a type system without losing human annotations](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-improve-mlwks_projtypesysmod)







 Type system creation guidelines 

The purpose of any type system in Knowledge Studio is to define how spans of text can be annotated. If you choose to create a type system from scratch, follow these guidelines.

Focus on creating an inventory of entity types and relation types that cover the information that is needed by the application in which the type system will be used. Do not cover things that are not needed. Do not split types or make distinctions that are not needed by the application. For example, if the source document contains the sentence Murder on the Orient Express made headlines, then how you define entity types to capture the key information in the sentence differs depending on the type of application that will use the model that you build with the type system.



* For a literary application, you might create types that capture this information:
To use a custom color, specify its hexadecimal color code (#fff0f7).
4. Click Create.
5. Repeat this process to add all of the entity types that you want the extractor to recognize.

If you aren't sure what to add for entity types, it might help to review the documents in the collection first. By reviewing the content, you can get a feel for which terms have significant meaning and look for logical ways to group such terms.







 Label significant terms 

From the Label documents view, find terms of significance in the documents from your collection and label them to indicate their entity types.

Before you begin labeling documents, decide whether you want to keep bulk labeling enabled. The bulk label feature is a great way to speed up the process of labeling your documents. When enabled, every term that you label is labeled automatically everywhere it occurs in the document. Otherwise, you must label each occurrence of the term one at a time.

If you decide that you don't want to bulk label examples, set the Bulk label entity examples switch to Off. For more information, see [Labeling examples in bulk](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-entity-extractorentity-extractor-bulk-label).

Zoom

![Shows the bulk label feature switcher.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/ee-bulk-label-enable.png)

Figure 3. Bulk labeling switcher



 Labeling tips 

Review these tips before you begin:



* The document collection that you label must contain a representative set of documents. The documents must have many and varied examples of the entity types that you want the entity extractor to recognize. If the collection you selected when you started to create the entity extractor does not meet the requirement, stop now and start over with a different document collection.
* Define entity types that are clearly distinct from one another.
However, many other response types are available, supporting multimedia and interactive content, plus control over the behavior of some channel integrations.

Although the generic format can be sent to any channel integration, not all channels support all response types, so a particular response might be ignored (or handled differently) by some channels. For more information, see [Channel integration support for response types](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-assistant-responses-jsonassistant-responses-json-integration-support).

At run time, output with multiple responses might be split into multiple message payloads. The channel integration sends these messages to the channel in sequence, but it is the responsibility of the channel to deliver these messages to the user; this can be affected by network or server issues.



 Adding responses 

To specify a response in the JSON editor, insert the appropriate JSON objects into the generic field of the step response. The following example shows output with two responses of different types (text and an image):

{
"generic":[
{
"response_type": "text",
"values":
{
"text_expression": {
"concat":
{
"scalar": "This is a text response."
}
]
}
}
]
},
{
"response_type": "image",
"source": "https://example.com/image.jpg",
"title": "Example image",
"description": "This is an image response."
}
]
}
Show more

For information about the available response types, see [Response types](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-assistant-responses-jsonassistant-responses-json-response-types).





 Targeting specific integrations 

If you plan to deploy your assistant to multiple channels, you might want to send different responses to different integrations based on the capabilities of each channel. The channels property of the generic response object provides a way to do this.
A workspace defines all the resources that are required to create a machine learning model, including training documents, the type system, dictionaries, and annotations that are added by human annotators. For more information about workspace creation, see [Creating a workspace](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-create-project).





 Procedure 



1. As a Knowledge Studio administrator, from your IBM Cloud [resources](https://cloud.ibm.com/resources) page, click the Knowledge Studio service instance under Services.
2. Click Launch tool from the Manage page.
3. Click Create Workspace.
4. Specify the details for the new workspace:



* In the Workspace name field, type My workspace.
* In the Workspace description field, type Watson Knowledge Studio tutorial workspace.
* In the Language of documents field, use the default value, English. The sample files you will be using for this tutorial are in English.



5. Click Create.







 Results 

The workspace is created and opens automatically.





 What to do next 

You can now start configuring the workspace resources, such as the type system.







 Lesson 3: Creating a type system 

In this lesson, you will learn how to upload and modify a type system within Knowledge Studio. You must create or upload a type system before you begin any annotation tasks.



 About this task 

For more information about type systems, see [Type systems](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-typesystemwks_typesystem).





 Procedure 



1. Download the [en-klue2-types.json !
When you create that class, all terms in this dictionary that are found in documents will be automatically annotated as a mention of the associated class type. For more information about dictionaries, see [Adding dictionaries to a workspace](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-dictionarieswks_projdictionaries).





 Procedure 



1. Download the [dictionary-items-month.csv![External link icon](https://cloud.ibm.com/docs-content/v1/content/50eddb8fd81f33092880335ff107a78ff5cd0f65/icons/launch-glyph.svg)](https://watson-developer-cloud.github.io/doc-tutorial-downloads/knowledge-studio/dictionary-items-month.csv) file to your computer. This file contains dictionary terms in CSV format that are suitable for uploading into a Knowledge Studio dictionary.
2. Click Assets > Dictionaries.
3. Click the Create Dictionary button to add a dictionary.
4. In the Name field, type Month dictionary and click Save to create the dictionary. The new dictionary is created and automatically opened for editing.
5. In the dictionary pane, click Upload.
6. Select the dictionary-items-month.csv file from your computer and click Upload.

The terms from the file are imported into the dictionary.









 Lesson 2: Adding sample documents 

In this lesson, you will learn how to add documents with linguistic patterns that illustrate the types of rules you want to define.



 About this task 

For more information about adding documents, see [Adding documents for defining rules](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks_rule_anno_add).





 Procedure 



1. Download the [documents-new.csv!
Payload: core.StringPtr("secret-data"),
},
},
})

v2 example

secretPrototypeModel :=&sm.ArbitrarySecretPrototype{
Description: core.StringPtr("Extended description for this secret."),
Labels: []string{"dev", "us-south"},
Name: core.StringPtr("example-arbitrary-secret"),
SecretGroupID: core.StringPtr("default")
SecretType: core.StringPtr("arbitrary")
Payload: core.StringPtr("secret-data"),
}

createSecretOptions := secretsManager.NewCreateSecretOptions(
secretPrototypeModel,
)

secret, _, err := secretsManager.CreateSecret(createSecretOptions)
}
response = secretsManager.create_secret(
secret_prototype=secret_prototype_model,
)







 Migrating Go applications 

If you use Go, review the following steps and examples to migrate to Secrets Manager API v2.



1. Update your go.mod file to use the latest version (2.0.0).
2. Update your import statements to use v2.
3. For each operation that you're using, remove the type and the metadata and resources wrapper from the resource JSON object.



The response type matches the created response. For example, with v2, you use ArbitrarySecret instead of SecretResourceIntf.



 Comparing v1 and v2 Go examples 

Compare the following examples to see the changes between the v1 Go code snippet and how the code must look like in v2.

v1 example

createRes, resp, err := secretsManager.CreateSecret(&sm.CreateSecretOptions{
SecretType: core.StringPtr(sm.CreateSecretOptionsSecretTypeArbitraryConst),
Metadata: &sm.CollectionMetadata{
CollectionType: core.StringPtr(sm.CollectionMetadataCollectionTypeApplicationVnd)
CollectionTotal: core.Int64Ptr(1)
},
Resources: []sm.SecretResourceIntf{
&sm.ArbitrarySecretResource{
Name: core.StringPtr("example-arbitrary-secret"),
Description: core.StringPtr("Extended description for this secret."),
Payload: core.StringPtr("secret-data"),
},
},
})

v2 example

secretPrototypeModel :=&sm.ArbitrarySecretPrototype{
Description: core.StringPtr("Extended description for this secret."),
Labels: []string{"dev", "us-south"},
Name: core.StringPtr("example-arbitrary-secret"),
type string The request type: <br>start - an initial request to Watson Assistant <br>speech_to_text - a request is triggered on speech recognition <br>dtmf - a request is triggered when DTMF collection completes <br>sms - a request is triggered when a SMS message is received from the caller <br>post_response_timeout - a request is triggered when the post response timer expires <br>redirect - a request is triggered when a call is redirected <br>transfer - a request is triggered when a call is transferred <br>transfer_failed - a request is triggered when a call transfer fails <br>final_utterance_timeout - a request is triggered when the final utterance timer expires <br>no_input_turn - a request is triggered when no inpout turn is enabled <br>sms_failure - a request is triggered when a SMS message can't be sent to the caller <br>speech_to_text_result_filtered - a request is triggered when an utterance is filtered due to low confidence level <br>mrcp_recognition_unsuccessful - a request is triggered when the MRCP recognition completes without a final utterance <br>network_warning - a request is triggered when a network error is detected <br>media_capability_change - a request is triggered when media capabilities change in the middle of a call 
 streaming_statistics JSON object Contains information and statistics related to the Speech to Text recognition. 



The request.streaming_statistics object contains the following keys:



Keys for the request.streaming_statistics object

 Key Type Description 

 transaction_id string A unique identifier of the transaction. 
 start_timestamp string. Time in the ISO format yyyy-MM-ddTHH:mm:ss.SSSZ Time when the transaction started. 
 stop_timestamp string. Time in the ISO format yyyy-MM-ddTHH:mm:ss.SSSZ Time when the transaction ended. 
 response_milliseconds number Latency in milliseconds between when silence is detected in the caller's speech and a final result from Speech to Text is received. 
 echo_detected Boolean Indicates whether an echo was detected. The value can be true or false.
Models 

A model represents a [type](https://cloud.ibm.com/docs/api-handbook?topic=api-handbook-types) of resource and defines the structure and behavior of objects that represent resources of that type. Such objects are called instances of the model. Every object returned or accepted by a service MUST be an instance of a model.

In designing an API, the system must be decomposed into resources and each resource carefully modeled. This process must be guided by how the consumer is expected to understand the system. It should not be dictated by how the system itself represents or stores objects internally. The mapping between models and internal structures — such as classes — must be well-behaved but it need not be one-to-one.

An API's models should be described and implemented consistently in the API's specification and documentation and in the service that provides the API. Where possible, automated tooling and testing should be used to ensure this consistency.



 Fields 

A model specifies the following about each field:



* Name
* Type
* Meaning: What the field represents
* Optionality: If the field is required to be present in every instance
* Nullability: If the field may contain an explicitly null value
* Mutability: If the field may be explicitly changed via the API
* Rules: Any constraints on the field's values in addition to ones imposed by its type





 Field types 

Types constrain the kinds of values a field can contain. Each field must support only one of the below types. Note that most field types are stricter than their base JSON type. Each type below is linked to more information.



* An [Identifier](https://cloud.ibm.com/docs/api-handbook?topic=api-handbook-typesidentifier) field contains a string value uniquely identifying a resource.
* A [Boolean](https://cloud.ibm.com/docs/api-handbook?topic=api-handbook-typesboolean) field contains a true or false value.
* UI
* CLI
* API
* Terraform







 Provisioning 

To create an IBM Cloud® Databases deployment, first create an IBM Cloud service instance. The service type is determined by the service ID. Specify the appropriate service ID when you create a new Cloud Databases deployment.

Provision a deployment by going to the service's catalog page, or by specifying the service ID in the CLI, API, or Terraform.



Table 1. Cloud Databases deployments

 Deployment Type Service ID 

 [IBM Cloud® Databases for DataStax](https://cloud.ibm.com/catalog/services/databases-for-cassandra) databases-for-datastax 
 [IBM Cloud® Databases for Elasticsearch](https://cloud.ibm.com/catalog/services/databases-for-elasticsearch) databases-for-elasticsearch 
 [IBM Cloud® Databases for EnterpriseDB](https://cloud.ibm.com/catalog/services/databases-for-enterprisedb) databases-for-enterprisedb 
 [IBM Cloud® Databases for etcd](https://cloud.ibm.com/catalog/services/databases-for-etcd) databases-for-etcd 
 [IBM Cloud® Databases for MongoDB](https://cloud.ibm.com/catalog/services/databases-for-mongodb) databases-for-mongodb 
 [IBM Cloud® Databases for PostgreSQL](https://cloud.ibm.com/catalog/services/databases-for-postgresql) databases-for-postgresql 
 [IBM Cloud® Databases for Redis](https://cloud.ibm.com/catalog/services/databases-for-redis) databases-for-redis 
 [IBM Cloud® Databases for MySQL](https://cloud.ibm.com/catalog/services/databases-for-mysql) databases-for-mysql 
 [IBM Cloud® Messages for RabbitMQ](https://cloud.ibm.com/catalog/services/messages-for-rabbitmq) messages-for-rabbitmq
The following are the differences between the Version 1 type system and the Version 2 type system.



* New entity types:



* Date
* Duration
* Measure
* Money
* Number*
* Percent*
* PhoneNumber*
* Ordinal
* Time
* URL*



* Removed entity types:



* Anatomy
* Award
* Broadcaster
* Company
* Crime
* Drug
* HealthCondition
* Movie
* MusicGroup
* NaturalEvent
* PrintMedia
* Quantity
* Sport
* SportingEvent
* TelevisionShow
* Vehicle



* New entity subtype:



* Quantity





* This entity type is not yet detectable in French text.





 January 2018 



 25 January 2018 

Custom model support for Chinese (Simplified)
: Chinese (Simplified) [custom model](https://cloud.ibm.com/docs/natural-language-understanding?topic=natural-language-understanding-customizing) support is now available for entities and relations.





 12 January 2018 

Concepts support for German
: Added support for German concepts.







 15 December 2017 

Language support improvements
: Dutch [custom model](https://cloud.ibm.com/docs/natural-language-understanding?topic=natural-language-understanding-customizing) support is now available for entities and relations.
: [French language support](https://cloud.ibm.com/docs/natural-language-understanding?topic=natural-language-understanding-language-supportfrench) now includes concepts.
: The French sentiment model has been improved to deliver better results.
: The language detection model is faster and detects more languages overall. For the complete list of languages, see [Detectable languages](https://cloud.ibm.com/docs/natural-language-understanding?topic=natural-language-understanding-detectable-languages).



* The following languages are new additions to the list:



* Belarusian (be)
* Bihari (bh)
* Dhivehi (dv)
* Galician (gl)
You must also create a .zip file that contains all of the files that are required to upload the analyzed data into a Knowledge Studio workspace.







 Mapping UIMA types to entity types 

Before you upload XMI files into a Knowledge Studio workspace, you must define mappings between the UIMA types and Knowledge Studio entity types.



 Before you begin 

The type system in your Knowledge Studio workspace must include the entity types that you want to map the UIMA types to.





 Procedure 

To map UIMA types to Knowledge Studio entity types, follow these steps:



1. Create a file named cas2di.tsv in the folder that contains the UIMA TypeSystem descriptor file, such as exported_typesystem.xml or TypeSystem.xml.
2. Open the cas2di.tsv file with a text editor. Each line in the file specifies a single mapping. The format of the mapping depends on which annotator's annotations you want to map:



* You can create mappings by using the basic format:

UIMA_Type_Name[TAB]WKS_Entity_Type

The following example defines mappings between UIMA types produced by the Named Entity Recognition annotator in Watson Explorer Content Analytics and entity types defined in a Knowledge Studio type system:

com.ibm.langware.Organization ORGANIZATION
com.ibm.langware.Person PERSON
com.ibm.langware.Location LOCATION

Another example defines a mapping between UIMA types produced by custom annotator that was created in Watson Explorer Content Analytics Studio and Knowledge Studio entity types:

com.ibm.Person PERSON
com.ibm.Date DATE
* You can create mappings based on facets that are used in the Pattern Matcher annotator or Dictionary Lookup annotator in Watson Explorer Content Analytics. In text analysis rule files (.pat), the facet is represented as the category attribute. To define a mapping, use the following syntax:

com.ibm.takmi.nlp.annotation_type.ContiguousContext:category={FACET_PATH}[TAB]{WKS_ENTITY_TYPE}
Types and formats 

Property and parameter schema MUST only use combinations of type and format defined in OpenAPI Specification [ref](https://github.com/OAI/OpenAPI-Specification/blob/master/versions/3.0.2.mddata-types), with the following additions:



Types and formats

 type      format        Comments                                                      

 string    identifier    identifier for a resource (see [Identifier in Types](https://cloud.ibm.com/docs/api-handbook/design/typesidentifier))        
 string    crn           the Cloud Resource Name for a resource (see [CRN in Types](https://cloud.ibm.com/docs/api-handbook/design/typescrn))  





  Specifying additional constraints 

Some constraints needed for input validation or output guarantees are not naturally expressed in OpenAPI. The following are recommendations for documenting such constraints.





  Date-time precision 

To document the precision of date-time properties in responses, use the following additional constraints:



  For second precision: 

minLength: 20
maxLength: 20





  For millisecond precision: 

minLength: 24
maxLength: 24







  String character sets 

Where practical, constrain the character set for string properties by using the schema's pattern. Regardless, describe the character set constraints in plain language in the schema's description.
CAST(2018-10-31 23:55:00 AS TIMESTAMP) CAST(2018-2-28 23:55:00 AS DATE) CAST(HELLO AS TIMESTAMP) 

 2018-10-31 23:55:00.0 2018-02-28 null 







 Boolean type 

The BOOLEAN type represents a domain with two values, true or false. Any numeric value that represents zero, for example, 0, 0.0, or 0.0E10, can be cast to false. Numeric values that represent a nonzero value, for example, 1, 1.0, 1.0E10, or 21474.83648 can be cast to true. The string value '0' can be cast to false and '1' can be cast to true. Any other string value is cast to false.





 Binary type 

A BINARY type represents an array of byte values. Thus, string values can be cast to type BINARY.





 Related references - dataType 

A dataType is referenced by the following clauses:



* [castExpression](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencecastExpression)
* [createTable](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referencecreateTable)









 Catalog management 

The following commands allow users to store table metadata catalog in the Data Engine catalog. By defining the tables, columns, and partitions in the catalog, you can use short table names in the SQL SELECT statements. Each instance of Data Engine has its own catalog, and table definitions are not visible from other instances. For more information, see [catalog management](https://cloud.ibm.com/docs/sql-query?topic=sql-query-getting_started_catalog).



 Create table 



 createTable 





 columnDefinition 

Create a table definition in the catalog based on the objects in the specified Object Storage location. The LOCATION option is mandatory. If a table or view with the same name exists in the same Data Engine instance, you receive an error, unless the IF NOT EXISTS clause is specified.