vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Types of artifacts 

First create or upload artifacts to use to train the models.

The models learn about the following types of artifacts that you add to the workspace:



*  Type system

A type system defines the entities and relationships between entities that matter to you. See [Establishing a type system](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-typesystem).
*  Dictionaries

A dictionary groups together words and phrases that should be treated equivalently by a model. See [Creating dictionaries](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-dictionaries).
*  Documents

Documents serve a different purpose depending on whether you are creating a machine learning model or a rule-based model. See the following topics for more information:



*  Machine learning model documents: [Adding documents for annotation](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-documents-for-annotation)
*  Rule-based model documents: [Adding documents for defining rules](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-wks_rule_anno_add)





You can upload many of these artifacts from external resources, including artifacts that you downloaded from other Knowledge Studio workspaces. See [Uploading resources from another workspace](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-exportimport) for information about uploading artifacts from other workspaces.

See [Summary of inputs, outputs, and limitations](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-create-projectwks_formats) for key facts about artifacts, such as size limits and supported file types for imported artifacts.
Types of artifacts 

First create or upload artifacts to use to train the models.

The models learn about the following types of artifacts that you add to the workspace:



*  Type system

A type system defines the entities and relationships between entities that matter to you. See [Establishing a type system](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-typesystem).
*  Dictionaries

A dictionary groups together words and phrases that should be treated equivalently by a model. See [Creating dictionaries](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-dictionaries).
*  Documents

Documents serve a different purpose depending on whether you are creating a machine learning model or a rule-based model. See the following topics for more information:



*  Machine learning model documents: [Adding documents for annotation](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-documents-for-annotation)
*  Rule-based model documents: [Adding documents for defining rules](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-wks_rule_anno_add)





You can upload many of these artifacts from external resources, including artifacts that you downloaded from other Knowledge Studio workspaces. See [Uploading resources from another workspace](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-exportimport) for information about uploading artifacts from other workspaces.

See [Summary of inputs, outputs, and limitations](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-create-projectwks_formats) for key facts about artifacts, such as size limits and supported file types for imported artifacts.
Types 

Types define particular kinds of values. Each field in a model MUST have exactly one type, and fields of some types cannot be optional, mutable, or (even in JSON merge patch requests) nullable. Types also govern how a value from (or for) a particular field can be represented in JSON and in query parameters.

The following types are described below:



* [Identifier](https://cloud.ibm.com/docs/api-handbook?topic=api-handbook-typesidentifier)
* [Boolean](https://cloud.ibm.com/docs/api-handbook?topic=api-handbook-typesboolean)
* [Integer](https://cloud.ibm.com/docs/api-handbook?topic=api-handbook-typesinteger)
* [Float](https://cloud.ibm.com/docs/api-handbook?topic=api-handbook-typesfloat)
* [String](https://cloud.ibm.com/docs/api-handbook?topic=api-handbook-typesstring)
* [Date](https://cloud.ibm.com/docs/api-handbook?topic=api-handbook-typesdate)
* [Date/Time](https://cloud.ibm.com/docs/api-handbook?topic=api-handbook-typesdatetime)
* [CRN](https://cloud.ibm.com/docs/api-handbook?topic=api-handbook-typescrn)
* [Enumeration](https://cloud.ibm.com/docs/api-handbook?topic=api-handbook-typesenumeration)
* [Model](https://cloud.ibm.com/docs/api-handbook?topic=api-handbook-typesmodel)
* [Array](https://cloud.ibm.com/docs/api-handbook?topic=api-handbook-typesarray)





 Identifier 

The identifier type contains a value which identifies a resource.
Establishing a type system 

The type system controls how content can be annotated.



 Type systems 

A type system defines things that are interesting in your domain content that you want to label with an annotation. The type system controls how content can be annotated by defining the types of entities that can be labeled and how relationships among different entities can be labeled. The model process manager typically works with subject matter experts for your domain to define the type system.

In Knowledge Studio, you can create a type system from scratch or upload an existing type system. To jump-start a workspace, you might want to upload a type system that was created for a similar domain. You can then edit the type system to add or remove entity types or redefine the relationship types.

A sample type system based on the KLUE type system is provided for you to use with the Knowledge Studio tutorials. KLUE stands for Knowledge from Language Understanding and Extraction and was derived by IBM Research based on the analysis of collections of news articles. [Download ![External link icon](https://cloud.ibm.com/docs-content/v1/content/50eddb8fd81f33092880335ff107a78ff5cd0f65/icons/launch-glyph.svg)](https://watson-developer-cloud.github.io/doc-tutorial-downloads/knowledge-studio/en-klue2-types.json) a sample KLUE type system.

Many industries, such in domains like metallurgy, geology, market intelligence, life science, electronic health records, and oncology publish dictionaries or ontologies of domain-specific terminology. Consider referencing this type of resource to get an idea of the types of entities you might want to define in your own type system.



 Mentions 

A mention is any span of text that you consider relevant in your domain data. For example, in a type system about automotive vehicles, occurrences of terms like airbag, Ford Explorer, and child restraint system might be relevant mentions.





 Entity types 

An entity type is how you categorize a real-world thing. An entity mention is an example of a thing of that type.
Establishing a type system 

The type system controls how content can be annotated.



 Type systems 

A type system defines things that are interesting in your domain content that you want to label with an annotation. The type system controls how content can be annotated by defining the types of entities that can be labeled and how relationships among different entities can be labeled. The model process manager typically works with subject matter experts for your domain to define the type system.

In Knowledge Studio, you can create a type system from scratch or upload an existing type system. To jump-start a workspace, you might want to upload a type system that was created for a similar domain. You can then edit the type system to add or remove entity types or redefine the relationship types.

A sample type system based on the KLUE type system is provided for you to use with the Knowledge Studio tutorials. KLUE stands for Knowledge from Language Understanding and Extraction and was derived by IBM Research based on the analysis of collections of news articles. You can download a sample KLUE type system from [here ![External link icon](https://cloud.ibm.com/docs-content/v1/content/icons/launch-glyph.svg)](https://watson-developer-cloud.github.io/doc-tutorial-downloads/knowledge-studio/en-klue2-types.json).

Many industries, such in domains like metallurgy, geology, market intelligence, life science, electronic health records, and oncology publish dictionaries or ontologies of domain-specific terminology. Consider referencing this type of resource to get an idea of the types of entities you might want to define in your own type system.



 Mentions 

A mention is any span of text that you consider relevant in your domain data. For example, in a type system about automotive vehicles, occurrences of terms like airbag, Ford Explorer, and child restraint system might be relevant mentions.





 Entity types 

An entity type is how you categorize a real-world thing. An entity mention is an example of a thing of that type. For example, the mention President Obama can be annotated as a PERSON entity type.
.collectionType("application/vnd.ibm.secrets-manager.secret+json")
.collectionTotal(Long.parseLong("1"))
.build();
ArbitrarySecretResource arbitrarySecretResource = new ArbitrarySecretResource.Builder()
.name("example-arbitrary-secret")
.description("Extended description for this secret.")
.payload("secret-data")
.build();
CreateSecretOptions createSecretOptions = new CreateSecretOptions.Builder()
.secretType ("arbitrary")
.resources(new java.util.ArrayList<>(Collections.singletonList(arbitrarySecretResource)))
.metadata(collectionMetadata)
.build();
Response<CreateSecret> createResp = sm.createSecret(createSecretOptions).execute();

v2 example

ArbitrarySecretPrototype arbitrarySecretResource = new ArbitrarySecretResource.Builder()
.name("example-arbitrary-secret")
.description("Extended description for this secret.")
.payload("secret-data")
.secretType("arbitrary")
.build();
CreateSecretOptions createSecretOptions = new CreateSecretOptions.Builder()
.secretPrototype(arbitrarySecretResource)
.build();
Response<Secret> createResp = sm.createSecret(createSecretOptions).execute();







 Migrating Python applications 

If you use Python, review the following steps and examples to migrate to Secrets Manager API v2.



1. Update your environment to use the latest version (2.0.0) by updating the requirements.txt or running pip install --upgrade "ibm-secrets-manager-sdk".
2. Update your import statements to use v2.
3. Remove the secret type, and the metadata and resources wrapper from your resource for each operation that you're using.
In this lesson, you will learn how to upload and modify a type system within Knowledge Studio. You must create or upload a type system before you begin any annotation tasks.



 About this task 

For more information about type systems, see [Type systems](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-typesystemwks_typesystem).





 Procedure 



1. Download the [en-klue2-types.json ![External link icon](https://cloud.ibm.com/docs-content/v1/content/148d3bd95f946aa1bb53ea1540475f522e6b61c9/icons/launch-glyph.svg)](https://watson-developer-cloud.github.io/doc-tutorial-downloads/knowledge-studio/en-klue2-types.json) file to your computer. This file contains an example KLUE type system.
2. Click Assets > Entity Types.
3. On the Entity Types page, click Upload.
4. Upload the en-klue2-types.json file from your computer. The uploaded type system is displayed in the table.
5. Browse the type system so you can see the data that was uploaded.
6. Edit an entity type:



1. Locate the MONEY entity type.
2. Double-click anywhere in the table row to edit the entity type.
3. In the Roles column, click the Delete a role icon ![The "Delete a role" icon.](images/wks_delete.png "The "Delete a role" icon.") next to the AWARD role.

![A screen capture of deleting a role from an entity type.](images/wks_tut_delete_role.png "A screen capture that shows the cursor hovering over the "Delete a role" icon."
type array type MUST be array. 
 items items MUST be present. 
 minItems minItems MUST be present. 
 maxItems maxItems MUST be present. 









 Model 

To learn about models, see the [Models](https://cloud.ibm.com/docs/api-handbook?topic=api-handbook-models) section. Each model—while itself a collection of fields with types—is also a type.

Because of this, one model can contain a field whose type is another model. For example, consider a Server Model with a processor field. This field's type could be the Processor Model. Consequently, an instance of the Server Model would contain an instance of the Processor Model in its processor field.
Required status per method

 Action Request Field required 

 Create a new category for a given type. POST ![Check mark icon](https://cloud.ibm.com/docs-content/v1/content/6f3c07c34058f637b1c86044ca4f9b24214330a7/images/checkmark-icon.svg) 
 Modify a category UPDATE ![Check mark icon](https://cloud.ibm.com/docs-content/v1/content/6f3c07c34058f637b1c86044ca4f9b24214330a7/images/checkmark-icon.svg) 









 Examples of using the categories configuration API 

The following are examples of how to use the categories configuration API.

In these examples, <SERVICE_KEY> is the [service key](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-service_keys) for your Log Analysis instance.



 Creating a category for boards 

The following sample creates a new category for boards named My new category.

curl POST https://api.us-south.logging.cloud.ibm.com/v1/config/categories/boards 
-H 'content-type: application/json' 
-H 'servicekey: <SERVICE_KEY>' 
-d '{"name": "My new category"}'





 Listing all defined categories for views 

The following sample lists all defined categories for views.

curl GET https://api.us-south.logging.cloud.ibm.com/v1/config/categories/views 
-H 'content-type: application/json' 
-H 'servicekey: <SERVICE_KEY>' 





 Deleting a category 

The following sample deletes a category.

curl DELETE https://api.us-south.logging.cloud.ibm.com/v1/config/categories/views/xxxxxxxxxx 
-H 'content-type: application/json' 
-H 'servicekey: <SERVICE_KEY>'
Creating projects 

A project is a convenient way to collect and manage the resources in your IBM Watson® Discovery application. You can assign a Project type and connect your data to the project by creating a collection.

Before you create a project, decide which project type best fits your needs.



 Project descriptions 



Project type use cases

 Need Goal Project type 

 Which document contains the answer to my question? Find meaningful information in sources that contain a mix of structured and unstructured data, and surface it in a stand-alone enterprise search application or in the search field of a business application. Document Retrieval 
 Where is the part of the contract that I need for my task? Quickly extract critical information from contracts. Document Retrieval for Contracts 
 I want the chatbot I'm building to use knowledge that I own. Give a virtual assistant quick access to technical information that is stored in various external data sources and document formats to answer customer questions. Conversational Search 
 I want to uncover insights I didn't know to ask about. Gain insights from pattern analysis or perform root cause analysis. Content Mining 



If you created the Discovery service as part of a IBM Cloud Pak for Data as a Service deployment, the Discovery project is separate and distinct from the deployment project that is displayed in IBM Cloud.

To create a project, complete the following steps:



1. Open the Projects page by selecting My Projects.
2. Click New project. Name your project, and then choose the project type.

For more information about each type, see [Project types](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-projectsproject-type).

Otherwise, choose None of the above and a Custom project type is created for you.
3. If you choose a Document Retrieval project type and your data sources are in English, decide whether to enable the Content Intelligence feature.

If your data source contains contracts, enable the feature by selecting Apply contracts enrichment. Scroll to see the checkbox, if necessary.
4. Click Next.
5. Choose and configure a data source or connect to an existing collection.
SchoolSportsTeam 
 Scientist 
 ShipBuilder 
 ShoppingCenter 
 ShortStory 
 SkiArea 
 Skyscraper 
 SoccerClub 
 Software 
 SoftwareDeveloper 
 SoftwareLicense 
 Soundtrack 
 SpaceAgency 
 Spacecraft 
 SpaceMission 
 Spaceport 
 Sport 
 SportsAssociation 
 SportsEquipment 
 SportsLeagueChampionship 
 SportsLeagueChampionshipEvent 
 SportsLeagueDraft 
 SportsOfficial 
 SportsTeam 
 Stadium 
 Star 
 StarSystem 
 StarSystemBody 
 Station 
 Supercouple 
 Symptom 
 Synthesizer 
 Telescope 
 TelescopePlatform 
 TelevisionShow 
 TennisPlayer 
 TennisTournament 
 Theater 
 TheaterCharacter 
 TheaterProduction 
 TopLevelDomainRegistry 
 TouristAttraction 
 Tower 
 TradeUnion 
 TransitLine 
 TransportOperator 
 TransportTerminus 
 TropicalCyclone 
 TVActor 
 TVCharacter 
 TVCrewRole 
 TVEpisode 
 TVNetwork 
 TVPersonality 
 TVProducer 
 TVSeason 
 TVWriter 
 TypeOfPlaceOfWorship 
 UnfinishedWork 
 UnitOfMass 
 UnitOfVolume 
 University 
 U.S.Congressperson 
 Vein 
 VideoGame 
 VideoGameDesigner 
 VideoGameDeveloper 
 VideoGameEngine 
 VideoGamePlatform 
 VideoGamePublisher 
 VisualArtist 
 WebBrowserExtension 
 Website 
 WineProducer 
 WineRegion 
 WorkOfFiction 
 Writer 







 Relations entity types 

For languages that use the Version 1 entity type system that also support relations, relations results contain entity types that are different from the entity types returned in entities results. To return the relations entity types in your entities results, you can specify one of the following public entity models in the model option for the entities feature to override the default entity detection model.



 Model ID Description 

 ar-news Arabic news 
 en-news English news 
 es-news Spanish news
Migrating Knowledge Studio solutions 

Use custom models and other resources that you created in Knowledge Studio by migrating them to Discovery.



 Using a model as is 

To start using your Knowledge Studio model immediately, export the model from Knowledge Studio and import it to Discovery as a machine learning enrichment.

When you import a Knowledge Studio model to use as is in Discovery, root-level entity types that were defined in the model can be recognized when they occur in your documents. Any mentions of entity subtypes that occur are identified as mentions of the parent entity type. The subtype entities themselves are not preserved. If you want the model to continue to distinguish between different subtypes of an entity, you must take extra steps. For more information, see [Retaining subtype information](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-migrate-wksmigrate-wks-subtypes).

You cannot continue to update a model that you import as an ML enrichment.

The following types of models can be imported and used as is:



* Rule-based models created in Knowledge Studio that find entities in documents based on rules that you define. (File format: .pear)
* Machine learning models created in Knowledge Studio that understand the linguistic nuances, meaning, and relationships specific to your industry (file format: .zip)



The models that you can add depend on your deployment type:



IBM Cloud

You can add models that were created with a IBM Watson® Knowledge Studio instance that is hosted in IBM Cloud only.
IBM Cloud Pak for Data

You can add models that were created with an instance of IBM Watson® Knowledge Studio that is hosted on IBM Cloud Pak® for Data or IBM Cloud.



For more information, see [Using imported ML models to find custom terms](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain-ml).





 Using a corpus as training data 

Discovery has an entity extractor tool that you can use to define a type system. The entity extractor user interface is similar to the Knowledge Studio user interface that is used to annotate documents that you add to corpus for a machine learning model.
Use imported ML models to find custom terms 

Use custom Machine Learning models that use rules or context to recognize and tag entities.

Add Machine Learning models that you created with IBM tools that you can use to define your own type system.

The type of models you can add depend on your deployment:



IBM Cloud Pak for Data

You can add models that were created with Watson Explorer Content Analytics Studio models, or with an instance of IBM Watson® Knowledge Studio that is hosted on IBM Cloud Pak® for Data or IBM Cloud. Starting with the 4.6.2 release, you can also add custom entity extractor models that were created in and exported from another instance of Discovery.
IBM Cloud

You can add models that were created with a IBM Watson® Knowledge Studio instance that is hosted in IBM Cloud only.

To use a Knowledge Studio model that was built with Knowledge Studio on IBM Cloud Pak for Data, migrate the ground truth to a IBM Cloud instance of Knowledge Studio. and then retrain the model.



The following types of models are supported:



* Rule-based models created in Knowledge Studio that find entities in documents based on rules that you define. (File format: .pear)
* Machine learning models created in Knowledge Studio that understand the linguistic nuances, meaning, and relationships specific to your industry (file format: .zip)
* Custom entity extractors that are created in and exported from Discovery. (File format: .ent)
IBM Cloud Pak for Data

Custom UIMA text analysis models created in Watson Explorer Content Analytics Studio. (File format: .pear)



From installed deployments, support for importing entity extractor models was added with the 4.6.2 release.

Discovery cannot identify entity subtypes that are defined by a Knowledge Studio model.

To add a Machine Learning model, complete the following steps:



1. Create the model and export it from the tool you use to create it.

For more information, see the following documentation:



* Knowledge Studio for IBM Cloud Pak® for Data



* [Creating a rule-based model](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-rule-annotator)