vania-janet/multiturn-rag-retrieval-data huggingface dataset:
[JSON](https://cloud.ibm.com/docs/language-translator?topic=language-translator-customizingjson) .json Custom JSON format for specifying aligned sentences and phrases. 
 [Microsoft Excel](https://cloud.ibm.com/docs/language-translator?topic=language-translator-customizingexcel) .xls, .xlsx Excel file with the first two columns for aligned sentences and phrases. The first row contains the language code. 



All formats have in common that they specify an alignment of translated text segments (sentences or phrases). You must encode all text data in UTF-8 format. For an example of how to convert a TMX file to UTF-8 format, see [Changing a TMX file to UTF-8 encoding](https://cloud.ibm.com/docs/language-translator?topic=language-translator-customizingchanging-tmx-file-to-utf-8).

You upload all training data files to the service as part of a multipart/form-data request. To indicate the type of the format (for example, TMX or CSV), either use the correct file extension (for example, .tmx or .csv) in the file name or set the content-type parameter for the respective part of the multipart/form-data request (for example, to --form "forced_glossary=@glossary;type=text/csv" for a CSV file).



 TMX 

One option to provide a glossary or corpus of aligned text segments for training the Language Translator service is to create a valid UTF-8 encoded document that conforms to the [Translation Memory Exchange (TMX) version 1.4 specification](http://www.ttt.org/oscarStandards/tmx/). TMX is an XML specification that is designed for machine-translation tools. Submit files to the service either with the file extension .tmx or a content-type of application/x-tmx+xml. The following example is a TMX-formatted glossary file with two translation units (<tu> elements):
"credentials": "Auto-generated service credentials",
"iam_apikey_description": "Auto generated apikey during resource-key operation for Instance - crn:v1:ibmcloud:public:language-translator:us-south:a/a11a1a11aa111a111aa1aa1111a1a111:1111a11-aaa1-11a1-111a-1a1111a11111::",
"iam_apikey_name": "auto-generated-apikey-aa1a1a1-aa1a-11a1-a1aa-aa11aa1a11a1",
"iam_role_crn": "crn:v1:ibmcloud:public:iam::::serviceRole:Manager",
"iam_serviceid_crn": "crn:v1:ibmcloud:public:iam-identity::a/a11a1a11aa111a111aa1aa1111a1a111::serviceid:ServiceId-11a1111-11a1-1111-1111-aaa11a11a11a",
"instance": "Language Translator-g2",
"url": "https://gateway.watsonplatform.net/language-translator/api"
}
}
}
]







 Installing from the Cloud Functions console 

Install the Language Translator package from the console.



1. In the Cloud Functions console, go to the [Create page](https://cloud.ibm.com/functions/create).
2. Select the namespace that you want to install the package into.
3. Click Install Packages.
4. Click the Watson Package group.
5. Click the Language Translator Package.
6. Click Install.
7. After the package has been installed you are redirected to the actions page and can search for your new package, which is named language-translator-v3.
8. To use the actions in the language-translator-v3 package, you must bind service credentials to the actions.
Real Estate Hotel Properties 
 Real Estate Houses 
 Real Estate Industrial Property 
 Real Estate Land and Farms 
 Real Estate Office Property 
 Real Estate Real Estate Buying and Selling 
 Real Estate Real Estate Renting and Leasing 
 Religion & Spirituality 
 Religion & Spirituality Agnosticism 
 Religion & Spirituality Spirituality 
 Religion & Spirituality Astrology 
 Religion & Spirituality Atheism 
 Religion & Spirituality Buddhism 
 Religion & Spirituality Christianity 
 Religion & Spirituality Hinduism 
 Religion & Spirituality Islam 
 Religion & Spirituality Judaism 
 Religion & Spirituality Sikhism 
 Science 
 Science Biological Sciences 
 Science Chemistry 
 Science Environment 
 Science Genetics 
 Science Geography 
 Science Geology 
 Science Physics 
 Science Space and Astronomy 
 Shopping 
 Shopping Coupons and Discounts 
 Shopping Flower Shopping 
 Shopping Gifts and Greetings Cards 
 Shopping Grocery Shopping 
 Shopping Holiday Shopping 
 Shopping Household Supplies 
 Shopping Lotteries and Scratchcards 
 Shopping Sales and Promotions 
 Shopping Children's Games and Toys 
 Sports 
 Sports American Football 
 Sports Boxing 
 Sports Cheerleading 
 Sports College Sports 
 Sports College Sports College Football 
 Sports College Sports College Basketball 
 Sports College Sports College Baseball 
 Sports Cricket 
 Sports Cycling 
 Sports Darts 
 Sports Disabled Sports 
 Sports Diving 
 Sports Equine Sports 
 Sports Equine Sports Horse Racing 
 Sports Extreme Sports 
 Sports Extreme Sports Canoeing and Kayaking 
 Sports Extreme Sports Climbing 
 Sports Extreme Sports Paintball 
 Sports Extreme Sports Scuba Diving 
 Sports Extreme Sports Skateboarding 
 Sports Extreme Sports Snowboarding 
 Sports Extreme Sports Surfing and Bodyboarding 
 Sports Extreme Sports Waterskiing and Wakeboarding 
 Sports Australian Rules Football 
 Sports Fantasy Sports 
 Sports Field Hockey 
 Sports Figure Skating 
 Sports Fishing Sports 
 Sports Golf 
 Sports Gymnastics 
 Sports Hunting and Shooting 
 Sports Ice Hockey 
 Sports Inline Skating 
 Sports Lacrosse 
 Sports Auto Racing 
 Sports Auto Racing Motorcycle Sports 
 Sports Martial Arts 
 Sports Olympic Sports 
 Sports Olympic Sports Summer Olympic Sports 
 Sports Olympic Sports Winter Olympic Sports 
 Sports Poker and Professional Gambling 
 Sports Rodeo 
 Sports Rowing 
 Sports Rugby 
 Sports Rugby Rugby League 
 Sports Rugby Rugby Union 
 Sports Sailing 
 Sports Skiing 
 Sports Snooker/Pool/Billiards
]
}



 Default stop word lists 

You can access the default stop words list for English from the [Watson Developer Cloud GitHub repository](https://github.com/watson-developer-cloud/doc-tutorial-downloads/tree/master/discovery-data/custom_stopwords_en.json).

For the following languages, Discovery uses the default stop words list that is defined by Apache Lucene. For more information about what words are included in the list, see the Lucene reference documentation:



* Arabic: [stopwords_ar.txt](https://github.com/apache/lucene/blob/main/lucene/analysis/common/src/resources/org/apache/lucene/analysis/ar/stopwords.txt)
* Czech: [stopwords_cs.txt](https://github.com/apache/lucene/blob/main/lucene/analysis/common/src/resources/org/apache/lucene/analysis/cz/stopwords.txt)
* Danish: [stopwords_da.txt](https://github.com/apache/lucene/blob/main/lucene/analysis/common/src/resources/org/apache/lucene/analysis/snowball/danish_stop.txt)
* Dutch: [stopwords_nl.txt](https://github.com/apache/lucene/blob/main/lucene/analysis/common/src/resources/org/apache/lucene/analysis/snowball/dutch_stop.txt)
* Finnish: [stopwords_fi.txt](https://github.com/apache/lucene/blob/main/lucene/analysis/common/src/resources/org/apache/lucene/analysis/snowball/finnish_stop.txt)
* Do not use backslashes, slashes, colons, equal signs, ampersands, or question marks in the name.
* Use a name that is unique among all custom language models that you own.



base_model_name (required string)
: The name of the base language model that is to be customized by the new custom model. You must use the name of a model that is returned by the GET /v1/models method. The new custom model can be used only with the base model that it customizes.

dialect (optional string)
: The dialect of the specified language that is to be used with the new custom model. For all languages, it is always safe to omit this field. The service automatically uses the language identifier from the name of the base model. For example, the service automatically uses en-US for all US English models.

If you specify the dialect for a new custom model, follow these guidelines:



* For non-Spanish previous-generation models and for next-generation models, you must specify a value that matches the five-character language identifier from the name of the base model.
* For Spanish previous-generation models, you must specify one of the following values:



* es-ES for Castilian Spanish (es-ES models)
* es-LA for Latin American Spanish (es-AR, es-CL, es-CO, and es-PE models)
* es-US for Mexican (North American) Spanish (es-MX models)





All values that you pass for the dialect field are case-insensitive.

description (optional string)
: A recommended description of the new custom model.



* Use a localized description that matches the language of the custom model.
* Include a maximum of 128 characters in the description.



The following example creates a new custom language model named Example model. The model is created for the base model en-US-BroadbandModel and has the description Example custom language model. The required Content-Type header specifies that JSON data is being passed to the method.

IBM Cloud
HistoricPlace 
 Hobby 
 HockeyCoach 
 HockeyConference 
 HockeyDivision 
 HockeyPlayer 
 HockeyTeam 
 Holiday 
 Hormone 
 Hospital 
 House 
 HumanLanguage 
 Illustrator 
 Industry 
 InfectiousDisease 
 Interest 
 InternetProtocol 
 Invention 
 Inventor 
 Island 
 JobTitle 
 Journal 
 Journalist 
 Judge 
 Kingdom 
 Lake 
 LanguageDialect 
 LanguageFamily 
 LanguageWritingSystem 
 LaunchSite 
 LegislativeCommittee 
 Legislature 
 Ligament 
 Lighthouse 
 LiterarySchoolOrMovement 
 Location 
 Magazine 
 ManufacturingPlant 
 MartialArt 
 MartialArtsOrganization 
 Mascot 
 Material 
 Mayor 
 MeansOfPropulsion 
 MedicalSpecialty 
 MedicalTreatment 
 MemberOfParliament 
 MeteorologicalService 
 MeteorShower 
 MilitaryConflict 
 MilitaryPerson 
 MilitaryPost 
 MilitaryUnit 
 Model 
 Monarch 
 Mountain 
 MultipartTVEpisode 
 Muscle 
 Museum 
 MusicalAlbum 
 MusicalArtist 
 MusicalGameSong 
 MusicalGroup 
 MusicalGroupMember 
 MusicalInstrumentCompany 
 MusicalPerformanceRole 
 MusicalRelease 
 MusicalTrack 
 MusicFestival 
 NaturalOrCulturalPreservationAgency 
 Nerve 
 Newspaper 
 NobleRank 
 NobleTitle 
 Non-ProfitOrganisation 
 Nutrient 
 OfficeHolder 
 OilField 
 OlympicAthlete 
 OlympicDemonstrationCompetition 
 OlympicEvent 
 OlympicEventCompetition 
 OlympicGames 
 Opera 
 OperaCharacter 
 OperaCompany 
 OperaHouse 
 OperaLibretto 
 OperatingSystem 
 OperatingSystemDeveloper 
 OrganismClassification 
 Organization 
 OrganizationCommittee 
 OrganizationSector 
 Park 
 ParliamentaryElection 
 PeriodicalPublisher 
 Person 
 PersonalAppearanceRole 
 Philosopher 
 Physician 
 Play 
 Poem 
 PoeticVerseForm 
 PoliticalDistrict 
 PoliticalIdeology 
 PoliticalParty
Example URL for an instance hosted in the Dallas location: api.us-south.language-translator.watson.cloud.ibm.com/instances/6bbda3b3-d572-45e1-8c54-22d6ed9e52c2

The previous public endpoint domain was watsonplatform.net.

For more information about the URLs, see the [API & SDK reference](https://cloud.ibm.com/apidocs/language-translator/language-translatorservice-endpoint).

These URLs do not introduce a breaking change. The new URLs work both for your existing service instances and for new instances. The original URLs continue to work on your existing service instances for at least one year (until December 2020).
* For more information about IAM, see [Authenticating to Watson services](https://cloud.ibm.com/docs/watson?topic=watson-iam).



New network and data security features
: Support for the following new network and data security features:



* Support for private network endpoints



* Users of Premium plans can create private network endpoints to connect to Language Translator over a private network. Connections to private network endpoints do not require public internet access. For more information, see [Public and private network endpoints](https://cloud.ibm.com/docs/watson?topic=watson-public-private-endpoints).



* Support for data encryption with customer-managed keys



* Users of new Premium and Dedicated instances can integrate IBM® Key Protect for IBM Cloud® with Language Translator to encrypt their data and manage encryption keys. For more information, see [Protecting sensitive information in your Watson service](https://cloud.ibm.com/docs/watson?topic=watson-keyservice).









 13 November 2019 

New Seoul location
: You can now create Language Translator instances in the Seoul, South Korea, location. As with other locations, the IBM Cloud Seoul location uses token-based Identity and Access Management (IAM) authentication.





 11 November 2019 

Customization and identification improvements
: Improved custom model training time.
Verify that the package is configured with your Natural Language Classifier service instance credentials.

ibmcloud fn package get natural-language-classifier-v1 parameters

Example output

ok: got package natural-language-classifier-v1, displaying field parameters
[
{
"key": "__bx_creds",
"value": {
"natural_language_classifier": {
"credentials": "Credentials-1",
"instance": "Watson Natural Language Classifier",
"password": "AAAA0AAAAAAA",
"url": "https://gateway.watsonplatform.net/natural_language_classifier/api",
"username": "00a0aa00-0a0a-12aa-1234-a1a2a3a456a7"
}
}
}
]







 Installing from the Cloud Functions console 

You can install your package from the Cloud Functions console.



1. In the Cloud Functions console, go to the [Create page](https://cloud.ibm.com/functions/create).
2. Select the namespace that you want to install the package into.
3. Click Install Packages.
4. Click the Watson Package group.
5. Click the Natural Language Classifier Package.
6. Click Install.
7. After the package is installed, you are redirected to the actions page and can search for your new package, which is named natural-language-classifier-v1.
8. To use the actions in the natural-language-classifier-v1 Package, you must bind service credentials to the actions.



* To bind service credentials to all actions in the package, follow steps 4 and 5 in the [CLI instructions](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-pkg_natlang_classifiernlclassifier_cli).
* To bind service credentials to individual actions, complete the following steps in the console.
Picheny, Michael, David Nahamoo, Vaibhava Goel, Brian Kingsbury, Bhuvana Ramabhadran, Steven J. Rennie, and George Saon. [Trends and Advances in Speech Recognition.](https://ieeexplore.ieee.org/document/6032775) IBM Journal of Research and Development, Vol. 55(5) (October 2011): pp. 2:1-2:18.
13. Saon, George, Zoltan Tueske, Daniel Bolanos, and Brian Kingsbury. [Advancing RNN Transducer Technology for Speech Recognition](https://arxiv.org/abs/2103.09935) Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (March 2021).
14. Saon, George, Zoltán Tüsky, and Kartik Audhkhasi. [Alignment-Length Synchronous Decoding for RNN Transducer.](https://ieeexplore.ieee.org/document/9053040) Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (May 2020).
15. Saon, George, Zoltan Tueske, Kartik Audhkhasi, and Brian Kingsbury. [Sequence Noise Injected Training for End-to-end Speech Recognition.](https://ieeexplore.ieee.org/document/8683706) Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (May 2019).
16. Saon, George, Gakuto Kurata, Tom Sercu, Kartik Audhkhasi, Samuel Thomas, Dimitrios Dimitriadis, Xiaodong Cui, Bhuvana Ramabhadran, Michael Picheny, Lynn-Li Lim, Bergul Roomi, and Phil Hall. [English Conversational Telephone Speech Recognition by Humans and Machines.]
[delete-classifier](https://cloud.ibm.com/apidocs/natural-language-classifierdelete-classifier) Action username, password, iam_access_token, iam_apikey, iam_url, headers, headers[X-Watson-Learning-Opt-Out], url, classifier_id Delete a classifier. 
 [get-classifier](https://cloud.ibm.com/apidocs/natural-language-classifierget-information-about-a-classifier) Action username, password, iam_access_token, iam_apikey, iam_url, headers, headers[X-Watson-Learning-Opt-Out], url, classifier_id Get information about a classifier. 
 [list-classifiers](https://cloud.ibm.com/apidocs/natural-language-classifierlist-classifiers) Action username, password, iam_access_token, iam_apikey, iam_url, headers, headers[X-Watson-Learning-Opt-Out], url List classifiers. 





 Creating a Natural Language Classifier service instance 

Before you install the package, you must create a Natural Language Classifier service instance and service credentials.



1. [Create a Natural Language Classifier service instance](https://cloud.ibm.com/catalog/services/natural_language_classifier).
2. When the service instance is created, service credentials are automatically created for you.







 Installing the Natural Language Classifier package 

After you have a Natural Language Classifier service instance, install the Natural Language Classifier package into your namespace.



 Installing from the Cloud Functions CLI 

Install the Natural Language Classifier package from the CLI. Be sure to [install the Cloud Functions plug-in for the IBM Cloud CLI](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-cli_install) first.

To install the Natural Language Classifier package, run the following command.



1. Clone the Natural Language Classifier package repo.
The bigger kitchen brought with it new chefs, but each one is faithful to the philosophy that made the Smith food truck so popular to begin with: deliver fresh, local produce in inventive and delicious ways. Join us for lunch or dinner seven days a week. Or order a cake from our bakery.
6. Let's add an image to the response also.

Click Add response type. Select Image from the drop-down list. In the Image source field, add https://www.ibmlearningcenter.com/wp-content/uploads/2018/02/IBM-Learning-Center-Food4.jpg.
7. Move the image response type up, so it is displayed in the response before the text is displayed. Click the Move up arrow to reorder the two response types.

![Shows the image response type listed before the text response type.](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/gs-ass-image-response-type.png)
8. Click ![Close](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/close.png) to close the edit view.







 Test the about_restaurant dialog node 

Test the intent by checking whether user utterances that are similar to, but not exactly the same as, the examples you added to the training data have successfully trained your assistant to recognize input with an about_restaurant intent.



1. Click the ![Try it](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/ask_watson.png) icon to open the "Try it out" pane.
2.
Use the DELETE /v1/customizations/{customization_id}/corpora/{corpus_name} method to remove an existing corpus from a custom language model.



* If the custom model is based on a next-generation model, the service deletes the corpus from the model.
* If the custom model is based on a previous-generation model, the service deletes the corpus from the model and removes OOV words that are associated with the corpus from the custom model's words resource. The service removes an OOV word from a custom model unless



* The word was also added by another corpus or by a grammar.
* The word was modified in some way with the POST /v1/customizations/{customization_id}/words or PUT /v1/customizations/{customization_id}/words/{word_name} method.





Removing a corpus does not affect the custom model until you train the model on its updated data by using the POST /v1/customizations/{customization_id}/train method. If you previously trained the model on the corpus successfully, information extracted from the corpus remains in the model and applies to speech recognition until you retrain the model.



 Delete a corpus example 

The following example deletes the corpus that is named corpus3 from the custom model with the specified customization ID:

IBM Cloud

curl -X DELETE -u "apikey:{apikey}" "{url}/v1/customizations/{customization_id}/corpora/corpus3"

IBM Cloud Pak for Data

curl -X DELETE --header "Authorization: Bearer {token}" "{url}/v1/customizations/{customization_id}/corpora/corpus3"
Execute terraform destroy in all directories in reverse order using the ./apply.sh command:

./apply.sh -d : :





 Expand the tutorial 

Your architecture may not be the same as the one presented, but will likely be constructed from the fundamental components discussed here. Ideas to expand this tutorial:



* Integrate incoming public Internet access using [IBM Cloud® Internet Services](https://cloud.ibm.com/docs/cis?topic=cis-getting-started).
* Add [Flow Logs for VPC capture](https://cloud.ibm.com/docs/vpc?topic=vpc-flow-logs) in the transit.
* Put each of the spokes in a separate account in an [enterprise](https://cloud.ibm.com/docs/account?topic=account-enterprise-tutorialaccount_groups_tutorial).
* Force some of the spoke to spoke traffic through the firewall and some not through the firewall.
* Replace the worker VSIs with [Red Hat OpenShift on IBM Cloud and VPC load balancer](https://cloud.ibm.com/docs/openshift?topic=openshift-vpc-lbaas).
* Force all out bound traffic through the firewall in the transit VPC and through [Public gateways](https://cloud.ibm.com/docs/vpc?topic=vpc-public-gateways) .







 Related content 



* [IBM Cloud for Financial Services](https://cloud.ibm.com/docs/framework-financial-services)
* Tutorial: [Best practices for organizing users, teams, applications](https://cloud.ibm.com/docs/solution-tutorials?topic=solution-tutorials-users-teams-applicationsusers-teams-applications)
* [How to deploy isolated workloads across multiple locations and regions](https://cloud.ibm.com/docs/solution-tutorials?topic=solution-tutorials-vpc-multi-region)
* You can now [retrieve the source file share for a replica file share](https://cloud.ibm.com/apidocs/vpc-betaget-share-source). Returned information also includes the replication schedule, role, status, and status reasons.
* You can now [split the source file from a replica share](https://cloud.ibm.com/apidocs/vpc-betadelete-share-source), which removes the replication relationship between a source share and its replica share, resulting in two independent read-write file shares. This action cannot be reversed. For more information, see [Remove the replication relationship](https://cloud.ibm.com/docs/vpc?topic=vpc-file-storage-manage-replication&interface=apifs-remove-replication).
* You can now [fail over to the replica file share](https://cloud.ibm.com/apidocs/vpc-betafailover-share), which reverses the replication relationship. You can optionally specify split for the fallback_policy to trigger a split if the failover operation fails or times out. For more information, see [Replication failover](https://cloud.ibm.com/docs/vpc?topic=vpc-file-storage-failover&interface=api).



File storage native tagging. Accounts that have been granted special approval to preview this feature can now specify user_tags when [creating a new file share](https://cloud.ibm.com/apidocs/vpc-betacreate-share) or [updating an existing file share](https://cloud.ibm.com/apidocs/vpc-betaupdate-share). Adding user tags to a file share helps you organize your resources. For details, see [Add user tags to a file share](https://cloud.ibm.com/docs/vpc?topic=vpc-file-storage-managing&interface=apifs-add-user-tags).







 22 March 2022 



 For all version dates 

VPN client-to-site servers update.