vania-janet/multiturn-rag-retrieval-data huggingface dataset:
"my_text": "is a text."
}
}





 String.toLowerCase() 

This method returns the original String converted to lowercase letters.

For this input:

"This is A DOG!"

This syntax:

{
"context": {
"input_lower_case": "<? input.text.toLowerCase() ?>"
}
}

Results in this output:

{
"context": {
"input_lower_case": "this is a dog!"
}
}





 String.toUpperCase() 

This method returns the original String converted to uppercase letters.

For this input:

"hi there".

This syntax:

{
"context": {
"input_upper_case": "<? input.text.toUpperCase() ?>"
}
}

Results in this output:

{
"context": {
"input_upper_case": "HI THERE"
}
}





 String.trim() 

This method trims any spaces at the beginning and the end of the string and returns the modified string.

For this Dialog runtime context:

{
"context": {
"my_text": " something is here "
}
}

This syntax:

{
"context": {
"my_text": "<? $my_text.trim() ?>"
}
}

Results in this output:

{
"context": {
"my_text": "something is here"
}
}





 java.lang.String support 

In addition to the built-in methods, you can use standard methods of the java.lang.String class.



 java.lang.String.format() 

You can apply the standard Java String format() method to text.
none "I am going to give her the <emphasis level='none'>book</emphasis>." Your browser does not support the audio tag. 
 moderate "I am going to <emphasis level='moderate'>give</emphasis> her the book." Your browser does not support the audio tag. 
 strong "I am going to <emphasis level='strong'>give</emphasis> her the book." Your browser does not support the audio tag. 
 reduced "I <emphasis level='reduced'>am going to give</emphasis> her the book." Your browser does not support the audio tag. 





 Emphasizing a word with SSML 

The following example HTTP request uses the <emphasis> element with a level of moderate (the default) to emphasize the word give in the sentence. The example uses the en-US_MichaelExpressive voice.

IBM Cloud

curl -X POST -u "apikey:{apikey}" --header "Content-Type: application/json" --header "Accept: audio/wav" --output strong-emphasis.wav --data "{"text":"I am going to <emphasis level='moderate'>give</emphasis> her the book."}" "{url}/v1/synthesize?voice=en-US_MichaelExpesssive"

IBM Cloud Pak for Data

curl -X POST --header "Authorization: Bearer {token}" --header "Content-Type: application/json" --header "Accept: audio/wav" --output strong-emphasis.wav --data "{"text":"I am going to <emphasis level='moderate'>give</emphasis> her the book."}" "{url}/v1/synthesize?voice=en-US_MichaelExpesssive"
Mix upper- and lowercase letters and use capitalization only when intended.
* Include each sentence of the corpus on its own line, and terminate each line with a carriage return. Including multiple sentences on the same line can degrade accuracy.
* Add personal names as discrete units on separate lines. Do not add the individual elements of a name on separate lines or as individual custom words, and do not include multiple names on the same line of a corpus. The following example shows the correct way to improve recognition accuracy for three names:

Gakuto Kutara
Sebastian Leifson
Malcolm Ingersol

Include additional contextual information where appropriate, for example, Doctor Sebastian Leifson or President Malcolm Ingersol. As with all words, duplicating the names multiple times and, if possible, in different contexts can improve recognition accuracy.
* Beware of typographical errors. The service assumes that typographical errors are new words. Unless you correct them before you train the model, the service adds them to the model's vocabulary. Remember the adage Garbage in, garbage out!
* More sentences result in better accuracy. But the service does limit a model to a maximum of 10 million total words and 90 thousand OOV words from all sources combined.



The service cannot generate a pronunciation for all words. After adding a corpus, you must validate the words resource to ensure that each OOV word's definition is complete and valid. For more information, see [Validating a words resource for previous-generation models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-corporaWordsvalidateModel).





 What happens when I add a corpus file? 

When you add a corpus file, the service analyzes the file's contents. It extracts any new (OOV) words that it finds and adds each OOV word to the custom model's words resource. To distill the most meaning from the content, the service tokenizes and parses the data that it reads from a corpus file. The following sections describe how the service parses a corpus file for each supported language.
CIS also compresses items based on the browser's UserAgent to speed up page loading time.

If you're already using gzip CIS honors your gzip settings as long as you're passing the details in a header from your web server for the files.

CIS only supports the content type "gzip" towards your origin server and can also only deliver content either gzip-compressed, brotli-compressed, or not compressed.

CIS's reverse proxy is also able to convert between compressed formats and uncompressed formats, meaning that it can pull content from a customer's origin server via gzip and serve it to clients uncompressed (or vice versa). This is done independently of caching.

The Accept-Encoding header is not respected and is removed.
The content of the input document is represented as a special view called Document.

The fields of a tuple must belong to one of the built-in data types of AQL:



* Boolean

A data type that has a true or false value.
* Float

A single-precision floating-point number.
* Integer

A 32-bit signed integer.
* ScalarList

A collection of values of the same scalar type (Integer, Float, Text, or Span). A value of data type ScalarList can be obtained as a result of the AQL built-in aggregate function List(), or as the result of a UDF.
* Span

A Span is a contiguous region of a Text object, which is identified by its beginning and ending offsets in the Text object. Assume that your input text is:

Amelia Earhart is a pilot.

The text across Span [0-6] is Amelia.

This Span can be visualized as:

0A1m2e3l4i5a6

Likewise, the text across Span [20-25] is pilot.

A Span of [x-x] represents the span between the end of a character and the beginning of the next character. In the previous example, [0-0] is an empty string before the character A. Likewise, a Span of [3-3] is an empty string between the characters e and l.

A value of type Span can be obtained as a result of an extract statement or a built-in scalar function, or a UDF.
* Text

The AQL data type to represent a character sequence. A text object contains a Unicode string, called its string value. When a string is formed as the concatenation of disjoint substrings of another Text object, it also contains reference to the original Text object and relevant offset mapping information. Two Text objects are considered equal to each other if all their components are correspondingly equal to each other.
* Comparing values of type Span and Text Prioritization factors affect comparisons between values of type Span and type Text.







 Comparing values of type Span and Text 

Prioritization factors affect comparisons between values of type Span and type Text.
"usage": {
"text_units": 2,
"text_characters": 15000,
"features": 3
}

There are 2 text units and 3 features in the request, so there are 2 × 3 = 6 NLU items.

Text units are counted seperately for each request. If one request analyzes 15,000 characters and another request analyzes 3,000 characters, that counts as 3 text units total. The first request has 2 text units and the second request has 1 text unit.





 FAQ 



 I want to extract Entities, Keywords and Categories for 50,000 documents with an average of 12,000 characters per document. What will be my estimated price on the Standard plan? 



* Step 1: Calculate number of text units per request
Text units = Number of groups of 10,000 characters or less
12,000 characters per request = 2 text units per request
* Step 2: Calculate the number of features per request
Entities, Keywords, Categories = 3 features
* Step 3: Calculate number of NLU items per request
NLU items = (Number of text unit) × (Number of features)
NLU items = 2 data units × 3 enrichments = 6 NLU items
* Step 4: Calculate total number of NLU items
Total number of NLU items = (Number of requests or documents) × (Number of NLU items)
Total number of NLU items = 50,000 requests × 6 NLU items = 300,000 NLU items
* Step 5: Estimate the price for the total number of NLU items
The first 250,000 NLU items are charged at $0.003 per item
Additional NLU items until the 5,000,000th NLU item are charged at $0.001 per item
Estimated price = (250,000 NLU items × $0.003) + (50,000 NLU items × $0.001)
Estimated price = $750 + $50



Total cost = $800





 How do I calculate the Standard Plan price for 15,000 NLU items?
I saw the story on yahoo dot com I saw the story on yahoo.com 
 Combinations The code is zero two four eight one and the date of service is May fifth two thousand and one The code is 02481 and the date of service is 5/5/2001 
 There are forty seven links on Yahoo dot com now There are 47 links on Yahoo.com now 







 Smart formatting results for long pauses 

In cases where an utterance contains long enough pauses of silence, the service can split the transcript into two or more final results. This affects the contents of the response, as shown in the following examples.



Table 3. Smart formatting example transcripts for long pauses

 Audio speech Formatted transcription results 

 My phone number is nine one four five five seven three three nine two My phone number is 914-557-3392 
 My phone number is nine one four ...pause... five five seven three three nine two My phone number is 914 <br>5573392 



For more information about specifying a pause interval that affects the service's response, see [End of phrase silence time](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-parsingsilence-time).





 Smart formatting example 

The following example requests smart formatting with a recognition request by setting the smart_formatting parameter to true. The following sections show the effects of smart formatting on the results of a request.

IBM Cloud

curl -X POST -u "apikey:{apikey}" --header "Content-Type: audio/flac" --data-binary @{path}audio-file.flac "{url}/v1/recognize?smart_formatting=true"

IBM Cloud Pak for Data

curl -X POST --header "Authorization: Bearer {token}" --header "Content-Type: audio/flac" --data-binary @{path}audio-file.flac "{url}/v1/recognize?smart_formatting=true"







 Numeric redaction 

The numeric redaction feature is beta functionality that is available for US English, Japanese, and Korean.
<br><br> * CSV file in UTF-8 format<br> * Text in UTF-8 format<br> * HTML<br> * PDF files (scanned and password-protected files are not supported)<br> * Microsoft Word DOC or DOCX files (password-protected files are not supported)<br> * ZIP file that contains documents downloaded from another workspace<br> * ZIP file that contains documents in UIMA CAS XMI format<br><br><br> ZIP archive file of documents <br><br> * 40,000 characters per document<br> * 10,000 documents per workspace<br> * 1,000 document sets (including annotation sets) per workspace<br> * 5 MB per file and 200 MB per upload (TXT, PDF, DOC, DOCX, and HTML files)<br><br><br> 
 Document annotation Manage human annotation. Annotate entities, relations, and coreference chains to create ground truth Annotation task Ground truth <br><br> * 256 active anntation tasks per workspace<br><br><br> 
 Training and refinement Train a supervised machine learning model to extract domain-specific information from unstructured text. Evaluate and improve a supervised machine learning model. You cannot create a semi-supervised or unsupervised machine learning model. You cannot do extensive feature engineering. Not applicable Machine learning model <br><br> * 1 machine learning model per workspace<br> * 10 model versions per workspace<br> * Maximum number of workspaces is determined by your deployment.<br> * The maximum number of training actions you can perform per month is determined by your deployment.<br><br><br> 
 Publication Export a machine learning model to use for performing text extraction in other Watson applications. Not applicable <br><br> * ZIP file<br><br><br> None 







 Rule-based model 

Table 3: Rule-based model



 Task Typical usage Supported input formats Supported output formats Limits and requirements 

 Rule editor Create or upload documents to the rule editor from which to define classes, regular expressions, and rules.
[Yes](https://cloud.ibm.com/docs-content/v1/content/ca9232b3702768f24bb897f48f10778fcda28718/watson-assistant/images/checkmark-icon.svg) 







 Fields 



 Name Type Description Required? 

 response_type string text Y 
 values list A list of one or more objects defining text response. Y 
 values.[n].text_expression object An object describing the text of the response. N 
 values.[n].text_expression.concat list A list of objects that form components of the text response. These objects can include scalar text strings and references to variables. N 
 selection_policy string How a response is selected from the list, if more than one response is specified. The possible values are sequential, random, and multiline. N 
 delimiter string The delimiter to output as a separator between responses. Used only when selection_policy=multiline. The default delimiter is newline (). N 







 Example 

This examples displays a greeting message to the user.

{
"generic": [
{
"response_type": "text",
"values":
{
"text_expression": {
"concat":
{
"scalar": "Hi, "
},
{
"variable": "step_472"
},
{
"scalar": ". How can I help you?"
}
]
}
}
],
"selection_policy": "sequential"
}
]
}
Show more







 text_to_speech 

Sends a command to the Text to Speech service instance used by the phone integration. These commands can dynamically change the configuration or behavior of the service during a conversation.



 Integration channel support 



 Phone 

 !
cf0a3d8b-4856-4aa5-be6a-88cca465eab0 my-test-key-20

 this is the same list sorted in KEY_ID order
 this is the order we expect when listing keys

10954149-0217-472d-8137-42fc330dbd03 my-test-key-16
16276369-ba81-4eb5-be92-f13512147ad4 my-test-key-14
215f592c-3e42-47b6-867c-3cc1cce8a8fd my-test-key-2
3a5a3b52-6942-463d-9f7a-5b216bbf5123 my-test-key-18
3b4b0511-00df-48e6-ad65-fad35bb6ce17 my-test-key-13

3d683d56-d340-40cb-b8b6-605dcfded01f my-test-key-15
3ee0d859-2968-4444-8e4d-3d28614f055c my-test-key-10
54dd2745-7718-487f-85e6-a67e758dd945 my-test-key-17
62090ba8-7363-408e-9edc-aa9fb69c09be my-test-key-12
768c9ceb-368b-46f7-9b1d-be211e036fe1 my-test-key-4

87679194-52d0-42af-81e1-bffd64f87315 my-test-key-7
90138be5-1dd1-4eea-b7c1-73cc8609e7f7 my-test-key-19
: Korean tokenizers have been updated to address informal language support. IBM continues to work on improvements to entity recognition and classification.

Emoji support
: Emojis added to intent examples, or as entity values, will now be correctly classified/extracted.

Only emojis that are included in your training data will be correctly and consistently identified; emoji support may not correctly classify similar emojis with different color tones or other variations.

Entity stemming (Beta - English only)
: The fuzzy matching beta feature recognizes entities and matches them based on the stem form of the entity value. For example, this feature correctly recognizes bananas as being similar to banana, and run being similar to running as they share a common stem form. For more information, see [Fuzzy matching](https://cloud.ibm.com/docs/assistant?topic=assistant-entitiesentities-fuzzy-matching).

Workspace import progress
: When you import a workspace from a JSON file, a tile for the workspace is displayed immediately, in which information about the progress of the import is displayed.

Reduced training time
: Multiple models are now trained in parallel, which noticeably reduces the training time for large workspaces.





 26 May 2017 

New API version
: The current API version is now 2017-05-26. This version introduces the following changes:



* The schema of ErrorResponse objects has changed. This change affects all endpoints and methods. For more information, see the [API Reference](https://cloud.ibm.com/apidocs/assistant).
* The internal schema used to represent dialog nodes in exported workspace JSON has changed. If you use the 2017-05-26 API to import a workspace that was exported using an earlier version, some dialog nodes might not import correctly. For best results, always import a workspace using the same version that was used to export it.







 25 May 2017 

Manage context variables
: You can now manage context variables in the "Try it out" pane. Click the Manage context link to open a new pane where you can set and check the values of context variables as you test the dialog.
You can use the customization interface to specify how the service pronounces unusual words that occur in your input text by creating language-specific custom dictionaries. For more information, see [Understanding customization](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-customIntro).





 15 July 2020 (Version 1.1.4) 

Red Hat OpenShift version 4.3 is going out of service
: IBM Cloud Pak for Data 3.0.1 is deprecating support for Red Hat OpenShift 4.3 on 1 September 2020. Red Hat OpenShift 4.3 is going out of service on 22 October 2020. IBM Cloud Pak for Data is introducing support for Red Hat OpenShift 4.5. IBM Cloud Pak for Data is recommending that clients upgrade to Red Hat OpenShift 4.5 before 22 October 2020. IBM Support will work with any customers who already installed IBM Cloud Pak for Data 3.0.1 on Red Hat OpenShift 4.3. New customers who want to install on Red Hat OpenShift 4.x are instructed to install Red Hat OpenShift 4.5.





 19 June 2020 (Version 1.1.4) 

Version 1.1.4 is available
: Text to Speech for IBM Cloud Pak for Data version 1.1.4 is now available. Installation and administration of the service include many changes. This version supports IBM Cloud Pak for Data versions 2.5 and 3.0.1, and Red Hat OpenShift versions 3.11 and 4.3. For more information about installing and managing the service, see [Installing Watson Text to Speech version 1.1.4](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-speech-install).

New neural voices
: The service now supports five new neural voices:



* US English: en-US_EmilyV3Voice, en-US_HenryV3Voice, en-US_KevinV3Voice, and en-US_OliviaV3Voice
* German: de-DE_ErikaV3Voice
Configuring support for Arabic 

Read these guidelines to understand how Knowledge Studio handles Arabic character shaping and numeric shaping in Arabic documents.



 About this task 

With respect to character shaping, the Arabic alphabet does not have capital letters, but letters can change shape depending on their position in the text string and the surrounding letters. Different operating systems and code page conversion programs handle letter shaping in different ways. Unshaped storage is a standard for Windows systems, and Knowledge Studio presumes that Arabic text is stored unshaped. If you want to upload shaped text into Knowledge Studio, you must first convert the text to unshaped form by using standard tools, such as the International Components for Unicode (ICU) API (see the ArabicShaping Class at [http://icu-project.org/apiref/icu4j/com/ibm/icu/text/ArabicShaping.html](http://icu-project.org/apiref/icu4j/com/ibm/icu/text/ArabicShaping.html)).

> Important: In some cases, lack of proper Arabic character shaping might cause content to be displayed incorrectly in the ground truth editor.

With respect to numeric shaping, Knowledge Studio treats numeric shaping as a storage-level property, similar to how Arabic content is handled on the iOS platform. Because a lot of Arabic content is created on platforms like Windows, which treat numeric shaping as a display-level property, you need to either convert content to make numeric shaping a storage-level property or use a Firefox browser when you use Knowledge Studio. Firefox supports the ability to set numeric shaping preferences explicitly at the browser level and enforce the appropriate display for all content shown in the browser.





 Procedure 

To configure numeric shaping in the Firefox browser:



1. In the browser URL field, enter about:config. If you are shown a warning from Firefox, click the action to disregard the warning and continue. For information about editing about:config properties, see [http://kb.mozillazine.org/About:config_entries](http://kb.mozillazine.org/About:config_entries).
The fields property in the JSON of a record schema is a case where its ordering is important. The Avro specification requires that a record’s fields are encoded and decoded in the order they appear in the schema that is used for the encode and decode operation.

As an example, consider the following three schemas.



 Schema 1 

{
"type": "record",
"name": "book",
"fields": [
{
"name": "title",
"type": "string"
},
{
"name": "author",
"type": "string"
}
]
}





 Schema 2 

{
"type": "record",
"name": "book",
"fields": [
{
"name": "author",
"type": "string"
},
{
"name": "title",
"type": "string"
}
]
}





 Schema 3 

{
"type": "record",
"name": "book",
"fields": [
{
"type": "string"
"name": "author",
},
{
"type": "string"
"name": "title",
}
]
}

Schema 1 and Schema 2 are distinct schemas and the registry stores them as separate schemas. They cannot be used interchangeably because they list the author and title fields in a different order. Data encoded with Schema 1 would not be decoded correctly if the decoding process used Schema 2.

When you use the SerDes to create the new schema in the order Schema 1, Schema 2, and Schema 3, the result would be two new schemas. Schema 1 and Schema 2 are different but Schema 3 is the equivalent of Schema 2.
The following image shows you how digressions into the #cuisine node (shown in the earlier illustration) are configured.

!]3. Click Apply.4. Use the "Try it out" pane to test the digression behavior.
Again, you cannot define the start and end of a digression. The user controls where and when digressions happen. You can only apply settings that determine how a single node participates in one. Because digressions are so unpredictable, it is hard to know how your configuration decisions will impact the overall conversation. To truly see the impact of the choices you made, you must test the dialog.
<-- </ol> -->The #reservation and #cuisine nodes represent two dialog branches that can participate in a single user-directed digression. The digression settings that are configured for each individual node are what make this type of digression possible at run time.!]<-- </section "id="section-dialog-runtime-enable-digressions" "> --><-- <section "id="section-dialog-runtime-digress-tips" "> --> Digression usage tips This section describes solutions to situations that you might encounter when using digressions.<-- <ul> --> * Custom return message: For any nodes where you enable returns from digressions away, consider adding wording that lets users know they are returning to where they left off in a previous dialog flow. In your text response, use a special syntax that lets you add two versions of the response.
If you do not take action, the same text response is displayed a second time to let users know they have returned to the node they digressed away from. You can make it clearer to users that they have returned to the original conversation thread by specifying a unique message to be displayed when they return.

For example, if the original text response for the node is, What's the order number?, then you might want to display a message like, Now let's get back to where we left off. What is the order number? when users return to the node.