vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Enter the following context variable name and value pair:



Order number context variable details

 Variable Value 

 $ordernumber 



The context variable value (<? @order_number.literal ?>) is a SpEL expression that captures the number that the user specifies that matches the pattern defined by the @order_number pattern entity. It saves it to the $ordernumber variable.
16. Add the following message in the response text field:

Ok. The order $ordernumber is canceled. We hope we get the opportunity to bake a cake for you sometime soon.

![Shows the order number child node details.](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/gs-ass-order-number-child.png)
17. Click ![Close](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/close.png) to close the edit view.
18. Add another node to capture the case where a user provides a number, but it is not a valid order number. Click the More![More options](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/kabob.png) icon on the @order_number node, and then select Add node below.
19. Type true into the If assistant recognizes field of this node.
20. Add the following message in the response text field:

I need the order number to cancel the order for you. If you don't know the order number, please call us at 958-234-3456 to cancel over the phone.

!
* For the purposes of this tutorial, where we want to explore how the Smart Document Understanding tool works, we'll choose to use the user-trained model.



If you don't choose a model, the text extraction model is applied automatically. With the text extraction model, most of the document content is treated as standard text and is indexed in the text field.
3. Click User-trained models, and then click Submit.

Zoom

![Shows the confirmation dialog after you choose the User-trained models option.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/tut-sdu-pick-user-trained.png)

Figure 10. Confirmation dialog for user-trained model
4. Click Apply changes and reprocess.

Zoom

![Shows the Apply changes and reprocess button is active.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/tut-sdu-extra-step.png)

Figure 11. Apply changes and reprocess button

After the evaluation process is complete, a representation of the document is displayed in the Smart Document Understanding tool.

Zoom

![Shows the PDF as it is displayed in the SDU tool.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/tut-sdu-tool.png)

Figure 12. PDF is displayed in the SDU tool

The tool shows you a view of the original document along with a representation of the document, where the text is replaced by blocks. The blocks represent field types.
Custom language models and grammars can also influence how and where the service splits a transcript.

The parameter causes the service to add an end_of_utterance field to each final result to indicate the motivation for the split: full_stop, silence, end_of_data, or reset.

For more information, see [Split transcript at phrase end](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-parsingsplit-transcript).





 12 December 2019 

Full support for IBM Cloud IAM
: The Speech to Text service now supports the full implementation of IBM Cloud Identity and Access Management (IAM). API keys for IBM Watson® services are no longer limited to a single service instance. You can create access policies and API keys that apply to more than one service, and you can grant access between services. For more information about IAM, see [Authenticating to Watson services](https://cloud.ibm.com/docs/watson?topic=watson-iam).

To support this change, the API service endpoints use a different domain and include the service instance ID. The pattern is api.{location}.speech-to-text.watson.cloud.ibm.com/instances/{instance_id}.



* Example HTTP URL for an instance hosted in the Dallas location:

https://api.us-south.speech-to-text.watson.cloud.ibm.com/instances/6bbda3b3-d572-45e1-8c54-22d6ed9e52c2
* Example WebSocket URL for an instance hosted in the Dallas location:

wss://api.us-south.speech-to-text.watson.cloud.ibm.com/instances/6bbda3b3-d572-45e1-8c54-22d6ed9e52c2



For more information about the URLs, see the [API & SDK reference](https://cloud.ibm.com/apidocs/speech-to-text/speech-to-textservice-endpoint).

These URLs do not constitute a breaking change.
* The service now supports the X-Watson-Metadata header and the DELETE /v1/user_data method. For more information, see [Information security](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-information-security).







 2 October 2017 

Changes to audio/l16 audio format
: For the audio/l16 format, you can now optionally specify the endianness of the audio that is returned. (You must already specify the sampling rate.) Examples are audio/l16;rate=22050;endianness=big-endian and audio/l16;rate=22050;endianness=little-endian; the default is big endian. For more information, see [Using audio formats](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-audio-formats).





 14 July 2017 

New support for MP3 (MPEG) audio format
: The service now supports the MP3 or Motion Picture Experts Group (MPEG) audio format. For more information about supported audio formats, see [Using audio formats](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-audio-formats).





 10 April 2017 

New support for Web Media (WebM) audio format
: The service now supports the Web Media (WebM) audio format with the Opus or Vorbis codec. The service now also supports the Ogg audio format with the Vorbis codec in addition to the Opus codec. For more information about supported audio formats, see [Using audio formats](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-audio-formats).

New support for Cross-Origin Resource Sharing
: The service now supports Cross-Origin Resource Sharing (CORS) to allow browser-based clients to call the service directly.
: Optional. The number of resources to skip. The minimum value is 0.

--sort ([]string)
: Optional. The sorting order. The default value is ["name"]. The list items must match the regular expression /^-?[a-z0-9_.\[\],-]+$/.

--all-pages (bool)
: Optional. Invoke multiple requests to display all pages of the collection for the associated resources for the target keystore.
* Example

ibmcloud hpcs uko associated-resources-for-target-keystore 
--id=93777bca-baef-4070-b9b5-a2e6079df1b4 
--uko-vault=5295ad47-2ce9-43c3-b9e7-e5a9482c362b 
--output json
* Output

The command returns the output similar to the following example:

{
"total_count": 3,
"limit": 20,
"offset": 0,
"first": {
"href": "https://uko.us-south.hs-crypto.cloud.ibm.com:9549/api/v4/managed_key/5295ad47-2ce9-43c3-b9e7-e5a9482c362b/associated_resources?limit=20"
},
"last": {
"href": "https://uko.us-south.hs-crypto.cloud.ibm.com:9549/api/v4/managed_key/5295ad47-2ce9-43c3-b9e7-e5a9482c362b/associated_resources?limit=20&offset=0"
},
"previous": {
But suppose speech recognition is performed with a grammar that expects the user to speak six digits in a single-phrase response. Because the service splits the transcript at the one-second pause, the results are empty. The grammar is applied to each final result, but neither result, "one two three" nor "four five six", contains six digits.

The second example uses the same audio but sets the end_of_phrase_silence_time to 1.5 seconds:

IBM Cloud

curl -X POST -u "apikey:{apikey}" --header "Content-Type: audio/wav" --data-binary @{path}audio-file.wav "{url}/v1/recognize?end_of_phrase_silence_time=1.5"

IBM Cloud Pak for Data

curl -X POST --header "Authorization: Bearer {token}" --header "Content-Type: audio/wav" --data-binary @{path}audio-file.wav "{url}/v1/recognize?end_of_phrase_silence_time=1.5"

Because this value is greater than the length of the speaker's pause, the service returns a single final result that contains the entire spoken phrase. A grammar that expects to find six digits recognizes this transcript.

{
"result_index": 0,
"results": [
{
"alternatives":
{
"confidence": 1.0,
"transcript": "one two three four five six "
}
],
"final": true
}
]
}







 Split transcript at phrase end 

The split_transcript_at_phrase_end parameter directs the service to split the transcript into multiple final results based on semantic features of the input. Setting the parameter to true causes the service to split the transcript at the conclusion of meaningful phrases such as sentences. The service bases its understanding of semantic features on the base language model that you use with the request along with a custom language model or grammar that you use. Custom language models and grammars can influence how and where the service splits a transcript.
'Apache HTTP Server',
'Eclipse',
'Google Android'
);

-- extract product names from input text
create view ProductNames as
extract
dictionary 'ProductNamesDict'
on D.text as name
from Document D;

-- gather context around product name mention
create view ProductNamesWithContext as
select
P.name as name,
LeftContext(P.name, 5) as leftctxt,
RightContext(P.name, 5) as rightctxt
from ProductNames P;

-- use a regex to identify products with version number mentions on either sides of the product mention
create view ProductsWithVersionNumbers as
(
select
P.name as productname,
P.leftctxt as productversion
from ProductNamesWithContext P
where ContainsRegex (/vd((.d)+)?/, P.leftctxt)
)
union all
(
select
P.name as productname,
P.rightctxt as productversion
from ProductNamesWithContext P
where ContainsRegex (/vd((.d)+)?/, P.rightctxt)
);
Show more





 Equals 

The Equals function takes two arguments of arbitrary type:

Equals(<arg1>, <arg2>)

Two spans are considered equal if they both begin and end at the same offsets and contain the same text. If either or both arguments are null, the function returns null.

The following example illustrates the use of the Equals function.

-- Select phone number spans whose text is equal to 001-543-2217
create view PhoneNumber as
select P.number as number
from PhoneNum P
where Equals('001-543-2217',GetText(P.number));





 Follows 

The Follows predicate function takes two span arguments and two integer arguments:

Follows(<span1>, <span2>, <minchar>, <maxchar>)
When true, a DTMF collection succeeded, and returns the expected number of digits. When false, a DTMF collection failed to collect the specified number of digits. Even when dtmf_collection_succeeded is false, all collected digits are passed to the dialog in the input string of the turn request. 
 is_dtmf boolean Whether the input to Watson Assistant is dual-tone multi-frequency signaling (DTMF). 
 speech_to_text_result object The final response from the Speech to Text service in JSON format, including the transcript and confidence score for the top hypothesis and any alternatives. The format matches exactly the format that is received from the Speech to Text service. (For more information, see the [Speech to Text API documentation](https://cloud.ibm.com/apidocs/speech-to-textrecognize).) 





 Example 

{
"input": {
"text": "agent ",
"integrations": {
"voice_telephony": {
"speech_to_text_result": {
"result_index": 0,
"stopTimestamp": "2021-09-29T17:43:31.036Z",
"transaction_ids": {
"x-global-transaction-id": "43dd6ce0-139a-4d76-95aa-86e03fcfc434",
"x-dp-watson-tran-id": "6e60695e-fed7-4efe-a376-0888b027d30f"
},
"results": [
{
"final": true,
"alternatives":
{
"transcript": "agent ",
"confidence": 0.78
}
]
}
],
"transactionID": "43dd6ce0-139a-4d76-95aa-86e03fcfc434",
"startTimestamp": "2021-09-29T17:43:29.436Z"
Configuring support for Arabic 

Read these guidelines to understand how Knowledge Studio handles Arabic character shaping and numeric shaping in Arabic documents.



 About this task 

With respect to character shaping, the Arabic alphabet does not have capital letters, but letters can change shape depending on their position in the text string and the surrounding letters. Different operating systems and code page conversion programs handle letter shaping in different ways. Unshaped storage is a standard for Windows systems, and Knowledge Studio presumes that Arabic text is stored unshaped. If you want to upload shaped text into Knowledge Studio, you must first convert the text to unshaped form by using standard tools, such as the International Components for Unicode (ICU) API (see the ArabicShaping Class at [http://icu-project.org/apiref/icu4j/com/ibm/icu/text/ArabicShaping.html ![External link icon](https://cloud.ibm.com/docs-content/v1/content/icons/launch-glyph.svg)](http://icu-project.org/apiref/icu4j/com/ibm/icu/text/ArabicShaping.html)).

> Important: In some cases, lack of proper Arabic character shaping might cause content to be displayed incorrectly in the ground truth editor.

With respect to numeric shaping, Knowledge Studio treats numeric shaping as a storage-level property, similar to how Arabic content is handled on the iOS platform. Because a lot of Arabic content is created on platforms like Windows, which treat numeric shaping as a display-level property, you need to either convert content to make numeric shaping a storage-level property or use a Firefox browser when you use Knowledge Studio. Firefox supports the ability to set numeric shaping preferences explicitly at the browser level and enforce the appropriate display for all content shown in the browser.





 Procedure 

To configure numeric shaping in the Firefox browser:



1. In the browser URL field, enter about:config. If you are shown a warning from Firefox, click the action to disregard the warning and continue.
Compression and optimization concepts 

IBM Cloud® Internet Services applies gzip and brotli compression to some types of content. CIS also compresses items based on the browser's UserAgent to speed up page loading time.

If you're already using gzip, CIS honors your gzip settings as long as you're passing the details in a header from your web server for the files.

CIS only supports the content type gzip towards your origin server and can also only deliver content either gzip-compressed, brotli-compressed, or not compressed.

CIS's reverse proxy is also able to convert between compressed formats and uncompressed formats, meaning that it can pull content from a customer's origin server through gzip and serve it to clients uncompressed (or reversed). This is done independently of caching.

The Accept-Encoding header is not respected and is removed.



 What gets compressed 

In addition to CIS's serving stale content and minification of CSS, JS, and HTML to speed up your site, CIS also provides gzip and brotli compression to help site owners.

CIS returns gzip or brotli encoded responses to compatible clients and browsers for the following content-types:

text/html
text/richtext
text/plain
text/css
text/x-script
text/x-component
text/x-java-source
text/x-markdown
application/javascript
application/x-javascript
text/javascript
text/js
image/x-icon
image/vnd.microsoft.icon
application/x-perl
application/x-httpd-cgi
text/xml
application/xml
application/xml+rss
application/vnd.api+json
application/x-protobuf
application/json
multipart/bag
multipart/mixed
application/xhtml+xml
font/ttf
font/otf
font/x-woff
image/svg+xml
application/vnd.ms-fontobject
application/ttf
If the input span contains any white space at the beginning or end, the Chomp function shrinks the span enough to remove the white space. The function then returns a new span with no leading or trailing white space. If the input span has no leading or trailing white space, then the Chomp function returns the same span. If the input span is null, then Chomp returns null.





 CombineSpans 

The CombineSpans function takes two spans as input and returns the shortest span that completely covers both input spans if the spans are based on the same text object.

CombineSpans(['IgnoreOrder',] <span1>, <span2>)

The CombineSpans function is sensitive to the order of its input spans, unless you use the IgnoreOrder parameter. When the optional IgnoreOrder parameter is used, the order of the two spans is ignored.

The following example illustrates the use of the CombineSpans function.

create view FullName as
select
CombineSpans('IgnoreOrder',F.name, L.name) as fullName
from
FirstName F,
LastName L
where
FollowsTok(F.name, L.name, 0,0);

The semantics of the function are as follows:



* If either span1 or span2 is null, or the two spans are over different Text objects, the function returns null.
* Otherwise, if span1 is smaller than span2 or the IgnoreOrder parameter is used, the function returns the shortest span that covers both span1 and span2.
* Otherwise (span1 is larger than span2 and the IgnoreOrder is not used), the function returns a runtime error.



Based on the definition of Span, the different scenarios of arguments to the CombineSpans function are as follows:



* Span 2 is always after span 1. In other words, left-to-right order is maintained:

CombineSpans([0,7], [3,7]) returns the span [0,7]
CombineSpans([0,7], [8,10]) returns the span [0,10]
If you want the expression to be rendered in its original data type, then do not surround it with text.

For example, you can add this expression to a dialog node response to return the entities that are recognized in the user input in String format:

The entities are <? entities ?>.

If the user specifies Hello now as the input, then the @sys-date and @sys-time entities are triggered by the now reference. The entities object is an array, but because the expression is included in text, the entities are returned in String format, like this:

The entities are 2018-02-02, 14:34:56.

If you do not include text in the response, then an array is returned instead. For example, if the response is specified as an expression only, not surrounded by text.

<? entities ?>

The entity information is returned in its native data type, as an array.

[
{
"entity":"sys-date","location":6,9],"value":"2018-02-02","confidence":1,"metadata":{"calendar_type":"GREGORIAN","timezone":"America/New_York"}
},
{
"entity":"sys-time","location":6,9],"value":"14:33:22","confidence":1,"metadata":{"calendar_type":"GREGORIAN","timezone":"America/New_York"}
}
]

As another example, the following $array context variable is an array, but the $string_array context variable is a string.

{
"context": {
"array": [
"one",
"two"
],
"array_in_string": "this is my array: $array"
}
}

If you check the values of these context variables in the Try it out pane, you will see their values specified as follows:

$array : ["one","two"]
When you release the mouse button, a sequence object is displayed on the canvas.
3. Right-click Clerical title and click Add After > Proximity Rule. To capture terms such as Archbishop of Canterbury, Robert Runcie, specify the minimum and maximum number of tokens between words, in this case 0-5.
4. Click the sequence on the canvas and then under Extractor Properties, open the General tab if it doesn't open automatically. Rename the sequence Clerics.

![Clerics extractor](https://cloud.ibm.com/docs-content/v1/content/50eddb8fd81f33092880335ff107a78ff5cd0f65/watson-knowledge-studio/images/clerics.jpg)



Example 2:

As a second example, select tweets that refer to Twitter names of industry analysts with a big data term. To accomplish this, create two dictionaries, one of Twitter usernames of analysts and a second of big data terms and combine them on the workspace canvas with a proximity of one to 25 tokens.

]







 Creating a union 

To combine extractors in a union:



1. Define two or more extractors on the canvas.
2. Click Output in the Extractor Properties pane to ensure that the number and names of output columns for each sequence or extractor are the same. In addition to having the same number of columns and identical column names, output columns in each sequence or extractor must have the same data type. In this context, data type is a span, number, string, character, date, or time. Note that you cannot edit these properties for an extractor or sequence while in a union.
3. On the canvas, without regard to order, click and drag one extractor or sequence above or below another one until the dark blue drop zone indicator indicates you can drop the extractor.

(images/analyst.jpg "Analyst Twitter extractor")Shows an indicator for creating a union with the Person extractor][!!]
4. Release the mouse button to drop the extractor.
5.
Autocorrecting user input 

Autocorrection fixes misspellings that users make in their requests. The corrected words are used to match to an action or an intent.

Autocorrection corrects user input in the following way:



* Original input: letme applt for a memberdhip
* Corrected input: let me apply for a membership



When your assistant evaluates whether to correct the spelling of a word, it does not rely on a simple dictionary lookup process. Instead, it uses a combination of natural language processing and probabalistic models to assess whether a term is, in fact, misspelled and should be corrected.

Autocorrection is enabled automatically for all English-language assistants. It is also available in French-language assistants, but is disabled by default. Autocorrection isn't available for any other languages.



 Disabling autocorrection 

If necessary, you can disable autocorrection for your assistant.

If you find that a domain-specific term is being corrected that shouldn't be, you can prevent the correction from happening by adding the term or phrase to your training data. For more details, see [Autocorrection rules](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-autocorrectionautocorrection-rules).

If you are using actions in your assistant, follow these steps to disable autocorrection:



1. On the Actions page, click Global settings![Gear icon](https://cloud.ibm.com/docs-content/v1/content/icons/settings.svg).
2. Click the Autocorrection tab.
3. Set the switch to Off, then click Save.



If you are using dialog in your assistant, follow these steps to disable autocorrection:



1. In the Options section, click Autocorrection.
2. Set the switch to Off.







 Testing autocorrection in dialog 

If you are using dialog, you can test autocorrection using Try it out.



1.
In addition, the extractor returns the entire match as reference, the first group as salutation, and the second group as name and carry forward the values of the given name, middle initial, and surname from the respective input views.

create view MiddleInitial as
extract regex /b([p{Lu}p{M}].s){1,5}b/
on between 1 and 10 tokens in D.text as initial
from Document D;

create view Person as
extract F.first as first,
M.initial as middle,
L.last as last,
pattern ('Mr.'|'Ms.'|'Miss')? (<F.first> <M.initial>? <L.last>)
return group 0 as reference
and group 1 as salutation
and group 2 as name
from FirstName F, MiddleInitial M, LastName L;

Since the subpattern expression ('Mr.'|'Ms.'|'Miss')? is optional, the value of the salutation output column is null when a salutation is not present in the text. Similarly, since the pattern subexpression <M.initial>? is optional, the value of the output column middle is null when a middle initial is not present.

Example 2: Sequence pattern with string match and match parameters

The goal of this example is to find occurrences of meeting notes for known projects by examining the title annotations of the document. Notice the with inline_match clause, which specifies that string matching is done over the match field of the view Title, as opposed to the entire document text.

create view Project as
extract
regex /[Pp]rojects?w/ on D.text as name
from Document D;

create view Title as
extract regex /[A-z]+./
on between 1 and 20 tokens in D.text as match
from Document D;

create view MeetingNote as
extract
pattern <'Meeting Notes:'[with case exact]> (<P.name>)
return group 0 as match
and group 1 as projectname