vania-janet/multiturn-rag-retrieval-data huggingface dataset:
For each group, count the number of tuples with a non-null firstname value. Produce a single output tuple for each such group, with two values, the name and the number of tuples in that group.



Example 2: Issues with grouping dissimilar fields

This example illustrates a statement that is not valid.

select GetText(P.firstname) as first,
GetText(P.lastname) as last,
Count(P.firstname) as occurrences
from Person P
group by GetText(P.firstname);

The inclusion of GetText(P.lastname) in the select list is not accepted, since tuples with the same firstname values might have different lastname values, which leads to ambiguity.







 The order by clause 

The optional order by clause directs the runtime component to order the output tuples that are produced by the select statement from each document based on the values of the order by list, which is a comma-delimited set of expressions



 Syntax 

select ...
[order by <order by list>]





 Description 



* <order by list>

Specifies a comma-delimited list of expressions.

The order is based on the values of a comma-delimited list of expressions. The order by clause supports expressions that return numeric (Integer or Float), Text, or Span data types. If an expression in the order by clause returns a type Span, the result tuples are compared by comparing the relevant span values. In the following example, the span values of the person field are compared.

order by P.person

The order by clause treats nulls as unordered (amongst themselves). Nulls are ordered lower than other objects.







 Examples 

Example 1: Order by multiple expressions

Assume that person is a field of type Span. The following order by clause specifies that the statement returns tuples within each document. They are ordered lexicographically by the text of the person field and then by the beginning of the person field.

order by GetText(P.person), GetBegin(P.person)







 The limit clause
* Include each sentence of the corpus on its own line, and terminate each line with a carriage return. Including multiple sentences on the same line can degrade accuracy.
* Add personal names as discrete units on separate lines. Do not add the individual elements of a name on separate lines or as individual custom words, and do not include multiple names on the same line of a corpus. The following example shows the correct way to improve recognition accuracy for three names:

Gakuto Kutara
Sebastian Leifson
Malcolm Ingersol

Include additional contextual information where appropriate, for example, Doctor Sebastian Leifson or President Malcolm Ingersol. As with all words, duplicating the names multiple times can improve recognition accuracy.
* Beware of typographical errors. The service assumes that typographical errors are new words. Remember the adage Garbage in, garbage out!
* More sentences result in better accuracy. But the service does limit a model to a maximum of 10 million total words from all sources combined.







 What happens when I add a corpus file? 

When you add a corpus file, the service analyzes the file's contents. To distill the most meaning from the content, the service tokenizes and parses the data that it reads from a corpus file. The following topics describe how the service parses a corpus file for each supported language.

Information for the following languages is not yet available: Arabic, Chinese, Czech, Hindi, and Swedish. If you need this information for your custom language model, contact your IBM Support representative.



 Parsing of Dutch, English, French, German, Italian, Portuguese, and Spanish 

The following information applies to all supported dialects of Dutch, English, French, German, Italian, Portuguese, and Spanish:



* Converts numbers to their equivalent words.



Table 1. Examples of number conversion

 Language Whole number Decimal number 

 Dutch 500 becomes vijfhonderd 0,15 becomes nul komma vijftien 
 English 500 becomes five hundred 0.15 becomes zero point fifteen 
 French 500 becomes cinq cents 0,15 becomes zéro virgule quinze 
 German 500 becomes fünfhundert 0,15 becomes null punkt fünfzehn
For more information, see [The asynchronous HTTP interface](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-async) and the [API & SDK reference](https://cloud.ibm.com/apidocs/speech-to-text).

New beta smart_formatting parameter for speech recognition
: A beta smart formatting feature that converts dates, times, series of digits and numbers, phone numbers, currency values, and Internet addresses into more conventional representations in final transcripts. You enable the feature by setting the smart_formatting parameter to true on a recognition request. The feature is beta functionality that is available for US English only. For more information, see [Smart formatting](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-formattingsmart-formatting).

New French broadband model
: The list of supported models for speech recognition now includes fr-FR_BroadbandModel for audio in the French language that is sampled at a minimum of 16 kHz. For more information, see [Previous-generation languages and models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-models).

New support for audio/basic audio format
: The list of supported audio formats now includes audio/basic. The format provides single-channel audio that is encoded by using 8-bit u-law (or mu-law) data that is sampled at 8 kHz. For more information, see [audio/basic format](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-audio-formatsaudio-formats-basic).

Speech recognition methods now return warnings for invalid parameters
: The various recognition methods can return a warnings response that includes messages about invalid query parameters or JSON fields that are included with a request. The format of the warnings changed.
[More options](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/kabob.png) icon on the input.text.find('d') node, and then select Add child node.
7. Type true into the If assistant recognizes field of this node.
8. Enable conditional responses by clicking Customize, scrolling down, and then switching the Multiple responses toggle to On.
9. Click Apply.
10. In the newly-added If assistant recognizes field, type @order_number, and in the Respond with field, type:

Ok. The order $ordernumber is canceled. We hope we get the opportunity to bake a cake for you sometime soon.
11. Click Add response.
12. In the If assistant recognizes field, type true, and in the Respond with field, type:

I need the order number to cancel the order for you. If you don't know the order number, please call us at 958-234-3456 to cancel over the phone.

![Shows the addition of a node that checks for numbers in the user input and responds with a hint about the order number format.](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/gs-ass-give-hint.png)
13. Click ![Close](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/close.png) to close the edit view.



Now, when you test, you can provide a set of number or a mix of numbers and text as input, and the dialog reminds you of the correct order number format.
[More options](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/kebab.png) icon on the input.text.find('d') node, and then select Add child node.
7. Type true into the If assistant recognizes field of this node.
8. Enable conditional responses by clicking Customize, scrolling down, and then setting the Multiple conditioned responses switch to On.
9. Click Apply.
10. In the newly-added If assistant recognizes field, type @order_number, and in the Respond with field, type:

OK. The order $ordernumber is canceled. We hope we get the opportunity to bake a cake for you sometime soon.
11. Click Add response.
12. In the If assistant recognizes field, type true, and in the Respond with field, type:

I need the order number to cancel the order for you. If you don't know the order number, please call us at 958-234-3456 to cancel over the phone.

![Shows the addition of a node that checks for numbers in the user input and responds with a hint about the order number format.](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/gs-ass-give-hint.png)
13. Click ![Close](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/close.png) to close the edit view.



Now, when you test, you can provide a set of number or a mix of numbers and text as input, and the dialog reminds you of the correct order number format.
The following example illustrates the use of the RightContextTok function.

create view Salutation as
extract
regex /Mr.|Ms.|Miss/
on D.text as salutation
from Document D;

--Select the token immediately following a Salutation span
create view NameCandidate as
select RightContextTok(S.salutation, 1) as name
from Salutation S;

For both functions, if either argument is null, the function returns null.





 Remap 

The Remap function takes a single span argument:

Remap(<span>)

If the input span is over a text object that was produced by transforming another text object , the Remap function converts the span into a span over the original "source" text.

For example, if the span N.name is over a detagged document that is produced by running HTML through the AQL detag statement, then

Remap(<N.name>)

returns an equivalent span over the original HTML.

If the span argument was produced by running the detag statement over an empty document, the function remaps the spans to the beginning of the document (in other words, Document.text[0-0]). Also, if the detag statement produces an empty string, the function remaps the spans to the beginning of the document. The only part of AQL that produces such derived text object is the detag statement.

The following example illustrates the use of the Remap function:

-- Detag the HTML document and annotate the anchor tags
detag Document.text as DetagedDoc
annotate
element 'a' as Anchor;

-- Remap the Anchor Tags
create view AnchorTag as
select Remap(A.match) as anchor
from Anchor A;

If the argument to Remap is not a derived text object or a span over a derived text object , the function generates an error. If the argument is null, the function returns null.





 SpanBetween
You use the HTTP GET /v1/models method to list information about all available models. The service returns an array of JSON objects that provides information about all of the models.

The order in which the service returns models can change from call to call. So, do not rely on an alphabetized or static list of models. Because the models are returned as an array of JSON objects, the order has no bearing on programmatic uses of the response.



 List all models example 

The following example lists all models that are supported by the service:

IBM Cloud

curl -X GET -u "apikey:{apikey}" "{url}/v1/models"

IBM Cloud Pak for Data

curl -X GET --header "Authorization: Bearer {token}" "{url}/v1/models"

The response is abbreviated to show only the first few models.

{
"models": [
{
"name": "pt-BR_NarrowbandModel",
"language": "pt-BR",
"url": "{url}/v1/models/pt-BR_NarrowbandModel",
"rate": 8000,
"supported_features": {
"custom_language_model": true,
"custom_acoustic_model": true,
"speaker_labels": true
},
"description": "Brazilian Portuguese narrowband model."
},
{
"name": "ko-KR_BroadbandModel",
"language": "ko-KR",
"url": "{url}/v1/models/ko-KR_BroadbandModel",
"rate": 16000,
"supported_features": {
"custom_language_model": true,
"custom_acoustic_model": true,
"speaker_labels": true
},
"description": "Korean broadband model."
},
{
"name": "fr-FR_BroadbandModel",
"language": "fr-FR",
If you encounter this issue, export the PPT file as a PDF file, and then upload the PDF file instead. Apply a user-trained Smart Document Understanding (SDU) model to the document, and then use the SDU tool to identify the tables in the document. The resulting model handles table boundaries properly and can extract text from the tables cleanly.





 PDF file troubleshooting tips 

Failed to parse document due to invalid encoding
: Enable OCR for the file.





 Enrichment troubleshooting tips 

Table Understanding: n input tables excluded by enrichment
: If tables in a document have inconsistent column and row spans or are too large for the system to process completely, the table understanding enrichment is not applied to them. Information from such tables cannot be returned in search results. If you want the table understanding enrichment to be applied to a table that was skipped, consider editing the table. Change a table with inconsistent column and row spans to have a simpler table format. Split a large table into many smaller tables.

To find the table where the enrichment was not applied, check the warning message. It lists the character offsets where the table begins and ends in the HTML representation of the document. To see the full warning message and get the document ID, click View all, and then make a note of the document ID. From the Improve and customize page, submit an empty search query to return all of the indexed documents. Look for the document ID. (You can change the search result settings to show the document ID as the result title.) Click the View passage in document link for your document, and then click Open advanced view. Choose to view the document as JSON and then look for the html field. Copy and paste the HTML representation of the document into a text editor. Look for the character offsets that were listed in the original warning message to find the table.
first to ensure that the intents themselves are as unique as possible, which helps your assistant attain better intent confidence scores.Keep in mind:<-- <ul> --> * For nodes that condition on intents, if your assistant is confident that the node's intent condition matches the user's intent, then the node might be included as a disambiguation option. * For nodes with boolean conditions (conditions that evaluate to either true or false), the node might be included as a disambiguation option if the condition evaluates to true. For example, when the node conditions on an entity type, if the entity is mentioned in the input that triggers disambiguation, then the node might be included. * The order of nodes in the tree hierarchy impacts disambiguation.
<-- <ul> -->

* It impacts whether disambiguation is triggered at all

<-- </ul> -->

Look at the scenario]Disambiguation example]!! ! ! ! ! ! ! ! that is used earlier to introduce disambiguation, for example. If the node that conditions on @sys-date was placed higher in the dialog tree than the nodes that condition on the Customer_Care_Cancel_Account and eCommerce_Cancel_Product_Order intents, disambiguation would never be triggered when a user enters, i must cancel it today. That's because your assistant would consider the date mention (today) to be more important than the intent references due to the placement of the corresponding nodes in the tree.

<-- <ul> -->

* It impacts which nodes are included in the disambiguation options list

<-- </ul> -->

Sometimes a node is not listed as a disambiguation option as expected. This can happen if a condition value is also referenced by a node that is not eligible for inclusion in the disambiguation list for some reason. For example, an entity mention might trigger a node that is situated earlier in the dialog tree but is not enabled for disambiguation.
-- Select phone number spans whose text is equal to 001-543-2217
create view PhoneNumber as
select P.number as number
from PhoneNum P
where Equals('001-543-2217',GetText(P.number));





 Follows 

The Follows predicate function takes two span arguments and two integer arguments:

Follows(<span1>, <span2>, <minchar>, <maxchar>)

The function returns TRUE if the number of characters between the end of span1 and the beginning of span2 is between minchar and maxchar, inclusive. If any argument is null, the function returns null.





 FollowsTok 

The FollowsTok predicate function is a version of Follows; however, the FollowsTok distance arguments are in terms of tokens instead of characters:

FollowsTok(<span1>, <span2>, <mintok>, <maxtok>)

The FollowsTok function returns TRUE if the number of tokens between the end of span1 and the beginning of span2 is between mintok and maxtok, inclusive. If any argument is null, the function returns null.





 GreaterThan 

The GreaterThan predicate function takes two arguments of arbitrary type:

GreaterThan(<arg1>, <arg2>)

The function returns TRUE if <arg1> is greater than <arg2>. The function returns FALSE if either argument is null.





 IsNull 

The IsNull function tests whether data is, or is not, null. It takes a single argument of any type and returns TRUE if the single argument is null, and FALSE otherwise. The behavior of this predicate and the already defined NotNull predicate is different from all other predicates that return null on null input.





 MatchesDict 

The MatchesDict function takes a dictionary (as in a dictionary extraction), an optional flag specification, and a span as arguments:

MatchesDict('<dictionary>', ['<flags>', ]<span>)

The MatchesDict function returns TRUE if the span exactly matches one or more of the terms in the dictionary. The flags can be Exact or IgnoreCase.
The user might include only the numbers or the letters too, but forget that they are meant to be uppercase. So, it would be a nice touch to give them a hint in such cases. If you want to be kind, add another node to the dialog tree that checks for numbers in the user input.



1. Find the @order_number node that is a child of the Ask order number node.
2. Click the More![More options](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/kabob.png) icon on the @order_number node, and then select Add node below.
3. In the condition field, add input.text.find('d'), which is a SpEL expression that says if you find one or more numbers in the user input, trigger this response.
4. In the text response field, add the following response:

The correct format for our order numbers is AAnnnnn. The A's represents 2 uppercase letters, and the n's represent 5 numbers. Do you have an order number in that format?
5. Click ![Close](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/close.png) to close the edit view.
6. Click the More![More options](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/kabob.png) icon on the input.text.find('d') node, and then select Add child node.
7. Type true into the If assistant recognizes field of this node.
8.
The user might include only the numbers or the letters too, but forget that they are meant to be uppercase. So, it would be a nice touch to give them a hint in such cases, correct? If you want to be kind, add another node to the dialog tree that checks for numbers in the user input.



1. Find the @order_number node that is a child of the Ask order number node.
2. Click the More![More options](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/kebab.png) icon on the @order_number node, and then select Add node below.
3. In the condition field, add input.text.find('d'), which is a SpEL expression that says if you find one or more numbers in the user input, trigger this response.
4. In the text response field, add the following response:

The correct format for our order numbers is AAnnnnn. The A's represents 2 uppercase letters, and the n's represent 5 numbers. Do you have an order number in that format?
5. Click ![Close](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/close.png) to close the edit view.
6. Click the More![More options](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/kebab.png) icon on the input.text.find('d') node, and then select Add child node.
7. Type true into the If assistant recognizes field of this node.
8.
If the check_satisfaction node has a node name and has disambiguation enabled, then the name for that jumped-to node is displayed in the disambiguation list. As a result, Check satisfaction is displayed in the disambiguation list to represent the store-location branch instead of the Get store location name from the root node.
* Hide root nodes that condition on an entity or context variable only
Only a node with a matched intent can trigger disambiguation. However, once it's triggered, any node with a condition that matches is included in the disambiguation list. When your dialog includes nodes that condition on entities, for example, the order of nodes in the tree hierarchy can become significant in unexpected ways.

<-- <ul> -->

* The order of nodes impacts whether disambiguation is triggered at all

<-- </ul> -->

Look at the scenario]Disambiguation example]!! ! ! ! ! !! !!Shows the Settings link after the description text for the node name field.Shows the Settings link after the description text for the node name field_.]!!
Shows the expanded disambiguation settings section where the toggle is.Shows the expanded disambiguation settings section where the toggle is_.]!
Plus or higher plans only] that is used earlier to introduce disambiguation, for example. If the node that conditions on @sys-date was placed higher in the dialog tree than the nodes that condition on the Customer_Care_Cancel_Account and eCommerce_Cancel_Product_Order intents, disambiguation would never be triggered when a user enters, i must cancel it today. That's because your assistant would consider the date mention (today) to be more important than the intent references due to the placement of the corresponding nodes in the tree.

<-- <ul> -->

* The order of nodes impacts which nodes are included in the disambiguation options list

<-- </ul> -->

Sometimes a node is not listed as a disambiguation option as expected.
Correcting user input 

Autocorrection fixes misspellings that users make in the utterances that they submit as user input. When autocorrection is enabled, the misspelled words are automatically corrected. And it is the corrected words that are used to evaluate the input. When given more precise input, your assistant can more often recognize entity mentions and understand the user's intent.

Autocorrection is enabled automatically for all new English-language dialog skills. It is available as a beta feature in French-language dialog skills. You must turn it on to use it with French-language skills. For more information about language support, see [Supported languages](https://cloud.ibm.com/docs/assistant?topic=assistant-language-support).

Autocorrection corrects user input in the following way:



* Orignal input: letme applt for a memberdhip
* Corrected input: let me apply for a membership



When your assistant evaluates whether to correct the spelling of a word, it does not rely on a simple dictionary lookup process. Instead, it uses a combination of Natural Language Processing and probabalistic models to assess whether a term is, in fact, misspelled and should be corrected.



 Turning autocorrection on or off 

Autocorrection helps your assistant understand user input. It is enabled automatically for some languages and available, but disabled in others.

If you decide to disable it, you must turn it off entirely. You cannot disable autocorrection for a single word or phrase.

If you find that a domain-specific term is being corrected that shouldn't be, you can prevent the correction from happening by adding the term or phrase to your training data. For more details, see [Autocorrection rules](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-dialog-runtime-spell-checkdialog-runtime-spell-check-rules).

To turn autocorrection on or off, complete the following steps:



1. Click the Skills icon !