vania-janet/multiturn-rag-retrieval-data huggingface dataset:
* text_normalized: If you provide customization input, the normalized version of the cell text according to the customization; otherwise, the same value as text.
* row_index_begin: The begin index of the cell's row location in the current table.
* row_index_end: The end index of the cell's row location in the current table.
* column_index_begin: The begin index of the cell's column location in the current table.
* column_index_end: The end index of the cell's column location in the current table.



* column_headers: An array of column-level cells, each applicable as a header to other cells in the same column as itself, of the current table. Each column header is defined as a collection of the following items:



* cell_id: The unique ID of the cell in the current table.
* location: The location of the cell in the input document as defined by its begin and end indexes.
* text: The textual contents of the cell from the input document without associated markup content.
* text_normalized: If you provide customization input, the normalized version of the cell text according to the customization; otherwise, the same value as text.
* row_index_begin: The begin index of the cell's row location in the current table.
* row_index_end: The end index of the cell's row location in the current table.
* column_index_begin: The begin index of the cell's column location in the current table.
* column_index_end: The end index of the cell's column location in the current table.



* body_cells: An array of cells that are not table header or column header or row header cells, of the current table with corresponding row and column header associations. Each body cell is defined as a collection of the following items:



* cell_id: The unique ID of the cell in the current table.
* location: The location of the cell in the input document as defined by its begin and end indexes.
* text: The textual contents of the cell from the input document without associated markup content.
* Consolidate tuples that pass the predicates according to the optional consolidation clause, and add the resulting tuples to the output.
* If the optional limit clause is present, limit the output to the specified number of tuples for each document.



The semantics of the from clause of an extract pattern statement are different from other forms of extract statements that do not have a pattern specification. If at least one of the views in the <from list> does not contain any tuples on a particular document, then the output of the extract statement is empty. This output is empty because the set of all combinations of tuples in the input views is empty.

In the special case of extract pattern statements, the from clause is a placeholder that declares the names of relations that are involved in the pattern specification. The semantics of the statement are driven only by the pattern specification. In particular, the output of the statement can be non-empty even when some of the input views are empty.





 Examples 

Example 1: Extracting phone numbers from a pre-defined view

This sample extract statement evaluates a regular expression for United States phone numbers across input text that is represented by the pre-defined view Document. The output is then restricted to the first three phone numbers that are identified per each document. Field names in the having clause refer to the aliases at the beginning of the extract statement.

create view PhoneNumbers as
extract
D.text as documentText,
regex /d{3}-d{3}-d{4}/ on D.text as phoneNumber
from Document D
having MatchesRegex(/d{3}./, phoneNumber)
limit 3;

Example 2: Extracting blocks of capitalized words

In this example, the extract statement identifies blocks of two to three capitalized words. In AQL, a block refers to a contiguous span of tokens, in this case two to three tokens. This example also uses a consolidation policy to exclude blocks that are contained within larger blocks from the output set of tuples.

create view CapitalizedWord as
extract
regex /[A-Z]/
with flags 'CANON_EQ'
on 1 token in D.text
as word
"transcript": "several tornadoes touch down as a line of severe thunderstorms swept through Colorado on Sunday "
},
{
"transcript": "several tornadoes touched down as a line of severe thunderstorms swept through Colorado on Sunday "
},
{
"transcript": "several tornadoes touch down as a line of severe thunderstorms swept through Colorado and Sunday "
}
],
"final": true
}
]
}
Show more





 Next steps 



* To try an example application that transcribes text from streaming audio input or from a file that you upload, see the [Speech to Text demo](https://www.ibm.com/demos/live/speech-to-text/sales/home).
* For more information about the service's interfaces and features, see [Service features](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-service-features).
* For more information about all methods of the service's interfaces, see the [API & SDK reference](https://cloud.ibm.com/apidocs/speech-to-text).
For more information about languages for which OCR v1 and OCR v2 are supported, see [Language support](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-language-support).

For a list of files types where you can apply OCR, see the [Supported file types](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionssupportedfiletypes) table.





 Enabling stemming for uncurated data IBM Cloud Pak for Data 

This feature is available from IBM Cloud Pak for Data deployments only. It was introduced with the 4.7.0 release.

You can configure Discovery to use stemming instead of lemmatization for normalization when you create a collection. This configuration is only occasionally useful when a collection contains data with many misspellings, missing accent marks, and grammatical errors.

Discovery normalizes words to enable faster recognition and matching of words and their various forms, such as plurals or alternative verb conjugations. By default, Discovery uses lemmatization to normalize words based on their meaning. Stemming normalizes words by using word stems only.

Lemmatization is more precise, but works best on curated data. If your data is not well curated, stemming might work better. The same word stem typically is detected whether or not a word is spelled correctly. However, lemmatization might not recognize a misspelled word or might misinterpret its meaning. As a result, the lemmatizer can add the wrong root word to represent the misspelled word in the index. A search against a stemmed version of a misspelled word is likely to return better results than a search against an incorrectly lemmatized word.

The following table shows examples of how some words are stemmed versus lemmatized.



Stemmer versus Lemmatizer comparison

 Surface form Lemmatized form Stemmed form 

 running run run 
 ran run ran 
 instructor instructor instruct 
 instruction instruction instruct 



As you can see from the examples, the lemmatizer captures the word meanings better than the stemmer. Both running and ran are recognized as different forms of the same root verb run.
(https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-corporaWordswordsResourceAmount).





 How does custom model upgrading work? 

When a new version of a previous-generation base model is released to improve the quality of speech recognition, you must upgrade any custom language and custom acoustic models that are based on that model to take advantage of the updates. When you upgrade a custom model, you do not need to upgrade its resources individually. The service upgrades the resources automatically. Custom model upgrading applies only to previous-generation models.



* For more information about upgrading a custom model, see [Upgrading custom models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-custom-upgrade).
* For more information about using an upgraded custom model for a speech recognition request, see [Using upgraded custom models for speech recognition](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-custom-upgrade-use).







 Can the Speech to Text service transcribe numbers as digits instead of strings? 

For US English, Japanese, and Spanish audio, you can use smart formatting to convert certain strings, such as digits and numbers, to more conventional representations. Smart formatting is beta functionality. For more information, see [Smart formatting](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-formattingsmart-formatting).
* ko-KR_Telephony



You do not need to upgrade custom models that are based on these models. For more information about all available next-generation models, see [Next-generation languages and models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-models-ng).





 25 May 2022 

New beta character_insertion_bias parameter for next-generation models
: All next-generation models now support a new beta parameter, character_insertion_bias, which is available with all speech recognition interfaces. By default, the service is optimized for each individual model to balance its recognition of candidate strings of different lengths. The model-specific bias is equivalent to 0.0. Each model's default bias is sufficient for most speech recognition requests.

However, certain use cases might benefit from favoring hypotheses with shorter or longer strings of characters. The parameter accepts values between -1.0 and 1.0 that represent a change from a model's default. Negative values instruct the service to favor shorter strings of characters. Positive values direct the service to favor longer strings of characters. For more information, see [Character insertion bias](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-parsinginsertion-bias).





 19 May 2022 

New Italian it-IT_Multimedia next-generation model
: The service now offers a next-generation multimedia model for Italian: it-IT_Multimedia. The new model is generally available. It does not support low latency, but it does support language model customization and grammars. For more information about all available next-generation models, see [Next-generation languages and models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-models-ng).

Updated Korean telephony and multimedia next-generation models
: The existing Korean next-generation models have been updated:



* The ko-KR_Telephony model has been updated for improved low-latency support for speech recognition.
<tmx version="1.4">
<header creationtool="" creationtoolversion=""
segtype="phrase" o-tmf="" adminlang="en"
srclang="en" datatype="PlainText" o-encoding="UTF-8" />
<body>
<tu>
<tuv xml:lang="en">
<seg>international business machines</seg>
</tuv>
<tuv xml:lang="fr">
<seg>International Business Machines</seg>
</tuv>
</tu>
<tu>
<tuv xml:lang="en">
<seg>patent</seg>
</tuv>
<tuv xml:lang="fr">
<seg>brevet</seg>
</tuv>
</tu>
</body>
</tmx>
Show more







 Parallel corpus customization 

Use a parallel corpus to provide additional translations for the base model to learn from. This helps to adapt the base model to a specific domain. How the resulting custom model translates text depends on the model's combined understanding of the parallel corpus and the base model.



* Encoding: UTF-8.
* Maximum length of translation pairs: 80 source words (longer input is ignored).
* Minimum number of translation pairs: 5000 for all corpora combined.
* Maximum number of translation pairs: 500,000 for all corpora combined.
* Maximum (cumulative) file size: 250 MB for all corpora combined.
* You can submit multiple parallel corpus files by repeating the parallel_corpus multipart form parameter.





 Parallel corpus example 

The following is a small piece of an English to French parallel corpus that was downloaded from the [MultiUN collection](http://opus.nlpl.eu/MultiUN.php) available on the OPUS open parallel corpus website.
Sorting 

Custom sorting MAY be implemented by any collection, but is likely to be far more useful for paginated collections where sort order impacts which resources fall on each page. It is OPTIONAL for any collection, paginated or not, to provide custom sorting.

[1]

Collections which provide custom sorting MUST do so via the sort query parameter. Depending on usefulness, a collection MAY support either single-field or multi-field sorting. The documentation for the collection should highlight any fields which are available for sorting, including first-level primitive fields and primitive fields of embedded resources. The . character MUST be used to delineate sub-resources and sub-fields.



  Single-field sorting 

Collections that support single-field sorting MUST allow the sort parameter to contain any one of the valid sort fields. For example, the following expression would sort accounts on company name (ascending):

GET /v2/accounts?sort=company_name

The client MUST also be able to prefix the field with a - character

[2], indicating a descending sort order. For example, the expression below would sort on the last name of the account owner, in descending order:

GET /v2/accounts?sort=-owner.last_name

If the - prefix is not provided, the sort MUST be done in ascending order.

An unrecognized or unsupported sort field SHOULD result in a 400 status code and appropriate error response model.





  Multi-field sorting 

Collections that support sorting on multiple fields MUST allow the sort parameter to contain a comma-separated sequence of fields (each, optionally, with a - prefix) in the same format as described above for single-field sorting. Sorts MUST be applied to the data set in the order that they are provided. For example, the expression below would sort accounts first on company name (ascending) and second on owner last name (descending):

GET /v2/accounts?sort=company_name,-owner.last_name

An unrecognized or unsupported sort field or a repeated sort field SHOULD result in a 400 status code and appropriate error response model.
* To select all text that includes a United Kingdom postal code formatted as A9 9AA, A99 9AA, AA99 9AA, A9A 9AA, or a9d AA9A 9AA (for example, M1 1AE, B33 8TH, DN55 1PT, W1A 0AX, CR2 6XH, and SW1A 2AA), specify:

[A-Z]{1,2}[1-9]?[A-Z]?s[0-9]{2,}|GIR 0AA
* To select all text that includes number plates for private and commercial vehicles in India formatted as AA-99-AA-9999, AA-99-AAA-9999 and, for states where the 0 is optional, AA-9-AA-9999 with separators that may be hyphens, spaces or not present (for example, TN-86-AF-1199, WB 06 F 5971, and DL4CAF4943), specify:

[A-Z]{2}(-| |)d{1,2}(-| |)[A-Z]{1,3}(-| |)d{4} (M1 1AE, B33 8TH, DN55 1PT, W1A 0AX, CR2 6XH, SW1A 2AA)









 Creating a sequence pattern 

Often, you must assess the text for patterns that provide context for the terms of interest. For example, you might want to distinguish references to military or law enforcement personnel from references to civilians. When designing an extractor, several methods are available to identify and extract information on the basis of patterns in text. These methods can be used in a single extractor, or you can combine extractors.

Before defining your own pattern to meet your needs, check to see if a provided extractor can give you the desired results.

To create a sequence pattern:



1.
],
"readings":
"ネコ"
] ,
"part_of_speech":"カスタム名詞"
}
Show more
With this custom dictionary, you can create rules with a single token. In this example, ibm発見 is tokenized as a single token, so it is not broken up into smaller units. { "tokenization_rules":

{
"text":"ibm発見",
"tokens":
"ibm発見"
],
"readings":
"ibm発見"
],
"part_of_speech":"カスタム名 詞"
},
...
] !
}
<-- <ul> --> * Tokenization occurs at both index and query time. * Tokens must be defined individually. * Do not insert spaces or the character in tokens and readings. Otherwise, they will not work. * A standard tokenization dictionary is used on all collections. If your collection is already indexed with that dictionary, you must reingest the documents in that collection after you upload a custom tokenization dictionary. * Uploading a tokenization dictionary is only available for private collections on Advanced and Premium plans. * Only one tokenization dictionary can be uploaded per collection; if a second tokenization dictionary is uploaded, it replaces the first. If that collection already contained documents, you must reingest them for the new custom tokenization dictionary to be applied. * The custom tokenization dictionary must be written in JSON, example file name: custom_tokenization_dictionary.json. * All custom tokenization dictionary terms must be lowercase. * The character limit for a custom tokenization dictionary is 1 million characters. * To disable tokenization, delete the tokenization dictionary and reingest your documents. * You cannot currently upload or delete a tokenization dictionary using the Discovery tooling; it must be done using the Discovery API. * Tokenization is performed on the query and multiple collection query methods. * Each tokenization dictionary is associated with a collection. When querying across multiple collections]] !
"disambiguation": {
"subtype":
"ProcessorManufacturer"
],
"name": "Advanced Micro Devices",
"dbpedia_resource": "http://dbpedia.org/resource/Advanced_Micro_Devices"
},
"count": 1
}
]
}
Show more







 17 November 2017 

Korean language support improved
: [Korean language support](https://cloud.ibm.com/docs/natural-language-understanding?topic=natural-language-understanding-language-supportkorean) has expanded to include the following features:



* Entities
* Keywords
* Semantic Roles







 30 July 2017 

Limit number of characters processed
: You can control cost by using the optional limit_text_characters parameter to limit the number of characters that are processed. For example,

{
"text": "The United States of America, commonly known as the United States (U.S.) or America, is a federal republic composed of 50 states, a federal district, five major self-governing territories, and various possessions. Forty-eight of the fifty states and the federal district are contiguous and located in North America between Canada and Mexico. The state of Alaska is in the northwest corner of North America, bordered by Canada to the east and across the Bering Strait from Russia to the west.",
"features": {
"entities": {}
},
"limit_text_characters": 50,
"return_analyzed_text": true
}



* Each character counts as one character, regardless of whether the character is a single-byte character or a multibyte character.
* For metering, one Natural Language Understanding Item continues to be one feature (also known as an enrichment) per one text unit. One text unit is 10,000 characters or less.
* For detailed pricing information, see [Natural Language Understanding](https://cloud.ibm.com/catalog/natural-language-understanding) in the IBM Cloud® Catalog.
Spanish (es) Advanced rules models, Built-in entities, Classifier (Document and Text), Custom entities, Dictionary, Document sentiment, Keywords, Machine Learning, Optical character recognition v1 (Installed), Optical character recognition v2 (Cloud-managed), Parts of speech, Phrase sentiment, Regular expressions, Smart Document Understanding, Stemmer, Table Understanding 
 Swedish (sv) Classifier (Document and Text), Custom entities, Dictionary, Optical character recognition v1, Parts of speech, Regular expressions, Smart Document Understanding, Stemmer, Table Understanding 



Optical character recognition (OCR) v2 was introduced in Cloud-managed service instances on 2 November 2022.



 English-only support 

The following features are currently supported in English only:



* [Document Retrieval for Contract project type](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-projectsdoc-retrieval-contracts)
IBM Cloud

[Patterns (beta)](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain-pattern)
The second action in the sequence translates the text from its original language into English.

In the premessage webhook configuration page, the following values are specified:



* URL: https://us-south.functions.appdomain.cloud/api/v1/web/e97d2516-5ce4-4fd9-9d05-acc3dd8ennn/default/translation_sequence
* Secret: none
* Header name: Content-Type
* Header value: application/json



The node.js code for the first web action in your sequence looks as follows:

let rp = require("request-promise");

function main(params) {
console.log(JSON.stringify(params))
if (params.payload.input.text !== '') {
const options = { method: 'POST',
url: 'https://api.us-south.language-translator.watson.cloud.ibm.com/instances/572b37be-09f4-4704-b693-3bc63869nnnn/v3/identify?version=2018-05-01',
auth: {
'username': 'apikey',
'password': 'nnn'
},
headers: {
"Content-Type":"text/plain"
},
body: [
params.payload.input.text
],
json: true,
};
return rp(options)
.then(res => {
//Set the language property of the incoming message to the language that was identified by Watson Language Translator.
params.payload.context.skills["actions skill"].user_defined["language"] = res.languages[0].language;
console.log(JSON.stringify(params))
return params;
})
}
else {
params.payload.context.skills["actions skill"].user_defined["language"] = 'none'
* [Previous-generation languages and models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-models)
* [Next-generation languages and models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-models-ng)







 What are the input audio formats? 

The service supports many audio formats (MIME types). Different formats support different sampling rates and other characteristics. By using a format that supports compression, you can maximize the amount of audio data that you can send with a request. For more information about the supported audio formats, see the following topics:



* [Audio terminology and characteristics](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-audio-terminology)
* [Supported audio formats](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-audio-formats)







 How much audio data can I submit to the service? 

The amount of audio that you can submit with a single speech recognition request depends on the interface that you are using:



* The WebSocket and synchronous HTTP interfaces accept a maximum of 100 MB of audio data.
* The asynchronous HTTP interface accepts a maximum of 1 GB of audio data.



For more information, see [Recognizing speech with the service](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-service-featuresfeatures-recognition).





 Can I transcribe speech from video files? 

You cannot transcribe speech from a multimedia file that contains both audio and video. To transcribe speech from a video file, you must separate the audio data from the video data. For more information, see [Transcribing speech from video files](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-audio-formatsaudio-formats-video).





 How can I improve transcription accuracy?