vania-janet/multiturn-rag-retrieval-data huggingface dataset:
IBM Cloud Schematics enables, to predictably [manage the resource lifecycle](https://cloud.ibm.com/docs/schematics?topic=schematics-manage-lifecycle) of your infrastructure by using Terraform. Drift occurs when the real-world state of your infrastructure differs from the state that is defined in your Terraform template configuration.

Terraform cannot detect drift in resources and attributes that are not managed or configured by using Terraform. For example, Terraform cannot not detect changes in a virtual machine that results from installing applications locally or by using configuration management tools like Chef or Ansible.





 Drift detection in IBM Cloud 

Drift detection for your Terraform automation workspaces is possible in IBM Cloud Schematics. You can use following three methods to check the drift detection.





 Drift detection using the UI 

You can initiate drift detection for workspaces from the Schematics Workspaces job page. It initiates a job to detect drift for the workspace and its deployed resources. During execution, the drift detection job is in progress, on completion it has a failure or success. To review the details of the drift job, you need to check the drift job log for the drift status.



 Viewing detect drift logs using the UI 

Use the following steps to view the drift job log.



1. From the [workspace dashboard](https://cloud.ibm.com/schematics/workspaces), select the workspace that you want check for drift.
2. Select and open your workspace.
3. Select the Actions drop down list.
4. Select Detect drift option to initiate the detect drift job.
5. During execution, the status shows in progress moving to a success or a job failure status on completion.
6. The drift status can be determined by reviewing the output of the job logon when the job is in a success state. A sample success job execution with detected drift is shown in the screen capture.

Review the success job log to identify the drift details.

2022/04/19 10:10:44 ----- Terraform DRIFT -----
For more information about checking the status of a grammar, see [Listing grammars for a custom language model](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-manageGrammarslistGrammars).





 Handling add grammar failures 

If its analysis of a grammar fails, the service sets the grammar's status to undetermined and includes an error field that describes the failure with the grammar's status. You can also use the GET /v1/customizations/{customization_id} method to check the status of the custom model. If addition of a grammar fails, the output includes an error message like the following:

{
. . .
"error": "{"code":500, "code_description":"Internal Server Error",
". . . Cannot compile grammar . . ."},
. . .
}

The status of the custom model is not affected by the error, but the grammar cannot be used with the model. You can address the problem by correcting the grammar file and repeating the request to add it to the custom model. Set the allow_overwrite parameter to true to overwrite the version of the grammar file that failed.

You can also delete the grammar from the custom model for the time being, and then correct and add the grammar file again in the future. For more information about deleting a grammar, see [Deleting a grammar from a custom language model](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-manageGrammarsdeleteGrammar).







 Validate the words from the grammar 

When you add a grammar file to a custom language model, the service parses the grammar to determine whether the grammar recognizes any words that are not already part of the service's base vocabulary. Such words are referred to as out-of-vocabulary (OOV) words. The service adds OOV words to the custom model's words resource. The purpose of the words resource is to define words that are not already present in the service's vocabulary.
For example, a container that is allowed to run in privileged mode is likely to be a more serious risk than a user who was mistakenly given list permissions for pods.

To help your team identify and respond to the most serious risks first, use KSPM tools, such as IBM Cloud Security and Compliance Center Workload Protection, and policies that can detect risks and categorize them by severity level.





 Don’t rely on KSPM alone 

KSPM is one part in a Kubernetes security strategy, but it isn't the only one. KSPM is not a substitute for runtime security, which helps detect active threats within your environment. Nor does KSPM address risks such as malware inside containers, which is a threat that a container image scanning can handle.

You must deploy a broad set of security tools for Kubernetes, such as IBM Cloud Security and Compliance Center Workload Protection. As part of a broader Kubernetes security strategy, KSPM allows teams to validate the security of Kubernetes configurations to find and remediate mistakes that can trigger a breach. By running continuous, automatic scans of Kubernetes configurations, admins can mitigate one of the most common areas of attack, human error. At the same time, you can automate compliance in even the most complex Kubernetes clusters.
Working with corpora and custom words for previous-generation models 

This information is specific to custom models that are based on previous-generation models. For information about corpora and custom words for custom models that are based on next-generation models, see [Working with corpora and custom words for next-generation models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-corporaWords-ng).

You can populate a custom language model with words by adding corpora or grammars to the model, or by adding custom words directly:



* Corpora - The recommended means of populating a custom language model with words is to add one or more corpora to the model. When you add a corpus, the service analyzes the file and automatically adds any new words that it finds to the custom model. Adding a corpus to a custom model allows the service to extract domain-specific words in context, which helps ensure better transcription results. For more information, see [Working with Corpora](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-corporaWordsworkingCorpora).
* Grammars - You can add grammars to a custom model to limit speech recognition to the words or phrases that are recognized by a grammar. When you add a grammar to a model, the service automatically adds any new words that it finds to the model, just as it does with corpora. For more information, see [Using grammars with custom language models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-grammars).
* Individual words - You can also add individual custom words to a model directly. The service adds the words to the model just as it does words that it discovers from corpora or grammars. When you add a word directly, you can specify multiple pronunciations and indicate how the word is to be displayed. You can also update existing words to modify or augment the definitions that were extracted from corpora or grammars.
An acronym is any word that consists of two or more consecutive uppercase letters. If the acronym contains one or more vowels, the service attempts to recognize the acronym as a series of individual characters and as a pronounced word.

For example, the acronym NASA can be read as four individual letters or pronounced as a word that sounds like nassa. The service checks for both during speech recognition. This enhances its ability to represent acronyms correctly in a transcript.









 Validating a words resource for next-generation models 

Especially when you add a corpus to a custom language model or add multiple custom words at one time, make sure to perform the following validation:



* Look for typographical and other errors in corpora. Especially when you add corpora, which can be large, mistakes are easily made. Make sure to review the corpus closely before adding it to the model.
* Look for typographical and other errors in custom words. Make sure to review closely custom words that you add directly to a model.
* Verify the sounds-like pronunciations. The service tries to generate sounds-like pronunciations for custom words for which none are specified. In most cases, these pronunciations are sufficient. But the service cannot generate a pronunciation for all words, so you must review the word's definition to ensure that it is complete and valid. Reviewing the pronunciations for accuracy is also recommended for words that have unusual spellings or are difficult to pronounce, and for acronyms and technical terms.



Typographical errors have the unintended consequence of modifying a custom model for nonexistent words, as do ill-formed HTML tags that are left in a corpus file.

To validate and, if necessary, correct a custom word for a custom model, use the following methods:



* List all of the words from a custom model by using the GET /v1/customizations/{customization_id}/words method or query an individual word with the GET /v1/customizations/{customization_id}/words/{word_name} method.
When you create a prompt, the text of the prompt must reasonably match the spoken audio. The service does its best to align the specified text with the audio, and it can often compensate for mismatches between the two. If the differences between the text and the audio are too pronounced, however, the service cannot create the prompt. The longer the prompt, the greater the chance for misalignment between its text and audio. Multiple shorter prompts are therefore preferable to a single long prompt.

There is always the chance that the service might fail to detect a mismatch or might not produce a prompt that completely satisfies your needs. Therefore, you must listen to and evaluate all prompts before using them in production. Evaluating a prompt also allows you to detect unusual words that the service might not pronounce correctly. And it lets you be certain that the service’s voice speaks the prompt exactly as you intend.

Always listen to and evaluate a prompt to determine its quality before using it in production. To evaluate a prompt, include only the single prompt in a speech synthesis request by using the following SSML extension, in this case for a prompt whose ID is goodbye:

<ibm:prompt id="goodbye"/>

You call the same methods to evaluate a custom prompt and to use it in an application. For example, the following request uses the POST /v1/synthesize method to synthesize the prompt:

IBM Cloud

curl -X POST -u "apikey:{apikey}" --header "Content-Type: application/json" --header "Accept: audio/wav" --data "{"text":"<ibm:prompt id='goodbye'/>"}" "{url}/v1/synthesize?customization_id=82f4809a-bf63-89a6-52ca-22731fe467ba&voice=en-US_AllisonV3Voice"

IBM Cloud Pak for Data
* Polish: [custom_stopwords_pl.json](https://github.com/watson-developer-cloud/doc-tutorial-downloads/tree/master/discovery-data/custom_stopwords_pl.json)



See [supported languages](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-language-support) for the list of the languages that are supported by Discovery.





 Defining query-time stop words 

To define stop words, complete the following steps:



1. Create a stop words file. The file must be a JSON file with the json file extension.

Follow these guidelines:



* Specify stop words in lowercase.
* In general, keep your list of stop words under 200 total words. The size limit is one million characters. However, if you specify too many terms, you might negatively affect search accuracy.



You can use the default English stop words list file, [custom_stopwords_en.json](https://github.com/watson-developer-cloud/doc-tutorial-downloads/tree/master/discovery-data/custom_stopwords_en.json), as a starting point when you build a custom stop word list in English.
2. From the navigation pane, open the Improve and customize page.
3. Expand Improve relevance from the Improvement tools pane.
4. Click Stopwords, and then click Upload stopwords for the collection.

Only one stop words list can be uploaded per collection. The stop words list that you upload augments the default stop words list for your collection; it does not replace the default list.
5. Click Done.



To disable a custom stop words file and revert to using the default stop words, delete the custom stop words file.
* For custom models that are based on next-generation models, [Using the display_as field](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-corporaWords-ngdisplayAs-ng)



* A source field that indicates how the word was added to the custom model's words resource.



* For custom models that are based on previous-generation models, the field includes the name of each corpus and grammar from which the service extracted the word. If you modified or added the word directly, the field includes the string user.
* For custom models that are based on next-generation models, this field shows only user for custom words that were added directly to the custom model. Words from corpora and grammars are not added to the words resource for custom models that are based on next-generation models.



* A count field that indicates the number of times the word is found across all corpora and grammars.



* For custom models that are based on previous-generation models, for example, if the word occurs five times in one corpus and seven times in another, its count is 12. If you add a custom word to a model before it is added by any corpora or grammars, the count begins at 1. If the word is added from a corpus or grammar first and later modified, the count reflects only the number of times it is found in corpora and grammars.
* For custom models that are based on next-generation models, the count field for any word is always 1.





If the service discovers one or more problems with a custom word's definition, the output includes an error field. The field provides an array that lists each problem element from the definition and a message that describes the problem.

An error can occur, for example, if you add a custom word with an invalid sounds_like field, one that violates one of the rules for adding a pronunciation. You cannot train a custom model whose words resource includes a word with an error. You must correct or delete the word before you can train the model.



 List all custom words example 

The following example lists all of the words, regardless of type, from the custom model with the specified customization ID.
* Spanish: [stopwords_es.txt](https://github.com/apache/lucene/blob/main/lucene/analysis/common/src/resources/org/apache/lucene/analysis/snowball/spanish_stop.txt)
* Swedish: [stopwords_sv.txt](https://github.com/apache/lucene/blob/main/lucene/analysis/common/src/resources/org/apache/lucene/analysis/snowball/swedish_stop.txt)
* Turkish: [stopwords_tr.txt](https://github.com/apache/lucene/blob/main/lucene/analysis/common/src/resources/org/apache/lucene/analysis/tr/stopwords.txt)



These default stop words are documented in TXT format, but if you want to augment the list and submit it for use by Discovery, you must submit a JSON file. To see an example of the syntax of stop words list file, see the custom English stop words list file.

For the remaining supported languages, no default stop words are used. You can specify a stop words list to use at query time for these languages. The list that you submit is not used when data is ingested.

Examples of stop word lists that you might want to apply at query time include:



* Japanese: [custom_stopwords_ja.json](https://github.com/watson-developer-cloud/doc-tutorial-downloads/tree/master/discovery-data/custom_stopwords_ja.json)
* Polish: [custom_stopwords_pl.json](https://github.com/watson-developer-cloud/doc-tutorial-downloads/tree/master/discovery-data/custom_stopwords_pl.json)



See [supported languages](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-language-support) for the list of the languages that are supported by Discovery.





 Defining query-time stop words 

To define stop words, complete the following steps:
: Object that describes the analyzer to be used or an object with the following fields:

Name
: Name of the analyzer. Valid values are standard, email, keyword, simple, whitespace, classic, and perfield.

Stopwords (optional)
: An array of stop words. Stop words are words that must not be indexed. If this array is specified, it overrides the default list of stop words. The default list of stop words depends on the analyzer. The standard analyzer includes the following list of stop words: a, an, and, are, as, at, be, but, by, for, if, in, into, is, it, no, not, of, on, or, such, that, the, their, then, there, these, they, this, to, was, will, and with.

Default (for the per field analyzer)
: Default language to use if no language is specified for the field.

Fields (for the per field analyzer)
: An object that specifies which language to use to analyze each field of the index. Field names in the object correspond to field names in the index, that is, the first parameter of the index function. The values of the fields are the languages to be used, for example english.

Index
: Function that handles the indexing.

Filters (optional, disallowed when partitioned is true)
: Filter functions.

Function name (one for each function)
: Function definition.

Validate_doc_update (optional, disallowed when partitioned is true)
: Update validation function.



 The options.partitioned field 

This field sets whether the created index is a partitioned or global index.

This field includes the following values:



Table 1. Values for the options.partitioned field

 Value Description Notes 

 true Create the index as partitioned. Can be used only in a partitioned database. 
 false Create the index as global. Can be used in any database. 



The default follows the partitioned setting for the database:



Table 2. Partition settings

 Database is partitioned? Default partitioned value Allowed values
New strict parameter for custom model training now available
: The training methods of the customization interface now include a strict query parameter that indicates whether training is to proceed if a custom model contains a mix of valid and invalid resources. By default, training fails if a custom model contains one or more invalid resources. Set the parameter to false to allow training to proceed as long as the model contains at least one valid resource. The service excludes invalid resources from the training.



* For more information about using the strict parameter with the POST /v1/customizations/{customization_id}/train method, see [Train the custom language model](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-languageCreatetrainModel-language) and [Training failures](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-languageCreatefailedTraining-language).
* For more information about using the strict parameter with the POST /v1/acoustic_customizations/{customization_id}/train method, see [Train the custom acoustic model](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-acoustictrainModel-acoustic) and [Training failures](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-acousticfailedTraining-acoustic).



New limits on maximum number of out-of-vocabulary words for custom language models
: You can now add a maximum of 90 thousand out-of-vocabulary (OOV) words to the words resource of a custom language model. The previous maximum was 30 thousand OOV words. This figure includes OOV words from all sources (corpora, grammars, and individual custom words that you add directly). You can add a maximum of 10 million total words to a custom model from all sources. For more information, see [How much data do I need?](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-corporaWordswordsResourceAmount).





 3 April 2019
Typographical errors in a corpus fole (or in custom words or a grammar file) have the unintended consequence of adding new words to a model's words resource, as do ill-formed HTML tags that are left in a corpus file.
* Verify the sounds-like pronunciations. The service tries to generate sounds-like pronunciations for OOV words automatically. In most cases, these pronunciations are sufficient. But the service cannot generate a pronunciation for all words, so you must review the word's definition to ensure that it is complete and valid. Reviewing the pronunciations for accuracy is also recommended for words that have unusual spellings or are difficult to pronounce, and for acronyms and technical terms.



To validate and, if necessary, correct a word for a custom model, regardless of how it was added to the words resource, use the following methods:



* List all of the words from a custom model by using the GET /v1/customizations/{customization_id}/words method or query an individual word with the GET /v1/customizations/{customization_id}/words/{word_name} method. For more information, see [Listing custom words from a custom language model](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-manageWordslistWords).
* Modify words in a custom model to correct errors or to add sounds-like or display-as values by using the POST /v1/customizations/{customization_id}/words or PUT /v1/customizations/{customization_id}/words/{word_name} method. For more information, see [Working with custom words for previous-generation models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-corporaWordsworkingWords).
* Delete extraneous words that are introduced in error (for example, by typographical or other mistakes in a corpus) by using the DELETE /v1/customizations/{customization_id}/words/{word_name} method.
* UI
* CLI
* API







 Detecting drift in workspaces 

IBM Cloud® Schematics enables users to deploy, manage, and manipulate infrastructure resources with Terraform based workspaces using declarative Infrastructure as Code (IaC) concepts. However, when a Terraform config is deployed and resources are created, it does not mean that the resources stay as declared by the config. Any change in the infrastructure state is called drift. It occurs when your deployed infrastructure differs from the wanted (declarative) state defined by your template configuration.

Drift can occur for many reasons. The most frequent cause is changes made to your infrastructure manually outside of Terraform automation. The Terraform state file of your deployed workspace is then no longer synchronized with your deployed infrastructure resources, and the workspace is said to be in drift.

Drift can happen for many reasons within the context of your configuration:



* Adding, or removing resources from the Template configuration without applying the changes.
* Changing template resource definitions.
* External to your template configuration, drift occurs when changes are made manually. For example, from a command line operation on a cloud resource, or change through the cloud console.
* Can occur through other automation tools.





 Example drift scenario 

A VSI instance is provisioned by using IBM Cloud Schematics and a configuration template. A DevOps cloud user can modify the provisioned VSI configuration by logging in to the Cloud console and modifying the boot volume of an instance or adding an Ethernet interface. These changes result in your infrastructure deployment in drift.

IBM Cloud Schematics enables, to predictably [manage the resource lifecycle](https://cloud.ibm.com/docs/schematics?topic=schematics-manage-lifecycle) of your infrastructure by using Terraform. Drift occurs when the real-world state of your infrastructure differs from the state that is defined in your Terraform template configuration.

Terraform cannot detect drift in resources and attributes that are not managed or configured by using Terraform. For example, Terraform cannot not detect changes in a virtual machine that results from installing applications locally or by using configuration management tools like Chef or Ansible.





 Drift detection in IBM Cloud