vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Target: Config terrafrom-1234567890-a12a for Project example-email-notification-project
Source: pipelinerun-123ab1ab-1a1a-1a11-1a12-ac1a1a12a12a
Triggered by: IBMid-12345ABCD1
Action URL: https://cloud.ibm.com/devops/pipelines/example
Date: Fri April 10 2023 09:15:08 GMT-0500 (Eastern Standard Time)

To receive detailed information about an event notification in your email, select the Add notification payload option when you create an Event Notifications subscription. Your email displays the [notification payload details](https://cloud.ibm.com/docs/secure-enterprise?topic=secure-enterprise-event-notifications-eventsevent-notifications-payload) that are associated with the event.





 Webhooks 

You can configure a webhook destination so that an incoming notification can be consumed programmatically by an app or service. For more information about setting up webhooks, check out the [Event Notifications documentation](https://cloud.ibm.com/docs/event-notifications?topic=event-notifications-en-destinations-webhook).







 Notification payload details 

Successful events that are generated by projects contain various fields that help you to identify the source and details of an event.

Event notifications from projects contain only metadata properties, such as names or identifiers of resources. Sensitive data, for example API keys or passwords, are not included in generated events.

The properties that are sent to Event Notifications vary depending on the event type. For example, if an check.pipeline.succeeded event takes place, projects sends a notification payload to Event Notifications that is similar to the following example.

{
"_id": "a7110265-88e5-4fda-8686-ae4d5bb3f86a",
"event": "check.pipeline.succeeded",
Integrating with IBM Cloud Monitoring 

You can send alerts to IBM Cloud Monitoring and manage them through the Events view section.



 Configuring an alert 

When you [configure an alert](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-alerts) you can have that alert sent to IBM Cloud Monitoring.



1. When configuring your alert, click ![Sysdig icon](https://cloud.ibm.com/docs-content/v1/content/f61dbc5d84bf0c89e477a05cab04c72f429e1b7a/log-analysis/images/sysdig.png).
2. Select if you want the alert to be sent when the condition exists (Presence) or does not exist (Absence).
3. Indicate the logging criteria when an alert should be sent. For example, when 100 lines matching in the view are logged in an hour. A graph will help you determine the number of log lines matching your specified criteria.
4. Select if the alert should be sent at the end of the selected period or immediately when the number of lines are logged.
5. Optionally you can specify a Custom schedule with alerting limited to a specified timezone, days of the week, or timeframe. To configure a Custom schedule:



1. Select on for Custom schedule.
2. Select the Timezone for the log entries.
3. Select the days of the week when alerts should be generated.
4. Optionally specify a time range for the selected days. A graph will help you determine the number of log entries for the timezone and time range.



6. Configure the Monitoring instance details.

Specify the API key. Set this field to the API token. For more information on how to get the token, see [Working with tokens](https://cloud.ibm.com/docs/monitoring?topic=monitoring-api_monitoring_token).
Valid options are:



* Hourly notifications
* Daily notifications at midnight UTC



Notifications are sent until the index rate reports a value that is below the thresholds that you have configured.

Individual recipients may mute alert emails.

You can also configure a custom schedule to set the days and hours that you want alerts to be send if the thresholds are exceeded.







 Configuring index rate alerts using the API 

You can configure index rates and what alerts are triggered using the [API.](https://cloud.ibm.com/docs/%7BDomainName%7D/apidocs/activity-trackerget-index-rate-alert)



 Creating or updating an index rate alert 

Use the PUT REST API command to create or update your index rate alerts. If no index rate alert is configured, the first PUT will create the index rate alert.

Run the following command:

curl -X PUT https://<ENDPOINT>/v1/config/index-rate -H 'content-type: application/json' -H 'servicekey: <SERVICE_KEY>' -d '{"max_lines":<MAX_LINES>,"max_z_score":<MAX_Z_SCORE>,"enabled":<ENABLED>,"channels":<CHANNELS>,"threshold_alert":"<THRESHOLD_ALERT>","frequency":"<FREQUENCY>"}'

Where:

ENDPOINT
: Is the [endpoint]/docs/activity-tracker?topic=activity-tracker-endpoints) for your IBM Cloud Activity Tracker instance.

SERVICE_KEY
: Is the [service key](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-service_keys) for your IBM Cloud Activity Tracker instance.

MAX_LINES
: Is the number of lines required in order to trigger the alert.

MAX_Z_SCORE
: Is the number of standard deviations above the 30 day average lines in order to trigger the alert.

ENABLED
: Specify if the index rate configuration is enabled (true) or disabled (false).

CHANNELS
Working with notification channels 

Configure notification channels to be notified when an alert is triggered. You manage notification channels through the Settings panel in the web UI.



 Configuring a notification channel 

Complete the following steps to add a notification channel:



1. Launch the web UI. For more information on how to launch the Web UI, see [Navigating to the Web UI](https://cloud.ibm.com/docs/monitoring?topic=monitoring-launchlaunch).
2. Click the user icon. This is the icon with the initials of the logged on user. Then click Settings.
3. Select Notification Channels.
4. Click Add Notification Channel.
5. Configure a notification channel:



1. Select the type of notification, for example, Webhook.
2. Enter the name of the channel.
3. Enable the Notify when Resolved condition to receive a notification when the alert is manually resolved by a user.
4. Enable the Notify when Acknowledged field to receive a notification when the alert is manually acknowledged by the user.
5. Configure the notification according to the selected notification type:



* For an Email notification channel, add the list of recipients, separated by commas.
* For a Slack notification channel, add the URL of the Slack channel.
* For Cloud Functions notification channel, specify the channel URL when you set up a IBM Cloud Function Channel. For more information, see [Configure IBM Cloud Function Channel](https://docs.sysdig.com/en/configure-ibm-cloud-functions-channel.html).
* For a Webhook notification channel, add the Webhook URL. Note: When an alert is triggered, the notification is sent as a POST in JSON format to your webhook endpoint. For more information, see [Configuring a Webhook channel](https://docs.sysdig.com/en/configure-a-webhook-channel.html).
When you set up alerts, make sure to allow your cluster enough time to self-heal. Because Kubernetes has self healing capabilities, configure your alerts only for the issues that arise over time. By observing your cluster over time, you can learn which issues Kubernetes can resolve itself and which issues require alerts to avoid downtime.

On 15 June 2022, the naming convention for IBM Cloud® Monitoring alerts is changing to a Prometheus compatible format. For more information, see the [Sysdig release notes](https://docs.sysdig.com/en/docs/release-notes/enhanced-metric-store/new-features-and-enhancements), [Mapping Legacy Sysdig Kubernetes Metrics with Prometheus Metrics](https://docs.sysdig.com/en/docs/sysdig-monitor/using-monitor/metrics/metrics-library/metrics-and-labels-mapping/mapping-legacy-sysdig-kubernetes-metrics-with-prometheus-metrics/), and [Mapping Classic Metrics with PromQL](https://docs.sysdig.com/en/docs/sysdig-monitor/using-monitor/metrics/metrics-library/metrics-and-labels-mapping/mapping-classic-metrics-with-promql-metrics/).

Depending on the size of your cluster, consider setting up alerts on the following levels:



* [Apps](https://cloud.ibm.com/docs/containers?topic=containers-health-monitorapp-level-alerts)
* [Worker nodes](https://cloud.ibm.com/docs/containers?topic=containers-health-monitorworker-node-level-alerts)
* [Cluster](https://cloud.ibm.com/docs/containers?topic=containers-health-monitorcluster-level-alerts)
* [Zone](https://cloud.ibm.com/docs/containers?topic=containers-health-monitorzone-level-alerts)
RAID alert "Spam"

Change the "global" section of the default config (/opt/Broadcom/mrmonitor/MegaMonitor/config-current.xml):

<global>

<severity level="FATAL">

<do-systemlog/>

<do-email/>

</severity>
<severity level="CRITICAL">

<do-email/>

<do-systemlog/>

</severity>

<severity level="WARNING">

<do-email/>

<do-systemlog/>

</severity>

<severity level="INFO">
<do-systemlog/>
</severity>

</global>

Show more

to read like this:

<global>

<severity level="FATAL">

<do-systemlog/>

<do-email/>

</severity>

<severity level="CRITICAL">

<do-email/>

<do-systemlog/>

</severity>

<severity level="WARNING">

<do-systemlog/>

</severity>

<severity level="INFO">
<do-systemlog/>

</severity>

</global>

Show more

Remove the "do-email" tag for level "WARNING". Alternatively, change the security level to "INFO".



 Common drive errors 

The most common driver errors are smart errors, hardware errors, and medium errors. You see these errors if a drive is failing. So, you need to replace the drive as soon as possible.

Although not unusual, aborted commands are another common error. But, if aborted commands increase in number (such as 100), open a support case.

Link errors can indicate that a cable might need reseated or replaced.



 Support case information 

The following information is needed when you open support case.



 Adaptec RAID cards
If not specified, the region logged into or targeted will be used.

--force
: Supresses the prompt and deletes the alert from the instance that is specified by --name.

--team <TEAM_NAME>
: The name of the IBM Cloud Monitoring team to be used for authorization. If no team is specified, the default team will be used.

--help | -h
: List options available for the command.





 Examples 

The following are examples using the ibmcloud monitoring alert delete command.

Delete an alert with ID 1234567 for the IBM Cloud Monitoring abc instance.

ibmcloud monitoring alert delete --name "IBM Cloud Monitoring abc" --id 1234567 --file alert.json

Delete an alert with ID 1234567 for the IBM Cloud Monitoring abc instance in the us-south region.

ibmcloud monitoring alert delete --name "IBM Cloud Monitoring abc" --id 1234567 --file alert.json --region us-south







 ibmcloud monitoring service-instances 

Use this command to list IBM Cloud Monitoring service instances.

ibmcloud monitoring service-instances [--region REGION] [--all-regions] [--g RESOURCE_GROUP] [--all-resource-groups] [--quiet] [--output FORMAT]



 Command options 

--region <REGION> | -r <REGION>
: Name of the region, for example, us-south or eu-gb. If not specified, the region logged into or targeted will be used.

--all-regions
: Services hosted across all regions.

-g <RESOURCE_GROUP>
: Resource Group associated with the hosted service.

--all-resource-groups
: Services hosted across all resource groups.

--quiet | -q
: Supresses verbose output.

--output <FORMAT>
: A comma-separated list of output preferences enclosed in double-quotes ("). If only a single preference is specified, the double-quotes can be omitted. Supported options are WIDE and JSON.
By default, an event has a state:



* Active: This state indicates that the circumstances that triggered the event remain in place, for example, a node continues to be down.
* OK: This state indicates that the situation is back to normal, for example, a node is up and running.



You manage events in the Events section of the web UI.



* You can view alert events through the Alert Events tab.
* You can view infrastructure-based events through the Custom events tab.
* You can view custom events through the Custom events tab.
* You can send custom events to any of your teams by using the [API token for that team](https://cloud.ibm.com/docs/monitoring?topic=monitoring-api_tokenapi_token). For more information, see [Custom events)](https://docs.sysdig.com/en/events.html).
* You can set the event as Resolved to notify other users that the issue has been addressed instead of waiting for the status to be set to OK.
<-- </section "id="section-update-certificate-alert-examples" "> --><-- </section "id="section-update-certificate-alert" "> --><-- <section "id="section-update-glb-healthcheck-alert" "> --> ibmcloud cis alert-policy glb-healthcheck-alert-update Update an alert policy for changes in health status for global load balancer, pools, and origins. ibmcloud cis alert-policy glb-healthcheck-alert-update POLICY_ID --name NAME] --emails EMAILS | --webhooks WEBHOOKS] --enabled (true | false)] --pools POOLS] --include-future-pools (true | false)] --health-status-trigger (healthy | unhealthy | either)] --event-source-trigger (pool | origin | either)] --description DESCRIPTION] -i, --instance INSTANCE] --output FORMAT] ! ! ! ! ! !
<-- <section "id="section-opt-update-glb-healthcheck-alert" "> --> Command options POLICY_ID: The ID of alert policy. Required.--name: The name of the alert policy.--description: The description for the alert policy.--emails: The email addresses for dispatching an alert notification. For example: --emails test1@cn.ibm.com,test2@cn.ibm.com--webhooks: The webhook ID that for dispatching an alert notification. For example: --webhook webhookID1,webhookID2--enabled: Whether or not the alert policy is enabled.--pools: The IDs of origin pool, if set to all, the all pool IDs will be used.--include-future-pools: Whether or not include the future pools. (default "false")--health-status-trigger: The trigger condition to fire the notification. Valid values: "healthy", "unhealthy", "either".
* Danger word detected: The customer uses words or phrases that match the Connect to agent step in the Trigger word detected action.
* Profanity detected: The customer repeatedly used words or phrases that match the Show warning step in the Trigger word detected action.



For more information about the Trigger word detected action, see [Detecting trigger words](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-trigger-words).

The Fallback action defines five conditional steps, one for each possible value of the Fallback reason variable. Each step sends a message to the customer, based on the error condition, and then uses the Connect to agent feature to transfer the conversation to a live agent. (For more information about this feature, see [Connecting to a live agent](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-human-agent).) You can modify these steps if you want to handle an error condition in a different way.
A preset is an alert template that you can attach to any number of views.

To reuse an alert configuration with different views and enforce notification channels across users that analyze data through that instance, configure alert presets.

When you send a notification, you can include log data as part of the notification.

Customize the data that is included in a notification for situations where the receiver of the notification does not have access to the log data.





 Define the archive strategy 

You might have different requirements that require archiving your data:



* Backup requirements to keep data for a period of time
* Data access requirements so that you can query the data after it is not available for search through the web UI
* Disaster recovery requirements
* Compliance requirements



logging as a service does not backup your data.

There are 2 types of data that you should consider archiving:



* Log data

By default, archiving of log data is not enabled for any logging instance.

When you enable archiving of your log data, notice that you are responsible for checking that your archived files are not corrupted, and for the maintenance of your archived files.
* Web UI resource definitions such as parsing templates, exclusion rules, views, screens, and dashboards.



You will want to archive your logging resource definitions and your log data.



 Backup the resource configurations of your logging instance 

In the logging web UI, you can define custom views, dashboards, parsing templates, screens, and exclusion rules that you can use to view and analyze data.

To reuse resource definitions that you define in your logging instance, you can export these resources from an IBM Log Analysis instance as a JSON file. Then, you can import the definitions into other logging instances. For example, you can reuse your logging resources across different environments for your stage, pre-production, and production logging instances. [Learn more](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-reuse_resource_definitions).

Backup logging resource definitions into a version control system such as a git repository where you can control access to the archived files and manage versions.





 Archive log data to a COS bucket
Removing alerts through the UI 

You can detach alerts graphically by detaching an alert from a view or by deleting a preset.



  Detaching all alerts from a view 

Complete the following steps to detach all the alerts that are configured for a view:



1.  [Navigate to the web UI through the IBM Cloud UI](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-launch).
2.  Identify the view with the alert.
3.  Select Detach alert.
4.  Confirm that you want to remove all the alerts. Click Detach.







  Detaching 1 alert from a view 

Complete the following steps to detach 1 alert that is configured for a view:



1.  [Navigate to the web UI through the IBM Cloud UI](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-launch).
2.  Identify the view with the alert.
3.  Select Edit alert.
4.  Select the alerdefinition that you want to delete.
5.  Click Delete Alert Channel.







  Deleting a preset (alert template) 

When you delete a preset, any alerts that are defined by using this preset are automatically deleted.

For more inform ation on how to delete a preset, see [Delete a preset](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-preset_uipreset_ui_delete).
Make sure to select a database in the same region as the monitoring instance.

You see output like the following:

Retrieving instances for all database types in all resource groups in all locations under IBM as …
OK
Name Location State
Databases for PostgreSQL-76 us-south inactive
testelastic eu-gb active
Databases for MySQL-9j us-south active

Now, use the name of your database to create the alert by using a command like:

curl --request POST --url https://<region>.monitoring.cloud.ibm.com/api/alerts --header "Authorization: Bearer $AUTH_TOKEN" --header 'Content-Type: application/json' --header "IBMInstanceID: $GUID" --data-raw '{
"alert":
{
"type": "MANUAL",
"name": "Disk Alert",
"description": "",
"enabled": true,
"severity": 1,
"timespan": 60000000,
"notificationChannelIds": [
<id_from_previous_step>
],
"filter": "ibm_service_instance_name in ("<db_instance_name_from_previous_step>")",
"condition": "max(max(ibm_databases_for_elasticsearch_disk_used_percent)) > 0.9"
}
}'
Show more

This command takes the max disk utilization of any member available, regardless of the number of members.





 Step 4: Check that your alert is created 

An alert is created whenever you reach 90% of disk size. You receive the alert to the same email that you created in the notification channel. You can also use the following command to check current active alerts:

ibmcloud monitoring alert list --name <monitoring instance name>

You now receive an alert whenever your Databases for Elasticsearch instance disk utilization exceeds 90%, so you can act before the disk is too full.





 Next Steps
For more information about configuring notifications, see [Working with notification channels](https://cloud.ibm.com/docs/monitoring?topic=monitoring-notificationsnotifications).
2. If you return to one of the dashboards that you created, you can click on the action menu in the upper right corner for any panel, and then click Create Alert from the drop-down menu.

Zoom

![IBM Cloud Monitoring alerts](https://cloud.ibm.com/docs-content/v1/content/c5288bed34c3820e3a5251d820ee91adcd2591a3/blockchain/images/configure-sysdig-alert.png)

Figure 4. IBM Cloud Monitoring alerts

After you configure the alert based on your custom triggers, scroll down to associate the notification channel that you configured above and select your notification preferences. Optionally, you can also enable IBM Cloud Monitoring capture which generates a trace file that you can use to analyze what happens in a host during a selected time frame.

For more information about the various options when configuring alerts in IBM Cloud Monitoring. see [IBM Cloud Monitoring alerts](https://cloud.ibm.com/docs/monitoring?topic=monitoring-alerts).



As a general guideline, we recommend that you configure alerts for when your node CPU, memory, and storage usage exceeds 80% so that you can take appropriate action. Feel free to adjust the alert settings according to your notification preferences.





 Summary 

This tutorial has provided some basic steps for getting started using IBM Cloud Monitoring to monitor your blockchain node resource allocation. You now have three dashboards that you can use to monitor the resource allocation of your peer, CA, and ordering nodes in your Kubernetes cluster. With a graduated tier purchase of the IBM Cloud Monitoring service, you can edit your charts and add more segmentations to further scope the data. You've also configured alerts for when nodes approach exceeding their allocated resources.
Optionally specify a time range for the selected days. A graph will help you determine the number of log entries for the timezone and time range.



6. Specify the Method & URL for the webhook.
7. If your webhook requires specific header information, specify that in Headers. Multiple headers can be added by clicking Add.

Headers must be specified in all lower-case characters.
8. In Body, specify, in JSON format, the information to be passed to the webhook. You can confirm if your JSON is correct by clicking Validate JSON. The following tokens can be specified in your JSON to access specific information about the view or matched lines that triggered the alert.



Table 1. JSON tokens

 Token Description 

 {{ name }} View name 
 {{ matches }} Number of matched lines 
 {{ lines }} Raw output of matched lines 
 {{ level }} Severity level 
 {{ url }} View URL, with first matched line when possible 
 {{ query }} View query 
 {{ app }} Apps from the first matched line 
 {{ host }} Hosts from the first matched line 
 {{ tag }} Tags from the first matched line 
 {{ line }} Text from the first matched line 
 {{ line_objects }} Array of line objects 
 {{ first_line_object }} First matched line object 



You can use dot or bracket notation to access array indexes and object properties. For example, {{ line_objects[0] }} or {{ first_line_object._line }}.
9. Click Save Alert.

The alert definition must be saved for alerts to be sent to your webhook.







 How to add additional fields to the webhook body 

If you need additional fields to be included in your webhook body and the JSON tokens that are provided are not enough, try the following:



1. Set the line template of the view where you configure a webhook notification so that it includes the additional data that you need in your webhook body.