vania-janet/multiturn-rag-retrieval-data huggingface dataset:
* The following example shows the format of data payload for sending notifications.

{
"data": {
"createTimestamp": 1557282940339,
"severity": "LOW",
"shortDescription": "examplestring"
}
}
* The following example shows the format of FCM message body to send notification to FCM devices.

{"message":{
"data":{
"alert": "examlestring", "delay_while_idle":true,"time_to_live":2,"collapse_key":"testCollapseKey","notification":{"title":"Match update","body":"Arsenal goal in added time, score is now 3-0"},"data":{"alert":"Notification alert message","url":"https","payload":{"mydevicearra":"cc75e4a6-edd8-3bec-a7c3-dfca6572a03b"]}}
}
}
}
* The following example shows the format of Chrome message body to send notification to FCM devices.

{"title":"Hello Chrome", "en_data":{"alert":"Hello Chrome Notification","title":"Chrome New Title","iconUrl":"https://","timeToLive":100,"payload":{"key":"value"}}}
* The following example shows the format of Firefox message body to send notification to FCM devices.

'{"title":"Hello Firefox", "en_data":{"alert":"Hello firefox Notification","title":"firefox New Title","iconUrl":"https://","timeToLive":100,"payload":{"key":"value"}}}'
* The following example shows the format of apns/chrome/fireox custom headers.

{
"test": "test header",
Configuring usage quota alerts 

By default the account owner is configured to receive alert emails. You can configure your systems to send email alerts, Slack notifications, or both.

To configure notifications when a quota threshold is exceeded, do the following:



1. Under Alert recipients select the desired recipient email addresses. There must be at least one email address configured if Slack notifcations are not configured.
2. If desired, select one or more Slack recipients to receive notifications. If you configure a Slack recipient, you can delete all email recipients if desired.
The following example shows possible output from the previous command.

action /whisk.system/alarms/alarm: Fire trigger when alarm occurs
(params: cron trigger_payload)

The /whisk.system/alarms/alarm feed takes two parameters:



* cron: A crontab specification of when to fire the trigger.
* trigger_payload: The payload parameter value to set in each trigger event.



3. Create a trigger that fires every minute.

ibmcloud fn trigger create everyOneMinute --feed /whisk.system/alarms/alarm -p cron "* * * * " -p trigger_payload "{"name":"Mork", "place":"Ork"}"

The following example shows possible output from the previous command.

ok: created trigger feed everyOneMinute
4. Create an app. Example hello.js:

function main(params) {
return {payload: 'Hello, ' + params.name + ' from ' + params.place};
}
5. Create an action.

ibmcloud fn action create hello hello.js
6. Create a rule that invokes the hello action every time the everyOneMinute trigger fires.

ibmcloud fn rule create myRule everyOneMinute hello

The following example shows possible output from the previous command.

ok: created rule myRule
7. Check that the action is being invoked by polling for activation logs.

ibmcloud fn activation poll

You can see that the activations occur every minute for the trigger, the rule, and the action. The action receives the parameters {"name":"Mork", "place":"Ork"} on every invocation.
Sending email alerts by using IBM Cloud Event Notifications 

In the IBM Cloud Monitoring service, you can configure single alerts and multi-condition alerts to notify support staff by email about problems that might require attention. Alerts can generate notifications to the IBM Cloud Event Notifications service to be handled as configured in that service.

[IBM Cloud Event Notifications is available in a limited number of regions](https://cloud.ibm.com/docs/event-notifications?topic=event-notifications-en-regions-endpoints). Your IBM Cloud Monitoring and IBM Cloud Event Notifications instances must be in the same region to communicate with one another. Because of this, support for integration with IBM Cloud Event Notifications is limited to the regions where IBM Cloud Event Notifications is supported and where IBM Cloud Monitoring is also installed.

For a step-by-step tutorial, see [Sending email notifications to IBM Cloud Event Notifications.](https://cloud.ibm.com/docs/monitoring?topic=monitoring-tutorial-en)

To configure 1 instance of the IBM Cloud Monitoring service to send notifications to IBM Cloud Event Notifications to be sent to recipients by email, do the following:



 Step 1. Provision an IBM Cloud Event Notifications instance 

Provision an [IBM Cloud Event Notifications instance.](https://cloud.ibm.com/docs/event-notifications?topic=event-notifications-en-create-en-instance)

The IBM Cloud Event Notifications instance must be provisioned in the same region as the Monitoring instance.

The number of events and filters that are available with IBM Cloud Event Notifications depends on the pricing plan selected. Review the limitations statement when creating your IBM Cloud Event Notifications instance for [plan limitations](https://cloud.ibm.com/catalog/services/event-notifications).





 Step 2. Define authorizations 

Configure an authorization that grants IBM Cloud Monitoring access to IBM Cloud Event Notifications. Choose 1 of the following options:
"123456ab7d8e9f0g12h2j34l5mn6op78"
]
}
}'

Where:



* -d is the array of attributes required to create the alert.



* name is the name of the alert.
* description (optional) is the description of the alert.
* enabled is the state of the alert (one of true, false).
* alert_type is the type of the alert (one of dos_attack_l7, load_balancing_pool_enablement_alert, clickhouse_alert_fw_anomaly, clickhouse_alert_fw_ent_anomaly, dedicated_ssl_certificate_event_type, universal_ssl_event_type, or load_balancing_health_alert).
* mechanisms is at least one of email, webhooks.
* filters is the list of all zones for the WAF alert.









 Advanced WAF alert command 

To configure an Advanced WAF alert, use the following command:

curl -X POST https://api.cis.cloud.ibm.com/v1/:crn/alerting/policies -H 'Content-Type: application/json' -H 'X-Auth-User-Token: Bearer xxxxxx' -d '{
"name": "WAF Alerter",
"description": "Send an email on spike in firewall events for WAF or browser integrity check",
"enabled": true,
"alert_type": "clickhouse_alert_fw_ent_anomaly",
"mechanisms": {
"email": [
{
"id": "sreteam@techcompany.com"
}
]
},
"filters": {
"services": [
"waf",
"bic"
],
"zones": [
"123456ab7d8e9f0g12h2j34l5mn6op78"
]
}
}'

Where:
[Panel options](https://cloud.ibm.com/docs-content/v1/content/220cdcbf6f4421bd7fee83fd1028048dba3f16c8/monitoring/platform/images/sysdig-platform-15.png)

Panel options

If you have multiple queries defined in a panel, you are prompted to select the metric for which you want to create an alert.
7. Configure the alert. Set the following fields:

Name: Enter a name for the alert.

Description: Add a description that other users can read to get more context. This field is optional.

Group: The alert group this alert will be part of. If not specified, the alert will be part of the default group.

Severity: Set the level of criticality of the alert. Valid values are High, Medium, Low, and Info.

Metric: This field is set to the metric that you have selected from the panel. Check that the metric and aggregation are the ones that you need.

Scope: This field is set to the scope that you have defined for the metric in the panel. Check that the scope is the one that you need.

Trigger: Define the condition and threshold value that must be evaluated. It also defines whether the alert sends a single alert or multiple alerts. Valid time scales are minute, hour, or day. A single alert fires an alert for the entire scope. Multiple Alerts are sent if 1 or more segments breach the threshold at once. An alert is sent for each segment that you specify.

Notification Channel: Enable 1 or more notification channels.









 Configuring an alert from the Alerts section 

You can define a metric alert directly from the Alerts section.

Complete the following steps to define an alert on a metric:



1. [Launch the monitoring UI](https://cloud.ibm.com/docs/monitoring?topic=monitoring-launch).
2. Verify that you have a notification channel that defines how you want to be notified.

You can enabled 1 or more notification channels when you configure an alert.
You can create different channel types, for example, an email and a Slack channel. Or, you can create multiple channels of the same type with different alert criteria.





Your preset can now be used by others to setup alerts.





 Deleting a preset 

You can manage presets through the ALERTS dashboard.

Complete the following steps to delete a preset from the ALERTS dashboard:



1. In the web UI, select the Settings icon ![Settings icon](https://cloud.ibm.com/docs-content/v1/content/6f3c07c34058f637b1c86044ca4f9b24214330a7/activity-tracker/alerts/images/admin.png).
2. Select ALERTS.
3. Hover the mouse over the Edit button of the preset that you want to delete. The Delete option displays.
4. Click Delete.
5. Confirm that you want to delete the preset by clicking Yes, delete.



When you delete a preset, any alerts that are defined by using this preset are automatically deleted.
[Configure alert](https://cloud.ibm.com/docs-content/v1/content/f61dbc5d84bf0c89e477a05cab04c72f429e1b7a/log-analysis/images/configalert.png)



1. Specify when the alert will be triggered, and an email sent. For this example specify When 5 or more matches appear within 30 seconds.
2. Select the email addresses of the Recipients who should receive the alert.
3. Click Save Alert.



9. Check the configured email for an alert.
10. When you are ready to disable the alert, click the view name and in the menu click Detach alerts.



![Detach alerts](https://cloud.ibm.com/docs-content/v1/content/f61dbc5d84bf0c89e477a05cab04c72f429e1b7a/log-analysis/images/detachalerts.png)





 Next steps 

Want to start fresh? Remove the following resources that you created as a part of this tutorial:



* Delete any views you will not need.
* Delete any dashboards and screens you will not need.
* Delete any parsing rules you will not need.
* Detach any alerts you will not need.
res = sdclient.get_alerts()
if not res[0]:
print("Failed to fetch existing alerts")

for alert in res['alerts']:
if alert['name'] == alert_name:
print("Deleting alert")
res = sdclient.delete_alert(alert)
if not res:
print("Alert deletion failed")
Show more





 Get all user alerts (GET) 

The following code shows the structure of a Python script that you can use to get information about all the alerts:

 Reference the Python client
from sdcclient import IbmAuthHelper, SdMonitorClient

 Add the monitoring instance information that is required for authentication
URL = <MONITORING-ENDPOINT>
APIKEY = <IAM_APIKEY>
GUID = <GUID>
ibm_headers = IbmAuthHelper.get_headers(URL, APIKEY, GUID)

 Instantiate the Python client
sdclient = SdMonitorClient(sdc_url=URL, custom_headers=ibm_headers)

json_res = sdclient.get_alerts()
print(json_res)
<-- </section "id="section-certificate-alert-examples" "> --><-- </section "id="section-certificate-alert" "> --><-- <section "id="section-create-glb-healthcheck-alert" "> --> ibmcloud cis alert-policy glb-healthcheck-alert-create Create an alert policy for changes in health status for global load balancer, pools, and origins. ibmcloud cis alert-policy glb-healthcheck-alert-create --name NAME (--emails EMAILS | --webhooks WEBHOOKS) --enabled (true | false) --pools POOLS --include-future-pools (true | false)] --health-status-trigger (healthy | unhealthy | either)] --event-source-trigger (pool | origin | either)] --description DESCRIPTION] -i, --instance INSTANCE] --output FORMAT] ! ! ! ! ! !
<-- <section "id="section-opt-create-glb-healthcheck-alert" "> --> Command options --name: The name of the alert policy.--description: The description for the alert policy.--emails: The email addresses for dispatching an alert notification. For example: --emails test1@cn.ibm.com,test2@cn.ibm.com--webhooks: The webhook ID that for dispatching an alert notification. For example: --webhook webhookID1,webhookID2--enabled: Whether or not the alert policy is enabled.--pools: The IDs of origin pool, if set to all, the all pool IDs will be used.--include-future-pools: Whether or not include the future pools. (default "false")--health-status-trigger: The trigger condition to fire the notification. Valid values: "healthy", "unhealthy", "either". (default "either")--event-source-trigger: The event source of trigger to fire the notification.
Table 18. Events for alerts

 Action Description 

 logdna.alert.create This event is generated when an alert is created as a preset. 
 logdna.alert.update This event is generated when an alert is updated. 
 logdna.alert.delete This event is generated when an alert is deleted. 



The following table lists custom fields that are included in these events:



Table 19. Custom fields for view actions

 Custom fields Description 

 requestData.alertId Defines the preset ID. 
 requestData.name Defines the name of the preset. 
 requestData.preset Defines whether the alert is defined as a preset. 
 requestData.channels List of channels that are configured in a preset. Each channel includes information about the notification method and the trigger conditions per method. 
 responseData.logdnaId Defines the logging ID that is associated with the IBM Log Analysis instance. 







 Dashboards 



Table 20. Events for dashboards

 Action Description 

 logdna.board.create This event is generated when a dashboard is created. 
 logdna.board.delete This event is generated when a dashboard is deleted. 
 logdna.board-graph.update This event is generated when a graph is added to a dashboard. 



The following table lists custom fields that are included in these events:



Table 21. Custom fields for boards

 Custom fields Description 

 requestData.boardId Defines the ID of the dashboard. 
 requestData.category Defines the category where the board is included. 
 requestData.title Defines the name of the dashboard. 
 requestData.graphId Defines the ID of a graph that is added to a board. 
 responseData.logdnaId Defines the logging ID that is associated with the IBM Log Analysis instance. 









 Viewing events 

Events that are generated by an instance of the IBM Log Analysis service are automatically forwarded to the IBM Cloud Activity Tracker service instance that is available in the same location. For more information, see [Cloud services locations](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-cloud_services_locations).

IBM Cloud Activity Tracker can have only one instance per location.
Integrating with the Monitoring service 

You can send alerts to the Monitoring service and manage them through the Events view section.



 Configuring a Monitoring alert 

When you [configure an alert](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-alertsconfigure-an-alert) you can have that alert sent to Monitoring.



1. When configuring your alert, click ![Sysdig icon](https://cloud.ibm.com/docs-content/v1/content/6f3c07c34058f637b1c86044ca4f9b24214330a7/activity-tracker/images/sysdig.png).
2. Select if you want the alert to be sent when the condition exists (Presence) or does not exist (Absence).
3. Indicate the logging criteria when an alert should be sent. For example, when 100 lines matching in the view are logged in an hour. A graph will help you determine the number of log lines matching your specified criteria.
4. Select if the alert should be sent at the end of the selected period or immediately when the number of lines are logged.
5. Optionally you can specify a Custom schedule with alerting limited to a specified timezone, days of the week, or timeframe. To configure a Custom schedule:



1. Select on for Custom schedule.
2. Select the Timezone for the log entries.
3. Select the days of the week when alerts should be generated.
4. Optionally specify a time range for the selected days. A graph will help you determine the number of log entries for the timezone and time range.



6. Configure the Monitoring instance details.

Specify the API key. Set this field to the API token. For more information on how to get the token, see [Working with tokens](https://cloud.ibm.com/docs/monitoring?topic=monitoring-api_token).
You can use this number as a threshold for sending alerts when the number of available pods is fewer than your requested number of replicas for your app. If you requested 3 replicas of your app, you can set an alert for when the number of available pods stays fewer than the requested replica count for a certain time duration, for example, a 30 minute duration. In this example, when the pod availability remains fewer than the 3 requested replicas, the alert is triggered.
6. In the Pod Availability tile, create an alert by clicking the Options menu and then Create alert. Set the alert parameters based on the threshold that you observed and your requested app replicas.
7. Open the Notification section of the alert menu and create or select an alert notification channel.
8. Save your alert.
9. Repeat these steps for all the apps that are deployed in your cluster.
10. Test the alert that you created by editing the threshold that you configured to manually trigger the alert. For example, if you set a pod availability alert for when the number of available pods is less than 3 for 5 minutes, edit your alert threshold to be less than 4 pods for 5 minutes and select the at least once option.
11. Verify that the alert is triggered after 5 minutes. Once you have verified the alert, reset the values to the values you configured earlier.







 Troubleshooting persistent storage when an IBM Cloud Monitoring alert is triggered 

When an alert is triggered, review the alert details in IBM Cloud Monitoring and review troubleshooting guides for persistent storage, apps, worker nodes, and clusters to find the root cause of the alert. The alerts that you set up might not be related to a storage volume issue, but to issues that occurred within your app, on the worker node, or in the cluster.



* [Troubleshooting persistent storage volumes](https://cloud.ibm.com/docs/openshift?topic=openshift-debug_storage_file).
* [Troubleshooting worker nodes](https://cloud.ibm.com/docs/openshift?topic=openshift-kubernetes-service-clics_worker_reload).
[Views icon](https://cloud.ibm.com/docs-content/v1/content/6f3c07c34058f637b1c86044ca4f9b24214330a7/activity-tracker/alerts/images/views.png).
2. Click the view name. Then, select Attach an alert.
3. Select View-specific alert.
4. Select the type of alert (Slack, Email, Webhook, or PagerDuty). If you selected the wrong alert type and you want to change it, click Delete Alert Channel.
5. Configure when you want the alert to be sent.

Select if you want the alert to be sent when the condition exists (Presence) or does not exist (Absence).

Indicate the criteria when an alert should be sent. For example, when 100 lines matching in the view are logged in an hour. A graph will help you determine the number of event lines matching your specified criteria.

Select if the alert should be sent at the end of the selected period or immediately when the number of lines are logged.

Optionally you can specify a Custom schedule with alerting limited to a specified timezone, days of the week, or timeframe. To configure a Custom schedule:

Select on for Custom schedule.

Select the Timezone for the event entries.

Select the days of the week when alerts should be generated.

Optionally specify a time range for the selected days. A graph will help you determine the number of event entries for the timezone and time range.
6. Depending on the type of alert you will also need to configure additional settings:

Slack: Specify your Webhook URL and the desired Message color.

Email: Specify the Recipients of the email and a Timezone. The timezone defines the timestamp value of each event that is included in the email. To see UTC timestamps, you can select (GMT +00:00) UTC.

PagerDuty: Specify the Service. If required, you will be prompted to connect to PagerDuty.
This alert will show all thresholds that have been exceeded during the period. If index rate alerts are configured on each separate threshold, an alert will be sent indicating the first threshold crossed (either Max lines/s, Max z-score, or both)





 Types of index rate alerts 

There are two types of index rate alerts:



* Max lines/s alerts: Use this alert to monitor and control the number of log lines per second that are indexed and stored for search.



* To monitor high flow rates, set an index rate threshold by configuring the max lines alert field so that you can get an alert when the value of lines per second is exceeded.
* When the average index rate changes over time, you must review and configure the threshold accordingly.



* Max z-score alerts: Use this alert to monitor and control anomalous flow rates.



* To monitor for spikes on logs and anomalous flow rates, set a threshold value by configuring the max z-score field so that you can get an alert when the hourly index rate is significantly higher than the monthly mean.
* The baseline value is calculated using data from the last 30 days.

After you provision an instance, you must wait a minimum of 30 days of data that has been indexed to use this feature.
* Each z-score represents 1 standard deviation that is based on the distribution of index rates as seen over the past 30 days.

0, 1 indicates a small spike of data.

3 indicates a spike of data that requires troubleshooting and control.

5 indicates a spike of data that requires quick action such as defining exclusion rules while you find out what source and app are causing a spike of data.



Setting a maximum of 3 standard deviations is a good starting point for alert triggering. You can then refine your alerting based on your needs.







 Launch the index rate alert page 

The index rate alert page offers an overview of your account's current rate of ingestion and indexing of searchable data. Through this page, you can see the ratio between the log lines that are ingested versus the number of log lines that are being stored.

To use the index rate feature on an instance, an administrator of the service must turn on the feature.
Managing presets through the UI 

To reuse an alert configuration, a service administrator can configure an alert preset (alert template).



 Creating a preset 

Complete the following steps to configure a preset:



1. In the web UI, select the Settings icon ![Settings icon](https://cloud.ibm.com/docs-content/v1/content/6f3c07c34058f637b1c86044ca4f9b24214330a7/activity-tracker/alerts/images/admin.png).
2. Select ALERTS.
3. Select Add Preset.
4. Specify a name for the preset.
5. Select the type of alert (Slack, Email, Webhook, or PagerDuty). If you selected the wrong alert type and you want to change it, click Delete Alert Channel.
6. Configure the type of alert.



1. Select if you want the alert to be sent when the condition exists (Presence) or does not exist (Absence).
2. Indicate the logging criteria when an alert should be sent. For example, when 100 lines matching in the view are logged in an hour. A graph will help you determine the number of log lines matching your specified criteria.
3. Select if the alert should be sent at the end of the selected period or immediately when the number of lines are logged.
4. Optionally you can specify a Custom schedule with alerting limited to a specified timezone, days of the week, or timeframe. To configure a Custom schedule:



1. Select on for Custom schedule.
2. Select the Timezone for the log entries.
3. Select the days of the week when alerts should be generated.
4. Optionally specify a time range for the selected days. A graph will help you determine the number of log entries for the timezone and time range.



5. Depending on the type of alert you will also need to configure additional settings: