Using XML Policies to Log and Analyze API Calls from Azure API Management
Tags: api, monitoring, analytics, azure
Published: 2022-05-16 11:43:14

---
title: Using XML Policies to Log and Analyze API Calls from Azure API Management
published: true
tags: api, monitoring, analytics, azure
canonical_url: https://www.moesif.com/blog/technical/azure-api-management/How-to-Monitor-Azure-API-Management-Performance-with-the-Moesif-Plugin/
---

Azure API Management (APIM) is a powerful platform that enables you to publish and scale APIs while ensuring they are secured. 
One of the great features of Azure APIM is that you can add plugins and transforms to your APIs without any code change or restarts.

These capabilities are deployed using [XML Policies](https://docs.microsoft.com/en-us/azure/api-management/api-management-howto-policies) which are a collection of statements. 
[Moesif API Observability](https://www.moesif.com/?language=azure-api-management&utm_campaign=DevTo&utm_source=devto&utm_medium=paid&utm_term=monitor-azure-api-management-performance-with-the-moesif-plugin) can be added in just a few minutes using an XML policy for APIM which makes it easy get visibility into API calls, even ones that are rejected and never reach your underlying service.

## What is Moesif API Observability for Azure APIM?
API Observability enables engineering and business to deeply understand how their APIs are used and identify what needs to be fixed before customers email support and overwhelm 
your team. Unlike classic API monitoring which usually probes an endpoint for typical "Red, Yellow, Green" status, API Observability 
enables leaders to observe any behavior happening with the API. Some examples include:

* Product owners to understand API usage and what to focus on
* Engineering leaders to stay informed of and troubleshoot API issues
* Security researchers to investigate API threats and prevent them

## How does this solution works

In order for API Observability to work, a monitoring agent needs to passively log API traffic to an observability service. This can be a custom build on a data warehouse like Azure Synapse Analytics or or a turnkey solution like Moesif. An API Gateway like Azure APIM provides a natural point to centralize API logs. Otherwise, each service would need to have it's own SDK.

This solution is deployed using an [Azure Resource Manager Template](https://azure.microsoft.com/en-us/resources/templates/201-api-management-logs-to-moesif-using-eventhub-webapp/), which automatically adds a few components to your Azure subscription:

|Component|Purpose|
|---------|-------|
|[API Management Logger](https://docs.microsoft.com/en-us/azure/api-management/api-management-howto-log-event-hubs)|Captures API logs from your API Management instance and inserts them into the EventHub|
|[Azure EventHub](https://docs.microsoft.com/en-us/azure/event-hubs/)|Buffers the raw API logs until ready to be consumed|
|[Azure WebApp](https://docs.microsoft.com/en-us/azure/app-service/)|Runs the [ApimEventProcessor](https://github.com/Moesif/ApimEventProcessor) app which reads from EventHub and sends logs to Moesif in batches|

A diagram is below showing how the solution is deployed in your Azure subscription.

![Azure API Management Gateway Logging Architecture Diagram](https://blog.moesif.com/images/posts/technical/azure/azure-api-management-logging-architecture-diagram.png)


Once the API logs are received by Moesif, the rest of the data processing and analytics is handled by the service. 

## Use cases

### Understand Customer API usage

A goal for [API analytics](https://www.moesif.com/features/api-analytics?utm_campaign=DevTo&utm_source=devto&utm_medium=paid&utm_term=monitor-azure-api-management-performance-with-the-moesif-plugin) is to gain an understanding on who is using your APIs and how they use them. To enable this, the integration will automatically associate API calls to specific user profiles. By default, The XML policy will also extract user info like the User Id, First Name, and Email from the 
[context.User](https://docs.microsoft.com/en-us/azure/api-management/api-management-policy-expressions#ContextVariables) object and saved as part of a user profile in Moesif. You can always add additional user properties using Moesif's [user tracking API](https://www.moesif.com/implementation/track-user-behaviors-with-http-api?platform=http-api)

A critical report is understanding which customers are using your APIs the most. Because we are tracking name and email, we can open a report in Moesif showing weekly API traffic by company name.

![Azure API Management Tracking Usage by Customer](https://blog.moesif.com/images/posts/product-management/api-metrics/API-usage-by-customer-over-time.png)

### Troubleshoot issues

With [high-cardinality, high-dimension API observability](https://www.moesif.com/features/api-logs?utm_campaign=DevTo&utm_source=devto&utm_medium=paid&utm_term=monitor-azure-api-management-performance-with-the-moesif-plugin), you can slice and dice your API logs by any number of fields such as the URI Route, HTTP headers, and even fields in the payload enabling you to drill into issues impacting customers. One such metric we recommend monitoring is the 90th percentile. Unlike average latency, by looking at 90th percentile latency, you can better see large variations in your latency which is usually worse for customers than an API that's consistently high latency. An API with seamlessly random latency spikes can wreak havoc in their own services.

To do this, go to _Events -> Time series_ and then select the metric _P90 Latency_. You can also understand this broken down by route or service. To do so, add a group by "_Request URI_." Moesif consolidate routes such that `/items/1` and `/items/2` will show up as `/items/:id` in the UI which makes it easier for your analysis. 

![Azure API Management Report on 90th Percentile Latency By Endpoint](https://blog.moesif.com/images/posts/technical/azure/azure-apim-90th-percentile-latency.svg)

### Research threats

As you expose more APIs to the internet used by customers, partners, and single page apps, your security risk goes up. Traditional mechanisms like browser fingerprinting and captchas don't work so you need to leverage advanced user behavior analytics to find suspicious users.

A common [API security threat](https://www.moesif.com/solutions/api-security?utm_campaign=DevTo&utm_source=devto&utm_medium=paid&utm_term=monitor-azure-api-management-performance-with-the-moesif-plugin) is not having proper protection from data scraping and intentional API abuse. An API provides direct access to your data which a hacker can use to scrape. One way to [detect customers abusing your API](https://www.moesif.com/solutions/api-security) is to look at the amount of data accessed per user. To create this metric, add a summation of _response.headers.Content-Length_ and then group it by user's name:

![Monitoring API Pagination Attacks](https://blog.moesif.com/images/posts/technical/api-security/content-length-by-company.png)

##  How to set up Azure APIM with Moesif API Observability

### 1. Start Azure Resource Deployment

Click the below button to start a Custom deployment with the Moesif Azure Resource Template.

[![Deploy to Azure](https://aka.ms/deploytoazurebutton)](https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FAzure%2Fazure-quickstart-templates%2Fmaster%2Fquickstarts%2Fmicrosoft.apimanagement%2Fapi-management-logs-to-moesif-using-eventhub-webapp%2Fazuredeploy.json)

### 2. Configure Parameters

Within the Azure Template Deployment panel, set the following properties:

![Create a Custom Deployment in Azure](https://docs.moesif.com/images/docs/integration/azure-api-management-create-custom-deployment.png)

* Set _Resource group_ to the same resource group that contains your exiting Azure APIM instance. This ensures the APIM logger, `moesif-log-to-event-hub`, is automatically created for you. 

* Set _Moesif Application Id_ to the one displayed after logging into your Moesif account. You can create a free one on [Moesif's website](https://www.moesif.com/?language=azure-api-management&utm_campaign=DevTo&utm_source=devto&utm_medium=paid&utm_term=monitor-azure-api-management-performance-with-the-moesif-plugin)

* Set _Existing Api Mgmt Name_ to the name of your Azure APIM instance. If blank, you will need to manually create the [APIM logger](https://docs.microsoft.com/en-us/azure/api-management/api-management-log-to-eventhub-sample#policy-declaration). 

Once done, click the _Review+create_ button at the bottom and finish the template creation wizard. 

> Occasionally, Azure reports a failed deployment due to slow propagation of new DNS settings even though everything was deployed successfully. We recommend proceeding with rest of process. If you still have issues after last step, [view troubleshooting](#troubleshooting).

### 3. Add XML Policy

Within the Azure portal, navigate to your existing Azure API Management instance.
Then, add the below XML policies to all products or APIs that you want API logging enabled. 

> It's recommended to add the XML policy globally for all APIs. Then, use Moesif [dynamic sampling](https://www.moesif.com/docs/platform/dynamic-sampling/?utm_campaign=DevTo&utm_source=devto&utm_medium=paid&utm_term=monitor-azure-api-management-performance-with-the-moesif-plugin) if you want to create rules that selectively sample or suppress data collection. Rules are dynamically enabled based on specific customer behaviors, regex rules, and more.

More info on editing APIM policies is available on the [Azure docs](https://docs.microsoft.com/en-us/azure/api-management/set-edit-policies)

```xml
<policies>
    <inbound>
        <base />
        <set-variable name="moesif-message-id" value="@(Guid.NewGuid())" />
        <log-to-eventhub logger-id="moesif-log-to-event-hub" partition-id="0">@{
var body = context.Request.Body?.As<string>(true);
var MAX_BODY_EH = 145000;
var origBodyLen = (null != body) ? body.Length : 0;
if (MAX_BODY_EH < origBodyLen){ body = body.Remove(MAX_BODY_EH); }
var headers = context.Request.Headers
    .Where(h => h.Key != "Ocp-Apim-Subscription-Key")
    .Select(h => string.Format("{0}: {1}", h.Key, String.Join(", ", h.Value).Replace("\"", "\\\""))).ToArray<string>();
var jwtToken = context.Request.Headers.GetValueOrDefault("Authorization","").AsJwt();
var userId = (context.User != null && context.User.Id != null) ? context.User.Id : (jwtToken != null && jwtToken.Subject != null ? jwtToken.Subject : string.Empty);
var cru = new JObject();
if (context.User != null) {
  cru.Add("Email", context.User.Email);
  cru.Add("Id", context.User.Id);
  cru.Add("FirstName", context.User.FirstName);
  cru.Add("LastName", context.User.LastName);}
var crus = System.Convert.ToBase64String(Encoding.UTF8.GetBytes(cru.ToString()));
var requestBody = (body != null ? System.Convert.ToBase64String(Encoding.UTF8.GetBytes(body)) : string.Empty);
return new JObject(
  new JProperty("event_type", "request"),
  new JProperty("message-id", context.Variables["moesif-message-id"]),
  new JProperty("method", context.Request.Method),
  new JProperty("ip_address", context.Request.IpAddress),
  new JProperty("uri", context.Request.OriginalUrl.ToString()),
  new JProperty("user_id", userId),
  new JProperty("contextRequestUser", crus),
  new JProperty("company_id", ""),
  new JProperty("request_headers", string.Join(";;", headers)),
  new JProperty("request_body", requestBody),
  new JProperty("contextTimestamp", context.Timestamp.ToString("o")),
  new JProperty("metadata", $@"")
  ).ToString();}</log-to-eventhub>
        <set-variable name="sent-moesif-request" value="@(true)" />
    </inbound>
    <backend>
        <forward-request follow-redirects="true" />
    </backend>
    <outbound>
        <base />
        <choose>
            <when condition="@(context.Variables.ContainsKey("sent-moesif-request") && !context.Variables.ContainsKey("sent-moesif-response"))">
                <log-to-eventhub logger-id="moesif-log-to-event-hub" partition-id="1">@{
var body = context.Response.Body?.As<string>(true);
var MAX_BODY_EH = 145000;
var origBodyLen = (null != body) ? body.Length : 0;
if (MAX_BODY_EH < origBodyLen){ body = body.Remove(MAX_BODY_EH);}
var headers = context.Response.Headers.Select(h => string.Format("{0}: {1}", h.Key, String.Join(", ", h.Value).Replace("\"", "\\\""))).ToArray<string>();
var responseBody = (body != null ? System.Convert.ToBase64String(Encoding.UTF8.GetBytes(body)) : string.Empty);
return new JObject(
  new JProperty("event_type", "response"),
  new JProperty("orig_body_len", origBodyLen),
  new JProperty("message-id", context.Variables["moesif-message-id"]),
  new JProperty("status_code", context.Response.StatusCode),
  new JProperty("response_headers", string.Join(";;", headers)),
  new JProperty("contextTimestamp", context.Timestamp.Add(context.Elapsed).ToString("o")),
  new JProperty("response_body", responseBody)
  ).ToString();}</log-to-eventhub>
                <set-variable name="sent-moesif-response" value="@(true)" />
            </when>
        </choose>
    </outbound>
    <on-error>
        <base />
        <choose>
            <when condition="@(context.Variables.ContainsKey("sent-moesif-request") && !context.Variables.ContainsKey("sent-moesif-response"))">
                <log-to-eventhub logger-id="moesif-log-to-event-hub" partition-id="1">@{
var body = context.Response.Body?.As<string>(true);
var MAX_BODY_EH = 145000;
var origBodyLen = (null != body) ? body.Length : 0;
if (MAX_BODY_EH < origBodyLen){ body = body.Remove(MAX_BODY_EH);}
var headers = context.Response.Headers.Select(h => string.Format("{0}: {1}", h.Key, String.Join(", ", h.Value).Replace("\"", "\\\""))).ToArray<string>();
var responseBody = (body != null ? System.Convert.ToBase64String(Encoding.UTF8.GetBytes(body)) : string.Empty);
return new JObject(
  new JProperty("event_type", "response"),
  new JProperty("orig_body_len", origBodyLen),
  new JProperty("message-id", context.Variables["moesif-message-id"]),
  new JProperty("status_code", context.Response.StatusCode),
  new JProperty("response_headers", string.Join(";;", headers)),
  new JProperty("contextTimestamp", context.Timestamp.Add(context.Elapsed).ToString("o")),
  new JProperty("response_body", responseBody)
  ).ToString();}</log-to-eventhub>
                <set-variable name="sent-moesif-response" value="@(true)" />
            </when>
        </choose>
    </on-error>
</policies>
```


### 4. Success!

With the Azure APIM integration done, you should see your [API logs show up in Moesif](https://www.moesif.com/features/api-logs?utm_campaign=DevTo&utm_source=devto&utm_medium=paid&utm_term=monitor-azure-api-management-performance-with-the-moesif-plugin). Make a few calls against your API Gateway domain and see them show up in Moesif's event log in real-time. You should see the status code, URL, and other HTTP parameters captured like the below screenshot:

![Azure API Management logs](https://blog.moesif.com/images/posts/technical/azure/azure-apim-api-log.svg)

## Identifying users

API calls are associated to users using the field `user_id`. The default XML policy extracts this from the `context.User.Id` or the JWT Token using the following logic:

```csharp
var jwtToken = context.Request.Headers.GetValueOrDefault("Authorization","").AsJwt();
var userId = (context.User != null && context.User.Id != null) ? context.User.Id : (jwtToken != null && jwtToken.Subject != null ? jwtToken.Subject : null);
```

You can modify what the userId is by changing these lines of code. 

## Adding user metadata

By default, the XML policy also saves some helpful user properties like Email and FirstName using the following code in the XML Policy:

```csharp
if (context.User != null) {
  cru.Add("Email", context.User.Email);
  cru.Add("FirstName", context.User.FirstName);
  cru.Add("LastName", context.User.LastName);
}
var crus = System.Convert.ToBase64String(Encoding.UTF8.GetBytes(cru.ToString()));
```
These will be saved with the user profile in Moesif that matches the defined `userId`. You can add additional fields from the `context.User` to meet your requirements by changing these lines of code. 

## Adding Event metadata

You can also save event metadata. Unlike user metadata, event metadata is specific to each API transaction and can contain helpful info not already logged by Moesif such as trace ids or environment variables. The `metadata` field should be a JSON encoded string.

## Sampling requests

This integration also supports [dynamic sampling](https://www.moesif.com/docs/platform/dynamic-sampling/). This enables you to selectively sample API calls based on user behavior 
to save on your Moesif subscription cost. Moesif will still extrapolate the original metrics.

![Dynamic Sampling Rules](https://blog.moesif.com/images/posts/technical/dynamic-sampling/sampling-and-suppression-overview.png)

## Advanced User Behavior API Analytics

You can leverage your integration beyond just looking at API calls in isolation and stitch your entire customer journey together. This approach makes it easier to see things like funnel reports on "_Time to First Hello World_" and "_Time to Value_."

Track user actions in your UI such as "Signed In" or "Viewed Docs" and start tracking user actions in your UI like "_Signed In_" or "_Viewed Docs_". This makes it easier to slice and dice API usage by customer traffics. In order to do so, add the [moesif-browser-js](https://www.moesif.com/implementation/track-user-behaviors-with-browser) to your UI and call the track method:

```javascript
moesif.track('Clicked Sign Up', {
  button_label: 'Get Started',
  sign_up_method: 'Google SSO'
});
```

Once done, the first thing you should do is generate a funnel report. In the below report, we created a [funnel analysis](https://www.moesif.com/solutions/api-product-management) composing of three steps.

1. The first step is a customer signing into your web application (a user action).
2. The second step is a single payment transaction via the API. Thus moving from step 1 to step 2 shows the conversion rate of sign ups to the first API call.
3. The third step is over 100 payment transactions. For this example, we consider this the "aha" moment demonstrating customer value. Moving from step 2 to step 3 shows the drop off of customers who made API calls who actually got to see real value.

![Funnel showing sign up to first API call](https://blog.moesif.com/images/posts/product-management/api-metrics/funnel-three-steps.svg)

## Conclusion

API observability is critical for engineering and business leaders to make informed decisions on what to focus on and where issues are. While you can roll your own API gateway, data processing pipeline, and a data warehouse, this can create a massive time sink for your engineering team. Using fully managed services like Azure API Management API Gateway and [Moesif API Analytics](https://www.moesif.com/features/api-analytics) can help you scale without being held back by legacy infrastructure. 

---

**Do you want to know how customers use your APIs? [Try Moesif API Analytics!](https://www.moesif.com/?utm_campaign=DevTo&utm_source=devto&utm_medium=paid&utm_term=monitor-azure-api-management-performance-with-the-moesif-plugin)**

---

This article was [originally written for the Moesif blog](https://www.moesif.com/blog/technical/azure-api-management/How-to-Monitor-Azure-API-Management-Performance-with-the-Moesif-Plugin/?utm_campaign=DevTo&utm_source=devto&utm_medium=paid&utm_term=monitor-azure-api-management-performance-with-the-moesif-plugin) by Derric Gilling, CEO and founder of Moesif API Analytics.