Unbearably slow API gateway calls between Azure App Services
Tags: help, azure, dotnet, csharp
Published: 2019-09-17 19:08:36

Hello everyone,

I'd like to ask for help debugging ASP.NET Core Web API.

That's the situation:

We've got an authorization service that exposes multiple RESTful APIs to check if authenticated user can access certain business objects. In my case the business is object is a concept of market and they are authorized by project. So API call would look as follows:
```
GET https://authorization.service.com/markets?project=123456
```
Successful response will return array of markets, unauthorized response will return a 403 status code.


It all works great when I call it through Swagger, call it from Postman - responses are quick and come back in tens of milliseconds.

Now my other service is calling this service in authorization filter and based on response either continues or responds with Forbidden.

I've implemented a reusable client for HTTP calls:
```csharp
public class ApiClient : IApiClient
{
    private readonly HttpClient _client;
    private readonly JsonSerializer _jsonSerializer;

    public ApiClient(HttpClient httpClient, JsonSerializer jsonSerializer)
    {
        _client = httpClient ?? throw new ArgumentNullException(nameof(httpClient));
        _jsonSerializer = jsonSerializer ?? throw new ArgumentNullException(nameof(jsonSerializer));
    }

    public async Task<T> GetDataAsync<T>(string uri, string accessToken)
        where T : class
    {
        var request = CreateRequest(uri, accessToken);
        var response = await _client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);

        if (!response.IsSuccessStatusCode)
        {
            return default(T);
        }

        using (var responseStream = await response.Content.ReadAsStreamAsync())
        using (var streamReader = new StreamReader(responseStream))
        using (var jsonTextReader = new JsonTextReader(streamReader))
        {
            return _jsonSerializer.Deserialize<T>(jsonTextReader);
        }
    }

    private HttpRequestMessage CreateRequest(string uri, string accessToken)
    {
        var locationUri = new Uri(uri);

        var message = new HttpRequestMessage(HttpMethod.Get, locationUri);

        message.Headers.Add("Authorization", $"Bearer {accessToken}");

        return message;
    }
}
```

My authorization filter looks as follows:
```csharp
public class AuthorizeProjectFilter : IAsyncActionFilter
{
    private readonly IApiClient _apiClient;
    private readonly IMarketsContext _marketsContext;

    public AuthorizeProjectFilter(IApiClient apiClient, IMarketsContext marketsContext)
    {
        _apiClient = apiClient ?? throw new ArgumentNullException(nameof(apiClient));
        _marketsContext = marketsContext ?? throw new ArgumentNullException(nameof(marketsContext));
    }

    public async Task OnActionExecutionAsync(ActionExecutingContext context, ActionExecutionDelegate next)
    {
        if (
            context.HttpContext.Request.Query.TryGetValue(nameof(RequestBase.Project), out var value) &&
            context.HttpContext.Request.Headers.TryGetValue("Authorization", out var authorizationHeader)
            )
        {
            var project = value.ToString();
            var accessToken = authorizationHeader.ToString().Split(" ")[1];

            var markets = await _apiClient.GetDataAsync<IEnumerable<Market>>($"{_authorizationServiceOptions.Authorization}/markets?project={project}", accessToken);

            var authorized = markets.Any();

            if (!authorized)
            {
                context.Result = new ForbidResult();
            }
            else
            {
                _marketsContext.Markets = markets;
                await next();
            }
        }
    }
}
```


And this works great locally, requests are quick when I call this service, however when I deploy this to Azure App Service - responses take 20+ seconds, sometimes more than a minute. App plan isn't also the cheapest one and it should be fast.

That doesn't seem like a cold start issue because I can keep on calling the service - it still remains slow.


I've run a trace on Azure diagnostics and got the following:
![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/5f3xh1ith5nty218am1f.png)

Seems like most of the time is spent on `BLOCKED_TIME` within `system.net.http` and I'm literally lost here.

Tracing has also caught some exceptions with the following message:
```
An attempt was made to access a socket in a way forbidden by its access permissions
```

Also my `HttpClient` is added using `IHttpClientFactory`:
```csharp
services
    .AddHttpClient<IApiClient, ApiClient>(client =>
    {
        client.DefaultRequestHeaders.Add("Accept", "application/json");
    })
    .ConfigurePrimaryHttpMessageHandler(() => new HttpClientHandler
    {
        UseProxy = false
    });
```

I'm literally lost now, I couldn't find anything on Google that would've helped me.

Please help :)