Intercepting HTTP Requests with NodeJS
Tags: node, javascript, webdev, tutorial
Published: 2021-09-07 02:20:02

## Intro
As part of a project at work, I had to develop a way to intercept and store HTTP traffic for any given backend application (microservice in this case). This would have been a fairly straightforward task, but our backend is composed of many services (and many repos). Therefore, the solution had to be as seamless as possible so that it could be easily integrated into any of the services.

## TLDR;
Using [`@mswjs/interceptors`](https://www.npmjs.com/package/@mswjs/interceptors) makes it straightforward to intercept HTTP traffic on your backend app.

## Intercepting HTTP Traffic
For my use case, there were two options I could think of for capturing the HTTP traffic:
<ol>
    <li>Create a library that wraps an HTTP client library like Axios</li>
    <li>Somehow intercept all HTTP traffic</li>
</ol>

In an ideal world, I would have chosen **option 1** since it would be the simplest. Unfortunately, the project I work on consists of many microservices owned by different teams. Therefore, it would make it difficult for everyone to go back and refactor their code to use this new library.

Therefore, my only option was really **option 2**.

### First attempt
My first attempt was ok, but far from perfect. After trying to intercept the traffic directly through the low-level [http](https://nodejs.org/api/http.html) module, I opted for a more higher-level solution. My idea was to [monkey patch](https://stackoverflow.com/a/5626250/1650910) Axios's request methods to inject my own logic before a request is sent and after the response is received.

```ts
function _instrumentAxios(axiosInstance: AxiosInstance) {
   axiosInstance.request = _instrumentHttpRequest(axiosInstance.request, axiosInstance);
   axiosInstance.get = _instrumentHttpRequest(axiosInstance.get, axiosInstance, "get");
   axiosInstance.post = _instrumentHttpRequest(axiosInstance.post, axiosInstance, "post");
   axiosInstance.put = _instrumentHttpRequest(axiosInstance.put, axiosInstance, "put");
   axiosInstance.patch = _instrumentHttpRequest(axiosInstance.patch, axiosInstance, "patch");
   axiosInstance.delete = _instrumentHttpRequest(axiosInstance.delete, axiosInstance, "delete");
   axiosInstance.options = _instrumentHttpRequest(axiosInstance.options, axiosInstance, "options");
}

function _instrumentHttpRequest(originalFunction: Function, thisArgument: any, method?: string) {
   return async function() {
      const {method: cleanedMethod, url, config: requestConfig, data} = _parseAxiosArguments(arguments, method);

      const requestEvent: HttpRequestEvent = {
         url,
         method: cleanedMethod,
         body: data,
         headers: requestConfig?.headers,
      };


      // Intentionally not waiting for a response to avoid adding any latency with this instrumentation
      doSomethingWithRequest(requestEvent);

      const res = await originalFunction.apply(thisArgument, arguments);


      const responseEvent: HttpResponseEvent = {
         url,
         method: cleanedMethod,
         body: res.data,
         headers: res.headers,
         statusCode: res.status,
      };

      doSomethingWithResponse(responseEvent);

      return res;
   };
}
```

This method worked fine, but then I accidentally bumped into a cleaner approach while reading the Axios docs.

### Second attempt
To my surprise, Axios actually offers an API for intercepting requests and responses!

```ts
import {createInterceptor, InterceptorApi, IsomorphicRequest, IsomorphicResponse} from "@mswjs/interceptors";
import {interceptXMLHttpRequest} from "@mswjs/interceptors/lib/interceptors/XMLHttpRequest";
import {interceptClientRequest} from "@mswjs/interceptors/lib/interceptors/ClientRequest";

function _instrumentAxios(axiosInstance: AxiosInstance) {
   axiosInstance.interceptors.request.use(_instrumentHttpRequest);
   axiosInstance.interceptors.response.use(_instrumentHttpResponse);
}

function _instrumentHttpRequest(requestConfig: AxiosRequestConfig): AxiosRequestConfig {
   const method = String(requestConfig.method);
   const headers = requestConfig.headers && {
      ...requestConfig.headers.common,
      ...requestConfig.headers[method],
   };

   const requestEvent: HttpRequestEvent = {
      headers,
      method,
      url: String(requestConfig.url),
      body: requestConfig.data,
   };


   // Intentionally not waiting for a response to avoid adding any latency with this instrumentation
   doSomethingWithRequest(requestEvent);

   return requestConfig;
}

function _instrumentHttpResponse(response: AxiosResponse): AxiosResponse {
   const responseEvent: HttpResponseEvent = {
      url: String(response.request?.url),
      method: String(response.request?.method),
      body: response.data,
      headers: response.headers,
      statusCode: response.status,
   };


   // Intentionally not waiting for a response to avoid adding any latency with this instrumentation
   doSomethingWithResponse(responseEvent);

   return response;
}
```

Ah! Much better. **However**, there is another complication to this approach that's also present in the first attempt: the interception has to be set for every Axios instance; this makes for a less than ideal developer experience. I initially assumed everyone used the default axios instance. However, it turns out that it's also possible to create new instances via [`axios.create()`](https://github.com/axios/axios#creating-an-instance). So back to the drawing board 😔

### Final solution
Before attempting to mess with the low-level `http` module, I decided to look for some existing solutions. And after digging around for some time, I stumbled into [`@mswjs/interceptors`](https://www.npmjs.com/package/@mswjs/interceptors). This library is very well documented and is TypeScript friendly.

```ts
function _instrumentHTTPTraffic() {
    const interceptor = createInterceptor({
      resolver: () => {}, // Required even if not used
      modules: [interceptXMLHttpRequest, interceptClientRequest],
   });

   interceptor.on("request", _handleHttpRequest);

   interceptor.on("response", _handleHttpResponse);

   interceptor.apply();
}

function _handleHttpRequest(request: IsomorphicRequest): void {
   const url = request.url.toString();
   const method = String(request.method);
   const headers = request.headers.raw();

   const requestEvent: HttpRequestEvent = {
      headers,
      method,
      url: request.url.toString(),
      body: request.body,
   };


   // Intentionally not waiting for a response to avoid adding any latency with this instrumentation
   doSomethingWithRequest(requestEvent);
}

function _handleHttpResponse(request: IsomorphicRequest, response: IsomorphicResponse): void {
   const url = request.url.toString();
   const headers = request.headers.raw();


   const responseEvent: HttpResponseEvent = {
      url: request.url.toString(),
      method: request.method,
      body: response.body,
      headers: response.headers.raw(),
      statusCode: response.status,
   };

   // Intentionally not waiting for a response to avoid adding any latency with this instrumentation
   doSomethingWithResponse(responseEvent);
}

```

### Caviats
Although the final solution is more general and is also agnostic to the client HTTP library used, there are some downsides:
- Since all HTTP traffic going thorugh the app is intercepted, there needs to be some logic in place to know which requests to ignore. E.g., instrumentation tools like NewRelic regularly send requests to capture metadata. This can add a lot of noise if not handled properly
- Dependence on another library. Whether this is a big deal depends on what the interception is used for. Probably not a big deal for most projects