Dealing with a janky backend
Tags: android, retrofit, okhttp
Published: 2020-08-04 18:40:24

---
title: Dealing with a janky backend
published: true
description: What to do when you have a noncompliant server operated by a megacorp
tags: android, retrofit, okhttp
cover_image: https://dev-to-uploads.s3.amazonaws.com/i/unt0yickfnf7qa0hdbh6.jpg
---

I have to deal with a third-party backend with the following properties:
1. It requires my request payload to be gzipped.
1. It does not like a "chunked" transfer encoding, so I have to provide the actual content length as a header.
1. It doesn't understand how to parse headers; it expects _exact_ matches with its requirements.
1. It sometimes responds with a gzipped payload, and other times does not. It depends on which physical server is hit.
1. It sometimes responds with a JSON array wrapping the main body (`[{ ... }]`) and other times with just the main body (`{ ... }`). It depends on which physical server is hit.

What follows is a true story. Some class names have been changed to protect the innocent.

![definition of janky: of extremely poor or unreliable quality](https://dev-to-uploads.s3.amazonaws.com/i/1lnh3qr61ucn5ihk3wyz.png)

Dictionary.com gets it.

# Gzipping a request body and providing the real content length

Gzipping your request body is actually straightforward, and the [official docs](https://square.github.io/okhttp/interceptors/#rewriting-requests) provide an exact recipe for doing so.

```java
final class GzipRequestInterceptor implements Interceptor {
  @Override public Response intercept(Interceptor.Chain chain) throws IOException {
    Request originalRequest = chain.request();
    if (originalRequest.body() == null || originalRequest.header("Content-Encoding") != null) {
      return chain.proceed(originalRequest);
    }

    Request compressedRequest = originalRequest.newBuilder()
      .header("Content-Encoding", "gzip")
      .method(originalRequest.method(), gzip(originalRequest.body()))
      .build();
    return chain.proceed(compressedRequest);
  }

  private RequestBody gzip(final RequestBody body) {
    return new RequestBody() {
      @Override public MediaType contentType() {
        return body.contentType();
      }

      @Override public long contentLength() {
        return -1; // We don't know the compressed length in advance!
      }

      @Override public void writeTo(BufferedSink sink) throws IOException {
        BufferedSink gzipSink = Okio.buffer(new GzipSink(sink));
        body.writeTo(gzipSink);
        gzipSink.close();
      }
    };
  }
}
```

If you return `-1` as the content length (as above), then OkHttp will automatically add the `Transfer-Encoding: chunked` header to your request. As mentioned earlier, my janky backend does not like such a transfer-encoding.

It is not immediately obvious how to provide a non-`-1` content length, and unfortunately there is no public recipe. However, a bit of searching online uncovered [this solution](https://github.com/square/okhttp/issues/350#issuecomment-123105641), which happens to live on OkHttp's Github issue tracker (which has been upvoted enough it probably warrants being an official recipe, but I digress):

```java
class GzipRequestInterceptor implements Interceptor {
  @Override public Response intercept(Chain chain) throws IOException {
    ...

    Request compressedRequest = originalRequest.newBuilder()
      .header("Content-Encoding", "gzip")
      .method(originalRequest.method(), forceContentLength(gzip(originalRequest.body())))
      .build();
    return chain.proceed(compressedRequest);
  }

  private RequestBody forceContentLength(final RequestBody requestBody) throws IOException {
    final Buffer buffer = new Buffer();
    requestBody.writeTo(buffer);

    return new RequestBody() {
      @Override
      public MediaType contentType() {
        return requestBody.contentType();
      }

      @Override
      public long contentLength() {
        return buffer.size();
      }

      @Override
      public void writeTo(BufferedSink sink) throws IOException {
        sink.write(buffer.snapshot());
      }
    };
  }

  private RequestBody gzip(final RequestBody body) {
    ...
  }
}
```

where you can see that the gzipped `RequestBody` has been wrapped in yet another `RequestBody` that writes out to a buffer so we can get the number of bytes via `buffer.size()`.

So, this solves my first two problems, which is that my request must be (1) gzipped and (2) provide a real content-length (not be chunked). :tada:

# Understanding HTTP header semantics, or, The quest for HTTP 200

This is the story of how I learned that Retrofit will automatically add/override a `Content-Type` header to be "application/json; charset=UTF-8".

Despite resolving the first two problems and (seemingly) meeting the spec, I was met with an HTTP 400 at every turn. The response gave no indication _what_ was causing the 400; all I knew was that the server did not like my request. I scrutinized every header, I double-checked the content length, I tweaked the payload in countless ways &mdash; always an opaque 400.

I learned that another team at a different company had found a solution using [Volley](https://developer.android.com/training/volley). I confess I thought that library dead long ago. Trying to be a team player, I made an attempt at this, but despair set in. Surely there was another way!

That's when I decided to use what I called "raw OkHttp." Rather than use Retrofit as a fancy wrapper for creating my requests, I built the request myself out of `JSONObject`s. When I finally launched the app with this approach, I watched in excited bewilderment as it actually returned a 200! The body completely failed to deserialize, of course,[<sup>1</sup>](#endnote-1) <a name="source-1" /> but I was a step closer to a solution.

Here's what that "raw" request looked like:

```kotlin
class RawApi(private val client: OkHttpClient) {
  fun makeRequest(theRequest: TheRequest): TheResponse? {
    client.newCall(newRequest(theRequest)).execute().use { response ->
      if (!response.isSuccessful) {
        throw IOException("Unexpected code $response")
      }
      return bodyFromResponse(response)
    }
  }

  private fun bodyFromResponse(response: Response): TheResponse {
    val body = if (response.header("Content-Encoding", "")!!.contains("gzip")) {
      GzipSource(response.body!!.byteStream().source())
        .use { source ->
          source.buffer().use {
            it.readUtf8()
          }
        }
    } else {
      response.body!!.string()
    }
    return fromJson(body)
  }

  private fun fromJson(body: String): TheResponse = when {
    body.startsWith("[") -> {
      // The form should be "[{ ... }]". I.e., a json object wrapped in a json array.
      val type = object : TypeToken<List<TheResponse>>() {}.type
      Gson().fromJson<List<TheResponse>>(body, type).first()
    }
    body.startsWith("{") -> {
      // The form should be "{ ... }". I.e., a json object, without an array wrapper.
      Gson().fromJson<TheResponse>(body, TheResponse::class.java)
    }
    else -> {
      throw IOException("Cannot deserialize this body. It should start with { or [. Was $body")
    }
  }

  private fun newRequest(theRequest: TheRequest): Request = Request.Builder()
    .url("https://some/url")
    .post(newRequestBody(theRequest))
    .build()

  private fun newRequestBody(theRequest: TheRequest): RequestBody {
    val request = JSONArray()
    val body = JSONObject()
    body.put("param-1", theRequest.param1)
    // ...etc...
    request.put(body)
    return object : RequestBody() {
      override fun contentType(): MediaType? {
        return "application/json".toMediaType()
      }
      override fun writeTo(sink: BufferedSink) {
        sink.writeUtf8(request.toString())
      }
    }
  }
}
```
Here you can see how I optionally decompress from gzip, and also differentially deserialize from either a JSON array or a JSON object. I'll return to this topic in just a moment.

So, how does this solution differ from the one using Retrofit? I compared the two requests (their respective headers and bodies), and saw that the Retrofit request had this header:

```
Content-Type: application/json; charset=UTF-8
```
whereas the raw OkHttp request had
```
Content-Type: application/json
```
And this is how I learned that this janky backend doesn't do proper header parsing, because when I modified my Retrofit request (using a network interceptor) to remove the charset from the header, my request worked &mdash; I got a 200.

:sob:

# Returning to Retrofit + OkHttp
## Optionally decompressing the response

First, you need an interceptor:

```kotlin
class GunzipInterceptor : Interceptor {
  override fun intercept(chain: Interceptor.Chain): Response {
    // Handle the response, which may or may not be gzipped
    val response = chain.proceed(gzipped)

    val contentEncoding: String? = response.header("Content-Encoding")
    if (contentEncoding == null || !contentEncoding.contains("gzip")) {
      // Not gzipped, so just proceed with the original response
      return response
    }

    // Gzipped, so decompress it and return new, uncompressed response body
    val originalResponseBody = response.body!!
    val gzipSource = GzipSource(originalResponseBody.byteStream().source()).buffer()
    val responseBody = decompressed(gzipSource, originalResponseBody.contentType())

    return response.newBuilder()
      .body(responseBody)
      .build()
    }
  }

  private fun decompressed(source: BufferedSource, contentType: MediaType?): ResponseBody {
    return object : ResponseBody() {
      override fun contentLength(): Long = source.buffer.size
      override fun contentType(): MediaType? = contentType
      override fun source(): BufferedSource = source
    }
  }
}
```
Here we trust that, if the server sends the `Content-Encoding: gzip` header, then the body truly is gzipped.[<sup>2</sup>](#endnote-2) <a name="source-2" />

## Differentially deserializing the response

Here we leverage a Gson `JsonDeserializer`, alongside a Retrofit `Converter.Factory`.

```kotlin
class ResponseDeserializer : JsonDeserializer<TheResponse> {
  override fun deserialize(
    json: JsonElement, typeOfT: Type?, context: JsonDeserializationContext?
  ): TheResponse {
    val jsonObject = when {
      json.isJsonObject -> json.asJsonObject
      json.isJsonArray -> json.asJsonArray.get(0).asJsonObject
      else -> error("json neither an array nor an object")
    }
    return Gson().fromJson<TheResponse>(jsonObject, TheResponse::class.java)
  }
}
```

## "Fixing" the content type

To be clear, there is nothing inherently wrong with Retrofit's behavior of appending "; charset=UTF-8" to the `Content-Type` header. It's part of [the standard](https://tools.ietf.org/html/rfc2388#section-4.5). However, as indicated, I have a janky server, so I need to remove the charset. Here's how you set the exact header you want your backend to see. Note that it _must_ be a network interceptor.

```kotlin
okHttpClientBuilder.
  // ...etc...
  .addNetworkInterceptor { chain ->
    val original = chain.request()
    val simplified = original.newBuilder()
      .header("Content-Type", "application/json")
      .build()
    return chain.proceed(simplified)
  }
  .build()
```

(Please see the [excellent documentation](https://square.github.io/okhttp/interceptors/) for a discussion of how a regular interceptor differs from a network interceptor.)

## Tying it all together

Build your `Retrofit` instance like so

```java
Gson gson = new GsonBuilder()
  .registerTypeAdapter(TheResponse.class, new ResponseDeserializer())
  .create();
GsonConverterFactory factory = GsonConverterFactory.create(gson);
return new Retrofit.Builder()
  .addConverterFactory(factory)
  .callFactory(okHttpClient)
  // ...etc...
  .build()
```

And now, finally, it *cough* just works.

# Special thanks

Special thanks to [Jesse Wilson](https://twitter.com/jessewilson) and [Jake Wharton](https://twitter.com/JakeWharton) for discussing this issue with me and helping me to resolve it. Only lost a bit of sanity in the process!

# Endnotes
<sup>1</sup><a name="endnote-1"/> Because the server is janky. [<small>up</small>](#source-1)
<sup>2</sup><a name="endnote-2"/> We shouldn't trust this, because the server is janky. [<small>up</small>](#source-2)