Adventures in Tracking Upload Progress With OkHttp and Retrofit
Tags: android, networking, okhttp, retrofit
Published: 2021-12-10 19:19:09

This article tells the story of how Stream’s Android team refined our progress tracking process during file uploads in the [Stream Chat Android SDK](https://github.com/GetStream/stream-chat-android/).

Our original implementation to track file upload progress worked, but it had some in-code usability and UX issues that we wanted to clean up. 

The following account gives an up-close look into the process we had, the problems we encountered, and what we did to improve. 

> **Warning:** As this is a story of mistakes we made and then corrected over time, **do not** use any of the initial or intermediate forms of code here in your own projects, only the final fixed version. 😉

## Uploading Files With Retrofit

First things first, let's see how you can upload a file to an API using [Retrofit](https://square.github.io/retrofit/). Most APIs will expect a multipart form to contain the file data. You can declare a method inside a Retrofit interface like the one below to support that operation: 

```kotlin
@Multipart
@POST("/channels/{type}/{id}/file")
fun sendFile(
    @Path("type") channelType: String,
    @Path("id") channelId: String,
    @Part file: MultipartBody.Part,
): RetrofitCall<UploadFileResponse>
```

To invoke this method, you can create a `Part` by using the `createFormData` function, like so:

```kotlin
fun sendFile(file: File) {
    val part = MultipartBody.Part.createFormData("file", file.name,
                   file.asRequestBody(file.getMediaType()))
    retrofitCdnApi
        .sendFile(channelType, channelId, part)
        .enqueue(...)
}
```

Then you just `enqueue` the Retrofit `Call` to run it, and done! That's a basic, working file upload.

## Counting in the Request Body

Time to add progress tracking. For our initial implementation, we used this callback interface:

```kotlin
public interface ProgressCallback {
    public fun onProgress(progress: Long)
    public fun onSuccess(file: String)
    public fun onError(error: ChatError)
}
```

We then created a `ProgressRequestBody` class, most likely based on [this StackOverflow answer](https://stackoverflow.com/a/33384551/4465208). 

This wraps a `file` and a `callback`, and implements the OkHttp [`RequestBody`](https://square.github.io/okhttp/4.x/okhttp/okhttp3/-request-body/) class.

```kotlin
internal class ProgressRequestBody(
    private val file: File,
    private val callback: ProgressCallback
) : RequestBody() {
    override fun contentType(): MediaType = file.getMediaType()

    override fun contentLength(): Long = file.length()

    override fun writeTo(sink: BufferedSink) {
        val total = file.length()
        val buffer = ByteArray(DEFAULT_BUFFER_SIZE)
        var uploaded = 0L
        FileInputStream(file).use { fis ->
            var read: Int
            val handler = Handler(Looper.getMainLooper())
            while (fis.read(buffer).also { read = it } != -1) {
                handler.post {
                    callback.onProgress((100 * uploaded / total))
                }
                uploaded += read.toLong()
                sink.write(buffer, 0, read)
            }
        }
    }

    companion object {
        private const val DEFAULT_BUFFER_SIZE = 2048
    }
}
```

Whenever `writeTo` is invoked to write this `RequestBody` to the network, we loop through the contents of the file manually, write the bytes, and invoke the `callback`, calculating the percentage of the upload completed so far.

This requires a modification to our API invocation, as we now have to create a `ProgressRequestBody` that we then embed in our `Part`, which will — remember this — contain the `ProgressCallback` instance that was passed in.

```kotlin
fun sendFile(file: File, callback: ProgressCallback) {
    val progressBody = ProgressRequestBody(file, callback)
    val part = MultipartBody.Part.createFormData("file", file.name, progressBody)

    retrofitCdnApi
        .sendFile(channelType, channelId, part)
        .enqueue(RetroProgressCallback(callback))
}
```

We also pass the callback to the `enqueue` call via a wrapper to adapt it to a Retrofit [`Callback`](https://square.github.io/retrofit/2.x/retrofit/retrofit2/Callback.html), which will run the `onFailure` and `onSuccess` methods of our original callback as needed.

```kotlin
internal class RetroProgressCallback(
    private val callback: ProgressCallback
) : Callback<UploadFileResponse> {
    override fun onFailure(call: Call<UploadFileResponse>, t: Throwable) {
        callback.onError(ChatError(cause = t))
    }

    override fun onResponse(
        call: Call<UploadFileResponse>,
        response: retrofit2.Response<UploadFileResponse>
    ) {
        val body = response.body()
        if (body == null) {
            onFailure(call, RuntimeException("File response is null"))
        } else {
            callback.onSuccess(body.file)
        }
    }
}
```

If you want to explore this implementation, you can find the old version of our code [here in the GitHub history](https://github.com/GetStream/stream-chat-android/tree/c592028765a7b541503c4a0976775cfd823c57f1/stream-chat-android-client/src/main/java/io/getstream/chat/android/client/api).

## Counting With a Custom Sink Implementation

This is okay, but we can make it a lot nicer. For this change, we essentially took the implementation from [Paulina Sadowska's post about the topic](https://medium.com/@PaulinaSadowska/display-progress-of-multipart-request-with-retrofit-and-rxjava-23a4a779e6ba).

The improvement is to (instead of handling a `FileInputStream` manually) create a custom OkHttp [`Sink`](https://square.github.io/okio/2.x/okio/okio/-sink/index.html) implementation (based on the handy [`ForwardingSink`](https://square.github.io/okio/2.x/okio/okio/-forwarding-sink/index.html)), which will perform the counting and corresponding progress callbacks for us. 

We'll also make `ProgressRequestBody` wrap a `RequestBody` and delegate to it whenever it needs to behave like a `RequestBody`.

```kotlin
internal class ProgressRequestBody(
    private val delegate: RequestBody,
    private val callback: ProgressCallback,
) : RequestBody() {
    override fun contentType(): MediaType? = delegate.contentType()
    override fun contentLength(): Long = delegate.contentLength()

    override fun writeTo(sink: BufferedSink) {
        val countingSink = CountingSink(sink).buffer()
        delegate.writeTo(countingSink)
        countingSink.flush()
    }

    private inner class CountingSink(delegate: Sink) : ForwardingSink(delegate) {
        private val handler = Handler(Looper.getMainLooper())
        private val total = contentLength()
        private var uploaded = 0L

        override fun write(source: Buffer, byteCount: Long) {
            super.write(source, byteCount)
            uploaded += byteCount

						handler.post { callback.onProgress(uploaded, total) }
        }
    }
}
```

For this improvement, we've updated our `ProgressCallback` interface to take the more meaningful `bytesUploaded` and `totalBytes` values during progress updates instead of a percentage. 

(We also added KDoc at this point to make the interface more obvious, which you can check out [in the GitHub repo](https://github.com/GetStream/stream-chat-android/blob/main/stream-chat-android-client/src/main/java/io/getstream/chat/android/client/utils/ProgressCallback.kt).)

```kotlin
public interface ProgressCallback {
    public fun onProgress(bytesUploaded: Long, totalBytes: Long)
    public fun onSuccess(url: String?)
    public fun onError(error: ChatError)
}
```

On the call site, we can now create a simple `RequestBody` first, and then wrap it with the `ProgressRequestBody` implementation:

```kotlin
val body = file.asRequestBody(file.getMediaType())
val progressBody = ProgressRequestBody(body, callback)
val part = MultipartBody.Part.createFormData("file", file.name, progressBody)
```

## The Logging Problem

A major problem we encountered was that we had several interceptors added to the `OkHttpClient` that we used for these uploads. The configuration looked something like this:

```kotlin
baseClientBuilder()
    .connectTimeout(timeout, TimeUnit.MILLISECONDS)
    .writeTimeout(timeout, TimeUnit.MILLISECONDS)
    .readTimeout(timeout, TimeUnit.MILLISECONDS)
    .addInterceptor(ApiKeyInterceptor(...))
    .addInterceptor(HeadersInterceptor(...))
    .addInterceptor(TokenAuthInterceptor(...))
    .addInterceptor(HttpLoggingInterceptor())
    .addInterceptor(
         CurlInterceptor { message ->
             logger().logI("CURL", message)
         }
    )
```

This is natural, and lots of apps have setups like this. What's notable here is that `HttpLoggingInterceptor` and `CurlInterceptor` will both log the request and call `requestBody.writeTo()` internally to do that. 

In the case of our file upload calls, this method is what contains our progress tracking implementation.

The end result is that whenever we make an upload call, we'll run the progress callback three times in a row: twice for the logging, and once when it's actually written to the network. 

This results in an... *interesting* experience for the user looking at the UI, where progress goes something like this:

```kotlin
0 15 45 50 100 0 25 37 57 87 100 0 20 67 100
```

> This only happened in debug builds, as we had the logging disabled in release builds, but it was still a problem.

Fixing this wasn't simple, as we wanted to keep these logging interceptors around. After some elaboration, [we begrudgingly added an ugly, ugly workaround](https://github.com/GetStream/stream-chat-android/pull/2342#discussion_r709018104), like this:

```kotlin
internal class ProgressRequestBody(
    private val delegate: RequestBody, ...
) : RequestBody() {

    var writeCount = 0

    override fun writeTo(sink: BufferedSink) {
        if (writeCount >= progressUpdatesToSkip) {
            val countingSink = CountingSink(sink).buffer()
            delegate.writeTo(countingSink)
            countingSink.flush()
        } else {
            delegate.writeTo(sink)
        }
        
        writeCount++
    }

    companion object {
        private val progressUpdatesToSkip = 2
    }
}
```

As you can see, we simply hardcoded that the first two times when the `ProgressRequestBody` is written, it shouldn't invoke callbacks.

What made this worse is that this value wasn't actually `2` like I assumed, because as mentioned above, we only log with these interceptors in debug builds. 

This meant that we had to make this a `var` and set it dynamically. In release builds it'd be `0`, and in debug builds, we'd increment it to `2`. Ouch.

That's bad enough, but then we also had the requirement to allow our users to set their own `OkHttpClient` instances for the SDK to use, where they could also add more interceptors of their own, which may or may not invoke `writeTo` in the request body (one or more times!). 

We could've somehow provided an additional API where they can increment `progressUpdatesToSkip` to account for this, but then they could also have interceptors that will *sometimes* read the body but not at other times, based on some dynamic condition... There's clearly no winning with this approach, and it's an awful rabbit hole to go down.

So, to quote myself from the workaround PR linked above:

> A real solution would be intercepting the call with an OkHttp network interceptor, but we can't pass in individual callbacks per different Retrofit upload calls with that approach (at least not easily).

So the problem was that we had no way to get the `callback` value from the call site that invokes the Retrofit method down to the interceptor attached to the underlying OkHttp client.

Turns out, thankfully, that I was wrong about that!

## OkHttp Tags

Enter OkHttp's [tags API](https://square.github.io/okhttp/4.x/okhttp/okhttp3/-request/-builder/tag/) (many thanks to [Jesse Wilson](https://twitter.com/jessewilson) for showing me this!). With this API, the library allows you to add arbitrary tags (objects) to your requests when building them. 

As the [docs](https://square.github.io/okhttp/4.x/okhttp/okhttp3/-request/-builder/tag/) say, you can:

> Use this API to attach timing, debugging, or other application data to a request so that you may read it in interceptors, event listeners, or callbacks.

When building the `Request`, you can pass in a `Class` as a key and then an object of that type as the associated value:

```kotlin
val request = Request.Builder()
    .post(requestBody)
    .tag(ProgressCallback::class.java, callback)
    .build()
```

And then later you can read these values from the `Request`:

```kotlin
val cb: ProgressCallback? = request.tag(ProgressCallback::class.java)
```

One small problem is that in our file upload code we create a `RequestBody` manually, but not the actual `Request` object, as that's created under the hood by Retrofit. 

Thankfully, the tagging API is also exposed through Retrofit, so you can add tags to methods using the [`@Tag` annotation](https://square.github.io/retrofit/2.x/retrofit/retrofit2/http/Tag.html) on parameters. We'll use this in the next section!

## The last-interceptor-swaparoo

The strategy then is the following:

1. Add the `ProgressCallback` instance as a tag when making the Retrofit call.
2. Create an interceptor *at the end of the interceptor chain* that will check each outgoing request and wrap its `RequestBody` into a `ProgressRequestBody` if the callback is present on it.

First, we'll update the Retrofit interface to accept a `progressCallback` parameter that will be used as a tag:

```kotlin
@Multipart
@POST("/channels/{type}/{id}/file")
fun sendFile(
    @Path("type") channelType: String,
    @Path("id") channelId: String,
    @Part file: MultipartBody.Part,
    @Tag progressCallback: ProgressCallback?,
): RetrofitCall<UploadFileResponse>
```

Then, we'll implement the interceptor that reads this tag and performs the wrapping of the `RequestBody` if needed:

```kotlin
internal class ProgressInterceptor : Interceptor {
    override fun intercept(chain: Interceptor.Chain): Response {
        val request = chain.request()

        val progressCallback = request.tag(ProgressCallback::class.java)
        if (progressCallback != null) {
            return chain.proceed(wrapRequest(request, progressCallback))
        }

        return chain.proceed(request)
    }

    private fun wrapRequest(request: Request, progressCallback: ProgressCallback): Request {
        return request.newBuilder()
            // Assume that any request tagged with a ProgressCallback is a POST
            // request and has a non-null body            
            .post(ProgressRequestBody(request.body!!, progressCallback))
            .build()
    }
}
```

Finally, we'll add this interceptor to our `OkHttpClient`, making sure it's the last one added. 

We'll add it as a [network interceptor](https://square.github.io/okhttp/interceptors/#network-interceptors), as we want it to be as close to the actual upload as possible, and it doesn't need to be involved in requests that don't go out to the network.

```kotlin
return baseClientBuilder()
    .addInterceptor(...)
    .addInterceptor(...)
    .addNetworkInterceptor(ProgressInterceptor())
```

This way nothing that previous interceptors do with the request will interfere with the progress reporting, as they'll still have the original `RequestBody` to work with, and the special progress-tracking wrapper is only added at the very last moment before it actually goes out to the network.

If you want to see this last change in detail, check out [the corresponding PR on GitHub](https://github.com/GetStream/stream-chat-android/pull/2386).

## Wrap-Up

So that's the implementation we ended up with for now! You can find all of this code in the [Chat Android SDK's GitHub repository](https://github.com/GetStream/stream-chat-android/) if you want to look at it in a real project.

There is one remaining issue with this progress tracking: Whenever the body gets written, it's only written into local buffers, and what we track is how fast we're writing to that buffer. This then still needs to make it over the network, which can take some time. 

So the upload progress tends to get to 100% relatively quickly and then the request will remain pending for a while as the network call completes. 

This is as close to the socket (so to say) as we can get with these APIs. For a bit more info and references, see [this GitHub discussion](https://github.com/GetStream/stream-chat-android/pull/2342#discussion_r709021729).

If you want to learn more about how Okio (and OkHttp, Retrofit, and Moshi) work super efficiently with data, watch [A Few Ok Libraries by Jake Wharton](https://www.youtube.com/watch?v=WvyScM_S88c). For an introduction to Moshi, check out [Say Hi to Moshi](https://zsmb.co/appearances/droidcon-london-2021/).