URL Expansion with Dart
Tags: dart, flutter, tutorial, http
Published: 2023-09-02 12:46:16

A few months ago, I was proud to announce during a team meeting that I had successfully deployed the deeplinking service for a [Flutter](https://flutter.dev/) app we were developing. However, just a few days later, I was jolted awake in the middle of the night by a flurry of Slack alerts. The deeplinking service was down, and customers couldn't pay using our payment links! Although I had thoroughly tested the service, I hadn't accounted for one thing: our recent switch to using shortened URLs through an [Iterable SMS](https://iterable.com/features/sms/) campaign.

In this article, I will share the solution that finally fixed the deeplinking service's shortcomings. We'll dive into how we developed a URL expander app using [Dart's](https://dart.dev/overview) [HttpClient](https://api.dart.dev/stable/3.1.0/dart-io/HttpClient-class.html) class, making our service more robust and reliable. First, we'll outline the app's structure, explaining its purpose and how it works. Then, we'll provide some practical usage examples to give you a clearer understanding of its capabilities. We'll also identify some initial limitations that we needed to address to make the tool more effective.

Next, we'll go through the steps of unit testing the app, ensuring that it meets our requirements and behaves as expected. Unit tests are crucial for confirming that the code functions properly, especially when changes are made in the future. Through this journey, we aim to provide you with a comprehensive understanding of how to tackle issues like the one we faced with deeplinking, thereby improving your own development practices.

## What is a URL expander?
We are likely very familiar with URL shortening services like [bit.ly](https://bitly.com/) which allow for users to generate shorter (sometimes custom, meaningful) URLs from long and potentially unwieldy URLs. However the task of resolving the original links is trivialised by modern browsers, which often resolve to original urls automatically, thus little attention is often paid to URL expansion.

However, in non-browser environments, like a Flutter mobile app, automatic URL resolution isn't a given. In these contexts, URL expanders become essential. You can think of a URL expander as a tool or service that resolves a shortened URL back to its original, longer form. In this article, we'll walk you through how to implement such a URL expander.

## The URL Expander
Our URL Expander is implemented as a Dart [CLI](https://en.wikipedia.org/wiki/Command-line_interface) app exposing a utility via which shortened URLs may be expanded to their original form.

### Overview of URL Expander App

The structure of the App patterns in agreement with the template for dart CLI apps as highlighted below.

```bash
.
├── CHANGELOG.md
├── README.md
├── analysis_options.yaml
├── bin
│   └── url_expander.dart
├── build
│   ├── test_cache
│   │   └── build
│   │       └── c075001b96339384a97db4862b8ab8db.cache.dill.track.dill
│   └── unit_test_assets
│       ├── AssetManifest.bin
│       ├── AssetManifest.json
│       ├── FontManifest.json
│       ├── NOTICES.Z
│       └── shaders
│           └── ink_sparkle.frag
├── coverage
│   ├── coverage.json
│   └── lcov.info
├── lib
│   └── url_expander.dart
├── pubspec.lock
├── pubspec.yaml
├── test
│   └── url_expander_test.dart
└── tree.txt

9 directories, 17 files
```

### Steps to Url Expansion

1. Get input URL (shortened).
2. Get input base URL for expected expanded URL.
3. Validate input URL.
4. IF input URL is invalid throw an exception; else
5. Open a HTTP HEAD request to fetch headers associated with the URL
6. Close request to receive resolved headers.
7. Extract location header from the response
8. IF location header is not found then throw an exception; else
9. Validate the location against the expected base URL for the expansion of input URL
10. IF validation fails for location but there is a redirect in the response; recursively goto 1; decrease times we can goto 1; else
11. IF validation fails and there is no redirect throw an exception; else
12. IF validation is successful for location return it as expanded URL

The `UrlExpander` encapsulates these steps, the `UrlExpander` class is described in greater detail in the following section.

### The `UrlExpander` class
The `UrlExpander` class is the central functional unit of our app. The UrlExpander class exposes a public method `getFullUrl`, a wrapper to the encapsulated `_extractFullUrl` method. The `UrlExpander` class also encapsulates the `_client` variable (it's HttpClient instance). The `_isValidUrl` method and the `_isValidFullUrl` method, which serve as checks and guards for licit expansions.

```dart
import 'dart:developer';
import 'dart:io';

class UrlExpander {
  final HttpClient _client = HttpClient();

  Future<String> getFullUrl(
    String url, {
    required String expandedBaseUrl,
  }) async {
    try {
      return await _extractFullUrl(url, expandedBaseUrl);
    } catch (e, t) {
      log('$e', error: e, stackTrace: t);
      throw Exception(e);
    }
  }

  _extractFullUrl(
    String url,
    String expandedBaseUrl, [
    int iterationsAllowed = 5,
  ]) async {
    try {
      if (!(iterationsAllowed >= 1)) throw "Max redirect limit reached";
      if (!_isValidUrl(url)) throw "Invalid URL";

      final uri = Uri.parse(url);

      final request = await _client.headUrl(uri);
      request.followRedirects = false;

      final response = await request.close();

      stdout.write(
          "\x1B[32m========\nHEADERS\n=========\n${response.headers}\n==========\x1B[0m\n");

      final fullUrl = response.headers.value(HttpHeaders.locationHeader);

      if ((fullUrl ?? '').isEmpty) throw "URL not found";

      final isValidUrl = _isValidFullUrl(
        fullUrl,
        expandedBaseUrl,
      );

      if (!isValidUrl && response.isRedirect) {
        return await _extractFullUrl(
          fullUrl!,
          expandedBaseUrl,
          --iterationsAllowed,
        );
      }

      if (!isValidUrl) throw 'Cannot fetch expanded URL';

      return fullUrl!;
    } catch (_) {
      rethrow;
    }
  }

  bool _isValidUrl(String? url) {
    if (url != null && Uri.tryParse(url) != null) {
      final RegExp urlRegex = RegExp(
        r"\b[(http(s)?):\/\/(www\.)?a-zA-Z0-9@:%._\+~#=]{2,256}\.[a-z]{2,6}\b([-a-zA-Z0-9@:%_\+.~#?&//=]*)\b",
        caseSensitive: false,
      );

      return urlRegex.hasMatch(url);
    }

    return false;
  }

  bool _isValidFullUrl(String? url, String baseUrl) {
    if (url == null) return false;
    return url.startsWith(baseUrl);
  }
}
```
#### The `_extractFullUrl` method
```dart
  _extractFullUrl(
    String url,
    String expandedBaseUrl, [
    int iterationsAllowed = 5,
  ]) async {
    try {
      if (!(iterationsAllowed >= 1)) throw "Max redirect limit reached";
      if (!_isValidUrl(url)) throw "Invalid URL";

      final uri = Uri.parse(url);

      final request = await _client.headUrl(uri);
      request.followRedirects = false;

      final response = await request.close();

      stdout.write(
          "\x1B[32m========\nHEADERS\n=========\n${response.headers}\n==========\x1B[0m\n");

      final fullUrl = response.headers.value(HttpHeaders.locationHeader);

      if ((fullUrl ?? '').isEmpty) throw "URL not found";

      final isValidUrl = _isValidFullUrl(
        fullUrl,
        expandedBaseUrl,
      );

      if (!isValidUrl && response.isRedirect) {
        return await _extractFullUrl(
          fullUrl!,
          expandedBaseUrl,
          --iterationsAllowed,
        );
      }

      if (!isValidUrl) throw 'Cannot fetch expanded URL';

      return fullUrl!;
    } catch (_) {
      rethrow;
    }
  }
```
The `_extractFullUrl` method is the central functional unit of the `UrlExpander` class. The method receives three arguments `url`, `expandedBaseUrl` and `iterationsAllowed`.

The `url` argument serves as the shortened URL input. The `expandedBaseUrl` serves as input to the associated `_isValidFullUrl` method (along with the `url` input), ensuring that only full urls that match the host expected for the expanded shortened URL are marked as valid. The `iterationsAllowed` is primarily given in [recursive](https://en.wikipedia.org/wiki/Recursion_(computer_science)) calls to prevent runaway/infinite recursions.

The stepwise operations of the `_extractUrl` method are as follows;

1. WHEN recursively called it check that the recursive iteration is permitted in a guard; if not an exception is thrown.
2. It checks that the input URL is valid in a guard else it throws an exception; then.
3. It generates `Uri` object instance from the input URL
4. It creates a `HttpClientRequest` instance `request` from an HTTP HEAD request on the generated `Uri`.
5. We disable automatic redirects on the request instance.
6. We create a `HttpClientResponse` instance from the result of closing the request.
7. We print the response header in green.
8. We extract the `fullUrl` string from the location header
9. IF `fullUrl` is null or empty we throw an exception; else
10. We validate `fullUrl` against its expected base URL denoted by the `expandedBaseUrl` argument; then
11. IF `fullUrl` is invalid but response has a redirect, we recursively call `_extractFullUrl`, reducing the value of the `iterationsAllowed` argument; else
12. IF `fullUrl` is invalid but there is no redirect in the response we throw an exception; else;
13. We return `fullUrl` as the output.

With these operations stated, we find it necessary to highlight the use of the CLI, this is shown below.

![Success Gif](http://g.recordit.co/6ctuOGk2RM.gif)

### Current gaps of the `_extractFullUrl` method

The current iteration of the `_extractFullUrl` method while apparently well forms, has a shortcoming. It fails to parse full URL inputs correctly, as full URLs may not return the location header from a `HEAD` request. This is highlighted below:

![Failure Gif](http://g.recordit.co/NMdlwifceQ.gif)

We can introduce a guard to force an early exit from the method when a valid full URL is input as shown below.

```dart
  _extractFullUrl(
    String url,
    String expandedBaseUrl, [
    int iterationsAllowed = 5,
  ]) async {
    try {
      if (!(iterationsAllowed >= 1)) throw "Max redirect limit reached";
      if (!_isValidUrl(url)) throw "Invalid URL";
      if (_isValidFullUrl(url, expandedBaseUrl)) return url;

      final uri = Uri.parse(url);

      final request = await _client.headUrl(uri);
      request.followRedirects = false;

      final response = await request.close();

      stdout.write(
          "\x1B[32m========\nHEADERS\n=========\n${response.headers}\n==========\x1B[0m\n");

      final fullUrl = response.headers.value(HttpHeaders.locationHeader);

      if ((fullUrl ?? '').isEmpty) throw "URL not found";

      final isValidUrl = _isValidFullUrl(
        fullUrl,
        expandedBaseUrl,
      );

      if (!isValidUrl && response.isRedirect) {
        return await _extractFullUrl(
          fullUrl!,
          expandedBaseUrl,
          --iterationsAllowed,
        );
      }

      if (!isValidUrl) throw 'Cannot fetch expanded URL';

      return fullUrl!;
    } catch (_) {
      rethrow;
    }
  }
```
Consequently the behaviour of the method changes, to return the valid full URL immediately as shown below.

![Full URL gif](http://g.recordit.co/sGj4XLlUco.gif)

## Testing the url expander
Unit tests were written to ensure that the url expander functions as expected in a plethora of scenarios, a presentation of these tests in action is presented below.

![tests](http://g.recordit.co/4k2DIfn4ca.gif)

The source code for these tests may be found [here](https://github.com/dirisujesse/url_expander/blob/main/test/url_expander_test.dart).

This article made a presentation on the implementation of a URL expander CLI app in Dart. I would like to thank [Kenneth Ngedo](https://stackoverflow.com/users/6859431/kenneth-ngedo) and [Samuel Abada](https://github.com/Mastersam07) for their reviews on earlier drafts of this article.

{% github dirisujesse/url_expander %}