How to Create a Flutter Document Rectification Plugin for Android and iOS
Tags: flutter, android, ios, programming
Published: 2023-02-10 07:47:58

Previously, we created a Flutter document rectification plugin for web, Windows, and Linux. In this article, we will add support for Android and iOS. As a result, you will be able to create a document rectification app that corrects the perspective of an image of a document on all platforms.

## Flutter Document Rectification SDK
[https://pub.dev/packages/flutter_document_scan_sdk](https://pub.dev/packages/flutter_document_scan_sdk)

## Development Environment
- Flutter 3.3.9
- Minimum Android SDK Version: 21
- Swift 5.0

## Add Support for Android and iOS in Flutter Plugin
To add support for Android and iOS, run the following command in the root directory of the project:

```bash
flutter create --org com.dynamsoft --template=plugin --platforms=android,ios .
```

After generating the platform-specific code, update the `pubspec.yaml` file:

```yaml
plugin:
    platforms:
      android:
        package: com.dynamsoft.flutter_document_scan_sdk
        pluginClass: FlutterDocumentScanSdkPlugin
      ios:
        pluginClass: FlutterDocumentScanSdkPlugin
      linux:
        pluginClass: FlutterDocumentScanSdkPlugin
      windows:
        pluginClass: FlutterDocumentScanSdkPluginCApi
      web:
        pluginClass: FlutterDocumentScanSdkWeb
        fileName: flutter_document_scan_sdk_web.dart
```

Since the Dart code and API remain unchanged, we only need to implement the native code for Android and iOS.


## Linking Third-party Libraries for Android and iOS in a Flutter Plugin

### Configure the Dynamsoft Document Normalizer SDK in Gradle and CocoaPods
For Android, open the `android/build.gradle` file and add the following dependency:

```gradle
rootProject.allprojects {
    repositories {
        maven {
            url "https://download2.dynamsoft.com/maven/aar"
        }
        google()
        mavenCentral()
    }
}

dependencies { 
    implementation 'com.dynamsoft:dynamsoftdocumentnormalizer:1.0.20' 
}
```

For iOS, open the `ios/flutter_document_scan_sdk.podspec` file and add the following dependency:

```ruby
s.dependency 'DynamsoftDocumentNormalizer', '1.0.20'
```

### Write Platform-Specific Code in Java and Swift

We write Java code in the `FlutterDocumentScanSdkPlugin.java` file and Swift code in the `SwiftFlutterDocumentScanSdkPlugin.swift` file. The SDK consists of two packages: `DynamsoftCore` and `DynamsoftDocumentNormalizer`. The `DynamsoftCore package` contains the classes and interfaces related to license management and image data, and the `DynamsoftDocumentNormalizer` package contains the classes and interfaces for document rectification.

1. Import the relevant SDK packages and classes:

    **Android**

    ```java
    import com.dynamsoft.core.ImageData;
    import com.dynamsoft.core.CoreException;
    import com.dynamsoft.core.LicenseManager;
    import com.dynamsoft.core.LicenseVerificationListener;
    import com.dynamsoft.core.EnumImagePixelFormat;
    import com.dynamsoft.core.Quadrilateral;

    import com.dynamsoft.ddn.DocumentNormalizer;
    import com.dynamsoft.ddn.DetectedQuadResult;
    import com.dynamsoft.ddn.DocumentNormalizerException;
    import com.dynamsoft.ddn.NormalizedImageResult;
    ```

    **iOS**
      
    ```swift
    import DynamsoftCore
    import DynamsoftDocumentNormalizer
    ```

2. Create an instance of the `DocumentNormalizer` class:

    **Android**

    ```java
    private DocumentNormalizer mNormalizer;
    try {
        mNormalizer = new DocumentNormalizer();
    } catch (DocumentNormalizerException e) {
        e.printStackTrace();
    }
    ```

    **iOS**

    ```swift
    var normalizer: DynamsoftDocumentNormalizer = DynamsoftDocumentNormalizer()
    ```

3. Set a [license key](https://www.dynamsoft.com/customer/license/trialLicense/?product=ddn) to activate the SDK:

    **Android**

    ```java
    LicenseManager.initLicense(
            license, activity,
                new LicenseVerificationListener() {
                    @Override
                    public void licenseVerificationCallback(boolean isSuccessful, CoreException e) {
                        if (isSuccessful)
                        {
                            result.success(0);
                        }
                        else {
                            result.success(-1);
                        }
                    }
                });
    ```

    The `initLicense()` method's second parameter is the `Activity` object. To obtain the `Activity` object in the Flutter Android plugin, we use the `ActivityAware` interface. The `FlutterDocumentScanSdkPlugin` class implements the `ActivityAware` interface. The `onAttachedToActivity` method is called when the `Activity` object is created, and the `onDetachedFromActivity` method is called when the `Activity` object is destroyed.

    ```java
    public class FlutterDocumentScanSdkPlugin implements FlutterPlugin, MethodCallHandler, ActivityAware {
    
        private void bind(ActivityPluginBinding activityPluginBinding) {
          activity = activityPluginBinding.getActivity();
        }

        @Override
        public void onAttachedToActivity(ActivityPluginBinding activityPluginBinding) {
          bind(activityPluginBinding);
        }

        @Override
        public void onDetachedFromActivity() {
          activity = null;
        }
    }
    
    ```

    **iOS**  

    ```swift
    public class SwiftFlutterDocumentScanSdkPlugin: NSObject, FlutterPlugin, LicenseVerificationListener {

        ...
        DynamsoftLicenseManager.initLicense(license, verificationDelegate: self)
        ...

        public func licenseVerificationCallback(_ isSuccess: Bool, error: Error?) {
            if isSuccess {
                completionHandlers.first?(0)
            } else{
                completionHandlers.first?(-1)
            }
        }
    }
    
    ```

4. Before calling the detection method, we can retrieve and configure [parameters](https://www.dynamsoft.com/document-normalizer/docs/parameters/parameter-organization-structure.html?ver=latest) for the detection algorithm.

    **Android**  
    ```java
    // Get parameters
    try {
            parameters = mNormalizer.outputRuntimeSettings("");
    } catch (Exception e) {}

    // Set parameters
    try {
            mNormalizer.initRuntimeSettingsFromString(params);
    } catch (DocumentNormalizerException e) {}
    ```

    **iOS**  
    ```swift
    // Get parameters
    let parameters = try? self.normalizer!.outputRuntimeSettings("")

    // Set parameters
    try? self.normalizer!.initRuntimeSettingsFromString(params)
    ```

5. Call the `detectQuad()` method to identify the document edge:

    **Android**
    ```java
    DetectedQuadResult[] detectedResults = mNormalizer.detectQuad(filename);
    if (detectedResults != null && detectedResults.length > 0) {
        for (int i = 0; i < detectedResults.length; i++) {
            Map<String, Object> map = new HashMap<>();

            DetectedQuadResult detectedResult = detectedResults[i];
            int confidence = detectedResult.confidenceAsDocumentBoundary;
            Point[] points = detectedResult.location.points;
            int x1 = points[0].x;
            int y1 = points[0].y;
            int x2 = points[1].x;
            int y2 = points[1].y;
            int x3 = points[2].x;
            int y3 = points[2].y;
            int x4 = points[3].x;
            int y4 = points[3].y;

            map.put("confidence", confidence);
            map.put("x1", x1);
            map.put("y1", y1);
            map.put("x2", x2);
            map.put("y2", y2);
            map.put("x3", x3);
            map.put("y3", y3);
            map.put("x4", x4);
            map.put("y4", y4);
            
            out.add(map);
        }
    }
    ```

    **iOS**  
    ```swift
    let detectedResults = try? self.normalizer!.detectQuadFromFile(filename)
                
    if detectedResults != nil {
        for result in detectedResults! {
            let dictionary = NSMutableDictionary()
            
            let confidence = result.confidenceAsDocumentBoundary
            let points = result.location.points as! [CGPoint]
            
            dictionary.setObject(confidence, forKey: "confidence" as NSCopying)
            dictionary.setObject(Int(points[0].x), forKey: "x1" as NSCopying)
            dictionary.setObject(Int(points[0].y), forKey: "y1" as NSCopying)
            dictionary.setObject(Int(points[1].x), forKey: "x2" as NSCopying)
            dictionary.setObject(Int(points[1].y), forKey: "y2" as NSCopying)
            dictionary.setObject(Int(points[2].x), forKey: "x3" as NSCopying)
            dictionary.setObject(Int(points[2].y), forKey: "y3" as NSCopying)
            dictionary.setObject(Int(points[3].x), forKey: "x4" as NSCopying)
            dictionary.setObject(Int(points[3].y), forKey: "y4" as NSCopying)
            
            out.add(dictionary)
        }
    }
    result(out)
    ```

6. Call the `normalizeFile()` method to crop the document based on its corners and correct its perspective:

    **Android**
    ```java
    Quadrilateral quad = new Quadrilateral();
    quad.points = new Point[4];
    quad.points[0] = new Point(x1, y1);
    quad.points[1] = new Point(x2, y2);
    quad.points[2] = new Point(x3, y3);
    quad.points[3] = new Point(x4, y4);
    mNormalizedImage = mNormalizer.normalize(filename, quad);

    if (mNormalizedImage != null) {
      ImageData imageData = mNormalizedImage.image;
      int width = imageData.width;
      int height = imageData.height;
      int stride = imageData.stride;
      int format = imageData.format;
      byte[] data = imageData.bytes;
      int length = imageData.bytes.length;
      int orientation = imageData.orientation;
    }
    ```

    **iOS**  
    ```swift
    let points = [CGPoint(x: x1, y: y1), CGPoint(x: x2, y: y2), CGPoint(x: x3, y: y3), CGPoint(x: x4, y: y4)]
    let quad = iQuadrilateral()
    quad.points = points

    if self.normalizedImage != nil {
        let imageData: iImageData = self.normalizedImage!.image
        let width = imageData.width
        let height = imageData.height
        let stride = imageData.stride
        let format = imageData.format
        let data = imageData.bytes
        let length = data!.count
        let orientation = imageData.orientation
    }
    ```

7. To transfer the image data of the rectified document to the Flutter side, we need to convert the image data (such as `RGB888`, `Grayscale` or `binary` ) to a `RGBA` byte array.
    - `RGB888`: A pixel is represented by three bytes. The order of the bytes is `R`, `G`, and `B`. 
    - `Grayscale`: Every byte represents the grayscale value of a pixel.
    - `Binary`: Every byte represents 8 pixels. The value of each bit is `0` or `1`. 

    **Android**
    ```java
    byte[] rgba = new byte[width * height * 4];
    
    if (format == EnumImagePixelFormat.IPF_RGB_888) {
      int dataIndex = 0;
      for (int i = 0; i < height; i++)
      {
          for (int j = 0; j < width; j++)
          {
              int index = i * width + j;

              rgba[index * 4] = data[dataIndex + 2];     // red
              rgba[index * 4 + 1] = data[dataIndex + 1]; // green
              rgba[index * 4 + 2] = data[dataIndex];     // blue
              rgba[index * 4 + 3] = (byte)255;                 // alpha
              dataIndex += 3;
          }
      }
    }
    else if (format == EnumImagePixelFormat.IPF_GRAYSCALED) {
      int dataIndex = 0;
      for (int i = 0; i < height; i++)
      {
          for (int j = 0; j < width; j++)
          {
              int index = i * width + j;
              rgba[index * 4] = data[dataIndex];
              rgba[index * 4 + 1] = data[dataIndex];
              rgba[index * 4 + 2] = data[dataIndex];
              rgba[index * 4 + 3] = (byte)255;
              dataIndex += 1;
          }
      }
    }
    else if (format == EnumImagePixelFormat.IPF_BINARY) {
      byte[] grayscale = new byte[width * height];
      binary2grayscale(data, grayscale, width, height, stride, length);

      int dataIndex = 0;
      for (int i = 0; i < height; i++)
      {
          for (int j = 0; j < width; j++)
          {
              int index = i * width + j;
              rgba[index * 4] = grayscale[dataIndex];
              rgba[index * 4 + 1] = grayscale[dataIndex];
              rgba[index * 4 + 2] = grayscale[dataIndex];
              rgba[index * 4 + 3] = (byte)255;
              dataIndex += 1;
          }
      }
    }

    void binary2grayscale(byte[] data, byte[] output, int width, int height, int stride, int length) {
      int index = 0;

      int skip = stride * 8 - width;
      int shift = 0;
      int n = 1;

      for (int i = 0; i < length; ++i)
      {
          byte b = data[i];
          int byteCount = 7;
          while (byteCount >= 0)
          {
              int tmp = (b & (1 << byteCount)) >> byteCount;

              if (shift < stride * 8 * n - skip)
              {
                  if (tmp == 1)
                      output[index] = (byte)255;
                  else
                      output[index] = 0;
                  index += 1;
              }

              byteCount -= 1;
              shift += 1;
          }

          if (shift == stride * 8 * n)
          {
              n += 1;
          }
      }
    }
    ```

    **iOS**  
    ```swift
    var rgba: [UInt8] = [UInt8](repeating: 0, count: width * height * 4)
                    
    if format == EnumImagePixelFormat.RGB_888 {
        var dataIndex = 0
        for i in 0..<height {
            for j in 0..<width {
                let index = i * width + j
                rgba[index * 4] = data![dataIndex + 2]     // red
                rgba[index * 4 + 1] = data![dataIndex + 1] // green
                rgba[index * 4 + 2] = data![dataIndex]     // blue
                rgba[index * 4 + 3] = 255                 // alpha
                dataIndex += 3
            }
        }
    }
    else if (format == EnumImagePixelFormat.grayScaled) {
        var dataIndex = 0
        for i in 0..<height {
            for j in 0..<width {
                let index = i * width + j
                rgba[index * 4] = data![dataIndex]
                rgba[index * 4 + 1] = data![dataIndex]
                rgba[index * 4 + 2] = data![dataIndex]
                rgba[index * 4 + 3] = 255
                dataIndex += 1
            }
        }
    }
    else if (format == EnumImagePixelFormat.binary) {
        var grayscale: [UInt8] = [UInt8](repeating: 0, count: width * height)
        
        var index = 0
        let skip = stride * 8 - width
        var shift = 0
        var n = 1
        
        for i in 0..<length {
            let b = data![i]
            var byteCount = 7
            while byteCount >= 0 {
                let tmp = (b & (1 << byteCount)) >> byteCount
                
                if (shift < stride * 8 * n - skip)
                {
                    if (tmp == 1) {
                        grayscale[index] = 255
                    }
                    else {
                        grayscale[index] = 0
                    }
                    index += 1
                }
                
                byteCount -= 1
                shift += 1
            }
            
            if (shift == stride * 8 * n)
            {
                n += 1
            }
        }
        
        var dataIndex = 0
        for i in 0..<height {
            for j in 0..<width {
                let index = i * width + j
                rgba[index * 4] = grayscale[dataIndex]
                rgba[index * 4 + 1] = grayscale[dataIndex]
                rgba[index * 4 + 2] = grayscale[dataIndex]
                rgba[index * 4 + 3] = 255
                dataIndex += 1
            }
        }
    }
    ```

## Test the Flutter Document Rectification Plugin on both Android and iOS

```bash
flutter run
```

**Document Edge Detection**

![Flutter document edge detection for Android and iOS](https://www.dynamsoft.com/codepool/img/2023/02/flutter-document-rectification-android-ios.jpg)


**Document Perspective Correction**

![Flutter document perspective correction for Android and iOS](https://www.dynamsoft.com/codepool/img/2023/02/flutter-document-rectification-correction-android-ios.jpg)

## Source Code
[https://github.com/yushulx/flutter_document_scan_sdk](https://github.com/yushulx/flutter_document_scan_sdk)