How to Turn Smartphone into a Peripheral Keyboard and Barcode QR Scanner in Flutter
Tags: flutter, dart, mobile, programming
Published: 2023-02-23 05:08:37

Barcode scanning technology is crucial for many enterprises, such as those in the supermarket, logistics, and warehousing industries. The data obtained from scanning barcodes is used to track inventory, manage orders, and process payments. Handheld barcode scanners are widely used as peripheral devices in these industries. However, traditional handheld laser barcode scanners can only detect one-dimensional codes, while more expensive scanners with an embedded camera are needed to detect two-dimensional codes like QR codes. Smartphones with built-in cameras can detect both one-dimensional and two-dimensional codes. Nowadays, smartphones have become ubiquitous. Everyone has at least one smartphone in their pocket. If smartphone can do the same job as a handheld barcode scanner, no additional hardware is needed. This article presents a solution that combines technologies like Flutter, Python, Bonjour, web sockets, PyAutoGUI, and Dynamsoft Barcode Reader SDK to help enterprises save costs by using smartphones instead of barcode scanners.

![no handheld barcode scanner](https://www.dynamsoft.com/codepool/img/2023/02/handheld-barcode-scanner.png)

## Supported Platforms
- Android
- iOS

## Demo Video: Smartphone as a Peripheral Keyboard and Barcode Scanner
The demo shows how to input text from an Android phone to Windows notepad and macOS notes simultaneously. 

<video src="https://user-images.githubusercontent.com/2202306/220560577-95e8b1a3-de36-48e1-926c-3a34a506fb82.mp4" controls="controls" muted="muted" style="max-height:640px; min-height: 200px; max-width: 100%;"></video>

## Overview of the Utilized Technologies and Why They Are Used
Before diving into the details of the solution, let’s first take a look at the technologies used in this solution.

- [Flutter](https://flutter.dev/): Flutter is a cross-platform UI framework. It enables developers build native applications for **Windows**, **Linux**, **macOS**, **Android**, **iOS** and **web** from a single codebase. Using Flutter can save a lot of time and effort for building the client user interface.
- [Python](https://www.python.org/): With Python, it only takes a few lines of code to implement network services and simulate keyboard input on operating systems.
- [Bonjour](https://developer.apple.com/bonjour/): Bonjour is a zero-configuration networking (zeroconf) protocol developed by Apple. It allows devices on the same network to discover each other. In this solution, Bonjour is used to discover the web socket server on the same network. 
- [WebSocket](https://en.wikipedia.org/wiki/WebSocket): Web sockets are a computer communications protocol that allows for bidirectional communication between a client and a server. In this solution, web sockets are used to communicate between mobile devices and PCs.
- [PyAutoGUI](https://pyautogui.readthedocs.io/en/latest/): PyAutoGUI is a Python module that can be used to automate mouse movements and keyboard presses. In this solution, PyAutoGUI is used to simulate keyboard input. 
- [Dynamsoft Barcode Reader SDK](https://www.dynamsoft.com/barcode-reader/overview/): Dynamsoft Barcode Reader SDK is a cross-platform barcode recognition SDK that can be used to recognize 1D and 2D barcodes in images. In this solution, Dynamsoft Barcode Reader SDK is used to recognize barcodes in images captured by the smartphone camera. 


## Building a Discoverable Web Socket Server with Bonjour in Python
Let's get started with the server-side implementation. The server-side implementation is written in Python. 

### Install the Required Python Modules

```bash
pip install pyautogui websockets zeroconf
```

### Get the Local IP Address in Python
There may be multiple network interfaces on your computer. A simple way to list all of them is to use the `ipconfig` command on Windows or the `ifconfig` command on Linux and macOS. The Python `subprocess.run()` method can be used to execute the command and get the output. The desired IP address can be obtained from the output by using a regular expression, such as `r'(\d+\.\d+\.\d+\.\d+)'`.

```python
import subprocess
import re
import sys

def get_ip_address():
    ip_address = ''
    if sys.platform == 'win32':
        result = subprocess.run(['ipconfig', '/all'], capture_output=True, text=True)   
    else:
        result = subprocess.run(['ifconfig'], capture_output=True, text=True)

    foundEthernet = False
    for line in result.stdout.split('\n'):
        if sys.platform == 'win32':
            match = re.search(r'Ethernet adapter Ethernet:', line)
            if match:
                foundEthernet = True
            
            if foundEthernet:
                match = re.search(r'IPv4 Address.*? : (\d+\.\d+\.\d+\.\d+)', line)
                if match:
                    ip_address = match.group(1)
                    break
        else:
            if sys.platform == "linux" or sys.platform == "linux2":
                match = re.search(r'eth0:', line)
            else:
                match = re.search(r'en0:', line)
            if match:
                foundEthernet = True
            
            if foundEthernet:
                match = re.search(r'(\d+\.\d+\.\d+\.\d+)', line)
                if match:
                    ip_address = match.group(1)
                    break

    return ip_address
```

**Note: You may need to modify the regular expression used in the above code to match the network interface name on your computer**.

### Implement the Web Socket Server
After obtaining the local IP address, the web socket server can be implemented with `asyncio` and `websockets`. 

```python
import asyncio
import websockets
import socket
import pyautogui
import signal

connected = set()
isShutdown = False

async def handle_message(websocket, path):
    async for message in websocket:
        if isinstance(message, str):
            pass
                
        elif isinstance(message, bytes):
            pass            
            

async def server(websocket, path):
    connected.add(websocket)
    try:
        await handle_message(websocket, path)
    except:
        connected.remove(websocket)
      

def ctrl_c(signum, frame):
    global isShutdown
    isShutdown = True
    
async def main(ip_address):
    global isShutdown
    # Start the server
    s = await websockets.serve(server, ip_address, 4000)

    while not isShutdown:
        await asyncio.sleep(1)
        
    s.close()
    await s.wait_closed()

signal.signal(signal.SIGINT, ctrl_c)
asyncio.run(main(ip_address_str))

```

We use `signal.signal()` to handle keyboard input for interrupting the infinite loop of the server.

### Simulate Keyboard Input with PyAutoGUI
Once a message is received from the mobile client, the server can simulate keyboard input with PyAutoGUI. 

```python
if message != '':
    if message == 'backspace':
        pyautogui.press('backspace')
    elif message == 'enter':
        pyautogui.press('enter')
    else:
        pyautogui.typewrite(message)
```

The `pyautogui.press` method can be used to simulate pressing a key on the keyboard. The `pyautogui.typewrite` method can be used to simulate typing text on the keyboard.

### Integrate Bonjour into the Web Socket Server
The last step is to make the web socket server discoverable with Bonjour. The `zeroconf` module can be used to integrate Bonjour into the web socket server. 

```python

from zeroconf import ServiceBrowser, ServiceInfo, ServiceListener, Zeroconf

class MyListener(ServiceListener):

    def update_service(self, zc: Zeroconf, type_: str, name: str) -> None:
        print(f"Service {name} updated")

    def remove_service(self, zc: Zeroconf, type_: str, name: str) -> None:
        print(f"Service {name} removed")

    def add_service(self, zc: Zeroconf, type_: str, name: str) -> None:
        info = zc.get_service_info(type_, name)
        print(f"Service {name} added, service info: {info}")

ip_address_str = get_ip_address()
ip_address = socket.inet_aton(ip_address_str)

info = ServiceInfo("_bonsoirdemo._tcp.local.", socket.gethostname().split('.')[0] +
                " Web Socket Server._bonsoirdemo._tcp.local.",
                port=7000, addresses=[ip_address])

zeroconf = Zeroconf()
# Register the Bonjour service
zeroconf.register_service(info)
listener = MyListener()
browser = ServiceBrowser(zeroconf, "_bonsoirdemo._tcp.local.", listener)

# Start the web socket server
signal.signal(signal.SIGINT, ctrl_c)
asyncio.run(main(ip_address_str))

# Unregister the Bonjour service
print("Unregistering Bonjour service")
zeroconf.unregister_service(info)
zeroconf.close()
```

The `socket.gethostname()` may return a host name that contains a `.` inside. The `.` must be filtered out, otherwise the Bonjour service name will be invalid.

## Building a Mobile App with Flutter, Bonjour, WebSocket and Dynamsoft Barcode Reader SDK
In this section, we will build a mobile app with Flutter, Bonjour, WebSocket and Dynamsoft Barcode Reader SDK. The mobile app can automatically discover the web socket server and send barcode data via the web socket connection.

### Install the Required Flutter Packages
```bash
flutter pub add bonsoir flutter_riverpod device_info web_socket_channel dynamsoft_capture_vision_flutter 
```

- For Android, change the minimum SDK version to 21 in `android/app/build.gradle`:

  ```gradle
  defaultConfig {
      minSdkVersion 21
  }
  ```

- For iOS, add the following description to `ios/Runner/Info.plist`:

  ```xml
  <key>NSCameraUsageDescription</key>
  <string>Can I use the camera please?</string>
  <key>NSLocalNetworkUsageDescription</key>
  <string>Looking for local tcp Bonjour service</string>
  <key>NSBonjourServices</key>
  <array>
    <string>_bonsoirdemo._tcp</string>
  ```

The value of `NSBonjourServices` must be the same as the service type of your Bonjour service.

### Implement Web Socket Connection and Bonjour Service Discovery in Flutter
We use Bonjour service discovery because we want to avoid manually entering the IP address of the web socket server.

The [example code of bonsoir](https://github.com/Skyost/Bonsoir/tree/master/bonsoir/example) is helpful for implementing the Bonjour service discovery. We can copy `app_service.dart`, `discovery.dart`, and `service_list.dart` files from the example code to our project, and then modify them to meet our needs.

- `app_service.dart`: Change the service type to `_bonsoirdemo._tcp`.

  ```dart

  static const String type = '_bonsoirdemo._tcp';
  ```

- `discovery.dart`: No change is needed.
- `service_list.dart`: Add an elevated button to trigger the web socket connection.

  ```dart
  class ItemWidgetState extends State<ItemWidget> {
    IOWebSocketChannel? _channel;
    String _connectAction = 'Connect';
    bool _connected = false;

    void _connect(String msg) {
      if (_connected) {
        print('disconnect to $msg');
        _channel!.sink.close(status.goingAway);
        channels.remove(_channel!);
        _connected = false;
        _connectAction = 'Connect';
        setState(() {});
        return;
      }

      _channel = IOWebSocketChannel.connect('ws://$msg');

      _channel!.ready.then((_) {
        channels.add(_channel!);
        print('connected to $msg');
        _connected = true;
        _connectAction = 'Disconnect';

        setState(() {});
        _channel!.stream.listen((message) {
          print('received: $message');
        }, onError: (error) {
          print('error: $error');
        }, onDone: () {
          if (_channel!.closeCode != null) {
            print('WebSocket is closed with code: ${_channel!.closeCode}');
          } else {
            print('WebSocket is closed');
          }
          _connected = false;
          _connectAction = 'Connect';
          setState(() {});
        });
      });
    }

    @override
    void dispose() {
      if (_channel != null) {
        _channel!.sink.close(status.goingAway);
        channels.remove(_channel!);
      }
      super.dispose();
    }

    @override
    Widget build(BuildContext context) {
      return Padding(
        padding: const EdgeInsets.all(8.0),
        child: Row(
          children: [
            Expanded(
              child: ListTile(
                title: Text(widget.service.name),
                subtitle: Text('IP : ${widget.service.ip}, Port: 4000'),
              ),
            ),
            ElevatedButton(
              onPressed: () {
                _connect('${widget.service.ip}:4000');
              },
              child: Text(_connectAction),
            ),
          ],
        ),
      );
    }
  }

  ```

### Implement Barcode QR Code Scanning in Flutter
The `dynamsoft_capture_vision_flutter` package also contains [excellent example code](https://github.com/Dynamsoft/capture-vision-flutter-samples/blob/main/barcode_reader_simple_sample/lib/main.dart), that guides you how to use the API of the Dynamsoft Barcode Reader SDK in Flutter. 

Here are the basic steps to implement barcode QR code scanning in Flutter:
1. Initialize the SDK with a [license key](https://www.dynamsoft.com/customer/license/trialLicense/?product=dbr).
    ```dart
    late final DCVCameraEnhancer _cameraEnhancer;
    late final DCVBarcodeReader _barcodeReader;

    final DCVCameraView _cameraView = DCVCameraView();

    @override
    void initState() {
      super.initState();
      WidgetsBinding.instance.addObserver(this);
      _sdkInit();
    }

    Future<void> _sdkInit() async {
      try {
        await DCVBarcodeReader.initLicense(
            'DLS2eyJoYW5kc2hha2VDb2RlIjoiMjAwMDAxLTE2NDk4Mjk3OTI2MzUiLCJvcmdhbml6YXRpb25JRCI6IjIwMDAwMSIsInNlc3Npb25QYXNzd29yZCI6IndTcGR6Vm05WDJrcEQ5YUoifQ==');
      } catch (e) {
        print(e);
      }
      _barcodeReader = await DCVBarcodeReader.createInstance();
      _cameraEnhancer = await DCVCameraEnhancer.createInstance();

      DBRRuntimeSettings currentSettings =
          await _barcodeReader.getRuntimeSettings();
      currentSettings.barcodeFormatIds =
          EnumBarcodeFormat.BF_ONED | EnumBarcodeFormat.BF_QR_CODE;

      currentSettings.expectedBarcodeCount = 0;
      await _barcodeReader
          .updateRuntimeSettingsFromTemplate(EnumDBRPresetTemplate.DEFAULT);
      await _barcodeReader.updateRuntimeSettings(currentSettings);

      _cameraEnhancer.setScanRegion(Region(
          regionTop: 30,
          regionLeft: 15,
          regionBottom: 70,
          regionRight: 85,
          regionMeasuredByPercentage: 1));

      _cameraView.overlayVisible = true;

      _cameraView.torchButton = TorchButton(
        visible: true,
      );

      await _barcodeReader.enableResultVerification(true);

      _barcodeReader.receiveResultStream().listen((List<BarcodeResult>? res) {
        if (mounted) {
          decodeRes = res ?? [];

          setState(() {});
        }
      });

      await _cameraEnhancer.open();

      _barcodeReader.startScanning();
    }
    ```

2. Add the camera view to the widget tree.
    ```dart
    Widget listItem(BuildContext context, int index) {
      BarcodeResult res = decodeRes[index];

      return ListTileTheme(
          textColor: Colors.white,
          // tileColor: Colors.green,
          child: ListTile(
            title: Text(res.barcodeFormatString),
            subtitle: Text(res.barcodeText),
          ));
    }

    @override
    Widget build(BuildContext context) {
      return Scaffold(
          appBar: AppBar(
            title: const Text('Barcode Scanner'),
          ),
          body: Stack(
            children: [
              Container(
                child: _cameraView,
              ),
              SizedBox(
                height: 100,
                child: ListView.builder(
                  itemBuilder: listItem,
                  itemCount: decodeRes.length,
                ),
              ),
            ],
          ));
    }
    ```
3. Send the barcode result to the server.
    ```dart
    Set<IOWebSocketChannel> channels = {};

    void sendMessage(String msg) {
      if (channels.isEmpty) {
        return;
      }

      for (final channel in channels) {
        channel.sink.add(msg);
      }
    }

    _barcodeReader.receiveResultStream().listen((List<BarcodeResult>? res) {
      if (mounted) {
        decodeRes = res ?? [];
        String msg = '';
        for (var i = 0; i < decodeRes.length; i++) {
          msg += '${decodeRes[i].barcodeText}\n';
        }
        if (msg.isNotEmpty && _ready) {
          _ready = false;
          sendMessage(msg);

          Future.delayed(const Duration(seconds: 2), () async {
            _ready = true;
          });
        }

        setState(() {});
      }
    });
    ```

You can now run the app on your Android or iOS device.

```bash
flutter run
```

![flutter barcode qr code scanner](https://www.dynamsoft.com/codepool/img/2023/02/flutter-barcode-qr-code-scanner.jpg)

![flutter bonjour web socket](https://www.dynamsoft.com/codepool/img/2023/02/flutter-peripheral-keyboard-qr-scanner.png)

## Source Code
[https://github.com/yushulx/flutter-peripheral-keyboard-barcode-scanner](https://github.com/yushulx/flutter-peripheral-keyboard-barcode-scanner)