Unprocessed data monitor in laravel.
Tags: php, laravel, tutorial
Published: 2022-04-27 20:03:31

Sometimes you need to select multiple values and process them in the background upon a button click. For example, like this screen:

![Screen where using checkboxes you are able to select an item and submit it for processing](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/kuo5u8y568rbnfw41ltg.png)

As you can see, I can select some items with specific ids and by clicking the **Process Them** button, I submit them for processing.

# Basic Algorithm

> NOTE this section contains stack - agnostic table structures and describes the general idea ignoring any table implementation.

Once I submit the data, I use the following table for the process:

```
Name: batch_process
-----------
batch_id: PK INTEGER AUTO INCREMENT
selected_ids: INTEGER ARRAY
batch_started: CURRENT TIMESTAMP (Optional Field)
batch_ended: TIMESTAMP (Optional Field)
```

Each record contains the submited ids in `selected_ids` field and a record is populated once I submit the data via post (Either via ajax or not).

Then I can dispatch a job-backround process where is processed. For each Item I process, I populate a table that has the following  structure (table structure can be modified depending the case)

```
Name: processed_items
-----
id: PK INTEGER AUTO INCREMENT
batch_id: FK references batch_process table
selected_id: INTEGER (Containing a single id from batch_process.selected_ids)
status: ENUM(FAILED, SUCCESS, PROCESSING)
error_msg: VARCHAR,
timestamp_started: TIMESTAMP
timestamp_ended: TIMESTAMP
```

Each time I process an id from a `batch_process` table record before processing it I create a record with populated these fields: 

* batch_id, With the batch Id provided
* selected_id, With an item from `batch_process.selected_ids`
* status=`PROCESSING`
* timestamp_started:  with the timestamp before record is created.

Once I finish processing the selected item I just need to update the `status` either with `SUCCESS` If the successfully has been processed or with `FAILED` if the item had fails to be processed, the failure reason is recorded in `error_msg` column. 

In case that an item either succeeds or fails to be processed, the column `timestamp_ended` is populated with the timestamp the processing has failed.

## Implementation

Once a form is submitted the following controller will be used:

```
class BatchProcessController extends Controller
{
  public function initiateBulkProcess(Request $request)
  {
    $ids = $request->post('ids');
    $now = Carbon::now()->format('Y-m-d H:i:s');

    $batch_id = DB::table('batch_process')->insertGetId([
'selected_ids' => "{".trim(implode(',',$ids),",")."}"
'batch_started' => date('Y-m-d H:i:s')
]);
    // We can also use a dedicated class for this but for me to explain it easily I use closure as a sample showcase.
    dispatch(function() use ($batch_id,$ids) {
      foreach($ids as $id){
        $log_id = DB::table('processed_items')->insertGetId([
            'batch_id' => $batch_id,
            'selected_id' => $id,
            'status' => 'PROCESSING',
            'timestamp_started' => date('Y-m-d H:i:s')
        ]);

        // Do some process on the item having $id
        $sucess = false;
        $error_msg = NULL'
        try {
          // processItem contains the logic for bulk processes 
          $sucess = prossessItem($id);
          $error_msg = $sucess ? NULL: "Uknown";
        } catch (Exception $e) {
          $sucess = false;
          $error_msg = $d->setMessage();
        }
      }
      
      $status = $sucess? "SUCESS":"FAILED";
  DB::table('processed_items')->update(['status'=>$status,'error_msg'=>$error_msg,'timestamp_ended'=>date('Y-m-d H:i:s')])->where('id',$log_id);

    });
  }

}

```

# What did we achieve

First, we managed to implement a mechanism that allows us to easily check what is processed and if the processed item is successful or not from user-initiated batch processes.

Secondly, we are able to extract more user info and statistics regarding the batch process frequency from our users. Furthermore, we can easily make watchdogs that reprocess for delayed processed items.

So we have a better logging and monitoring regarding of who processes and what is processed.