Using Async in Ruby on Rails for CSV export
Tags: csv, ruby, async, webdev
Published: 2024-07-09 11:10:14

In this article, we'll go over the methods used to achieve async CSV export with Ruby on Rails.

Problem: When a large amount of data is exported and the server is unable to process it quickly after receiving client requests, a timeout error may occasionally occur with CSV exports.

The solution is to move the CSV export to a worker thread, process it there so that it doesn't slow down the main thread, and notify the client when the desired CSV file is prepared.

We implemented a general solution using the Command design pattern, which will enable us to run the same code for many types of CSV format outputs.

Technology used
- Heroku
- Ruby On Rails 
- Sidekiq as a worker thread
- Redis for keeping the state of csv export processing
- Filestack or any other cloud storage service

Here, two enums with the following structure are used.

```ruby
  EXPORTABLE_REDIS_STATUSES = {
    processing: 'Processing',
    complete: 'Processed',
  }.freeze

  EXPORTABLE_REDIS_KEYS = {
    members_csv: 'MEMBERS_CSV_GENERATORS',
    all_tasks_csv: 'ALL_TASKS_CSV',
    my_tasks_csv: 'MY_TASKS_CSV',
  }.freeze
```

Here, we have two states: `processing` and `processed`. One is used to launch a worker thread, and the other is used to produce a CSV file once the task is completed.

The second enum is only used as a filter to prevent csv exports that aren't registered with the application.

This is how the command invoker class, which acts as a sidekiq worker, looks.

```ruby
class Exports::ExportableCommandJob < ApplicationJob
  after_enqueue do |job|
    uuid = job.arguments.first[:uuid]
    redis_key = redis_collection_key(job.arguments.first[:redis_key])

    REDIS.hset(
      redis_key,
      uuid,
      { status: Constants::EXPORTABLE_REDIS_STATUSES[:processing] }.to_json
    )
  end

  def perform(uuid:, redis_key:, command:, params: {}, cleanup_interval: nil)
    params = JSON.parse(params).symbolize_keys unless params.is_a?(Hash)
    command = command.constantize.new(params)
    redis_key = redis_collection_key(redis_key)

    file_data = command.call

    tmp_file = Tempfile.new('upload', encoding: 'ascii-8bit')
    tmp_file << file_data
    tmp_file.flush
    tmp_file.rewind

    file_name = command.file_name
    uploaded_file = UploadFileService::UploadableFile.new(file: tmp_file, filename: file_name)
    details = UploadFileService.upload_file(uploaded_file)
    tmp_file.unlink

    file_path = details.metadata[:fileurl]

    generator = JSON.parse(REDIS.hget(redis_key, uuid))
    generator['status'] = Constants::EXPORTABLE_REDIS_STATUSES[:complete]
    generator['exportable'] = file_path
    REDIS.hset(redis_key, uuid, generator.to_json)
  end

  after_perform do |job|
    uuid = job.arguments.first[:uuid]
    redis_key = redis_collection_key(job.arguments.first[:redis_key])

    ExportableCleanup.set(wait: job.arguments.first[:cleanup_interval] || 1.hour)
                     .perform_later(uuid: uuid, redis_key: redis_key)
  end

  private

  def redis_collection_key(key)
    redis_key = key.to_sym
    Constants::EXPORTABLE_REDIS_KEYS[redis_key] || key
  end
end
```
We used the uuid and redis_key to set the job's status to processing after it was queuing, allowing us to monitor its progress at any time.

We accept a command class name and its arguments via params in the perform method, allowing us to invoke a function and expect the presence of CSV data. After that, we store the data in a temporary file and upload it using a fileStack service or any cloud storage service.
We obtain the file's url after putting it on the cloud, and we use it to set Redis to change the task state from `processing` to `processed`. The client can now request to get updated when the CSV export is completed and to receive generated URL for downloading.

For filtering purposes, the private method `redis_collection_key` has been used here. 

In the end `after_perform` schedules a cleanup task, as shown in this example.

```ruby
class ExportableCleanup < ApplicationJob
  def perform(uuid:, redis_key:)
    exportable_json = REDIS.hget(redis_key, uuid)
    unless exportable_json.nil?
      generator = JSON.parse(exportable_json)
      file_url = generator['exportable']
      UploadFileService.remove_file(file_url) unless file_url.blank?
    end

    REDIS.hdel(redis_key, uuid)
  end
end
```

Here it just removes data from Redis and file from cloud storage.


The invoker class call looks like this

```ruby
def export_csv_async(args, redis_key)
  uuid = SecureRandom.uuid

  Exports::ExportableCommandJob.perform_later(
    uuid: uuid,
    redis_key: redis_key,
    command: 'CsvExportDataGenerator',
    params: args.to_h.to_json,
  )

  uuid
end
```
which returns the uuid and will return it to the client so it can make the state checking calls as described above.

This is a simple action which checks process state in `Redis`
```ruby
class ExportableGeneratorsController < ActionController::API
    include HttpErrorHandling

    before_action :load_resource

    def show
      render json: { status: @generator['status'], fileUrl: @generator['exportable'] }
    end

    private

    def load_resource
      @exportable_key = Constants::EXPORTABLE_REDIS_KEYS[params[:key].to_sym]
      gen = REDIS.hget(@exportable_key, params[:uuid])
      return not_found('Process not found') if gen.nil?

      @generator = JSON.parse(gen)
    end
  end
```