Backup Service Fabric Stateful Service Data into ADL
Tags: azureservicefabric
Published: 2020-07-15 15:38:29

In this blog post, we will learn about how to restore the stateful service data from the Azure Data Lake. You might be getting a doubt about why do we need to maintain a backup for the reliable service fabric stateful service data. There might be few scenarios when we might need to back up the data.

#### Few scenarios are like

- If we need to do any maintenance activity as Service Fabric Cluster upgrade.
- If there is any change in the schema of the state object.

Let's see with a simple example. I will be using the Visual Studio to create the sample Service Fabric application.

### Prerequisites

1.  Should have an active Azure subscription and configured an Azure Data Lake(ADL) storage service to restore the data into it.

    For more info on ADL in Azure [Azure DataLake](https://azure.microsoft.com/en-in/resources/videos/creating-your-first-adls-gen2-data-lake/)

### Demo

### 1. Create the Sample Service Fabric application in Visual Studio

- Select Stateless Service Fabric application
  ![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/hbn27cwu4l5a81u3shtc.png)
- Select the API Project
  ![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/l27gyl6ky2bzfhyepvdb.png)
- Once created, Right-click on Service Fabric project and select add Service Fabric project and select Stateful Service.
  ![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/m25xq9k4imkhoc04up3b.png)

### 2. Create a Timer Adapter to inject into the Stateful service

- Interface

```csharp
    public interface ITimerAdapter
    {
        void Start();
        public double Interval { get; set; }
        event ElapsedEventHandler Elapsed;
        void Stop();
    }
```

- Implementation class

```csharp
    public class TimerAdapter : Timer, ITimerAdapter
    {
    }
```

### 3. Now we need to create the helper to to interact with ADL.

- This helper will have following methods

  - **CreateFileSystem** - Used to create the file system inside the ADL container.

  ```csharp
          public async Task CreateFileSystem()
        {
            try
            {
                DataLakeServiceClient serviceClient = GetDataLakeServiceClient();

                await serviceClient.CreateFileSystemAsync("file-system-name");
            }
            catch (RequestFailedException ex)
            {
                if (ex.Status != 409)
                    throw;
            }
        }
  ```

  - **UploadFile** - Used to upload the file which we wanted to upload into the ADL.

  ```csharp
          public async Task UploadFile(byte[] content, string directory, string fileName)
        {
            DataLakeFileSystemClient fileSystemClient = new DataLakeFileSystemClient(
                new Uri("<your ADL URI>/file-system-name"),
                new StorageSharedKeyCredential(_accountName, _accountKey));

            DataLakeDirectoryClient directoryClient = fileSystemClient.GetDirectoryClient(directory);
            DataLakeFileClient fileClient = await directoryClient.CreateFileAsync(fileName);

            MemoryStream ms = new MemoryStream(content);

            await fileClient.CreateAsync();

            if (content.Length > 0)
            {
                await fileClient.AppendAsync(ms, 0);
                await fileClient.FlushAsync(ms.Length);
            }
        }
  ```

  - **DownloadFile** - Used to download the file which we backed up in ADL and restore into state.

  ```csharp
          public async Task<byte[]> DownloadFile(string directoryName, string fileName)
        {
            DataLakeFileSystemClient fileSystemClient = new DataLakeFileSystemClient(
                new Uri("<URI of ADL>/my-file-system"),
                new StorageSharedKeyCredential(_accountName, _accountKey));

            DataLakeDirectoryClient directoryClient =  fileSystemClient.GetDirectoryClient(directoryName);

            DataLakeFileClient fileClient = directoryClient.GetFileClient(fileName);

            Response<FileDownloadInfo> downloadResponse = await fileClient.ReadAsync();

            MemoryStream ms = new MemoryStream();
            downloadResponse.Value.Content.CopyTo(ms);

            return ms.ToArray();
        }
  ```

  For more information on how to create this helper you can refer this link [.NET to manage directories, files, and ACLs in Azure Data Lake Storage Gen2](https://docs.microsoft.com/en-us/azure/storage/blobs/data-lake-storage-directory-file-acl-dotnet)

### 4. Inject both TImer Adapter and DataLake Helper into the Stateful service class

- Inject TImer Adapter and DataLake Helper into the constructor of the stateful service class.

```csharp
        private readonly ITimerAdapter _timer;
        private readonly IDataLakeHelper _dataLakeHelper;
        public Stateful1(StatefulServiceContext context, ITimerAdapter timerAdapter, IDataLakeHelper dataLakeHelper)
            : base(context)
        {
            _dataLakeHelper = dataLakeHelper ?? new DataLakeHelper();
            _timer = timerAdapter ?? new TimerAdapter();
        }
```

- The StartBackup callback method will look like below

```csharp
        private void StartBackup(object sender, System.Timers.ElapsedEventArgs e)
        {
            var emp = new Employee()
            {
                Id = Guid.NewGuid(),
                Name = "Backup Test",
                Email = "backyp@test.com"
            };

            //Convert the object into byte[].
            byte[] bytes;
            var bf = new BinaryFormatter();
            using (var ms = new MemoryStream())
            {
                bf.Serialize(ms, emp);
                bytes = ms.ToArray();
            }

            //First we need to create the FileSystem
            _dataLakeHelper.CreateFileSystem();

            //Now upload the File into that file system.
            _dataLakeHelper.UploadFile(bytes, "Employee", "Employee.txt");
        }
```

- The Restore method will look like

```csharp
        private async Task Restore()
        {
            const string directoryName = "Employee";
            const string fileName = "Employee.txt";

            //Download the file which we uploaded as backup inside the directory "Employee" with a name "Employee.txt"
            var bytes = _dataLakeHelper.DownloadFile(directoryName, fileName).Result;

            // Deserialize the bytes into the Employee Object.
            Employee emp;
            var bf = new BinaryFormatter();
            await using (var ms = new MemoryStream(bytes))
            {
                emp = (Employee) bf.Deserialize(ms);

            }

            //Restore into the reliable dictionary.
            var dictionary = await StateManager.GetOrAddAsync<IReliableDictionary<string, Employee>>("Employee");

            using var tx = StateManager.CreateTransaction();
            await dictionary.AddOrUpdateAsync(tx, emp.Id.ToString(), emp, (s, d) => emp);

            await tx.CommitAsync();
        }
```

- Use RunAsync Method of the Stateful service to register the Timer. Also start the timer and create a event handler to call when the timer elapsed. Also use the Restore to restore into the state.

```csharp
        protected override async Task RunAsync(CancellationToken cancellationToken)
        {
            await Restore();

            _timer.Interval = 3000; //This is in MillSeconds
            _timer.Start();
            _timer.Elapsed += StartBackup;
        }
```