A Concise Guide to Using Multiple Azure Storage Accounts from a Single Spring Boot Application
Tags: azure, springboot, azurestorage, java
Published: 2022-07-18 20:36:16

Spring projects in general are opinionated: 80-90% of use cases are handled "by default", and code is often much more concise than would be required otherwise due to Spring's preference of convention over configuration. These and other "opinions" can result in dramatically less code to write and maintain and as a result, more focused impact.

In the vast majority of cases where Azure Storage is used from an application, there is no compelling advantage to using more than a single Azure storage account. But there are edge cases, and having the ability to use multiple Azure Storage accounts from a single app - even if we might only need that capability around 10% of the time - could provide an incredibly useful extension of our storage superpowers.

This article is the result of a collaboration with Shi li Chen.

## It's all about resources

The Spring Framework defines the `Resource` interface and provides several implementations built upon `Resource` to facilitate developer access to low-level resources. In order to handle a particular kind of resource, two things are required:

- A `Resource` implementation
- A `ResourcePatternResolver` implementation

A Spring application evaluates resources in question using one or more registered resolvers. When the type of resource is identified, the appropriate `Resource` implementation is used to access and/or manipulate the underlying resource.

If the implementations built into Spring Framework don't fulfill your use case, it's fairly straightforward to add support for additional types of resources by defining your own implementations of `AbstractResource` and `ResourcePatternResolver` interfaces.

This article will introduce the Spring Resource, review Spring Cloud Azure's implementation of Spring's `Resource` (especially with regard to Azure Storage Account considerations and limitations), and consider how to expand said implementation to address those edge cases in which it would be useful to access multiple Azure Storage Accounts from a single Spring Boot application.

## Getting resourceful

We've already mentioned that the Spring Framework defines several useful [Resource implementations](https://docs.spring.io/spring-framework/docs/current/reference/html/core.html#resources-implementations). As of this writing, the default types are:

- `UrlResource`
- `ClassPathResource`
- `FileSystemResource`
- `PathResource`
- `ServletContextResource`
- `InputStreamResource`
- `ByteArrayResource`

As mentioned earlier, each resource will have a corresponding resource resolver.

Enabling your Spring Boot application to use a custom `Resource` requires the following actions:

- Implement the `Resource` interface by extending `AbstractResource`
- Implement the `ResourcePatternResolver` interface to resolve the custom resource type
- Register the implementation of `ResourcePatternResolver` as a bean

NOTE: Your resolver must be added to the default resource loader's resolver set using the `org.springframework.core.io.DefaultResourceLoader#addProtocolResolver` method, but this code is present in `AbstractAzureStorageProtocolResolver`; extending that class to create your implementation accomplishes this on your behalf unless you choose to override its `setResourceLoader` method.

A `ResourceLoader` attempts to resolve each `Resource` by comparing its defined location/format with all registered protocol pattern resolvers until a non-null resource is returned. If no match is found, the `Resource` will be evaluated against Spring's built-in pattern resolvers.

## Spring resources in Spring Cloud Azure

Spring Cloud Azure provides two Spring resource and resource pattern resolver implementations. In this article, we only discuss the implementation of the Azure Storage Blob resource. You can examine the source code for Spring Cloud Azure `Resources` at [Spring Cloud Azure](https://github.com/Azure/azure-sdk-for-java/tree/spring-cloud-azure-starter-storage-blob_4.2.0/sdk/spring/spring-cloud-azure-core/src/main/java/com/azure/spring/cloud/core/resource) and related documentation at [Resource Handling](https://microsoft.github.io/spring-cloud-azure/4.2.0/reference/html/index.html#resource-handling).

NOTE: We use [Spring Cloud Azure Starter Storage Blob version 4.2.0](https://mvnrepository.com/artifact/com.azure.spring/spring-cloud-azure-starter-storage-blob/4.2.0) for analysis and experiments.

## Implementation of `AbstractResource`

The abstract implementation `AzureStorageResource` for Spring Cloud Azure primarily defines the format of the Azure storage resource protocol and accommodates the unique attributes of the Azure Storage Account service, e.g. the container name and file name. It is important to note that `AzureStorageResource` is decoupled from the Azure Storage SDK.

The Spring Framework interface `WritableResource` represents the underlying API we build upon to read from and write to the Azure Storage resource.

```
abstract class AzureStorageResource extends AbstractResource implements WritableResource {

    private boolean isAzureStorageResource(@NonNull String location) {
        ......
    }

    String getContainerName(String location) {
        ......
    }

    String getContentType(String location) {
        ......
    }

    String getFilename(String location) {
        ......
    }

    abstract StorageType getStorageType();
}
```

The `StorageBlobResource` is Spring Cloud Azure Storage Blob's implementation of the abstract class `AbstractResource`.
We can see `StorageBlobResource` uses the `BlobServiceClient` from the Azure Storage Blob SDK to implement all abstract methods, relying on the service client to interact with the Azure Storage Blob service.

```
public final class StorageBlobResource extends AzureStorageResource {
    private final BlobServiceClient blobServiceClient;
    private final BlobContainerClient blobContainerClient;
    private final BlockBlobClient blockBlobClient;
    
    public StorageBlobResource(BlobServiceClient blobServiceClient, String location, Boolean autoCreateFiles,
                               String snapshot, String versionId, String contentType) {
        ......
        this.blobContainerClient = blobServiceClient.getBlobContainerClient(getContainerName(location));
        BlobClient blobClient = blobContainerClient.getBlobClient(getFilename(location));
        this.blockBlobClient = blobClient.getBlockBlobClient();
    }

    @Override
    public OutputStream getOutputStream() throws IOException {
        try {
            ......
            return this.blockBlobClient.getBlobOutputStream(options);
        } catch (BlobStorageException e) {
            throw new IOException(MSG_FAIL_OPEN_OUTPUT, e);
        }
    }
    
    ......
    
    @Override
    StorageType getStorageType() {
        return StorageType.BLOB;
    }
}
```

## Implementation of `ResourcePatternResolver`

Spring Cloud Azure provides an abstract implementation `AbstractAzureStorageProtocolResolver`. This class incorporates general processing of the Azure storage resource protocol, exposes specific capabilities of the Azure Storage Account service, and adds the requisite logic to the default resource loader. Like `AzureStorageResource`, the `AbstractAzureStorageProtocolResolver` is also not coupled to the Azure Storage SDK.

```
public abstract class AbstractAzureStorageProtocolResolver implements ProtocolResolver, ResourcePatternResolver,
    ResourceLoaderAware, BeanFactoryPostProcessor {

    protected final AntPathMatcher matcher = new AntPathMatcher();

    protected abstract StorageType getStorageType();

    protected abstract Resource getStorageResource(String location, Boolean autoCreate);

    protected ConfigurableListableBeanFactory beanFactory;

    protected abstract Stream<StorageContainerItem> listStorageContainers(String containerPrefix);

    protected abstract StorageContainerClient getStorageContainerClient(String name);

    @Override
    public void setResourceLoader(ResourceLoader resourceLoader) {
        if (resourceLoader instanceof DefaultResourceLoader) {
            ((DefaultResourceLoader) resourceLoader).addProtocolResolver(this);
        } else {
            LOGGER.warn("Custom Protocol using azure-{}:// prefix will not be enabled.", getStorageType().getType());
        }
    }

    @Override
    public void postProcessBeanFactory(ConfigurableListableBeanFactory beanFactory) throws BeansException {
        this.beanFactory = beanFactory;
    }

    @Override
    public Resource resolve(String location, ResourceLoader resourceLoader) {
        if (AzureStorageUtils.isAzureStorageResource(location, getStorageType())) {
            return getResource(location);
        }
        return null;
    }

    @Override
    public Resource[] getResources(String pattern) throws IOException {
        Resource[] resources = null;

        if (AzureStorageUtils.isAzureStorageResource(pattern, getStorageType())) {
            if (matcher.isPattern(AzureStorageUtils.stripProtocol(pattern, getStorageType()))) {
                String containerPattern = AzureStorageUtils.getContainerName(pattern, getStorageType());
                String filePattern = AzureStorageUtils.getFilename(pattern, getStorageType());
                resources = resolveResources(containerPattern, filePattern);
            } else {
                return new Resource[] { getResource(pattern) };
            }
        }
        if (null == resources) {
            throw new IOException("Resources not found at " + pattern);
        }
        return resources;
    }

    @Override
    public Resource getResource(String location) {
        Resource resource = null;

        if (AzureStorageUtils.isAzureStorageResource(location, getStorageType())) {
            resource = getStorageResource(location, true);
        }

        if (null == resource) {
            throw new IllegalArgumentException("Resource not found at " + location);
        }
        return resource;
    }

    /**
     * Storage container item.
     */
    protected static class StorageContainerItem {
        private final String name;
        ......
    }

    protected static class StorageItem {

        private final String container;
        private final String name;
        private final StorageType storageType;
        ......
    }

    protected interface StorageContainerClient {
        
        ......
    }
}
```

The resource resolver `AzureStorageBlobProtocolResolver` is Spring Cloud Azure Storage Blob's implementation of `ResourcePatternResolver`. It encapsulates resources according to the location or storage item pattern based on `BlobServiceClient` and returns the associated `StorageBlobResource`.

```
public final class AzureStorageBlobProtocolResolver extends AbstractAzureStorageProtocolResolver {

    private BlobServiceClient blobServiceClient;
    @Override
    protected StorageType getStorageType() {
        return StorageType.BLOB;
    }

    @Override
    protected Resource getStorageResource(String location, Boolean autoCreate) {
        return new StorageBlobResource(getBlobServiceClient(), location, autoCreate);
    }

    private BlobServiceClient getBlobServiceClient() {
        if (blobServiceClient == null) {
            blobServiceClient = beanFactory.getBean(BlobServiceClient.class);
        }
        return blobServiceClient;
    }
}
```

## Opinions

As mentioned at the beginning of this post, the default capabilities fulfill the requirements admirably in the vast majority of circumstances. But in accordance with the Spring ethos, Spring Cloud Azure Starter Storage Blob was designed to seamlessly address 80-90% of use cases "out of the box", while still allowing for remaining (edge) cases with some extra effort.

As written, the storage blob resource supports multiple container operations using the same storage account. The salient point is that the blob paths under different containers can be properly resolved into `StorageBlobResource` objects. Combining the earlier code for `StorageBlobResource`, the blob resource must hold a blob service client, and if `blobServiceClient.getBlobContainerClient(getContainerName(location))` successfully returns a `BlobServiceClient`, the blob resource can be resolved and retrieved.

The `BlobServiceClient` bean represents an Azure Storage Account in the Azure Storage Blob SDK, meaning that the current implementation does not support simultaneous availability using multiple Azure Storage Accounts.

## Developing an extended version of Spring Cloud Azure Starter Storage Blob

For those rare cases in which it might be useful to simultaneously access multiple Azure Storage accounts from the same application, there is a way to make that happen. To demonstrate this capability, let's create a new library called `spring-cloud-azure-starter-storage-blob-extend`. The only external dependency for this new library is the existing `spring-cloud-azure-starter-storage-blob`.

## Extend the Storage Blob properties

While the primary goal is to support multiple storage accounts, a secondary design goal is to use a similar structure to `AzureStorageBlobProperties` in order to minimize the learning curve and to retain Spring Cloud Azure 4.0's out of the box authentication features.

```
public class ExtendAzureStorageBlobsProperties {

    public static final String PREFIX = "spring.cloud.azure.storage.blobs";

    private boolean enabled = true;

    private final List<AzureStorageBlobProperties> configurations = new ArrayList<>();

    public boolean isEnabled() {
        return enabled;
    }

    public void setEnabled(boolean enabled) {
        this.enabled = enabled;
    }

    public List<AzureStorageBlobProperties> getConfigurations() {
        return configurations;
    }
}
```

## Dynamically register Storage Blob beans

Since there will be multiple Storage Account configurations, we must name the beans corresponding to each storage account. The cleanest approach is to simply use the account name as the bean name.

Now, let's dynamically register these beans with the Spring context.

```
@Configuration(proxyBeanMethods = false)
@ConditionalOnProperty(value = { "spring.cloud.azure.storage.blobs.enabled"}, havingValue = "true")
public class ExtendStorageBlobsAutoConfiguration implements BeanDefinitionRegistryPostProcessor, EnvironmentAware {

    private Environment environment;

    public static final String EXTEND_STORAGE_BLOB_PROPERTIES_BEAN_NAME = "extendAzureStorageBlobsProperties";

    @Override
    public void postProcessBeanFactory(ConfigurableListableBeanFactory beanFactory) throws BeansException {
        AzureGlobalProperties azureGlobalProperties =
            Binder.get(environment)
                  .bind(AzureGlobalProperties.PREFIX, AzureGlobalProperties.class)
                  .orElse(new AzureGlobalProperties());
        ExtendAzureStorageBlobsProperties blobsProperties =
            Binder.get(environment)
                  .bind(ExtendAzureStorageBlobsProperties.PREFIX, ExtendAzureStorageBlobsProperties.class)
                  .orElseThrow(() -> new IllegalArgumentException("Can not bind the azure storage blobs properties."));
        // merge properties
        for (AzureStorageBlobProperties azureStorageBlobProperties : blobsProperties.getConfigurations()) {
            AzureStorageBlobProperties transProperties = new AzureStorageBlobProperties();
            AzureGlobalPropertiesUtils.loadProperties(azureGlobalProperties, transProperties);
            copyAzureCommonPropertiesIgnoreTargetNull(transProperties, azureStorageBlobProperties);
        }

        DefaultListableBeanFactory factory = (DefaultListableBeanFactory) beanFactory;
        registryBeanExtendAzureStorageBlobsProperties(factory, blobsProperties);
        blobsProperties.getConfigurations().forEach(blobProperties -> registryBlobBeans(factory, blobProperties));
    }

    private void registryBeanExtendAzureStorageBlobsProperties(DefaultListableBeanFactory beanFactory,
                                                               ExtendAzureStorageBlobsProperties blobsProperties) {
        BeanDefinitionBuilder beanDefinitionBuilder = BeanDefinitionBuilder.genericBeanDefinition(ExtendAzureStorageBlobsProperties.class,
            () -> blobsProperties);
        AbstractBeanDefinition rawBeanDefinition = beanDefinitionBuilder.getRawBeanDefinition();
        beanFactory.registerBeanDefinition(EXTEND_STORAGE_BLOB_PROPERTIES_BEAN_NAME, rawBeanDefinition);
    }

    private void registryBlobBeans(DefaultListableBeanFactory beanFactory, AzureStorageBlobProperties blobProperties) {
        String accountName = getStorageAccountName(blobProperties);
        Assert.hasText(accountName, "accountName can not be null or empty.");
        registryBeanStaticConnectionStringProvider(beanFactory, blobProperties, accountName);
        registryBeanBlobServiceClientBuilderFactory(beanFactory, blobProperties, accountName);
        registryBeanBlobServiceClientBuilder(beanFactory, accountName);
        registryBeanBlobServiceClient(beanFactory, accountName);
        registryBeanBlobContainerClient(beanFactory, blobProperties, accountName);
        registryBeanBlobClient(beanFactory, blobProperties, accountName);
    }

    private void registryBeanBlobServiceClientBuilder(DefaultListableBeanFactory beanFactory,
                                                      String accountName) {
        BeanDefinitionBuilder beanDefinitionBuilder = BeanDefinitionBuilder.genericBeanDefinition(BlobServiceClientBuilder.class,
            () -> {
                BlobServiceClientBuilderFactory builderFactory =
                    beanFactory.getBean(accountName + BlobServiceClientBuilderFactory.class.getSimpleName(),
                        BlobServiceClientBuilderFactory.class);
                return builderFactory.build();
            });
        AbstractBeanDefinition rawBeanDefinition = beanDefinitionBuilder.getRawBeanDefinition();
        beanFactory.registerBeanDefinition(
            accountName + BlobServiceClientBuilder.class.getSimpleName(), rawBeanDefinition);
    }
    
    ......
    
    @Override
    public void setEnvironment(Environment environment) {
        this.environment = environment;
    }
}
```

## Extend the `AzureStorageBlobProtocolResolver`

The next task is to make any container resolvable by the same resource pattern resolver. Specifying a storage blob resource location such as **azure-blob-accountname://containername/test.txt**, the resolver will use that to locate the appropriate `BlobServiceClient` bean by Azure Storage Account name and return the storage resource.

```
public class ExtendAzureStorageBlobProtocolResolver extends ExtendAbstractAzureStorageProtocolResolver {
    private final Map<String, BlobServiceClient> blobServiceClientMap = new HashMap<>();

    @Override
    protected Resource getStorageResource(String location, Boolean autoCreate) {
        return new ExtendStorageBlobResource(getBlobServiceClient(location), location, autoCreate);
    }

    private BlobServiceClient getBlobServiceClient(String locationPrefix) {
        String storageAccount = ExtendAzureStorageUtils.getStorageAccountName(locationPrefix, getStorageType());
        Assert.notNull(storageAccount, "storageAccount can not be null.");
        String accountKey = storageAccount.toLowerCase(Locale.ROOT);
        if (blobServiceClientMap.containsKey(accountKey)) {
            return blobServiceClientMap.get(accountKey);
        }

        BlobServiceClient blobServiceClient = beanFactory.getBean(
            accountKey + BlobServiceClient.class.getSimpleName(), BlobServiceClient.class);
        Assert.notNull(blobServiceClient, "blobServiceClient can not be null.");
        blobServiceClientMap.put(accountKey, blobServiceClient);
        return blobServiceClient;
    }
}
```

Again, you need to add the bean `ExtendAzureStorageBlobProtocolResolver` to the Spring context.

## Testing the Spring Cloud Azure Starter Storage Blob Extend

You can use [start.spring.io](https://start.spring.io/) to generate a Spring Boot 2.6.7 or greater project with Azure Storage support (or build on [this storage blob sample](https://github.com/Azure-Samples/azure-spring-boot-samples/tree/spring-cloud-azure_v4.2.0/storage/spring-cloud-azure-starter-storage-blob/storage-blob-sample) if you prefer).

Add the extending starter dependency to the _pom.xml_ file:

```
<dependency>
  <groupId>com.azure.spring.extend</groupId>
  <artifactId>spring-cloud-azure-starter-storage-blob-extend</artifactId>
  <version>1.0-SNAPSHOT</version>
</dependency>
```

Delete the _src/main/resources/application.properties_ file or add the following configuration file _application-extend.yml_, which enables multiple storage account usage:

_application-extend.yml_

```
spring:
  cloud:
    azure:
      storage:
        blob:
          enabled: false
        blobs:
          enabled: true
          configurations:
            - account-name: ${FIRST_ACCOUNT}
              container-name: ${FIRST_CONTAINER}
              account-key: ${ACCOUNT_KEY_OF_FIRST_ACCOUNT}
            - account-name: ${SECOND_ACCOUNT}
              container-name: ${SECOND_CONTAINER}
              account-key: ${ACCOUNT_KEY_OF_SECOND_ACCOUNT}
```

NOTE: You must provide values for the environment variables above (listed in all capital letters) with active Azure Storage Account resource information.

Add class `com.azure.spring.extend.sample.storage.resource.extend.SampleDataInitializer` with the following body:

```
@Profile("extend")
@Component
public class SampleDataInitializer implements CommandLineRunner {
    final static Logger logger = LoggerFactory.getLogger(SampleDataInitializer.class);

    private final ConfigurableEnvironment env;

    private final ExtendAzureStorageBlobProtocolResolver resolver;
    private final ExtendAzureStorageBlobsProperties properties;

    public SampleDataInitializer(ConfigurableEnvironment env, ExtendAzureStorageBlobProtocolResolver resolver,
                                 ExtendAzureStorageBlobsProperties properties) {
        this.env = env;
        this.resolver = resolver;
        this.properties = properties;
    }

    /**
     * This is used to initialize some data for each Azure Storage Account Blob container.
     */
    @Override
    public void run(String... args) {
        properties.getConfigurations().forEach(this::writeDataByStorageAccount);
    }

    private void writeDataByStorageAccount(AzureStorageBlobProperties blobProperties) {
        String containerName = blobProperties.getContainerName();
        if (!StringUtils.hasText(containerName) || blobProperties.getAccountName() == null) {
            return;
        }

        String accountName = getStorageAccountName(blobProperties);
        logger.info("Begin to initialize the {} container of the {} account", containerName, accountName);
        long currentTimeMillis = System.currentTimeMillis();
        String fileName = "fileName-" + currentTimeMillis;
        String data = "data" + currentTimeMillis;
        Resource storageBlobResource = resolver.getResource("azure-blob-" + accountName + "://" + containerName +"/" + fileName + ".txt");
        try (OutputStream os = ((WritableResource) storageBlobResource).getOutputStream()) {
            os.write(data.getBytes());
            logger.info("Write data to container={}, fileName={}.txt", containerName, fileName);
        } catch (IOException e) {
            logger.error("Write data exception", e);
        }
        logger.info("End to initialize the {} container of the {} account", containerName, accountName);
    }
}
```

Run the sample with following Maven command:

`mvn clean spring-boot:run -Dspring-boot.run.profiles=extend`

Finally, verify the expected outcome. Your console should display the following output:

```
c.a.s.e.s.s.r.e.SampleDataInitializer    : Begin to initialize the container first of the account firstaccount.
c.a.s.e.s.s.r.e.SampleDataInitializer    : Write data to container=first, fileName=fileName-1656641340271.txt
c.a.s.e.s.s.r.e.SampleDataInitializer    : End to initialize the container first of the account firstaccount.
c.a.s.e.s.s.r.e.SampleDataInitializer    : Begin to initialize the container second of the account secondaccount.
c.a.s.e.s.s.r.e.SampleDataInitializer    : Write data to container=second, fileName=fileName-1656641343572.txt
c.a.s.e.s.s.r.e.SampleDataInitializer    : End to initialize the container second of the account secondaccount.
```

All sample project code is published at the repository [spring-cloud-azure-starter-storage-blob-extend-sample](https://github.com/moarychan/spring-cloud-azure-starter-storage-blob-extend-sample/tree/main/spring-cloud-azure-starter-storage-blob-extend-sample).

Within this extended application, it's still possible to revert to the original, single storage account usage of Spring Cloud Azure Starter Storage Blob by adding the following configuration file _application-current.yml_:

```
spring:
  cloud:
    azure:
      storage:
        blob:
          account-name: ${FIRST_ACCOUNT}
          container-name: ${FIRST_CONTAINER}
          account-key: ${ACCOUNT_KEY_OF_FIRST_ACCOUNT}
current:
  second-container: ${SECOND_CONTAINER}
```

NOTE: You must set or replace the listed environment variable assigned values with active Azure Storage Account resource information.

Run the sample with following Maven command:

`mvn clean spring-boot:run -Dspring-boot.run.profiles=current`

To verify correct operation using a single storage account, compare terminal output with that listed here:

```
c.a.s.e.s.s.r.c.SampleDataInitializer    : StorageApplication data initialization of 'first-container' begin ...
c.a.s.e.s.s.r.c.SampleDataInitializer    : Write data to container=first-container, fileName=fileName1656641162614.txt
c.a.s.e.s.s.r.c.SampleDataInitializer    : StorageApplication data initialization of 'first-container' end ...
c.a.s.e.s.s.r.c.SampleDataInitializer    : StorageApplication data initialization of 'second-container' begin ...
c.a.s.e.s.s.r.c.SampleDataInitializer    : Write data to container=second-container, fileName=fileName1656641165411.txt
c.a.s.e.s.s.r.c.SampleDataInitializer    : StorageApplication data initialization of 'second-container' end ...
```

## Conclusion

Implementing a specific resource type and corresponding pattern resolver is relatively simple, largely thanks to clear documentation, the many built-in implementations, common usage within the Spring technology stack.

One point that warrants attention is the protocol definition for the resource, e.g. the Azure Storage Blob Resource. We must note whether we are using **azure-blob://** or **azure-blob-[account-name]://** and plan app capabilities accordingly. Additionally, since the identifier of a network resource must be uniquely identifiable, the latter location format may result in a much longer name and also exposes the name of the storage account. These tradeoffs need to be evaluated in light of requirements and risk profile.

## References & Useful Resources

- [Latest Spring Resources documentation](https://docs.spring.io/spring-framework/docs/current/reference/html/core.html#resources)
- [Extended starter for Spring Cloud Azure Starter Storage Blob and sample on GitHub](https://github.com/moarychan/spring-cloud-azure-starter-storage-blob-extend-sample)