Image Vector Similarity Search with Azure Computer Vision and PostgreSQL
Tags: azure, python, ai, vectordatabase
Published: 2023-07-05 16:41:32

Recently, vector embeddings and vector databases have become increasingly popular, enabling the development of new intelligent applications such as vector similarity search. Since I had no prior experience with these concepts, I decided to explore them and document my journey for you.

In this article, you will explore the Image Retrieval functionality of Azure Computer Vision 4.0, which is powered by the Florence Foundation model. You will:

* Explore the concept of vector embeddings and vector similarity search.
* Understand how the Image Retrieval APIs of Azure Computer Vision work.
* Use the Image Retrieval APIs in Python to search a collection of images to find those that are most similar to a given query image or text prompt.
* Use the Azure Cosmos DB for PostgreSQL to store and query vector data.

Before starting to build your image vector similarity system, follow these steps:

* Sign up for either an [Azure free account](https://azure.microsoft.com/free/?WT.mc_id=AI-MVP-5004971) or an [Azure for Students account](https://azure.microsoft.com/free/students/?WT.mc_id=AI-MVP-5004971). If you already have an active subscription, you can use it.

* Create a Cognitive Services resource in the Azure portal.
    
    > The Image Retrieval APIs are available in the following regions: East US, France Central, Korea Central, North Europe, Southeast Asia, West Europe, West US. To run the Image Retrieval demo in the Vision Studio, your resource must belong to the **East US** region.

* Install Python 3.x, Visual Studio Code, Jupyter Notebook and Jupyter Extension for Visual Studio Code.

> Do you want to start coding right away? The source code, image files, and their vector embeddings can all be found on my [GitHub repository](https://github.com/sfoteini/image-vector-search-azure-postgresql).

## What are vector embeddings?

Let's start by understanding vector embeddings. In simple terms, vector embeddings are numerical representations of data, such as images, text, videos, and audio. These vectors are high-dimensional dense vectors, with each dimension containing information about the original content. By translating data into dense vectors, we can use them to perform tasks like clustering, recommendation, or image search. Additionally, these dense representations allow computers to understand the semantic similarity between two objects. That is, if we represent objects like images, videos, documents, or audio as vector embeddings, we can quantify their semantic similarity by their proximity in a vector space. For example, the embedding vector of an image of a cat will be more similar to the embedding vector of another cat or the word "meow" than that of a picture of a dog or the word "woof".

![Graph of similar vector embeddings.](https://sfoteini.github.io/images/post/image-vector-similarity-search-with-azure-computer-vision-and-postgresql/similarity-vectors_hua18b5d1264c65e2761c20da22ff280a5_56484_649x392_fit_q95_box_3.png)

### Generating dense vectors

There are lots of ways to generate vector embeddings of data. To vectorize your data in Azure, you can use the Azure OpenAI Embeddings models for text documents or the Azure Cognitive Services for Vision Image Retrieval APIs for images and text. The latter is a multi-modal embedding model that enables you to vectorize image and text data and build image search systems.

If you're interested in learning more about vector embeddings and how they're created, check out the post ["Vector Similarity Search: From Basic to Production"](https://mlops.community/vector-similarity-search-from-basics-to-production/) on the MLOps community website!

## What is vector similarity search?

Vector embeddings capture the semantic similarity of objects, allowing for the development of semantic similarity search systems. Vector similarity search works by searching for images based on the content of the image itself, instead of relying only on manually assigned keywords, tags, or other metadata, as keyword-based search systems do. This approach is usually more efficient and accurate than traditional search techniques, which depend heavily on the user's capability to find the best search terms.

In a vector search system, the vector embedding of a user's query is compared to a set of pre-stored vector embeddings to find a list of vectors that are the most similar to the query vector. Since embeddings that are numerically close are also semantically similar, we can measure the semantic similarity by using a distance metric, such as cosine or Euclidean distance.

In this post, we will be utilizing the cosine similarity metric to measure the similarity between two vectors. This metric is equal to the cosine of the angle between the two vectors, or, equivalently, the dot product of the vectors divided by the product of their magnitudes. The cosine similarity ranges from -1 to 1, with a value close to 1 indicating that the vectors are very similar. To help illustrate this, here is a Python function that calculates the cosine similarity between two vectors.

```python
def get_cosine_similarity(vector1, vector2):
    """
    Get the cosine similarity between two vectors
    """
    dot_product = 0
    length = min(len(vector1), len(vector2))

    for i in range(length):
        dot_product += vector1[i] * vector2[i]

    magnitude1 = math.sqrt(sum(x * x for x in vector1))
    magnitude2 = math.sqrt(sum(x * x for x in vector2))

    return dot_product / (magnitude1 * magnitude2)
```

## Do image retrieval using Azure Computer Vision

Azure Computer Vision 4.0's Image Retrieval APIs, powered by the Florence Foundation model, allow for the vectorization of images and text queries. This vectorization converts images and text into coordinates in a 1024-dimensional vector space, enabling users to search a collection of images using text and/or images without the need for metadata, such as image tags, labels, or captions.

Azure Computer Vision provides two Image Retrieval APIs for vectorizing image and text queries: the Vectorize Image API and the Vectorize Text API. The diagram below shows a typical image retrieval process using these APIs.

![A typical image retrieval system using the Image Retrieval APIs of Azure Computer Vision.](https://learn.microsoft.com/en-us/azure/cognitive-services/computer-vision/media/image-retrieval.png "Image source: [Azure Computer Vision Image Retrieval – Microsoft Docs](https://learn.microsoft.com/en-us/azure/cognitive-services/computer-vision/concept-image-retrieval?WT.mc_id=AI-MVP-5004971)")

### Image Retrieval in Python

Navigate to my [GitHub repository](https://github.com/sfoteini/image-vector-search-azure-postgresql) and download the source code of this article. Open the `image-embedding.ipynb` Jupyter Notebook in Visual Studio Code and explore the code.

Using the Vectorize Image API, this code generates vector embeddings for 200 images and exports them into a JSON file. After that, two image vector similarity search processes are highlighted: image-to-image search and text-to-image search.

* In the image-to-image search process, the reference image is converted into a vector embedding using the Vectorize Image API and the cosine distance is used to measure the similarity between the query vector and the vector embeddings of our image collection. The top matched images are retrieved and displayed alongside the reference image.

* In the text-to-image search process, the text query is converted into a vector embedding using the Vectorize Text API and the most similar images are retrieved and displayed based on the cosine similarity between the query vector and their vectors.

> This example is inspired by the Serge Retkowsky’s [Azure Computer Vision in a day workshop](https://github.com/retkowsky/Azure-Computer-Vision-in-a-day-workshop).

## What is a vector database?

Vector databases are a specialized type of database that are designed for handling data as high-dimensional vectors, or embeddings. Vector databases are distinct from traditional databases in that they are optimized to store and query vectors with a large number of dimensions, which can range from tens to thousands, depending on the complexity of the data and the transformation function applied. The diagram below illustrates the workflow of a basic vector search system.

![Workflow of a vector search system.](https://sfoteini.github.io/images/post/image-vector-similarity-search-with-azure-computer-vision-and-postgresql/vector-search-flow_huf8a169ce5211bcd5ad8e6422d4d5b0fa_107385_853x401_fit_q95_box_3.png)

Let's analyze this further.

1. First, we use an embedding model to generate vector embeddings for our raw data, such as text, images, videos, or audio.
2. These vectors are stored in the vector database along with a reference to the original data and/or other metadata (optional).
3. When a query is issued by an application, we use the same embedding model to generate embeddings for the query. This query can be of the same data type as our dataset (e.g., an image to search for similar images) or of a different data type (e.g., text to search for similar images).
4. We then use the query vector to search for similar vector embeddings in the database. To determine the similarity between any two vectors, we must use a similarity measure, such as cosine similarity, to calculate the distance between them in the high-dimensional vector space.
5. The similarity search will output a list of vectors that are most similar to the query vector. The raw data associated with each vector can then be accessed.

As of July 3, 2023, the following Azure services are available for storing and querying vector data.

1. [Azure PostgreSQL Database](https://learn.microsoft.com/azure/postgresql/flexible-server/how-to-use-pgvector?WT.mc_id=AI-MVP-5004971) and [Azure Cosmos DB for PostgreSQL](https://learn.microsoft.com/azure/cosmos-db/postgresql/howto-use-pgvector?WT.mc_id=AI-MVP-5004971)
2. [Azure Cosmos DB for MongoDB vCore](https://learn.microsoft.com/azure/cosmos-db/mongodb/vcore/vector-search?WT.mc_id=AI-MVP-5004971)
3. [Azure Cache for Redis Enterprise](https://techcommunity.microsoft.com/t5/azure-developer-community-blog/introducing-vector-search-similarity-capabilities-in-azure-cache/ba-p/3827512?WT.mc_id=AI-MVP-5004971)
4. [Azure Data Explorer](https://techcommunity.microsoft.com/t5/azure-data-explorer-blog/azure-data-explorer-for-vector-similarity-search/ba-p/3819626?WT.mc_id=AI-MVP-5004971)
5. [Azure Cognitive Search – Vector Search (private preview)](https://github.com/Azure/cognitive-search-vector-pr/)

## Vector search with Azure Cosmos DB for PostgreSQL

For this post, I wanted to try out Azure Cosmos DB for PostgreSQL in which vector similarity search is enabled by the `pgvector` extension. In the sections below, I will show you how to enable the extension, create a table to store vector data, and query the vectors.

Let's get started!

### Create an Azure Cosmos DB for PostgreSQL cluster

1. Sign in to the [Azure portal](https://portal.azure.com/) and select **+ Create a resource**.
2. Search for **Azure Cosmos DB**. On the **Azure Cosmos DB** screen, select **Create**.
3. On the **Which API best suits your workload?** screen, select **Create** on the **Azure Cosmos DB for PostgreSQL** tile.
4. On the **Create an Azure Cosmos DB for PostgreSQL cluster** form fill out the following values:
    * **Subscription**: Select your subscription.
    * **Resource group**: Select an existing resource group or create a new one.
    * **Cluster name**: Enter a name for your Azure Cosmos DB for PostgreSQL cluster.
    * **Location**: Choose one of the available regions.
    * **Scale**: You can leave Scale as its default value or select the optimal number of nodes as well as compute, memory, and storage configuration.
    * **PostgreSQL version**: Choose a PostgreSQL version such as **15**.
    * **Database name**: You can leave database name at its default value `citus`.
    * **Administrator account**: The admin username must be `citus`. Select a password that will be used for `citus` role to connect to the database.
    
    ![Screenshot of the Basic tab of the Create an Azure Cosmos DB for PostgreSQL cluster form.](https://sfoteini.github.io/images/post/image-vector-similarity-search-with-azure-computer-vision-and-postgresql/create-cluster_hudfd1789368a8c9f0e3f3a78a257f7a8d_45051_640x570_fit_q95_box_3.png)

5. On the **Networking** tab, select **Allow public access from Azure services and resources within Azure to this cluster** and create your preferred firewall rule.

    ![Screenshot of the Networking tab of the Create an Azure Cosmos DB for PostgreSQL cluster form.](https://sfoteini.github.io/images/post/image-vector-similarity-search-with-azure-computer-vision-and-postgresql/create-cluster-firewall-rules_hu2a8f1f5c7b53b58edfa9f71fedd0667f_52240_760x549_fit_q95_box_3.png)

6. Navigate to the **Review + Create** tab and then select **Create** to create the cluster.
7. Once the deployment is complete, select **Go to resource**.
8. Save the **Coordinator name**, **Database name**, and **Admin username** found on the **Overview** tab of your cluster, as you will need them in a subsequent step.

### Enable the pgvector extension

The `pgvector` extension adds vector similarity search capabilities to your PostgreSQL database. To use the extension, you have to first create it in your database. To enable the extension, we will use the PSQL Shell.

1. Select the **Quick Start** tab on the left pane of your Azure resource and then select the **PostgreSQL Shell** tile.
2. Enter your password to connect to your database.
3. The `pgvector` extension can be enabled using the following command:

    ```sql
    SELECT CREATE_EXTENSION('vector');
    ```

### Create a table to store vector data

On the PSQL Shell, run the following command to create a table to store vector data.

```sql
CREATE TABLE imagevectors(
    file TEXT PRIMARY KEY,
    embedding VECTOR(1024)
    );
```

The table has two columns: one for the image file paths of type `TEXT` and one for the corresponding vector embeddings of type `VECTOR(1024)`.

### Create the Python application

Create a new `.ipynb` file and open it in Visual Studio Code. Create a `.env` file and save the following variables:

| Variable name | Variable value |
| --- | --- |
| CV_KEY | Azure Cognitive Services key |
| CV_ENDPOINT | Azure Cognitive Services endpoint |
| POSTGRES_HOST | PostgreSQL cluster coordinator name |
| POSTGRES_DB_NAME | PostgreSQL cluster database name |
| POSTGRES_USER | PostgreSQL cluster admin username |
| POSTGRES_PASSWORD | PostgreSQL cluster admin password |

> The source code, image files, and their vector embeddings can all be found on my [GitHub repository](https://github.com/sfoteini/image-vector-search-azure-postgresql).

1. Import the following libraries and 3 functions from the `azurecv.py` file.

    ```python
    import os
    import glob
    import json
    import psycopg2
    from psycopg2 import pool
    from dotenv import load_dotenv
    import pandas as pd
    import csv
    from io import StringIO
    import math
    from azurecv import text_embedding, image_embedding, display_image_grid
    ```

2. Insert a new code cell and add the following code, which will load the image file paths and the vector embeddings generated in a previous section.

    ```python
    # images
    images_folder = "images"
    image_files = glob.glob(images_folder + "/*")

    # embeddings
    output_folder = "output"
    emb_json = os.path.join(output_folder, "embeddings.json")
    with open(emb_json) as f:
        image_embeddings = json.load(f)

    print(f"Total number of images: {len(image_files)}")
    print(f"Number of imported vector embeddings: {len(image_embeddings)}")
    ```

3. In a new code cell, create a Pandas Dataframe with two columns: one for the image file paths and one for the corresponding vector embeddings (converted to a string representation).

    ```python
    df_files = pd.DataFrame(image_files, columns=['file'])
    df_embeddings = pd.DataFrame([str(emb) for emb in image_embeddings], columns=['embedding'])
    df = pd.concat([df_files, df_embeddings], axis=1)
    df.head(5)
    ```

4. Then, you will connect to the Azure Cosmos DB for PostgreSQL cluster. The following code forms a connection string using the environment variables for your Azure Cosmos DB for PostgreSQL cluster and creates a connection pool to your Postgres database. After that, a `cursor` object is created, which can be used to execute SQL queries with the `execute()` method.

    ```python
    # Load environment variables
    load_dotenv()
    host = os.getenv("POSTGRES_HOST")
    dbname = os.getenv("POSTGRES_DB_NAME")
    user = os.getenv("POSTGRES_USER")
    password = os.getenv("POSTGRES_PASSWORD")
    sslmode = "require"
    table_name = "imagevectors"
        
    # Build a connection string from the variables
    conn_string = "host={0} user={1} dbname={2} password={3} sslmode={4}".format(host, user, dbname, password, sslmode)

    postgreSQL_pool = psycopg2.pool.SimpleConnectionPool(1, 20, conn_string)
    if (postgreSQL_pool):
        print("Connection pool created successfully")

    # Get a connection from the connection pool
    conn = postgreSQL_pool.getconn()
    cursor = conn.cursor()
    ```

5. Let's add some data to our table. Inserting data into a PostgreSQL table can be done in several ways. I chose not to use the `pandas.DataFrame.to_sql()` method for inserting data into our PostgreSQL table because it is relatively slow. Instead, I use the `COPY FROM STDIN` command to add the data.

    > This method is inspired by the post [A Fast Method to Bulk Insert a Pandas DataFrame into Postgres](https://ellisvalentiner.com/post/a-fast-method-to-insert-a-pandas-dataframe-into-postgres/) by Ellis Michael Valentiner.

    The following code creates a temporary table with the same columns as the `imagevectors` table, and the data from the dataframe is copied into it. The data from the temporary table is then inserted into the `imagevectors` table, taking into account any potential conflicts that may arise due to duplicate keys (which could happen if the code is run multiple times).

    ```python
    sio = StringIO()
    writer = csv.writer(sio)
    writer.writerows(df.values)
    sio.seek(0)

    cursor.execute("CREATE TEMPORARY TABLE tmp (file TEXT PRIMARY KEY, embedding VECTOR(1024)) ON COMMIT DROP;")

    cursor.copy_expert("COPY tmp FROM STDIN CSV", sio)

    cursor.execute(f"""INSERT INTO {table_name} (file, embedding)
                    SELECT * FROM tmp
                    ON conflict (file) DO NOTHING;""")

    conn.commit()
    ```

6. To view the first 10 rows inserted into the table, run the following code:

    ```python
    # Fetch all rows from table
    cursor.execute(f"SELECT * FROM {table_name} limit 10;")
    rows = cursor.fetchall()

    # Print all rows
    for row in rows:
        print(f"Data row = ({row[0]}, {row[1]})")
    ```

7. After populating the table with vector data, you can use this image collection to search for images that are most similar to a reference image or a text prompt. The workflow is summarized as follows:
    * Use the Vectorize Image API or the Vectorize Text API to generate vector embeddings of an image or text, respectively.
    * To calculate similarity and retrieve images, use `SELECT` statements and the built-it vector operators of the PostgreSQL database.
    * Display the retrieved images using the `display_image_grid()` function.

    ![Vector similarity search workflow.](https://sfoteini.github.io/images/post/image-vector-similarity-search-with-azure-computer-vision-and-postgresql/vector-search-flow-azurecv_hu561b19e891a59eece5a27397f5fa79e6_80394_1140x279_fit_q95_box_3.png)

8. Let’s understand how a simple `SELECT` statement works. Consider the following query:

    ```sql
    SELECT * FROM imagevectors ORDER BY embedding <=> '[0.003, …, 0.034]' LIMIT 5
    ```

    This query computes the cosine distance (`<=>`) between the given vector (`[0.003, …, 0.034]`) and the vectors stored in the `imagevectors` table, sorts the results by the calculated distance, and returns the five most similar images (`LIMIT 5`).

    The PostgreSQL `pgvector` extension provides 3 operators that can be used to calculate similarity:

    | Operator | Description |
    | --- | --- |
    | `<->` | Euclidean distance |
    | `<#>` | Negative inner product |
    | `<=>` | Cosine distance |

9. An example of a text-to-image search process can be found below. For an example of an image-to-image search, please refer to the [Jupyter Notebook](https://github.com/sfoteini/image-vector-search-azure-postgresql/blob/main/image-search-postgresql.ipynb) in my GitHub repository.

    ```python
    # Generate the embedding of the text prompt
    txt = "a seahorse"
    txt_emb = text_embedding(txt, endpoint, key)

    # Vector search
    topn = 9
    cursor.execute(f"SELECT * FROM {table_name} ORDER BY embedding <=> %s LIMIT {topn}", (str(txt_emb),))

    # Display the results
    rows = cursor.fetchall()
    for row in rows:
        print(row)

    # Display the similar images
    images = [row[0] for row in rows]
    captions = [f"Top {i+1}: {os.path.basename(images[i])}" for i in range(len(images))]
    ncols = 3
    nrows = math.ceil(len(images)/ncols)
    display_image_grid(images, captions, 'Search results for: "' + txt + '"', nrows, ncols)
    ```

## Summary and next steps

In this post, we explored the concepts of “embeddings”, “vector search”, and “vector database” and created a simple image vector similarity search system with the Azure Computer Vision Image Retrieval APIs and Azure Cosmos DB for PostgreSQL. If you'd like to dive deeper into this topic, here are some helpful resources.

* [Azure Computer Vision Image Retrieval – Microsoft Docs](https://learn.microsoft.com/azure/cognitive-services/computer-vision/concept-image-retrieval?WT.mc_id=AI-MVP-5004971)
* [How to use pgvector on Azure Cosmos DB for PostgreSQL – Microsoft Docs](https://learn.microsoft.com/azure/cosmos-db/postgresql/howto-use-pgvector?WT.mc_id=AI-MVP-5004971)

You can also check out my previous blog posts about Azure Computer Vision 4.0 (Florence model):
* [Explore Azure Computer Vision 4.0 (Florence model)](https://sfoteini.github.io/blog/explore-azure-computer-vision-4-florence-model/)
* [Extract text from images with Azure Computer Vision 4.0 Read OCR](https://sfoteini.github.io/blog/extract-text-from-images-with-azure-computer-vision-4-read-ocr/)

---

👋 **Hi, I am Foteini Savvidou!**
An Electrical and Computer Engineering student and Microsoft AI MVP (Most Valuable Professional) from Greece.

🌈 [LinkedIn](https://www.linkedin.com/in/foteini-savvidou) | [Blog](https://sfoteini.github.io/) | [GitHub](https://github.com/sfoteini)