Building Multilingual Image Search with Marqo
Tags: marqo, tutorial, vectordatabase, search
Published: 2024-01-28 09:33:22

### Introduction

#### Multilingual Image Search

For global businesses, multilingual image search can prove to be a critical component for enhancing accessibility of their search platforms and expand their content reach. As e-commerce and digital platforms grow internationally, multilingual capabilities become essential in tapping into new markets and demographics.

#### Why Marqo?

Marqo differentiates itself by integrating state-of-the-art multilingual CLIP models with dense retrieval techniques, facilitating accurate and context-aware image searches across 100 languages. 

In this article, we will explore steps to set up and build multilingual image search. We'll be covering end-to-end flow from System Requirements to implementation to Use cases.

### Prerequisites
Before implementing a multilingual image search with Marqo, ensure your system meets the following requirements:

#### System Requirements
1. **Python Environment**: Requires Python 3.7 or later.
2. **Hardware Requirements**:
   * CPU: Modern multi-core processor (e.g., Intel Core i5/i7/i9, AMD Ryzen 5/7/9) with high clock speed for efficient processing.
   * GPU: Recommended for performance acceleration, especially beneficial for larger datasets. CUDA-compatible GPUs are ideal.
3. **Memory**: Minimum of 8GB RAM; more is beneficial for large datasets.

#### Setup and installation

Lets start by downloading and installing Marqo:
1. Install Docker. Follow [Docker Docs](https://docs.docker.com/get-docker/).
2. Use Docker to run Marqo server instance:
```
docker pull marqoai/marqo:latest
docker rm -f marqo
docker run --name marqo -it -p 8882:8882 marqoai/marqo:latest
```

##### Setting up Marqo Client (in Python)
```
pip install marqo
```
_Note:_ 
1. If you're using `Marqo 2.0.0`, you'll need to use `marqo1` package.
2. Find more detailed instructions on setting up Marqo [here](https://docs.marqo.ai/2.0.0/).

### Code Walkthrough
In this section, we will go through implementation steps and code snippets for Multilingual image search with Marqo.

#### Initializing Search Index
Firstly, we initialize the Marqo search index with the desired settings.

##### Code Snippet
```
import marqo

def initialize_search_index(client_url, index_name, settings):
    # Initialize Marqo client with the provided URL
    mq = marqo.Client(url=client_url)
    try:
        # Attempt to create a new search index with specified settings
        mq.create_index(index_name, settings_dict=settings)
    except marqo.errors.MarqoWebError as e:
        # Check if index already exists to avoid duplication
        if e.code == 'index_already_exists':
            print(f"Index {index_name} already exists. Skipping creation.")
        else:
            # Raise other exceptions
            raise
    return mq
```

#### Explanation
* **Client Initialization**: Establishes connection to Marqo server.
* **Index Creation**: Tries to create a new index; handles case where index already exists.

### Populating the Index with Image Data

We populate the index with images and their descriptions in multiple languages (English & Spanish).

#### Code Snippet
```
def populate_index_with_images(mq, index_name, image_data):
    # Add image data to the specified index
    return mq.index(index_name).add_documents(
        image_data,
        tensor_fields=['description', 'description_es'],  # Fields for dense retrieval
        device="cpu",  # Specify the device for processing (CPU in this case)
        client_batch_size=1  # Process data in individual batches
    )
```
#### Explanation
* **Document Addition**: Adds images and descriptions to the index.
* **Tensor Fields**: Fields used for embedding and retrieval.
* **Batch Processing**: Efficient data ingestion in batches.

### Performing Searches in Different Languages

This step involves searching the index using queries in multiple languages.

#### Code Snippet
```
def perform_search(mq, index_name, query_text, language_code):
    # Choose the search field based on language code
    search_field = "description_es" if language_code == "es" else "description"
    
    # Perform the search with the given query
    search_results = mq.index(index_name).search(
        q=query_text,
        attributes_to_retrieve=[search_field, "url"],  # Fields to retrieve in results
        limit=1  # Limit the number of results
    )
    return search_results
```

#### Explanation
* **Field Selection**: Dynamically selects the search field based on language (`description_es`, `description`).
* **Executing Query**: Searches the index with the specified query and retrieves relevant fields.

### Displaying Search Results
Finally, we extract and display image URLs from the search results.

#### Code Snippet
```
def display_image_urls(search_results):
    # Iterate through each hit in the search results
    for hit in search_results.get('hits', []):
        # Print the URL of the image
        print(hit.get('url'))
```

#### Explanation
* **Result Iteration**: Loops through each hit in the results.
* **URL Display**: Outputs the URL of each image.

### Putting It All Together (Example Implementation)

This section demonstrates the `main` function, which ties the previously discussed functionalities into a complete workflow, implementing a client for this functionality.

#### Code Snippet
```
def main():
    client_url = "http://localhost:8882"
    index_name = "multilingual-image-search"
    settings = {
        "treatUrlsAndPointersAsImages": True,
        "model": "multilingual-clip/XLM-Roberta-Large-Vit-L-14",
        "normalizeEmbeddings": True,
    }

    # Initialize Marqo client and index
    mq = initialize_search_index(client_url, index_name, settings)

    # Image data format: List of dictionaries, each with 'url', 'description', and 'description_es'
    image_data = [
        # Example:
        # {"url": "http://example.com/image1.jpg", 
        #  "description": "Description in English", 
        #  "description_es": "Descripción en español"}
        # ... [Add your image data with descriptions here] ...
    ]

    # Populate the index with images
    populate_index_with_images(mq, index_name, image_data)

    # Perform searches in different languages
    english_results = perform_search(mq, index_name, "A streetwise ape with a flair for the dramatic", "en")
    spanish_results = perform_search(mq, index_name, "Un simio callejero con un sentido dramático", "es")

    # Print and display search results
    print("Search results for English query:", english_results)
    print("Search results for Spanish query:", spanish_results)

    print("\nImage URL from English query:")
    display_image_urls(english_results)

    print("\nImage URL from Spanish query:")
    display_image_urls(spanish_results)

if __name__ == '__main__':
    main()
```
#### Explanation
* **Initialization**: Sets up the client URL and index name, and defines the index settings.
* **Index Population**: Loads the image data into the Marqo index.
* **Search Execution**: Performs searches using English and Spanish queries.
* **Result Display**: Outputs the search results and displays the image URLs.

**_Note:_** Find the full implementation on [Github](https://github.com/aryamihirs/marqo-demo).

#### *Best practices and Optimization techniques*

##### Optimize Index Settings
* **Select the Right Model**: Choose a model that best fits your dataset. For instance, `multilingual-clip/XLM-Roberta-Large-Vit-L-14` is ideal for diverse linguistic content.
* **Normalization**: Use the `normalizeEmbeddings` setting for more accurate distance measurements in vector space, improving search relevance.

##### Efficient Data Management
* **Batch Processing**: When populating the index, adjust `client_batch_size` based on your system's capabilities to balance between speed and resource usage.
* **Regular Index Updates**: Keep your index up-to-date with new images and descriptions to maintain search relevance and accuracy.

##### Handle Multilingual Content Effectively
* **Field Mapping**: Use language-specific fields (like `description_es` for Spanish) to store and retrieve language-specific content.
* **Language Detection**: Implement automatic language detection in your application to choose the appropriate search field dynamically.

##### Performance Tuning
* **Hardware Utilization**: If using a GPU, ensure Marqo is [configured correctly](https://docs.marqo.ai/2.0.0/Guides/using_marqo_with_a_gpu/) to leverage it, significantly speeding up indexing and search operations.
* **Query Optimization**: Refine your queries based on user feedback and search patterns. Short, concise queries often yield better results.

##### Utilize Marqo's Advanced Features
* **Synonyms and Stopwords**: Explore advanced features like synonyms and stopwords to refine the search experience.
* **Custom Ranking**: Use custom ranking parameters to tailor search results based on specific criteria like popularity or recency.

### Use Cases for Multilingual Image Search

Multilingual image search powered by Marqo offers specialized applications across many industries. Let's explore some specific use cases:

#### E-Commerce

* **Global Product Discovery**: Online shoppers from different regions can search for products like _red evening dress_ or _晚礼服_ in their native languages, significantly enhancing user experience.
* **Localized Marketing**: Retailers can present products with images that resonate culturally with each linguistic market, like showcasing traditional attire in searches from specific regions.
* **Customer Reviews and Q&A**: Multilingual search can help in navigating through customer reviews and questions in various languages, aiding in informed purchase decisions.

#### Stock Photo Libraries

* **Broad Accessibility**: Photographers can tag their work in multiple languages, enabling users worldwide to discover relevant images, like searching for `日出` or `sunrise` to find similar thematic content.
* **Cultural Archives**: Stock libraries can serve as digital archives, preserving cultural imagery that's searchable in multiple languages, thus aiding cultural preservation and education.

#### Media and Entertainment

* **Targeted Content Distribution**: Distributors can use multilingual image search to categorize films or shows based on culturally relevant themes, optimizing content for different linguistic audiences.
* **Promotional Campaigns**: Movie studios can create promotional campaigns with imagery that resonates differently with each language-speaking audience, enhancing global reach and relatability.

### Conclusion and External References

#### Recap & Key Points

In this guide, we explored the implementation of multilingual image search using Marqo, starting from the System Requirements and installation, to a detailed code walkthrough.

We emphasized on best practices, error handling, and optimization strategies to enhance search performance. The exploration of diverse use cases across sectors like e-commerce, stock photos, and Media & Entertainment highlighted the versatility and impact of Marqo.

#### Further Exploration

To explore Marqo further, here are some valuable resources:

* **Marqo Documentation**: Dive deeper into Marqo's capabilities and features by visiting [Marqo's official documentation](https://docs.marqo.ai/2.0.0/).
* **Multilingual CLIP Models**: Understand the underlying models used in Marqo for multilingual search by reading about [Multilingual CLIP models](https://docs.marqo.ai/2.0.0/Guides/Models-Reference/dense_retrieval/#multilingual-clip).
* **Community and Support**: Engage with the Marqo community and seek support for specific issues on [GitHub](https://github.com/marqo-ai/marqo).
* **Case Studies and Articles**: Gain insights from practical implementations and case studies, such as the Medium article on [implementing text-to-image search using Marqo](https://medium.com/@wanli19940222/how-to-implement-text-to-image-search-on-marqo-in-5-lines-of-code-448f75bed1da).